performance/io-threads: Remove code duplication

Change-Id: Ic905cc6074c796efce2972857b79ab53700a2de4 BUG: 1065657 Signed-off-by: Pranith Kumar K <pkarampu@redhat.com> Reviewed-on: http://review.gluster.org/7010 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Anand Avati <avati@redhat.com>
author: Pranith Kumar K <pkarampu@redhat.com> 2014-02-15 17:54:38 +0530
committer: Anand Avati <avati@redhat.com> 2014-02-24 18:56:06 -0800
commit: 868287aa5f70bdfcf7f56663e042d74fe1acf0db (patch)
tree: f84a67b72a8dac9c7d9eb80e80cc83d65d59165e /extras/hook-scripts
parent: 3a5914e421d10bca363f020b748e8648a4819495 (diff)
0 files changed, 0 insertions, 0 deletions
diff --git a/.clang-format b/.clang-format
new file mode 100644
index 00000000000..84c2efe3fad
--- /dev/null
+++ b/.clang-format
@@ -0,0 +1,107 @@
+---
+Language:        Cpp
+# BasedOnStyle:  Chromium
+AccessModifierOffset: -1
+AlignAfterOpenBracket: Align
+AlignConsecutiveAssignments: false
+AlignConsecutiveDeclarations: false
+AlignEscapedNewlines: Right
+AlignOperands:   true
+AlignTrailingComments: true
+AllowAllParametersOfDeclarationOnNextLine: false
+AllowShortBlocksOnASingleLine: false
+AllowShortCaseLabelsOnASingleLine: false
+AllowShortFunctionsOnASingleLine: Inline
+AllowShortIfStatementsOnASingleLine: false
+AllowShortLoopsOnASingleLine: false
+AlwaysBreakAfterDefinitionReturnType: All
+AlwaysBreakAfterReturnType: All
+AlwaysBreakBeforeMultilineStrings: true
+AlwaysBreakTemplateDeclarations: true
+BinPackArguments: true
+BinPackParameters: true
+BraceWrapping:
+  AfterClass:      false
+  AfterControlStatement: false
+  AfterEnum:       false
+  AfterFunction:   true
+  AfterNamespace:  false
+  AfterObjCDeclaration: false
+  AfterStruct:     false
+  AfterUnion:      false
+  BeforeCatch:     false
+  BeforeElse:      false
+  IndentBraces:    false
+  SplitEmptyFunction: true
+  SplitEmptyRecord: true
+  SplitEmptyNamespace: true
+BreakBeforeBinaryOperators: None
+BreakBeforeBraces: Linux
+BreakBeforeInheritanceComma: false
+BreakBeforeTernaryOperators: true
+BreakConstructorInitializersBeforeComma: false
+BreakConstructorInitializers: BeforeColon
+BreakAfterJavaFieldAnnotations: false
+BreakStringLiterals: true
+ColumnLimit:     80
+CommentPragmas:  '^ IWYU pragma:'
+CompactNamespaces: false
+ConstructorInitializerAllOnOneLineOrOnePerLine: true
+ConstructorInitializerIndentWidth: 4
+ContinuationIndentWidth: 4
+Cpp11BracedListStyle: true
+DerivePointerAlignment: false
+DisableFormat:   false
+ExperimentalAutoDetectBinPacking: false
+FixNamespaceComments: true
+ForEachMacros:
+  - foreach
+  - Q_FOREACH
+  - BOOST_FOREACH
+IncludeCategories:
+  - Regex:           '^<.*\.h>'
+    Priority:        1
+  - Regex:           '^<.*'
+    Priority:        2
+  - Regex:           '.*'
+    Priority:        3
+IncludeIsMainRegex: '([-_](test|unittest))?$'
+IndentCaseLabels: true
+IndentWidth:     4
+IndentWrappedFunctionNames: false
+JavaScriptQuotes: Leave
+JavaScriptWrapImports: true
+KeepEmptyLinesAtTheStartOfBlocks: false
+MacroBlockBegin: ''
+MacroBlockEnd:   ''
+MaxEmptyLinesToKeep: 1
+NamespaceIndentation: None
+ObjCBlockIndentWidth: 2
+ObjCSpaceAfterProperty: false
+ObjCSpaceBeforeProtocolList: false
+PenaltyBreakAssignment: 200
+PenaltyBreakBeforeFirstCallParameter: 1
+PenaltyBreakComment: 300
+PenaltyBreakFirstLessLess: 120
+PenaltyBreakString: 1000
+PenaltyExcessCharacter: 1000000
+PenaltyReturnTypeOnItsOwnLine: 200
+PointerAlignment: Right
+ReflowComments:  true
+SortIncludes:    false
+SortUsingDeclarations: true
+SpaceAfterCStyleCast: false
+SpaceAfterTemplateKeyword: true
+SpaceBeforeAssignmentOperators: true
+SpaceBeforeParens: ControlStatements
+SpaceInEmptyParentheses: false
+SpacesBeforeTrailingComments: 2
+SpacesInAngles:  false
+SpacesInContainerLiterals: true
+SpacesInCStyleCastParentheses: false
+SpacesInParentheses: false
+SpacesInSquareBrackets: false
+Standard:        Auto
+TabWidth:        8
+UseTab:          Never
+...
diff --git a/.github/ISSUE_TEMPLATE b/.github/ISSUE_TEMPLATE
new file mode 100644
index 00000000000..386ed2d8dd5
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE
@@ -0,0 +1,30 @@
+<!-- Please use this template while reporting an issue, providing as much information as possible. Failure to do so may result in a delayed response. Thank you! -->
+
+**Description of problem:**
+
+
+**The exact command to reproduce the issue**:
+
+
+**The full output of the command that failed**:
+<details>
+
+
+
+</details>
+
+**Expected results:**
+
+
+**Additional info:**
+
+
+**- The output of the `gluster volume info` command**:
+<details>
+
+
+
+</details>
+
+**- The operating system / glusterfs version**:
+
diff --git a/xlators/features/glupy/src/__init__.py.in b/.github/PULL_REQUEST_TEMPLATE
index e69de29bb2d..e69de29bb2d 100644
--- a/xlators/features/glupy/src/__init__.py.in
+++ b/.github/PULL_REQUEST_TEMPLATE
diff --git a/.github/RELEASE_TRACKER_TEMPLATE b/.github/RELEASE_TRACKER_TEMPLATE
new file mode 100644
index 00000000000..502bbd5556c
--- /dev/null
+++ b/.github/RELEASE_TRACKER_TEMPLATE
@@ -0,0 +1,12 @@
+<!-- Please use this template while creating a tracker issue -->
+
+**Description of problem:**
+A tracker issue to track the issues that will be fixed as a part of this release
+
+
+**Major or minor release**:
+
+
+**Release version**:
+
+
diff --git a/.github/stale.yml b/.github/stale.yml
new file mode 100644
index 00000000000..460e327c6ea
--- /dev/null
+++ b/.github/stale.yml
@@ -0,0 +1,25 @@
+# Number of days of inactivity before an issue becomes stale
+daysUntilStale: 210
+# Number of days of inactivity before a stale issue is closed
+daysUntilClose: 15
+# Issues with these labels will never be considered stale
+exemptLabels:
+  - pinned
+  - security
+# Label to use when marking an issue as stale
+staleLabel: wontfix
+
+# Comment to post when marking an issue as stale. Set to `false` to disable
+markComment: >
+  Thank you for your contributions.
+
+  Noticed that this issue is not having any activity in last ~6 months! We
+  are marking this issue as stale because it has not had recent activity.
+
+  It will be closed in 2 weeks if no one responds with a comment here.
+  
+
+# Comment to post when closing a stale issue. Set to `false` to disable
+closeComment: >
+  Closing this issue as there was no update since my last update on issue.
+  If this is an issue which is still valid, feel free to open it.
diff --git a/.gitignore b/.gitignore
index 29d72edcd63..fc5ba586f8e 100644
--- a/.gitignore
+++ b/.gitignore
@@ -3,11 +3,17 @@ autom4te.cache
 build
 config.*
 configure
+cscope.*
+tags
 depcomp
+INSTALL
 install-sh
 ltmain.sh
+Makefile
 Makefile.in
 missing
+stamp-h1
+stamp-h2
 test-driver
 *compile
 *.gcda
@@ -23,53 +29,97 @@ test-driver
 *.patch
 .libs
 .deps
-Makefile
-stamp-h1
+.dirstamp
+
 # Softlinks to test and log
 log
 *.vol
+.clang-format
+
+# cmocka unit tests
+*.log
+*.trs
+*_unittest
 
 # Generated files
+site.h
 *.py[co]
-tests/env.rc
-dht_layout_unittest
-mem_pool_unittest
-tests/utils/arequal-checksum
 api/examples/__init__.py
 api/examples/setup.py
-xlators/features/glupy/src/__init__.py
-contrib/argp-standalone/libargp.a
-contrib/uuid/uuid_types.h
+api/src/gfapi.map
+cli/src/gluster
+contrib/fuse-util/fusermount-glusterfs
+extras/geo-rep/gsync-sync-gfid
+extras/geo-rep/schedule_georep.py
+extras/snap_scheduler/conf.py
 extras/init.d/glusterd-Debian
+extras/init.d/glusterd-FreeBSD
 extras/init.d/glusterd-Redhat
 extras/init.d/glusterd-SuSE
-extras/init.d/glusterd-FreeBSD
 extras/init.d/glusterd.plist
-extras/systemd/glusterd.service
 extras/ocf/glusterd
 extras/ocf/volume
+extras/run-gluster.tmpfiles
+extras/systemd/glusterd.service
+extras/systemd/gluster-ta-volume.service
+extras/systemd/glusterfssharedstorage.service
 extras/who-wrote-glusterfs/gitdm
-extras/geo-rep/gsync-sync-gfid
+geo-replication/.tox
+geo-replication/gsyncd.conf
+geo-replication/src/gsyncd
+geo-replication/src/peer_gsec_create
+geo-replication/src/peer_mountbroker
+geo-replication/src/peer_mountbroker.py
 geo-replication/src/set_geo_rep_pem_keys.sh
-geo-replication/syncdaemon/configinterface.py
+geo-replication/src/peer_georep-sshkey.py
+geo-replication/syncdaemon.egg-info
+geo-replication/syncdaemon/conf.py
+geo-replication/tests/unit/.coverage
+geo-replication/tests/unit/cover
+geo-replication/tests/unit/coverage.xml
+geo-replication/tests/unit/nosetests.xml
+geo-replication/tests/unit/results.html
 glusterfs-api.pc
 glusterfs.spec
-glusterfsd/src/glusterfsd
 glusterfsd/src/glusterd
 glusterfsd/src/glusterfs
+glusterfsd/src/glusterfsd
+glusterfsd/src/gf_attach
+heal/src/glfsheal
 libgfchangelog.pc
 libglusterfs/src/graph.lex.c
 libglusterfs/src/y.tab.c
 libglusterfs/src/y.tab.h
+libglusterfs/src/defaults.c
+libglusterfs/src/cli1-xdr.h
+libglusterfs/src/protocol-common.h
 libtool
+# copied XDR for cyclic libglusterfs <-> rpc-header dependency
 run-tests.sh
+!tests/basic/fuse/Makefile
+!tests/basic/gfapi/Makefile
+tests/env.rc
+tests/utils/arequal-checksum
+xlators/features/glupy/src/__init__.py
+xlators/features/glupy/src/setup.py
 xlators/mount/fuse/utils/mount.glusterfs
 xlators/mount/fuse/utils/mount_glusterfs
-xlators/features/glupy/src/setup.py
-geo-replication/src/peer_add_secret_pub
-geo-replication/src/peer_gsec_create
-geo-replication/src/gsyncd
-cli/src/gluster
-contrib/fuse-util/fusermount-glusterfs
-geo-replication/src/gsyncd
-api/src/gfapi.map
+extras/peer_add_secret_pub
+tools/gfind_missing_files/gcrawler
+tools/glusterfind/glusterfind
+tools/glusterfind/src/tool.conf
+# Eventing
+events/src/eventsapiconf.py
+extras/systemd/glustereventsd.service
+events/src/eventtypes.py
+libglusterfs/src/eventtypes.h
+extras/init.d/glustereventsd-Debian
+extras/init.d/glustereventsd-FreeBSD
+extras/init.d/glustereventsd-Redhat
+tools/setgfid2path/src/gluster-setgfid2path
+xlators/features/cloudsync/src/cloudsync-autogen-fops.c
+xlators/features/cloudsync/src/cloudsync-autogen-fops.h
+xlators/features/utime/src/utime-autogen-fops.c
+xlators/features/utime/src/utime-autogen-fops.h
+tests/basic/metadisp/ftruncate
+xlators/features/metadisp/src/fops.c
diff --git a/.mailmap b/.mailmap
index d4a13b40065..141a1667ffa 100644
--- a/.mailmap
+++ b/.mailmap
@@ -10,15 +10,20 @@ Amar Tumballi <amarts@redhat.com> <amar@gluster.com> <amar@del.gluster.com>
 Anand Avati <avati@redhat.com> <avati@gluster.com> <avati@dev.gluster.com> <avati@amp.gluster.com> <avati@blackhole.gluster.com>
 Anush Shetty <ashetty@redhat.com> <anush@gluster.com>
 Csaba Henk <csaba@redhat.com> <csaba@gluster.com> <csaba@lowlife.hu> <csaba@zresearch.com>
+Günther Deschner <gd@redhat.com> <gd@samba.org>
 Harshavardhana <fharshav@redhat.com> <harsha@gluster.com> <harsha@zresearch.com> <harsha@dev.gluster.com> <harsha@harshavardhana.net>
+Ji-Hyeon Gim <potatogim@gluesys.com> <potatogim@potatogim.net>
 Justin Clift <justin@gluster.org> <jclift@redhat.com>
-Kaleb S. KEITHLEY <kkeithle@redhat.com> <kkeithle@f16node1.kkeithle.usersys.redhat.com>
+Kaleb S. KEITHLEY <kkeithle@redhat.com> <kkeithle@f16node1.kkeithle.usersys.redhat.com> <kkeithle@linux.keithley.org>
 Kaushal M <kaushal@redhat.com> <kaushal@gluster.com>
 Kaushik BV <kbudiger@redhat.com> <kaushikbv@gluster.com>
 Krishna Srinivas <ksriniva@redhat.com> <krishna@gluster.com> <krishna@zresearch.com> <krishna@guest-laptop>
 Krishnan Parthasarathi <kparthas@redhat.com> <kp@gluster.com>
 Louis Zuckerman <louiszuckerman@gmail.com> <me@louiszuckerman.com>
 M S Vishwanath Bhat <vbhat@redhat.com> <msvbhat@gmail.com> <vishwanath@gluster.com>
+Michael Adam <madam@redhat.com> <obnox@samba.org>
+Oleksandr Natalenko <oleksandr@natalenko.name> <o.natalenko@lanet.ua>
+Patrick Uiterwijk <puiterwijk@fedoraproject.org> <patrick@puiterwijk.org>
 Pavan Sondur <pavan@gluster.com> <pavan@dev.gluster.com>
 Pete Zaitcev <zaitcev@kotori.zaitcev.us> <zaitcev@yahoo.com>
 Pranith Kumar K <pkarampu@redhat.com> <pranithk@gluster.com>
@@ -27,9 +32,12 @@ Raghavendra Bhat <raghavendra@redhat.com> <raghavendrabhat@gluster.com>
 Raghavendra G <rgowdapp@redhat.com> <raghavendra@gluster.com> <raghavendra@zresearch.com>
 Rahul C S <rahulcs@redhat.com> <rahulcssjce@gmail.com>
 Rajesh Amaravathi <rajesh@redhat.com> <rajesh@gluster.com> <rajesh.amaravathi@gmail.com>
+Ravishankar N <ravishankar@redhat.com> <root@ravi2.(none)>
+Sakshi Bansal <sabansal@redhat.com> <sabansal@localhost.localdomain>
 Shehjar Tikoo <shehjart@gluster.com> <shehjart@zresearch.com>
 Venky Shankar <vshankar@redhat.com> <venky@gluster.com>
 Vijay Bellur <vbellur@redhat.com> <vijay@gluster.com> <vijay@dev.gluster.com>
 Vijaykumar Koppad <vkoppad@redhat.com> <vijaykumar.koppad@gmail.com>
+Vijaikumar Mallikarjuna <vmallika@redhat.com>
 Vikas Gorur <vikas@gluster.com> <vikas@zresearch.com>
 shishir gowda <sgowda@redhat.com> <shishirng@gluster.com>
diff --git a/.testignore b/.testignore
new file mode 100644
index 00000000000..fe8f838bf2b
--- /dev/null
+++ b/.testignore
@@ -0,0 +1,64 @@
+.github/ISSUE_TEMPLATE
+.github/PULL_REQUEST_TEMPLATE
+.github/stale.yml
+.gitignore
+.mailmap
+.testignore
+.clang-format
+rfc.sh
+submit-for-review.sh
+AUTHORS
+CONTRIBUTING.md
+COPYING-GPLV2
+COPYING-LGPLV3
+ChangeLog
+INSTALL
+MAINTAINERS
+NEWS
+README.md
+THANKS
+COMMITMENT
+api/examples/README
+api/examples/getvolfile.py
+api/src/README.Symbol_Versions
+build-aux/checkpatch.pl
+contrib/fuse-lib/COPYING.LIB
+contrib/fuse-util/COPYING
+contrib/macfuse/COPYING.txt
+doc/*
+extras/FreeBSD/README.FreeBSD
+extras/Solaris/README.solaris
+extras/Ubuntu/README.Ubuntu
+extras/benchmarking/README
+extras/cliutils/README.md
+extras/command-completion/README
+extras/create_new_xlator/README.md
+extras/glusterfs.vim
+extras/glusterfs-logrotate
+extras/glusterfs-georep-logrotate
+extras/init.d/glusterd-Debian.in
+extras/init.d/glusterd-FreeBSD.in
+extras/init.d/glusterd-Redhat.in
+extras/init.d/glusterd-SuSE.in
+extras/init.d/glusterd.plist.in
+extras/init.d/glustereventsd-Debian.in
+extras/init.d/glustereventsd-FreeBSD.in
+extras/init.d/glustereventsd-Redhat.in
+extras/init.d/rhel5-load-fuse.modules
+extras/logger.conf.example
+extras/snap_scheduler/README.md
+extras/test/ld-preload-test/README
+extras/who-wrote-glusterfs/*
+extras/distributed-testing/*
+geo-replication/syncdaemon/README.md
+geo-replication/test-requirements.txt
+rpc/xdr/src/.gitignore
+tests/README.md
+xlators/experimental/README.md
+xlators/experimental/dht2/README.md
+xlators/experimental/dht2/TODO.md
+xlators/experimental/posix2/README.md
+xlators/experimental/posix2/TODO.md
+xlators/features/glupy/doc/README.md
+xlators/features/glupy/doc/TESTING
+xlators/features/glupy/doc/test.vol
diff --git a/COMMITMENT b/COMMITMENT
new file mode 100644
index 00000000000..16b75efcf29
--- /dev/null
+++ b/COMMITMENT
@@ -0,0 +1,46 @@
+Common Cure Rights Commitment
+Version 1.0
+
+Before filing or continuing to prosecute any legal proceeding or claim
+(other than a Defensive Action) arising from termination of a Covered
+License, we commit to extend to the person or entity ('you') accused
+of violating the Covered License the following provisions regarding
+cure and reinstatement, taken from GPL version 3. As used here, the
+term 'this License' refers to the specific Covered License being
+enforced.
+
+    However, if you cease all violation of this License, then your
+    license from a particular copyright holder is reinstated (a)
+    provisionally, unless and until the copyright holder explicitly
+    and finally terminates your license, and (b) permanently, if the
+    copyright holder fails to notify you of the violation by some
+    reasonable means prior to 60 days after the cessation.
+
+    Moreover, your license from a particular copyright holder is
+    reinstated permanently if the copyright holder notifies you of the
+    violation by some reasonable means, this is the first time you
+    have received notice of violation of this License (for any work)
+    from that copyright holder, and you cure the violation prior to 30
+    days after your receipt of the notice.
+
+We intend this Commitment to be irrevocable, and binding and
+enforceable against us and assignees of or successors to our
+copyrights.
+
+Definitions
+
+'Covered License' means the GNU General Public License, version 2
+(GPLv2), the GNU Lesser General Public License, version 2.1
+(LGPLv2.1), or the GNU Library General Public License, version 2
+(LGPLv2), all as published by the Free Software Foundation.
+
+'Defensive Action' means a legal proceeding or claim that We bring
+against you in response to a prior proceeding or claim initiated by
+you or your affiliate.
+
+'We' means each contributor to this repository as of the date of
+inclusion of this file, including subsidiaries of a corporate
+contributor.
+
+This work is available under a Creative Commons Attribution-ShareAlike
+4.0 International license (https://creativecommons.org/licenses/by-sa/4.0/).
diff --git a/CONTRIBUTING b/CONTRIBUTING
deleted file mode 100644
index 7bccd88d7e5..00000000000
--- a/CONTRIBUTING
+++ /dev/null
@@ -1,25 +0,0 @@
-        Developer's Certificate of Origin 1.1
-
-        By making a contribution to this project, I certify that:
-
-        (a) The contribution was created in whole or in part by me and I
-            have the right to submit it under the open source license
-            indicated in the file; or
-
-        (b) The contribution is based upon previous work that, to the best
-            of my knowledge, is covered under an appropriate open source
-            license and I have the right under that license to submit that
-            work with modifications, whether created in whole or in part
-            by me, under the same open source license (unless I am
-            permitted to submit under a different license), as indicated
-            in the file; or
-
-        (c) The contribution was provided directly to me by some other
-            person who certified (a), (b) or (c) and I have not modified
-            it.
-
-        (d) I understand and agree that this project and the contribution
-            are public and that a record of the contribution (including all
-            personal information I submit with it, including my sign-off) is
-            maintained indefinitely and may be redistributed consistent with
-            this project or the open source license(s) involved.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
new file mode 100644
index 00000000000..65fc3497104
--- /dev/null
+++ b/CONTRIBUTING.md
@@ -0,0 +1,114 @@
+# GlusterFS project Contribution guidelines
+
+## Development Workflow
+
+We follow most of the details as per the [document here](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests). If you are not aware of the github workflow, it is recommended to go through them before continuing here.
+
+
+#### Get the Repository setup
+
+0. Fork Repository
+   - Fork [GlusterFS repository](https://github.com/gluster/glusterfs/fork).
+
+1. Clone Repository
+   - Clone the glusterfs repo freshly from github using below steps.
+
+```
+   git clone git@github.com:${username}/glusterfs.git
+   cd glusterfs/
+   git remote add upstream git@github.com:gluster/glusterfs.git
+```
+
+About two tasks are one time for the life time. You can continue to use the same repository for all the work in future.
+
+#### Development & Other flows
+
+0. Issue:
+   - Make sure there is an issue filed for the task you are working on.
+   - If it is not filed, open the issue with all the description.
+   - If it is a bug fix, add label "Type:Bug".
+   - If it is an RFC, provide all the documentation, and request for "DocApproved", and "SpecApproved" label.
+
+1. Code:
+   - Start coding
+   - Build and test locally
+   - Make sure clang-format is installed and is run on the patch.
+
+2. Keep up-to-date
+   - GlusterFS is a large project with many developers, so there would be one or the other patch everyday.
+   - It is critical for developer to be up-to-date with `devel` repo to be Conflict-Free when PR is opened.
+   - Git provides many options to keep up-to-date, below is one of them
+```
+   git fetch upstream
+   git rebase upstream/devel
+```
+   - It is recommended you keep pushing to your repo every day, so you don't loose any work.
+     - It can be done by `./rfc.sh` (or `git push origin HEAD:issueNNN`)
+
+2. Commit Message / PR description:
+   - The name of the branch on your personal fork can start with issueNNNN, followed by anything of your choice.
+   - PRs continue to have the title of format "component: \<title\>", like it is practiced now.
+   - When you open a PR, having a reference Issue for the commit is mandatory in GlusterFS.
+   - Commit message can have, either `Fixes: #NNNN` or `Updates: #NNNN` in a separate line in the commit message.
+     - Here, NNNN is the Issue ID in glusterfs repository.
+   - Each commit needs the author to have the "Signed-off-by: Name \<email\>" line.
+     - Can do this by `-s` option for `git commit`.
+   - If the PR is not ready for review, apply the label `work-in-progress`.
+     - Check the availability of "Draft PR" is present for you, if yes, use that instead.
+
+3. Tests:
+   - All the required smoke tests would be auto-triggered.
+     - Developers get a chance to retrigger the smoke tests using **"/recheck smoke"** as comment.
+   - The "regression" tests would be triggered by a comment **"/run regression"** from developers in the [@gluster-maintainers](https://github.com/orgs/gluster/teams/gluster-maintainers) group.
+     - Ask for help as comment in PR if you have any questions about the process!
+
+4. Review Process:
+   - `+2` : is equivalent to "Approve" from the people in the maintainer's group.
+   - `+1` : can be given by a maintainer/reviewer by explicitly stating that in the comment.
+   - `-1` : provide details on required changes and pick "Request Changes" while submitting your review.
+   - `-2` : done by adding the `DO-NOT-MERGE` label.
+
+   - Any further discussions can happen as comments in the PR.
+
+5. Making changes:
+   - There are 2 approaches to submit changes done after addressing review comments.
+     - Commit changes as a new commit on top of the original commits in the branch, and push the changes to same branch (issueNNNN)
+     - Commit changes into the same commit with `--amend` option, and do a push to the same branch with `--force` option.
+
+6. Merging:
+   - GlusterFS project follows 'Squash and Merge' method
+     - This is mainly to preserve the historic Gerrit method of one patch in `git log` for one URL link.
+     - This also makes every merge a complete patch, which has passed all tests.
+   - The merging of the patch is expected to be done by the maintainers.
+     - It can be done when all the tests (smoke and regression) pass.
+     - When the PR has 'Approved' flag from corresponding maintainer.
+   - If you feel there is delay, feel free to add a comment, discuss the same in Slack channel, or send email.
+
+## By contributing to this project, the contributor would need to agree to below.
+
+### Developer's Certificate of Origin 1.1
+
+By making a contribution to this project, I certify that:
+
+(a) The contribution was created in whole or in part by me and I
+    have the right to submit it under the open source license
+    indicated in the file; or
+
+(b) The contribution is based upon previous work that, to the best
+    of my knowledge, is covered under an appropriate open source
+    license and I have the right under that license to submit that
+    work with modifications, whether created in whole or in part
+    by me, under the same open source license (unless I am
+    permitted to submit under a different license), as indicated
+    in the file; or
+
+(c) The contribution was provided directly to me by some other
+    person who certified (a), (b) or (c) and I have not modified
+    it.
+
+(d) I understand and agree that this project and the contribution
+    are public and that a record of the contribution (including all
+    personal information I submit with it, including my sign-off) is
+    maintained indefinitely and may be redistributed consistent with
+    this project or the open source license(s) involved.
+
diff --git a/INSTALL b/INSTALL
index 9709727c358..a56390e54fb 100644
--- a/INSTALL
+++ b/INSTALL
@@ -19,7 +19,6 @@ Installation Instructions
    georeplication       : yes
    Linux-AIO            : yes
    Enable Debug         : no
-   systemtap            : no
    Block Device xlator  : yes
    glupy                : yes
    Use syslog           : yes
@@ -45,4 +44,4 @@ Installation completed :-)
 Make sure your version is the latest from the release, and the one you 
 just installed :-)
 
-For more information on GlusterFS installation refer# http://gluster.org/community/documentation/index.php/Building_GlusterFS
+For more information on GlusterFS installation refer# http://docs.gluster.org/en/latest/Developer-guide/Building-GlusterFS/
diff --git a/MAINTAINERS b/MAINTAINERS
index 96aa5746764..953e8755fd9 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -11,7 +11,7 @@ consult gluster-devel@gluster.org and not any specific individual privately.
 
 Descriptions of section entries:
 
-	M: Mail patches to: FullName <address@domain>
+	M: Main contact that knows and takes care of this area
 	L: Mailing list that is relevant to this area
 	W: Web-page with status/info
 	Q: Patchwork web based patch tracking system site
@@ -46,181 +46,409 @@ Descriptions of section entries:
 	      matches patches or files that contain one or more of the words
 	      printk, pr_info or pr_err
 	   One regex pattern per line.  Multiple K: lines acceptable.
+        P: Peer for a component
 
 
 General Project Architects
 --------------------------
 M: Amar Tumballi <amarts@gmail.com>
-M: Anand Avati <avati@redhat.com>
-M: Jeff Darcy <jdarcy@redhat.com>
-M: Kaleb S. Keithley <kkeithle@redhat.com>
-M: Vijay Bellur <vbellur@redhat.com>
-
+M: Xavier Hernandez  <xhernandez@redhat.com>
+P: Pranith Karampuri <pranith.karampuri@phonepe.com>
+P: Atin Mukherjee <amukherj@redhat.com>
 
 xlators:
 --------
+Access Control List (ACL)
+M: Raghavendra Talur <rtalur@redhat.com>
+P: Jiffin Tony Thottan <jthottan@redhat.com>
+S: Maintained
+F: xlators/system/posix-acl/
+
+Arbiter
+M: Ravishankar N <ravishankar@redhat.com>
+P: Pranith Karampuri <pranith.karampuri@phonepe.com>
+S: Maintained
+F: xlators/features/arbiter/
+
 Automatic File Replication (AFR)
-M: Pranith Karampuri <pkarampu@redhat.com>
+M: Pranith Karampuri <pranith.karampuri@phonepe.com>
+M: Ravishankar N <ravishankar@redhat.com>
+P: Karthik US <ksubrahm@redhat.com>
 S: Maintained
 F: xlators/cluster/afr/
 
+Barrier
+M: Raghavendra Bhat <rabhat@redhat.com>
+P: Atin Mukherjee <amukherj@redhat.com>
+S: Maintained
+F: xlators/features/barrier
+
+BitRot
+M: Kotresh HR <khiremat@redhat.com>
+P: Raghavendra Bhat <rabhat@redhat.com>
+S: Maintained
+F: xlators/features/bit-rot/
+
+Changelog
+M: Aravinda V K <avishwan@redhat.com>
+P: Kotresh HR <khiremat@redhat.com>
+S: Maintained
+F: xlators/features/changelog/
+
 Distributed Hashing Table (DHT)
-M: Shishir Gowda <gowda.shishir@gmail.com>
+P: Susant Palai <spalai@redhat.com>
 S: Maintained
 F: xlators/cluster/dht/
 
-Performance
-M: Raghavendra Gowdappa <rgowdapp@redhat.com>
+Erasure Coding
+M: Pranith Karampuri <pranith.karampuri@phonepe.com>
+M: Xavier Hernandez  <xhernandez@redhat.com>
+P: Ashish Pandey <aspandey@redhat.com>
 S: Maintained
-F: xlators/performance/
+F: xlators/cluster/ec/
+
+Error-gen
+M: Raghavendra Talur <rtalur@redhat.com>
+S: Maintained
+F: xlators/debug/error-gen/
+
+FUSE Bridge
+M: Csaba Henk <chenk@redhat.com>
+P: Niels de Vos <ndevos@redhat.com>
+S: Maintained
+F: xlators/mount/
 
 Index
-M: Pranith Karampuri <pkarampu@redhat.com>
+M: Pranith Karampuri <pranith.karampuri@phonepe.com>
+P: Ravishankar N <ravishankar@redhat.com>
 S: Maintained
 F: xlators/features/index/
 
-Quota
-M: Krishnan Parthasarathi <kparthas@redhat.com>
-M: Raghavendra Gowdappa <rgowdapp@redhat.com>
+IO Cache
+P: Mohammed Rafi KC <rafi.kavungal@iternity.com>
 S: Maintained
-F: xlators/features/quota/
+F: xlators/performance/io-cache/
+
+IO Statistics
+M: Krutika Dhananjay <kdhananj@redhat.com>
+M: Shyam Ranganathan <srangana@redhat.com>
+S: Maintained
+F: xlators/debug/io-stats/
+
+IO threads
+M: Pranith Karampuri <pranith.karampuri@phonepe.com>
+P: Ravishankar N <ravishankar@redhat.com>
+S: Maintained
+F: xlators/performance/io-threads/
+
+Leases
+M: Poornima G <pgurusid@redhat.com>
+P: Niels de Vos <ndevos@redhat.com>
+P: Soumya Koduri <skoduri@redhat.com>
+S: Maintained
+F: xlators/features/leases/
+
+Locks
+M: Krutika Dhananjay <kdhananj@redhat.com>
+P: Xavier Hernandez  <xhernandez@redhat.com>
+S: Maintained
+F: xlators/features/locks/
 
 Marker
-M: Raghavendra Gowdappa <rgowdapp@redhat.com>
+M: Kotresh HR <khiremat@redhat.com>
 S: Maintained
 F: xlators/features/marker/
 
-Changelog
-M: Venky Shankar <vshankar@redhat.com>
+Meta
+M: Mohammed Rafi KC <rafi.kavungal@iternity.com>
 S: Maintained
-F: xlators/features/changelog/
+F: xlators/features/meta/
 
-Block Device
-M: M. Mohan Kumar <mohan@in.ibm.com>
+Metadata-cache
+M: Poornima G <pgurusid@redhat.com>
+P: Soumya Koduri <skoduri@redhat.com>
 S: Maintained
-F: xlators/storage/bd/
+F: xlators/performance/md-cache/
 
-FUSE Bridge
-M: Anand Avati <avati@redhat.com>
-M: Brian Foster <bfoster@redhat.com>
-M: Csaba Henk <chenk@redhat.com>
+Negative-lookup Cache
+M: Poornima G <pgurusid@redhat.com>
+P: Pranith Karampuri <pranith.karampuri@phonepe.com>
 S: Maintained
-F: xlators/mount/
+F: xlators/performance/nl-cache/
 
-NFS
-M: Niels de Vos <ndevos@redhat.com>
-S: Maintained
+gNFS
+M: Jiffin Tony Thottan <jthottan@redhat.com>
+P: Xie Changlong <xiechanglong@cmss.chinamobile.com>
+P: Amar Tumballi <amarts@gmail.com>
+S: Odd Fixes
 F: xlators/nfs/server/
 
-Locks
-M: Pranith Karampuri <pkarampu@redhat.com>
+Open-behind
 S: Maintained
-F: xlators/features/locks/
+F: xlators/performance/open-behind/
 
+Posix:
+M: Raghavendra Bhat <raghavendra@redhat.com>
+P: Kotresh HR <khiremat@redhat.com>
+P: Krutika Dhananjay <kdhananj@redhat.com>
+S: Maintained
+F: xlators/storage/posix/
+
+Quick-read
+S: Maintained
+F: xlators/performance/quick-read/
+
+Quota
+M: Shyamsundar Ranganathan <srangana@redhat.com>
+P: Hari Gowtham <hgowtham@redhat.com>
+S: Maintained
+F: xlators/features/quota/
+
+Read-ahead
+P: Csaba Henk <chenk@redhat.com>
+S: Maintained
+F: xlators/performance/read-ahead/
+
+Readdir-ahead
+S: Maintained
+F: xlators/performance/readdir-ahead/
+
+Sharding
+M: Krutika Dhananjay <kdhananj@redhat.com>
+P: Xavier Hernandez  <xhernandez@redhat.com>
+S: Maintained
+F: xlators/features/shard/
+
+Trash
+M: Anoop C S <anoopcs@redhat.com>
+M: Jiffin Tony Thottan <jthottan@redhat.com>
+S: Maintained
+F: xlators/features/trash/
+
+Upcall
+M: Poornima G <pgurusid@redhat.com>
+M: Soumya Koduri <skoduri@redhat.com>
+P: Niels de Vos <ndevos@redhat.com>
+S: Maintained
+F: xlators/features/upcall/
+
+Write-behind
+P: Csaba Henk <chenk@redhat.com>
+S: Maintained
+F: xlators/performance/write-behind/
+
+Write Once Read Many
+P: Karthik US <ksubrahm@redhat.com>
+S: Maintained
+F: xlators/features/read-only/
+
+Cloudsync
+M: Susant Kumar Palai <spalai@redhat.com>
+S: Maintained
+F: xlators/features/cloudsync/
 
 Other bits of code:
 -------------------
+
+Doc
+M: Humble Chirammal <hchiramm@redhat.com>
+M: Raghavendra Talur <rtalur@redhat.com>
+S: Maintained
+F: doc/
+
 Geo Replication
-M: Venky Shankar <vshankar@redhat.com>
+M: Aravinda V K <avishwan@redhat.com>
+M: Kotresh HR <khiremat@redhat.com>
+M: Sunny Kumar <sunkumar@redhat.com>
 S: Maintained
 F: geo-replication/
 
-Glupy
-M: Justin Clift <justin@gluster.org>
+Glusterfind
+M: Aravinda VK <avishwan@redhat.com>
 S: Maintained
-F: xlators/features/glupy/
+F: tools/glusterfind/
 
 libgfapi
-M: Anand Avati <avati@redhat.com>
+M: Niels de Vos <ndevos@redhat.com>
+P: Poornima G <pgurusid@redhat.com>
+P: Shyamsundar Ranganathan <srangana@redhat.com>
+P: Soumya Koduri <skoduri@redhat.com>
 S: Maintained
 F: api/
 
-Management Daemon
-M: Krishnan Parthasarathi <kparthas@redhat.com>
-M: Kaushal Madappa <kmadapp@redhat.com>
+libglusterfs
+M: Amar Tumballi <amarts@gmail.com>
+M: Xavier Hernandez  <xhernandez@redhat.com>
+M: Jeff Darcy <jeff@pl.atyp.us>
+P: Kaleb Keithley <kkeithle@redhat.com>
+P: Niels de Vos <ndevos@redhat.com>
+P: Pranith Karampuri <pranith.karampuri@phonepe.com>
+P: Shyamsundar Ranganathan <srangana@redhat.com>
+S: Maintained
+F: libglusterfs/
+
+xxhash
+M: Aravinda VK <avishwan@redhat.com>
+M: Kotresh HR <khiremat@redhat.com>
+P: Yaniv Kaul <ykaul@redhat.com>
+S: Maintained
+F: contrib/xxhash/
+T: https://github.com/Cyan4973/xxHash.git
+
+Management Daemon - glusterd
+M: Atin Mukherjee <amukherj@redhat.com>
+M: Mohit Agrawal <moagrawa@redhat.com>
+M: Sanju Rakonde <srakonde@redhat.com>
 S: Maintained
 F: cli/
-F: xlators/mgmt/
+F: xlators/mgmt/glusterd/
+
+Protocol
+M: Niels de Vos <ndevos@redhat.com>
+P: Mohammed Rafi KC <rafi.kavungal@iternity.com>
+S: Maintained
+F: xlators/protocol/
 
 Remote Procedure Call subsystem
-M: Raghavendra Gowdappa <rgowdapp@redhat.com>
-M: Amar Tumballi <amarts@gmail.com>
-M: Anand Avati <avati@redhat.com>
+P: Mohit Agrawal <moagrawa@redhat.com>
 S: Maintained
-F: rpc/
+F: rpc/rpc-lib/
+F: rpc/xdr/
 
+Snapshot
+M: Raghavendra Bhat <raghavendra@redhat.com>
+P: Mohammed Rafi KC <rafi.kavungal@iternity.com>
+P: Sunny Kumar <sunkumar@redhat.com>
+S: Maintained
+F: xlators/mgmt/glusterd/src/glusterd-snap*
+F: extras/snap-scheduler.py
+
+Socket subsystem
+P: Krutika Dhananjay <kdhananj@redhat.com>
+P: Milind Changire <mchangir@redhat.com>
+P: Mohammed Rafi KC <rafi.kavungal@iternity.com>
+P: Mohit Agrawal <moagrawa@redhat.com>
+S: Maintained
+F: rpc/rpc-transport/socket/
 
-Distribution Specific:
-----------------------
-Debian Packaging
-M: Patrick Matthäi <pmatthaei@debian.org>
-M: Louis Zuckerman <me@louiszuckerman.com>
+Testing - .t framework
+M: Raghavendra Talur <rtalur@redhat.com>
 S: Maintained
-W: http://packages.qa.debian.org/g/glusterfs.html
+F: tests/
 
-Fedora Packaging
-M: glusterfs-owner@fedoraproject.org
-M: Humble Chirammal <hchiramm@redhat.com>
-M: Kaleb Keithley <kkeithle@redhat.com>
-M: Lalatendu Mohanty <lmohanty@redhat.com>
+Utilities
+M: Aravinda VK <avishwan@redhat.com>
+P: Niels de Vos <ndevos@redhat.com>
+P: Raghavendra Talur <rtalur@redhat.com>
+P: Sachidanda Urs <surs@redhat.com>
+S: Maintained
+F: extras/
+
+Events APIs
+M: Aravinda VK <avishwan@redhat.com>
+S: Maintained
+F: events/
+F: libglusterfs/src/events*
+F: libglusterfs/src/eventtypes*
+F: extras/systemd/glustereventsd*
+
+Distribution Specific:
+----------------------
+Build:
 M: Niels de Vos <ndevos@redhat.com>
+M: Hari Gowtham <hgowtham@redhat.com>
+P: Anoop C S <anoopcs@redhat.com>
+P: Raghavendra Talur <rtalur@redhat.com>
+P: Rinku Kothiya <rkothiya@redhat.com>
 S: Maintained
-W: https://apps.fedoraproject.org/packages/glusterfs
-T: http://pkgs.fedoraproject.org/git/glusterfs.git
 
-FreeBSD port
-M: Harshavardhana <harsha@harshavardhana.net>
+Debian packages on download.gluster.org
+M: packaging@gluster.org
+M: Kaleb Keithley <kkeithle@redhat.com>
+P: Sheetal Pamecha <spamecha@redhat.com>
+P: Shwetha Acharya <sacharya@redhat.com>
 S: Maintained
+W: http://download.gluster.org/pub/gluster/glusterfs/LATEST/Debian/Debian.README
+T: https://github.com/gluster/glusterfs-debian.git
 
-MacOS X port
-M: Dennis Schafroth <dennis@schafroth.com>
-M: Harshavardhana <harsha@harshavardhana.net>
+OpenSuSE
+M: packaging@gluster.org
+M: Kaleb Keithley <kkeithle@redhat.com>
+P: Sheetal Pamecha <spamecha@redhat.com>
+P: Shwetha Acharya <sacharya@redhat.com>
 S: Maintained
+W: https://build.opensuse.org/repositories/home:glusterfs
+W: https://download.gluster.org/pub/gluster/glusterfs/LATEST/SuSE/SuSE.README
+T: https://github.com/gluster/glusterfs-suse.git
 
-NetBSD port
-M: Emmanuel Dreyfus <manu@netbsd.org>
+Packages for the CentOS Storage SIG
+M: centos-devel@centos.org
+M: Niels de Vos <ndevos@redhat.com>
+P: Kaleb Keithley <kkeithle@redhat.com>
 S: Maintained
-W: http://pkgsrc.se/filesystems/glusterfs
+W: https://wiki.centos.org/SpecialInterestGroup/Storage/Gluster
+T: https://github.com/CentOS-Storage-SIG/glusterfs.git
 
-Ubuntu Packaging
-M: Louis Zuckerman <me@louiszuckerman.com>
+Ubuntu PPA
+M: packaging@gluster.org
+M: Kaleb Keithley <kkeithle@redhat.com>
+P: Sheetal Pamecha <spamecha@redhat.com>
+P: Shwetha Acharya <sacharya@redhat.com>
 S: Maintained
+W: https://launchpad.net/~gluster
 W: http://download.gluster.org/pub/gluster/glusterfs/LATEST/Ubuntu/Ubuntu.README
-
+T: https://github.com/gluster/glusterfs-debian.git
 
 Related projects
 ----------------
-Gluster Openstack Swift
-M: Luis Pabon <lpabon@redhat.com>
+Gluster Block
+M: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
+M: Xiubo Li <xiubli@redhat.com>
 S: Maintained
-T: https://github.com/gluster/gluster-swift.git
+T: https://github.com/gluster/gluster-block.git
 
-GlusterFS Hadoop HCFS plugin
-M: Jay Vyas <jvyas@redhat.com>
+GlusterFS core-utils
+M: Anoop C S <anoopcs@redhat.com>
 S: Maintained
-T: https://github.com/gluster/glusterfs-hadoop.git
+T: https://github.com/gluster/glusterfs-coreutils.git
 
 NFS-Ganesha FSAL plugin
-M: Anand Subramanian <ansubram@redhat.com>
+M: Jiffin Tony Thottan <jthottan@redhat.com>
+M: Kaleb Keithley <kkeithle@redhat.com>
+M: Soumya Koduri <skoduri@redhat.com>
 S: Maintained
 T: git://github.com/nfs-ganesha/nfs-ganesha.git
 F: src/nfs-ganesha~/src/FSAL/FSAL_GLUSTER/
 
 QEMU integration
-M: Bharata B Rao <bharata@linux.vnet.ibm.com>
+M: Niels de Vos <ndevos@redhat.com>
+M: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
 S: Maintained
 T: git://git.qemu.org/qemu.git
 F: block/gluster.c
 
 Samba VFS plugin
+M: Anoop C S <anoopcs@redhat.com>
 M: Raghavendra Talur <rtalur@redhat.com>
-M: Chris Hertel <chertel@redhat.com>
-M: Jose Rivera <jrivera@redhat.com>
-M: Ira Cooper <icooper@redhat.com>
+M: Michael Adam <madam@redhat.com>
+M: Poornima G <pgurusid@redhat.com>
 S: Maintained
 T: git://git.samba.org/samba.git
 F: source3/modules/vfs_glusterfs.c
 
+Storhaug
+M: Jose A. Rivera <jarrpa@redhat.com>
+P: Kaleb Keithley <kkeithle@redhat.com>
+S: Maintained
+T: https://github.com/linux-ha-storage/storhaug.git
+
+Testing - Glusto-Tests
+M: Jonathan Holloway <jholloway@redhat.com>
+M: Vijay Bhaskar Reddy Avuthu <vavuthu@redhat.com>
+M: Akarsha Rai <akrai@redhat.com>
+S: Maintained
+T: https://github.com/gluster/glusto-tests.git
+
 Wireshark dissectors
 M: Niels de Vos <ndevos@redhat.com>
 S: Maintained
@@ -228,3 +456,55 @@ W: https://forge.gluster.org/wireshark
 T: http://code.wireshark.org/git/wireshark
 F: epan/dissectors/packet-gluster*
 
+Infrastructure
+--------------
+
+Platform
+M: Michael Scherer <misc@redhat.com>
+P: Shyamsundar Ranganathan <srangana@redhat.com>
+P: Amar Tumballi <amarts@gmail.com>
+
+Continuous Integration
+M: Michael Scherer <misc@redhat.com>
+M: Deepshikha Khandelwal <dkhandel@redhat.com>
+P: Niels de Vos <ndevos@redhat.com>
+
+Special Thanks
+--------------
+
+GlusterFS would not be possible without the contributions of:
+
+
+M: Vijay Bellur <vbellur@redhat.com>
+M: Jeff Darcy <jeff@pl.atyp.us>
+M: Shreyas Siravara <sshreyas@fb.com>
+M: Kaushal M <kaushal@redhat.com>
+M: Nigel Babu
+M: Prashanth Pai
+P: Sanoj Unnikrishnan
+P: Milind Changire <mchangir@redhat.com>
+P: Sunil Kumar Acharya <sheggodu@redhat.com>
+M: Samikshan Bairagya <samikshan@gmail.com>
+M: Chris Hertel
+M: M. Mohan Kumar <mohan@in.ibm.com>
+M: Shishir Gowda <gowda.shishir@gmail.com>
+M: Brian Foster <bfoster@redhat.com>
+M: Anand Avati <avati@cs.stanford.edu>
+M: Dennis Schafroth <dennis@schafroth.com>
+M: Harshavardhana <harsha@harshavardhana.net>
+M: Krishnan Parthasarathi
+M: Justin Clift <justin@gluster.org>
+M: Venky Shankar <vshankar@redhat.com>
+M: Shravan Chandrashekar <shravantc99@gmail.com>
+M: Joseph Fernandes
+M: Vijaikumar Mallikarjuna
+M: Anand Subramanian
+M: Bharata B Rao <bharata@linux.vnet.ibm.com>
+M: Rajesh Joseph
+M: Dan Lambright
+M: Jay Vyas
+M: Luis Pabon
+M: Ira Cooper
+M: Shwetha Panduranga
+M: Nithya Balachandran
+M: Raghavendra Gowdappa
diff --git a/Makefile.am b/Makefile.am
index 187bfd5ed16..98ea5c1038d 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -1,45 +1,55 @@
+SOURCES = site.h
+
 EXTRA_DIST = autogen.sh \
-	COPYING-GPLV2 COPYING-LGPLV3 \
-	INSTALL README AUTHORS THANKS NEWS \
+	COPYING-GPLV2 COPYING-LGPLV3 COMMITMENT \
+	INSTALL README.md AUTHORS THANKS NEWS \
 	glusterfs.spec glusterfs-api.pc.in libgfchangelog.pc.in \
 	run-tests.sh \
 	build-aux/pkg-version \
-	build-aux/xdrgen \
-        contrib/argp-standalone \
-        contrib/umountd \
+	contrib/umountd \
 	$(shell find $(top_srcdir)/tests -type f -print)
 
-SUBDIRS = $(ARGP_STANDALONE_DIR) libglusterfs rpc api xlators glusterfsd \
-	$(FUSERMOUNT_SUBDIR) doc extras cli heal @SYNCDAEMON_SUBDIR@ \
-	@UMOUNTD_SUBDIR@
+
+SUBDIRS = $(ARGP_STANDALONE_DIR) libglusterfs rpc libglusterd api \
+	glusterfsd xlators $(FUSERMOUNT_SUBDIR) doc extras cli heal \
+	@SYNCDAEMON_SUBDIR@ @UMOUNTD_SUBDIR@ tools events
 
 pkgconfigdir = @pkgconfigdir@
 pkgconfig_DATA = glusterfs-api.pc libgfchangelog.pc
 
-CLEANFILES =
+CLEANFILES = glusterfs-api.pc libgfchangelog.pc contrib/umountd/Makefile
+
+clean-local:
+	find . -name '*.o' -o -name '*.lo' -o -name '.Po' | xargs rm -f
 
 gitclean: distclean
 	find . -name Makefile.in -exec rm -f {} \;
-	find . -name Makefile -exec rm -f {} \;
 	find . -name mount.glusterfs -exec rm -f {} \;
+	find . -name .deps -o -name .libs | xargs rm -rf
 	rm -fr autom4te.cache
 	rm -f missing aclocal.m4 config.h.in config.guess config.sub ltmain.sh install-sh configure depcomp
-	-rm -fr $(CONTRIBDIR)/argp-standalone/autom4te.cache
-	-rm -f $(CONTRIBDIR)/argp-standalone/aclocal.m4
-	-rm -f $(CONTRIBDIR)/argp-standalone/config.h.in
-	-rm -f $(CONTRIBDIR)/argp-standalone/configure
-	-rm -f $(CONTRIBDIR)/argp-standalone/depcomp
-	-rm -f $(CONTRIBDIR)/argp-standalone/install-sh
-	-rm -f $(CONTRIBDIR)/argp-standalone/missing
 
+# dist-hook gets executed with 'make dist', this is the only target getting
+# executed, a dist-hook in other Makefile.am files seem to get ignored.
 dist-hook: gen-VERSION gen-ChangeLog
-.PHONY: gen-VERSION gen-ChangeLog
+	-rm -fr $(distdir)/contrib/umountd/.deps
+	-rm -f $(distdir)/events/src/eventtypes.py
+	-rm -f $(distdir)/tests/env.rc
+	-cp -f $(top_srcdir)/build-aux/config.sub.dist $(distdir)/config.sub
+	-cp -f $(top_srcdir)/build-aux/config.guess.dist $(distdir)/config.guess
+
+.PHONY: gen-VERSION gen-ChangeLog clang-check
+
+clang-check:
+	@$(top_srcdir)/extras/clang-checker.sh
 
 gen-ChangeLog:
 	(cd $(srcdir) && git diff && echo ===== git log ==== && git log) > $(distdir)/ChangeLog
 
+.PHONY : gen-VERSION
 gen-VERSION:
-	if test -d .git; then                           \
-	  $(top_srcdir)/build-aux/pkg-version --full    \
-	    > $(distdir)/VERSION;                       \
+	if test -d $(top_srcdir)/.git; then             \
+	  cd $(top_srcdir);                             \
+	  ./build-aux/pkg-version --full                \
+	    > $(abs_top_builddir)/$(distdir)/VERSION;   \
 	fi
diff --git a/README b/README
deleted file mode 100644
index ce648743ee9..00000000000
--- a/README
+++ /dev/null
@@ -1 +0,0 @@
-For more info, please visit http://www.gluster.org/.
diff --git a/README.md b/README.md
new file mode 100644
index 00000000000..9d68e033782
--- /dev/null
+++ b/README.md
@@ -0,0 +1,46 @@
+# Gluster
+  Gluster is a software defined distributed storage that can scale to several
+  petabytes. It provides interfaces for object, block and file storage.
+
+## Development
+  The development workflow is documented in [Contributors guide](CONTRIBUTING.md)
+
+## Documentation
+  The Gluster documentation can be found at [Gluster Docs](http://docs.gluster.org).
+
+## Deployment
+  Quick instructions to build and install can be found in [INSTALL](INSTALL) file.
+
+## Testing
+
+  GlusterFS source contains some functional tests under `tests/` directory. All
+  these tests are run against every patch submitted for review. If you want your
+  patch to be tested, please add a `.t` test file as part of your patch submission.
+  You can also submit a patch to only add a `.t` file for the test case you are
+  aware of.
+
+  To run these tests, on your test-machine, just run `./run-tests.sh`. Don't run
+  this on a machine where you have 'production' glusterfs is running, as it would
+  blindly kill all gluster processes in each runs.
+
+  If you are sending a patch, and want to validate one or few specific tests, then
+  run a single test by running the below command.
+
+```
+  bash# /bin/bash ${path_to_gluster}/tests/basic/rpc-coverage.t
+```
+
+  You can also use `prove` tool if available in your machine, as follows.
+
+```
+  bash# prove -vmfe '/bin/bash' ${path_to_gluster}/tests/basic/rpc-coverage.t
+```
+
+
+## Maintainers
+  The list of Gluster maintainers is available in [MAINTAINERS](MAINTAINERS) file.
+
+## License
+  Gluster is dual licensed under [GPLV2](COPYING-GPLV2) and [LGPLV3+](COPYING-LGPLV3).
+
+  Please visit the [Gluster Home Page](http://www.gluster.org/) to find out more about Gluster.
diff --git a/api/examples/getvolfile.py b/api/examples/getvolfile.py
index 184586c632d..3b2c8ab5a15 100755
--- a/api/examples/getvolfile.py
+++ b/api/examples/getvolfile.py
@@ -1,43 +1,45 @@
-#!/usr/bin/python
+#!/usr/bin/python3
 
+from __future__ import print_function
 import ctypes
 import ctypes.util
 
-api = ctypes.CDLL(ctypes.util.find_library("gfapi"))
+api = ctypes.CDLL("libgfapi.so")
 api.glfs_get_volfile.argtypes = [ctypes.c_void_p,
                                  ctypes.c_void_p,
                                  ctypes.c_ulong]
-api.glfs_get_volfile.restype = ctypes.c_long;
+api.glfs_get_volfile.restype = ctypes.c_long
 
-def get_volfile (host, volume):
-	# This is set to a large value to exercise the "buffer not big enough"
-	# path.  More realistically, you'd just start with a huge buffer.
-	BUF_LEN = 0
-	fs = api.glfs_new(volume)
-	#api.glfs_set_logging(fs,"/dev/stderr",7)
-	api.glfs_set_volfile_server(fs,"tcp",host,24007)
-	api.glfs_init(fs)
-	vbuf = ctypes.create_string_buffer(BUF_LEN)
-	vlen = api.glfs_get_volfile(fs,vbuf,BUF_LEN)
-	if vlen < 0:
-		vlen = BUF_LEN - vlen
-		vbuf = ctypes.create_string_buffer(vlen)
-		vlen = api.glfs_get_volfile(fs,vbuf,vlen)
-	api.glfs_fini(fs)
-	if vlen <= 0:
-		return vlen
-	return vbuf.value[:vlen]
+
+def get_volfile(host, volume):
+    # This is set to a large value to exercise the "buffer not big enough"
+    # path.  More realistically, you'd just start with a huge buffer.
+    BUF_LEN = 0
+    fs = api.glfs_new(volume)
+    # api.glfs_set_logging(fs,"/dev/stderr",7)
+    api.glfs_set_volfile_server(fs, "tcp", host, 24007)
+    api.glfs_init(fs)
+    vbuf = ctypes.create_string_buffer(BUF_LEN)
+    vlen = api.glfs_get_volfile(fs, vbuf, BUF_LEN)
+    if vlen < 0:
+        vlen = BUF_LEN - vlen
+        vbuf = ctypes.create_string_buffer(vlen)
+        vlen = api.glfs_get_volfile(fs, vbuf, vlen)
+    api.glfs_fini(fs)
+    if vlen <= 0:
+        return vlen
+    return vbuf.value[:vlen]
 
 if __name__ == "__main__":
-	import sys
+    import sys
 
-	try:
-		res = apply(get_volfile,sys.argv[1:3])
-	except:
-		print "fetching volfile failed (volume not started?)"
+    try:
+        res = get_volfile(*sys.argv[1:3])
+    except:
+        print("fetching volfile failed (volume not started?)")
 
-	try:
-		for line in res.split('\n'):
-			print line
-	except:
-		print "bad return value %s" % res
+    try:
+        for line in res.split('\n'):
+            print(line)
+    except:
+        print("bad return value %s" % res)
diff --git a/api/examples/glfsxmp.c b/api/examples/glfsxmp.c
index 7180a997f13..a55616ef739 100644
--- a/api/examples/glfsxmp.c
+++ b/api/examples/glfsxmp.c
@@ -1,1598 +1,1811 @@
 #include <stdio.h>
 #include <stdlib.h>
 #include <errno.h>
-#include "api/glfs.h"
-#include "api/glfs-handles.h"
+#include <glusterfs/api/glfs.h>
+#include <glusterfs/api/glfs-handles.h>
 #include <string.h>
 #include <time.h>
 
+#define TEST_STR_LEN 2048
 
 int
-test_dirops (glfs_t *fs)
+test_dirops(glfs_t *fs)
 {
-        glfs_fd_t *fd = NULL;
-        char buf[512];
-        struct dirent *entry = NULL;
-
-        fd = glfs_opendir (fs, "/");
-        if (!fd) {
-                fprintf (stderr, "/: %s\n", strerror (errno));
-                return -1;
-        }
-
-        fprintf (stderr, "Entries:\n");
-        while (glfs_readdir_r (fd, (struct dirent *)buf, &entry), entry) {
-                fprintf (stderr, "%s: %lu\n", entry->d_name, glfs_telldir (fd));
-        }
-
-        glfs_closedir (fd);
-        return 0;
+    glfs_fd_t *fd = NULL;
+    char buf[512];
+    struct dirent *entry = NULL;
+
+    fd = glfs_opendir(fs, "/");
+    if (!fd) {
+        fprintf(stderr, "/: %s\n", strerror(errno));
+        return -1;
+    }
+
+    fprintf(stderr, "Entries:\n");
+    while (glfs_readdir_r(fd, (struct dirent *)buf, &entry), entry) {
+        fprintf(stderr, "%s: %lu\n", entry->d_name, glfs_telldir(fd));
+    }
+
+    glfs_closedir(fd);
+    return 0;
 }
 
-
 int
-test_xattr (glfs_t *fs)
+test_xattr(glfs_t *fs)
 {
-        char *filename = "/filename2";
-        char buf[512];
-        char *ptr;
-        int ret;
-
-        ret = glfs_setxattr (fs, filename, "user.testkey", "testval", 8, 0);
-        fprintf (stderr, "setxattr(%s): %d (%s)\n", filename, ret,
-                 strerror (errno));
-
-        ret = glfs_setxattr (fs, filename, "user.testkey2", "testval", 8, 0);
-        fprintf (stderr, "setxattr(%s): %d (%s)\n", filename, ret,
-                 strerror (errno));
-
-        ret = glfs_listxattr (fs, filename, buf, 512);
-        fprintf (stderr, "listxattr(%s): %d (%s)\n", filename, ret,
-                 strerror (errno));
-        if (ret < 0)
-                return -1;
-
-        for (ptr = buf; ptr < buf + ret; ptr++) {
-                printf ("key=%s\n", ptr);
-                ptr += strlen (ptr);
-        }
-
-        return 0;
+    char *filename = "/filename2";
+    char *linkfile = "/linkfile";
+    glfs_fd_t *fd = NULL;
+    char buf[512];
+    char *ptr;
+    int ret;
+
+    ret = glfs_setxattr(fs, filename, "user.testkey", "testval", 8, 0);
+    fprintf(stderr, "setxattr(%s): %d (%s)\n", filename, ret, strerror(errno));
+
+    ret = glfs_setxattr(fs, filename, "user.testkey2", "testval", 8, 0);
+    fprintf(stderr, "setxattr(%s): %d (%s)\n", filename, ret, strerror(errno));
+
+    ret = glfs_getxattr(fs, filename, "user.testkey", buf, 512);
+    fprintf(stderr, "getxattr(%s): %d (%s)\n", filename, ret, strerror(errno));
+    if (ret < 0)
+        return -1;
+
+    ret = glfs_listxattr(fs, filename, buf, 512);
+    fprintf(stderr, "listxattr(%s): %d (%s)\n", filename, ret, strerror(errno));
+    if (ret < 0)
+        return -1;
+
+    ret = glfs_symlink(fs, "filename", linkfile);
+    fprintf(stderr, "symlink(%s %s): %s\n", filename, linkfile,
+            strerror(errno));
+    if (ret < 0)
+        return -1;
+
+    ret = glfs_readlink(fs, linkfile, buf, 512);
+    fprintf(stderr, "readlink(%s) : %d (%s)\n", filename, ret, strerror(errno));
+    if (ret < 0)
+        return -1;
+
+    ret = glfs_lsetxattr(fs, filename, "user.testkey3", "testval", 8, 0);
+    fprintf(stderr, "lsetxattr(%s) : %d (%s)\n", linkfile, ret,
+            strerror(errno));
+    if (ret < 0)
+        return -1;
+
+    ret = glfs_llistxattr(fs, linkfile, buf, 512);
+    fprintf(stderr, "llistxattr(%s): %d (%s)\n", filename, ret,
+            strerror(errno));
+    if (ret < 0)
+        return -1;
+
+    ret = glfs_lgetxattr(fs, filename, "user.testkey3", buf, 512);
+    fprintf(stderr, "lgetxattr(%s): %d (%s)\n", linkfile, ret, strerror(errno));
+    if (ret < 0)
+        return -1;
+
+    for (ptr = buf; ptr < buf + ret; ptr++) {
+        printf("key=%s\n", ptr);
+        ptr += strlen(ptr);
+    }
+
+    ret = glfs_removexattr(fs, filename, "user.testkey2");
+    fprintf(stderr, "removexattr(%s): %d (%s)\n", filename, ret,
+            strerror(errno));
+
+    fd = glfs_open(fs, filename, O_RDWR);
+    fprintf(stderr, "open(%s): (%p) %s\n", filename, fd, strerror(errno));
+
+    ret = glfs_fsetxattr(fd, "user.testkey2", "testval", 8, 0);
+    fprintf(stderr, "fsetxattr(%s): %d (%s)\n", filename, ret, strerror(errno));
+
+    ret = glfs_fgetxattr(fd, "user.testkey2", buf, 512);
+    fprintf(stderr, "fgetxattr(%s): %d (%s)\n", filename, ret, strerror(errno));
+
+    ret = glfs_flistxattr(fd, buf, 512);
+    fprintf(stderr, "flistxattr(%s): %d (%s)\n", filename, ret,
+            strerror(errno));
+    if (ret < 0)
+        return -1;
+
+    for (ptr = buf; ptr < buf + ret; ptr++) {
+        printf("key=%s\n", ptr);
+        ptr += strlen(ptr);
+    }
+
+    ret = glfs_fremovexattr(fd, "user.testkey2");
+    fprintf(stderr, "fremovexattr(%s): %d (%s)\n", filename, ret,
+            strerror(errno));
+
+    glfs_close(fd);
+
+    return 0;
 }
 
-
 int
-test_chdir (glfs_t *fs)
+test_chdir(glfs_t *fs)
 {
-        int ret = -1;
-        char *topdir = "/topdir";
-        char *linkdir = "/linkdir";
-        char *subdir = "./subdir";
-        char *respath = NULL;
-        char pathbuf[4096];
-
-        ret = glfs_mkdir (fs, topdir, 0755);
-        if (ret) {
-                fprintf (stderr, "mkdir(%s): %s\n", topdir, strerror (errno));
-                return -1;
-        }
-
-        respath = glfs_getcwd (fs, pathbuf, 4096);
-        fprintf (stdout, "getcwd() = %s\n", respath);
-
-        ret = glfs_symlink (fs, topdir, linkdir);
-        if (ret) {
-                fprintf (stderr, "symlink(%s, %s): %s\n", topdir, linkdir, strerror (errno));
-                return -1;
-        }
-
-        ret = glfs_chdir (fs, linkdir);
-        if (ret) {
-                fprintf (stderr, "chdir(%s): %s\n", linkdir, strerror (errno));
-                return -1;
-        }
-
-        respath = glfs_getcwd (fs, pathbuf, 4096);
-        fprintf (stdout, "getcwd() = %s\n", respath);
-
-        respath = glfs_realpath (fs, subdir, pathbuf);
-        if (respath) {
-                fprintf (stderr, "realpath(%s) worked unexpectedly: %s\n", subdir, respath);
-                return -1;
-        }
-
-        ret = glfs_mkdir (fs, subdir, 0755);
-        if (ret) {
-                fprintf (stderr, "mkdir(%s): %s\n", subdir, strerror (errno));
-                return -1;
-        }
-
-        respath = glfs_realpath (fs, subdir, pathbuf);
-        if (!respath) {
-                fprintf (stderr, "realpath(%s): %s\n", subdir, strerror (errno));
-        } else {
-                fprintf (stdout, "realpath(%s) = %s\n", subdir, respath);
-        }
-
-        ret = glfs_chdir (fs, subdir);
-        if (ret) {
-                fprintf (stderr, "chdir(%s): %s\n", subdir, strerror (errno));
-                return -1;
-        }
-
-        respath = glfs_getcwd (fs, pathbuf, 4096);
-        fprintf (stdout, "getcwd() = %s\n", respath);
-
-        respath = glfs_realpath (fs, "/linkdir/subdir", pathbuf);
-        if (!respath) {
-                fprintf (stderr, "realpath(/linkdir/subdir): %s\n", strerror (errno));
-        } else {
-                fprintf (stdout, "realpath(/linkdir/subdir) = %s\n", respath);
-        }
-
-        return 0;
+    int ret = -1;
+    char *dir = "/dir";
+    char *topdir = "/topdir";
+    char *linkdir = "/linkdir";
+    char *linkdir2 = "/linkdir2";
+    char *subdir = "./subdir";
+    char *respath = NULL;
+    char pathbuf[4096];
+
+    ret = glfs_mkdir(fs, topdir, 0755);
+    fprintf(stderr, "mkdir(%s): %s\n", topdir, strerror(errno));
+    if (ret)
+        return -1;
+
+    ret = glfs_mkdir(fs, dir, 0755);
+    fprintf(stderr, "mkdir(%s): %s\n", dir, strerror(errno));
+    if (ret)
+        return -1;
+
+    respath = glfs_getcwd(fs, pathbuf, 4096);
+    fprintf(stdout, "getcwd() = %s\n", respath);
+
+    ret = glfs_symlink(fs, "topdir", linkdir);
+    if (ret) {
+        fprintf(stderr, "symlink(%s, %s): %s\n", topdir, linkdir,
+                strerror(errno));
+        return -1;
+    }
+
+    ret = glfs_chdir(fs, linkdir);
+    if (ret) {
+        fprintf(stderr, "chdir(%s): %s\n", linkdir, strerror(errno));
+        return -1;
+    }
+
+    respath = glfs_getcwd(fs, pathbuf, 4096);
+    fprintf(stdout, "getcwd() = %s\n", respath);
+
+    respath = glfs_realpath(fs, subdir, pathbuf);
+    if (respath) {
+        fprintf(stderr, "realpath(%s) worked unexpectedly: %s\n", subdir,
+                respath);
+        return -1;
+    }
+
+    ret = glfs_mkdir(fs, subdir, 0755);
+    if (ret) {
+        fprintf(stderr, "mkdir(%s): %s\n", subdir, strerror(errno));
+        return -1;
+    }
+
+    respath = glfs_realpath(fs, subdir, pathbuf);
+    if (!respath) {
+        fprintf(stderr, "realpath(%s): %s\n", subdir, strerror(errno));
+    } else {
+        fprintf(stdout, "realpath(%s) = %s\n", subdir, respath);
+    }
+
+    ret = glfs_chdir(fs, subdir);
+    if (ret) {
+        fprintf(stderr, "chdir(%s): %s\n", subdir, strerror(errno));
+        return -1;
+    }
+
+    respath = glfs_getcwd(fs, pathbuf, 4096);
+    fprintf(stdout, "getcwd() = %s\n", respath);
+
+    respath = glfs_realpath(fs, "/linkdir/subdir", pathbuf);
+    if (!respath) {
+        fprintf(stderr, "realpath(/linkdir/subdir): %s\n", strerror(errno));
+    } else {
+        fprintf(stdout, "realpath(/linkdir/subdir) = %s\n", respath);
+    }
+
+    return 0;
 }
 
 #ifdef DEBUG
 static void
-peek_stat (struct stat *sb)
+peek_stat(struct stat *sb)
 {
-        printf ("Dumping stat information:\n");
-        printf ("File type:                ");
-
-        switch (sb->st_mode & S_IFMT) {
-                case S_IFBLK:  printf ("block device\n");            break;
-                case S_IFCHR:  printf ("character device\n");        break;
-                case S_IFDIR:  printf ("directory\n");               break;
-                case S_IFIFO:  printf ("FIFO/pipe\n");               break;
-                case S_IFLNK:  printf ("symlink\n");                 break;
-                case S_IFREG:  printf ("regular file\n");            break;
-                case S_IFSOCK: printf ("socket\n");                  break;
-                default:       printf ("unknown?\n");                break;
-        }
-
-        printf ("I-node number:            %ld\n", (long) sb->st_ino);
-
-        printf ("Mode:                     %lo (octal)\n",
-                (unsigned long) sb->st_mode);
-
-        printf ("Link count:               %ld\n", (long) sb->st_nlink);
-        printf ("Ownership:                UID=%ld   GID=%ld\n",
-                (long) sb->st_uid, (long) sb->st_gid);
-
-        printf ("Preferred I/O block size: %ld bytes\n",
-                (long) sb->st_blksize);
-        printf ("File size:                %lld bytes\n",
-                (long long) sb->st_size);
-        printf ("Blocks allocated:         %lld\n",
-                (long long) sb->st_blocks);
-
-        printf ("Last status change:       %s", ctime(&sb->st_ctime));
-        printf ("Last file access:         %s", ctime(&sb->st_atime));
-        printf ("Last file modification:   %s", ctime(&sb->st_mtime));
-
-        return;
+    printf("Dumping stat information:\n");
+    printf("File type:                ");
+
+    switch (sb->st_mode & S_IFMT) {
+        case S_IFBLK:
+            printf("block device\n");
+            break;
+        case S_IFCHR:
+            printf("character device\n");
+            break;
+        case S_IFDIR:
+            printf("directory\n");
+            break;
+        case S_IFIFO:
+            printf("FIFO/pipe\n");
+            break;
+        case S_IFLNK:
+            printf("symlink\n");
+            break;
+        case S_IFREG:
+            printf("regular file\n");
+            break;
+        case S_IFSOCK:
+            printf("socket\n");
+            break;
+        default:
+            printf("unknown?\n");
+            break;
+    }
+
+    printf("I-node number:            %ld\n", (long)sb->st_ino);
+
+    printf("Mode:                     %lo (octal)\n",
+           (unsigned long)sb->st_mode);
+
+    printf("Link count:               %ld\n", (long)sb->st_nlink);
+    printf("Ownership:                UID=%ld   GID=%ld\n", (long)sb->st_uid,
+           (long)sb->st_gid);
+
+    printf("Preferred I/O block size: %ld bytes\n", (long)sb->st_blksize);
+    printf("File size:                %lld bytes\n", (long long)sb->st_size);
+    printf("Blocks allocated:         %lld\n", (long long)sb->st_blocks);
+
+    printf("Last status change:       %s", ctime(&sb->st_ctime));
+    printf("Last file access:         %s", ctime(&sb->st_atime));
+    printf("Last file modification:   %s", ctime(&sb->st_mtime));
+
+    return;
 }
 
 static void
-peek_handle (unsigned char *glid)
+peek_handle(unsigned char *glid)
 {
-        int i;
+    int i;
 
-        for (i = 0; i < GFAPI_HANDLE_LENGTH; i++)
-        {
-                printf (":%02x:", glid[i]);
-        }
-        printf ("\n");
+    for (i = 0; i < GFAPI_HANDLE_LENGTH; i++) {
+        printf(":%02x:", glid[i]);
+    }
+    printf("\n");
 }
-#else /* DEBUG */
+#else  /* DEBUG */
 static void
-peek_stat (struct stat *sb)
+peek_stat(struct stat *sb)
 {
-        return;
+    return;
 }
 
 static void
-peek_handle (unsigned char *id)
+peek_handle(unsigned char *id)
 {
-        return;
+    return;
 }
 #endif /* DEBUG */
 
-glfs_t    *fs = NULL;
-char      *full_parent_name = "/testdir", *parent_name = "testdir";
+glfs_t *fs = NULL;
+char *full_parent_name = "/testdir", *parent_name = "testdir";
 
 void
-test_h_unlink (void)
+test_h_unlink(void)
 {
-        char               *my_dir = "unlinkdir";
-        char               *my_file = "file.txt";
-        char               *my_subdir = "dir1";
-        struct glfs_object *parent = NULL, *leaf = NULL, *dir = NULL,
-                           *subdir = NULL, *subleaf = NULL;
-        struct stat         sb;
-        int                 ret;
-
-        printf ("glfs_h_unlink tests: In Progress\n");
-
-        /* Prepare tests */
-        parent = glfs_h_lookupat (fs, NULL, full_parent_name, &sb);
-        if (parent == NULL) {
-                fprintf (stderr, "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
-                         full_parent_name, NULL, strerror (errno));
-                printf ("glfs_h_lookupat tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        dir = glfs_h_mkdir (fs, parent, my_dir, 0644, &sb);
-        if (dir == NULL) {
-                fprintf (stderr, "glfs_h_mkdir: error creating %s: from (%p),%s\n",
-                         my_dir, parent, strerror (errno));
-                printf ("glfs_h_unlink tests: FAILED\n");
-                goto out;
-        }
-
-        leaf = glfs_h_creat (fs, dir, my_file, O_CREAT, 0644, &sb);
-        if (leaf == NULL) {
-                fprintf (stderr, "glfs_h_creat: error creating %s: from (%p),%s\n",
-                         my_file, dir, strerror (errno));
-                printf ("glfs_h_unlink tests: FAILED\n");
-                goto out;
-        }
-
-        subdir = glfs_h_mkdir (fs, dir, my_subdir, 0644, &sb);
-        if (subdir == NULL) {
-                fprintf (stderr, "glfs_h_mkdir: error creating %s: from (%p),%s\n",
-                         my_subdir, dir, strerror (errno));
-                printf ("glfs_h_unlink tests: FAILED\n");
-                goto out;
-        }
-
-        subleaf = glfs_h_creat (fs, subdir, my_file, O_CREAT, 0644, &sb);
-        if (subleaf == NULL) {
-                fprintf (stderr, "glfs_h_creat: error creating %s: from (%p),%s\n",
-                         my_file, subdir, strerror (errno));
-                printf ("glfs_h_unlink tests: FAILED\n");
-                goto out;
-        }
-
-        /* unlink non empty directory */
-        ret = glfs_h_unlink (fs, dir, my_subdir);
-        if ((ret && errno != ENOTEMPTY) || (ret == 0)) {
-                fprintf (stderr, "glfs_h_unlink: error unlinking %s: it is non empty: %s\n",
-                         my_subdir, strerror (errno));
-                printf ("glfs_h_unlink tests: FAILED\n");
-                goto out;
-        }
-
-        /* unlink regular file */
-        ret = glfs_h_unlink (fs, subdir, my_file);
-        if (ret) {
-                fprintf (stderr, "glfs_h_unlink: error unlinking %s: from (%p),%s\n",
-                         my_file, subdir, strerror (errno));
-                printf ("glfs_h_unlink tests: FAILED\n");
-                goto out;
-        }
-
-        /* unlink directory */
-        ret = glfs_h_unlink (fs, dir, my_subdir);
-        if (ret) {
-                fprintf (stderr, "glfs_h_unlink: error unlinking %s: from (%p),%s\n",
-                         my_subdir, dir, strerror (errno));
-                printf ("glfs_h_unlink tests: FAILED\n");
-                goto out;
-        }
-
-        /* unlink regular file */
-        ret = glfs_h_unlink (fs, dir, my_file);
-        if (ret) {
-                fprintf (stderr, "glfs_h_unlink: error unlinking %s: from (%p),%s\n",
-                         my_file, dir, strerror (errno));
-                printf ("glfs_h_unlink tests: FAILED\n");
-                goto out;
-        }
-
-        /* unlink non-existent regular file */
-        ret = glfs_h_unlink (fs, dir, my_file);
-        if ((ret && errno != ENOENT) || (ret == 0)) {
-                fprintf (stderr, "glfs_h_unlink: error unlinking non-existent %s: invalid errno ,%d, %s\n",
-                         my_file, ret, strerror (errno));
-                printf ("glfs_h_unlink tests: FAILED\n");
-                goto out;
-        }
-
-        /* unlink non-existent directory */
-        ret = glfs_h_unlink (fs, dir, my_subdir);
-        if ((ret && errno != ENOENT) || (ret == 0)) {
-                fprintf (stderr, "glfs_h_unlink: error unlinking non-existent %s:  invalid errno ,%d, %s\n",
-                         my_subdir, ret, strerror (errno));
-                printf ("glfs_h_unlink tests: FAILED\n");
-                goto out;
-        }
-
-        /* unlink directory */
-        ret = glfs_h_unlink (fs, parent, my_dir);
-        if (ret) {
-                fprintf (stderr, "glfs_h_unlink: error unlinking %s: from (%p),%s\n",
-                         my_dir, dir, strerror (errno));
-                printf ("glfs_h_unlink tests: FAILED\n");
-                goto out;
-        }
-
-        printf ("glfs_h_unlink tests: PASSED\n");
+    char *my_dir = "unlinkdir";
+    char *my_file = "file.txt";
+    char *my_subdir = "dir1";
+    struct glfs_object *parent = NULL, *leaf = NULL, *dir = NULL,
+                       *subdir = NULL, *subleaf = NULL;
+    struct stat sb;
+    int ret;
+
+    printf("glfs_h_unlink tests: In Progress\n");
+
+    /* Prepare tests */
+    parent = glfs_h_lookupat(fs, NULL, full_parent_name, &sb, 0);
+    if (parent == NULL) {
+        fprintf(stderr,
+                "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
+                full_parent_name, NULL, strerror(errno));
+        printf("glfs_h_lookupat tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    dir = glfs_h_mkdir(fs, parent, my_dir, 0755, &sb);
+    if (dir == NULL) {
+        fprintf(stderr, "glfs_h_mkdir: error creating %s: from (%p),%s\n",
+                my_dir, parent, strerror(errno));
+        printf("glfs_h_unlink tests: FAILED\n");
+        goto out;
+    }
+
+    leaf = glfs_h_creat(fs, dir, my_file, O_CREAT, 0644, &sb);
+    if (leaf == NULL) {
+        fprintf(stderr, "glfs_h_creat: error creating %s: from (%p),%s\n",
+                my_file, dir, strerror(errno));
+        printf("glfs_h_unlink tests: FAILED\n");
+        goto out;
+    }
+
+    subdir = glfs_h_mkdir(fs, dir, my_subdir, 0755, &sb);
+    if (subdir == NULL) {
+        fprintf(stderr, "glfs_h_mkdir: error creating %s: from (%p),%s\n",
+                my_subdir, dir, strerror(errno));
+        printf("glfs_h_unlink tests: FAILED\n");
+        goto out;
+    }
+
+    subleaf = glfs_h_creat(fs, subdir, my_file, O_CREAT, 0644, &sb);
+    if (subleaf == NULL) {
+        fprintf(stderr, "glfs_h_creat: error creating %s: from (%p),%s\n",
+                my_file, subdir, strerror(errno));
+        printf("glfs_h_unlink tests: FAILED\n");
+        goto out;
+    }
+
+    /* unlink non empty directory */
+    ret = glfs_h_unlink(fs, dir, my_subdir);
+    if ((ret && errno != ENOTEMPTY) || (ret == 0)) {
+        fprintf(stderr,
+                "glfs_h_unlink: error unlinking %s: it is non empty: %s\n",
+                my_subdir, strerror(errno));
+        printf("glfs_h_unlink tests: FAILED\n");
+        goto out;
+    }
+
+    /* unlink regular file */
+    ret = glfs_h_unlink(fs, subdir, my_file);
+    if (ret) {
+        fprintf(stderr, "glfs_h_unlink: error unlinking %s: from (%p),%s\n",
+                my_file, subdir, strerror(errno));
+        printf("glfs_h_unlink tests: FAILED\n");
+        goto out;
+    }
+
+    /* unlink directory */
+    ret = glfs_h_unlink(fs, dir, my_subdir);
+    if (ret) {
+        fprintf(stderr, "glfs_h_unlink: error unlinking %s: from (%p),%s\n",
+                my_subdir, dir, strerror(errno));
+        printf("glfs_h_unlink tests: FAILED\n");
+        goto out;
+    }
+
+    /* unlink regular file */
+    ret = glfs_h_unlink(fs, dir, my_file);
+    if (ret) {
+        fprintf(stderr, "glfs_h_unlink: error unlinking %s: from (%p),%s\n",
+                my_file, dir, strerror(errno));
+        printf("glfs_h_unlink tests: FAILED\n");
+        goto out;
+    }
+
+    /* unlink non-existent regular file */
+    ret = glfs_h_unlink(fs, dir, my_file);
+    if ((ret && errno != ENOENT) || (ret == 0)) {
+        fprintf(stderr,
+                "glfs_h_unlink: error unlinking non-existent %s: invalid errno "
+                ",%d, %s\n",
+                my_file, ret, strerror(errno));
+        printf("glfs_h_unlink tests: FAILED\n");
+        goto out;
+    }
+
+    /* unlink non-existent directory */
+    ret = glfs_h_unlink(fs, dir, my_subdir);
+    if ((ret && errno != ENOENT) || (ret == 0)) {
+        fprintf(stderr,
+                "glfs_h_unlink: error unlinking non-existent %s:  invalid "
+                "errno ,%d, %s\n",
+                my_subdir, ret, strerror(errno));
+        printf("glfs_h_unlink tests: FAILED\n");
+        goto out;
+    }
+
+    /* unlink directory */
+    ret = glfs_h_unlink(fs, parent, my_dir);
+    if (ret) {
+        fprintf(stderr, "glfs_h_unlink: error unlinking %s: from (%p),%s\n",
+                my_dir, dir, strerror(errno));
+        printf("glfs_h_unlink tests: FAILED\n");
+        goto out;
+    }
+
+    printf("glfs_h_unlink tests: PASSED\n");
 
 out:
-        if (dir)
-                glfs_h_close (dir);
-        if (leaf)
-                glfs_h_close (leaf);
-        if (subdir)
-                glfs_h_close (subdir);
-        if (subleaf)
-                glfs_h_close (subleaf);
-        if (parent)
-                glfs_h_close (parent);
-
-        return;
+    if (dir)
+        glfs_h_close(dir);
+    if (leaf)
+        glfs_h_close(leaf);
+    if (subdir)
+        glfs_h_close(subdir);
+    if (subleaf)
+        glfs_h_close(subleaf);
+    if (parent)
+        glfs_h_close(parent);
+
+    return;
 }
 
 void
-test_h_getsetattrs (void)
+test_h_getsetattrs(void)
 {
-        char               *my_dir = "attrdir";
-        char               *my_file = "attrfile.txt";
-        struct glfs_object *parent = NULL, *leaf = NULL, *dir = NULL;
-        struct stat         sb, retsb;
-        int                 ret, valid;
-        struct timespec     timestamp;
-
-        printf("glfs_h_getattrs and setattrs tests: In Progress\n");
-
-        /* Prepare tests */
-        parent = glfs_h_lookupat (fs, NULL, full_parent_name, &sb);
-        if (parent == NULL) {
-                fprintf (stderr, "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
-                         full_parent_name, NULL, strerror (errno));
-                printf ("glfs_h_lookupat tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        dir = glfs_h_mkdir (fs, parent, my_dir, 0644, &sb);
-        if (dir == NULL) {
-                fprintf (stderr, "glfs_h_mkdir: error creating %s: from (%p),%s\n",
-                         my_dir, parent, strerror (errno));
-                printf ("glfs_h_unlink tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        leaf = glfs_h_creat (fs, dir, my_file, O_CREAT, 0644, &sb);
-        if (leaf == NULL) {
-                fprintf (stderr, "glfs_h_creat: error creating %s: from (%p),%s\n",
-                         my_file, dir, strerror (errno));
-                printf ("glfs_h_unlink tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        ret = glfs_h_getattrs (fs, dir, &retsb);
-        if (ret != 0) {
-                fprintf (stderr, "glfs_h_getattrs: error %s: from (%p),%s\n",
-                         my_dir, dir, strerror (errno));
-                printf ("glfs_h_getattrs and setattrs tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&retsb);
-        /* TODO: Compare stat information */
-
-        retsb.st_mode = 00666;
-        retsb.st_uid = 1000;
-        retsb.st_gid = 1001;
-        ret = clock_gettime (CLOCK_REALTIME, &timestamp);
-        if(ret != 0) {
-                fprintf (stderr, "clock_gettime: error %s\n", strerror (errno));
-                printf ("glfs_h_getattrs and setattrs tests: FAILED\n");
-                goto out;
-        }
-        retsb.st_atim = timestamp;
-        retsb.st_mtim = timestamp;
-        valid = GFAPI_SET_ATTR_MODE | GFAPI_SET_ATTR_UID | GFAPI_SET_ATTR_GID |
-        GFAPI_SET_ATTR_ATIME | GFAPI_SET_ATTR_MTIME;
-        peek_stat (&retsb);
-
-        ret = glfs_h_setattrs (fs, dir, &retsb, valid);
-        if (ret != 0) {
-                fprintf (stderr, "glfs_h_setattrs: error %s: from (%p),%s\n",
-                         my_dir, dir, strerror (errno));
-                printf ("glfs_h_getattrs and setattrs tests: FAILED\n");
-                goto out;
-        }
-
-        memset(&retsb, 0, sizeof (struct stat));
-        ret = glfs_h_stat (fs, dir, &retsb);
-        if (ret != 0) {
-                fprintf (stderr, "glfs_h_stat: error %s: from (%p),%s\n",
-                         my_dir, dir, strerror (errno));
-                printf ("glfs_h_getattrs and setattrs tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&retsb);
-
-        printf ("glfs_h_getattrs and setattrs tests: PASSED\n");
+    char *my_dir = "attrdir";
+    char *my_file = "attrfile.txt";
+    struct glfs_object *parent = NULL, *leaf = NULL, *dir = NULL;
+    struct stat sb, retsb;
+    int ret, valid;
+    struct timespec timestamp;
+
+    printf("glfs_h_getattrs and setattrs tests: In Progress\n");
+
+    /* Prepare tests */
+    parent = glfs_h_lookupat(fs, NULL, full_parent_name, &sb, 0);
+    if (parent == NULL) {
+        fprintf(stderr,
+                "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
+                full_parent_name, NULL, strerror(errno));
+        printf("glfs_h_lookupat tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    dir = glfs_h_mkdir(fs, parent, my_dir, 0755, &sb);
+    if (dir == NULL) {
+        fprintf(stderr, "glfs_h_mkdir: error creating %s: from (%p),%s\n",
+                my_dir, parent, strerror(errno));
+        printf("glfs_h_unlink tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    leaf = glfs_h_creat(fs, dir, my_file, O_CREAT, 0644, &sb);
+    if (leaf == NULL) {
+        fprintf(stderr, "glfs_h_creat: error creating %s: from (%p),%s\n",
+                my_file, dir, strerror(errno));
+        printf("glfs_h_unlink tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    ret = glfs_h_getattrs(fs, dir, &retsb);
+    if (ret != 0) {
+        fprintf(stderr, "glfs_h_getattrs: error %s: from (%p),%s\n", my_dir,
+                dir, strerror(errno));
+        printf("glfs_h_getattrs and setattrs tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&retsb);
+    /* TODO: Compare stat information */
+
+    retsb.st_mode = 00666;
+    retsb.st_uid = 1000;
+    retsb.st_gid = 1001;
+    ret = clock_gettime(CLOCK_REALTIME, &timestamp);
+    if (ret != 0) {
+        fprintf(stderr, "clock_gettime: error %s\n", strerror(errno));
+        printf("glfs_h_getattrs and setattrs tests: FAILED\n");
+        goto out;
+    }
+    retsb.st_atim = timestamp;
+    retsb.st_mtim = timestamp;
+    valid = GFAPI_SET_ATTR_MODE | GFAPI_SET_ATTR_UID | GFAPI_SET_ATTR_GID |
+            GFAPI_SET_ATTR_ATIME | GFAPI_SET_ATTR_MTIME;
+    peek_stat(&retsb);
+
+    ret = glfs_h_setattrs(fs, dir, &retsb, valid);
+    if (ret != 0) {
+        fprintf(stderr, "glfs_h_setattrs: error %s: from (%p),%s\n", my_dir,
+                dir, strerror(errno));
+        printf("glfs_h_getattrs and setattrs tests: FAILED\n");
+        goto out;
+    }
+
+    memset(&retsb, 0, sizeof(struct stat));
+    ret = glfs_h_stat(fs, dir, &retsb);
+    if (ret != 0) {
+        fprintf(stderr, "glfs_h_stat: error %s: from (%p),%s\n", my_dir, dir,
+                strerror(errno));
+        printf("glfs_h_getattrs and setattrs tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&retsb);
+
+    printf("glfs_h_getattrs and setattrs tests: PASSED\n");
 out:
-        if (parent)
-                glfs_h_close (parent);
-        if (leaf)
-                glfs_h_close (leaf);
-        if (dir)
-                glfs_h_close (dir);
-
-        return;
+    if (parent)
+        glfs_h_close(parent);
+    if (leaf)
+        glfs_h_close(leaf);
+    if (dir)
+        glfs_h_close(dir);
+
+    return;
 }
 
 void
-test_h_truncate (void)
+test_h_truncate(void)
 {
-        char               *my_dir = "truncatedir";
-        char               *my_file = "file.txt";
-        struct glfs_object *root = NULL, *parent = NULL, *leaf = NULL;
-        struct stat         sb;
-        glfs_fd_t          *fd = NULL;
-        char                buf[32];
-        off_t               offset = 0;
-        int                 ret = 0;
-
-        printf("glfs_h_truncate tests: In Progress\n");
-
-        /* Prepare tests */
-        root = glfs_h_lookupat (fs, NULL, full_parent_name, &sb);
-        if (root == NULL) {
-                fprintf (stderr, "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
-                         full_parent_name, NULL, strerror (errno));
-                printf ("glfs_h_truncate tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        parent = glfs_h_mkdir (fs, root, my_dir, 0644, &sb);
-        if (parent == NULL) {
-                fprintf (stderr, "glfs_h_mkdir: error creating %s: from (%p),%s\n",
-                         my_dir, root, strerror (errno));
-                printf ("glfs_h_truncate tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        leaf = glfs_h_creat (fs, parent, my_file, O_CREAT, 0644, &sb);
-        if (leaf == NULL) {
-                fprintf (stderr, "glfs_h_creat: error creating %s: from (%p),%s\n",
-                         my_file, parent, strerror (errno));
-                printf ("glfs_h_truncate tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        fd = glfs_h_open (fs, leaf, O_RDWR);
-        if (fd == NULL) {
-                fprintf (stderr, "glfs_h_open: error on open of %s: %s\n",
-                         my_file, strerror (errno));
-                printf ("glfs_h_truncate tests: FAILED\n");
-                goto out;
-        }
-
-        memcpy (buf, "abcdefghijklmnopqrstuvwxyz012345", 32);
-        ret = glfs_write (fd, buf, 32, 0);
-
-        /* run tests */
-        /* truncate lower */
-        offset = 30;
-        ret = glfs_h_truncate (fs, leaf, offset);
-        if (ret != 0) {
-                fprintf (stderr, "glfs_h_truncate: error creating %s: from (%p),%s\n",
-                         my_file, parent, strerror (errno));
-                printf ("glfs_h_truncate tests: FAILED\n");
-                goto out;
-        }
-        ret = glfs_h_getattrs (fs, leaf, &sb);
-        if (ret != 0) {
-                fprintf (stderr, "glfs_h_getattrs: error for %s (%p),%s\n",
-                         my_file, leaf, strerror (errno));
-                printf ("glfs_h_truncate tests: FAILED\n");
-                goto out;
-        }
-        if (sb.st_size != offset) {
-                fprintf (stderr, "glfs_h_truncate: post size mismatch\n");
-                printf ("glfs_h_truncate tests: FAILED\n");
-                goto out;
-        }
-
-        /* truncate higher */
-        offset = 32;
-        ret = glfs_h_truncate (fs, leaf, offset);
-        if (ret != 0) {
-                fprintf (stderr, "glfs_h_truncate: error creating %s: from (%p),%s\n",
-                         my_file, parent, strerror (errno));
-                printf ("glfs_h_truncate tests: FAILED\n");
-                goto out;
-        }
-        ret = glfs_h_getattrs (fs, leaf, &sb);
-        if (ret != 0) {
-                fprintf (stderr, "glfs_h_getattrs: error for %s (%p),%s\n",
-                         my_file, leaf, strerror (errno));
-                printf ("glfs_h_truncate tests: FAILED\n");
-                goto out;
-        }
-        if (sb.st_size != offset) {
-                fprintf (stderr, "glfs_h_truncate: post size mismatch\n");
-                printf ("glfs_h_truncate tests: FAILED\n");
-                goto out;
-        }
-
-        /* truncate equal */
-        offset = 30;
-        ret = glfs_h_truncate (fs, leaf, offset);
-        if (ret != 0) {
-                fprintf (stderr, "glfs_h_truncate: error creating %s: from (%p),%s\n",
-                         my_file, parent, strerror (errno));
-                printf ("glfs_h_truncate tests: FAILED\n");
-                goto out;
-        }
-        ret = glfs_h_getattrs (fs, leaf, &sb);
-        if (ret != 0) {
-                fprintf (stderr, "glfs_h_getattrs: error for %s (%p),%s\n",
-                         my_file, leaf, strerror (errno));
-                printf ("glfs_h_truncate tests: FAILED\n");
-                goto out;
-        }
-        if (sb.st_size != offset) {
-                fprintf (stderr, "glfs_h_truncate: post size mismatch\n");
-                printf ("glfs_h_truncate tests: FAILED\n");
-                goto out;
-        }
-
-        printf ("glfs_h_truncate tests: PASSED\n");
+    char *my_dir = "truncatedir";
+    char *my_file = "file.txt";
+    struct glfs_object *root = NULL, *parent = NULL, *leaf = NULL;
+    struct stat sb;
+    glfs_fd_t *fd = NULL;
+    char buf[32];
+    off_t offset = 0;
+    int ret = 0;
+
+    printf("glfs_h_truncate tests: In Progress\n");
+
+    /* Prepare tests */
+    root = glfs_h_lookupat(fs, NULL, full_parent_name, &sb, 0);
+    if (root == NULL) {
+        fprintf(stderr,
+                "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
+                full_parent_name, NULL, strerror(errno));
+        printf("glfs_h_truncate tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    parent = glfs_h_mkdir(fs, root, my_dir, 0755, &sb);
+    if (parent == NULL) {
+        fprintf(stderr, "glfs_h_mkdir: error creating %s: from (%p),%s\n",
+                my_dir, root, strerror(errno));
+        printf("glfs_h_truncate tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    leaf = glfs_h_creat(fs, parent, my_file, O_CREAT, 0644, &sb);
+    if (leaf == NULL) {
+        fprintf(stderr, "glfs_h_creat: error creating %s: from (%p),%s\n",
+                my_file, parent, strerror(errno));
+        printf("glfs_h_truncate tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    fd = glfs_h_open(fs, leaf, O_RDWR);
+    if (fd == NULL) {
+        fprintf(stderr, "glfs_h_open: error on open of %s: %s\n", my_file,
+                strerror(errno));
+        printf("glfs_h_truncate tests: FAILED\n");
+        goto out;
+    }
+
+    memcpy(buf, "abcdefghijklmnopqrstuvwxyz012345", 32);
+    ret = glfs_write(fd, buf, 32, 0);
+
+    /* run tests */
+    /* truncate lower */
+    offset = 30;
+    ret = glfs_h_truncate(fs, leaf, offset);
+    if (ret != 0) {
+        fprintf(stderr, "glfs_h_truncate: error creating %s: from (%p),%s\n",
+                my_file, parent, strerror(errno));
+        printf("glfs_h_truncate tests: FAILED\n");
+        goto out;
+    }
+    ret = glfs_h_getattrs(fs, leaf, &sb);
+    if (ret != 0) {
+        fprintf(stderr, "glfs_h_getattrs: error for %s (%p),%s\n", my_file,
+                leaf, strerror(errno));
+        printf("glfs_h_truncate tests: FAILED\n");
+        goto out;
+    }
+    if (sb.st_size != offset) {
+        fprintf(stderr, "glfs_h_truncate: post size mismatch\n");
+        printf("glfs_h_truncate tests: FAILED\n");
+        goto out;
+    }
+
+    /* truncate higher */
+    offset = 32;
+    ret = glfs_h_truncate(fs, leaf, offset);
+    if (ret != 0) {
+        fprintf(stderr, "glfs_h_truncate: error creating %s: from (%p),%s\n",
+                my_file, parent, strerror(errno));
+        printf("glfs_h_truncate tests: FAILED\n");
+        goto out;
+    }
+    ret = glfs_h_getattrs(fs, leaf, &sb);
+    if (ret != 0) {
+        fprintf(stderr, "glfs_h_getattrs: error for %s (%p),%s\n", my_file,
+                leaf, strerror(errno));
+        printf("glfs_h_truncate tests: FAILED\n");
+        goto out;
+    }
+    if (sb.st_size != offset) {
+        fprintf(stderr, "glfs_h_truncate: post size mismatch\n");
+        printf("glfs_h_truncate tests: FAILED\n");
+        goto out;
+    }
+
+    /* truncate equal */
+    offset = 30;
+    ret = glfs_h_truncate(fs, leaf, offset);
+    if (ret != 0) {
+        fprintf(stderr, "glfs_h_truncate: error creating %s: from (%p),%s\n",
+                my_file, parent, strerror(errno));
+        printf("glfs_h_truncate tests: FAILED\n");
+        goto out;
+    }
+    ret = glfs_h_getattrs(fs, leaf, &sb);
+    if (ret != 0) {
+        fprintf(stderr, "glfs_h_getattrs: error for %s (%p),%s\n", my_file,
+                leaf, strerror(errno));
+        printf("glfs_h_truncate tests: FAILED\n");
+        goto out;
+    }
+    if (sb.st_size != offset) {
+        fprintf(stderr, "glfs_h_truncate: post size mismatch\n");
+        printf("glfs_h_truncate tests: FAILED\n");
+        goto out;
+    }
+
+    printf("glfs_h_truncate tests: PASSED\n");
 out:
-        if (fd)
-                glfs_close (fd);
-        if (root)
-                glfs_h_close (root);
-        if (parent)
-                glfs_h_close (parent);
-        if (leaf)
-                glfs_h_close (leaf);
-
-        return;
+    if (fd)
+        glfs_close(fd);
+    if (root)
+        glfs_h_close(root);
+    if (parent)
+        glfs_h_close(parent);
+    if (leaf)
+        glfs_h_close(leaf);
+
+    return;
 }
 
 void
-test_h_links (void)
+test_h_links(void)
 {
-        char               *my_dir = "linkdir";
-        char               *my_file = "file.txt";
-        char               *my_symlnk = "slnk.txt";
-        char               *my_lnk = "lnk.txt";
-        char               *linksrc_dir = "dir1";
-        char               *linktgt_dir = "dir2";
-        struct glfs_object *root = NULL, *parent = NULL, *leaf = NULL,
-                           *dirsrc = NULL, *dirtgt = NULL, *dleaf = NULL;
-        struct glfs_object *ln1 = NULL;
-        struct stat         sb;
-        int                 ret;
-        char               *buf = NULL;
-
-        printf("glfs_h_link(s) tests: In Progress\n");
-
-        /* Prepare tests */
-        root = glfs_h_lookupat (fs, NULL, full_parent_name, &sb);
-        if (root == NULL) {
-                fprintf (stderr, "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
-                         full_parent_name, NULL, strerror (errno));
-                printf ("glfs_h_link(s) tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        parent = glfs_h_mkdir (fs, root, my_dir, 0644, &sb);
-        if (parent == NULL) {
-                fprintf (stderr, "glfs_h_mkdir: error creating %s: from (%p),%s\n",
-                         my_dir, root, strerror (errno));
-                printf ("glfs_h_link(s) tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        leaf = glfs_h_creat (fs, parent, my_file, O_CREAT, 0644, &sb);
-        if (leaf == NULL) {
-                fprintf (stderr, "glfs_h_creat: error creating %s: from (%p),%s\n",
-                         my_file, parent, strerror (errno));
-                printf ("glfs_h_link(s) tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        dirsrc = glfs_h_mkdir (fs, parent, linksrc_dir, 0644, &sb);
-        if (dirsrc == NULL) {
-                fprintf (stderr, "glfs_h_mkdir: error creating %s: from (%p),%s\n",
-                         linksrc_dir, parent, strerror (errno));
-                printf ("glfs_h_link(s) tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        dirtgt = glfs_h_mkdir (fs, parent, linktgt_dir, 0644, &sb);
-        if (dirtgt == NULL) {
-                fprintf (stderr, "glfs_h_mkdir: error creating %s: from (%p),%s\n",
-                         linktgt_dir, parent, strerror (errno));
-                printf ("glfs_h_link(s) tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        dleaf = glfs_h_creat (fs, dirsrc, my_file, O_CREAT, 0644, &sb);
-        if (dleaf == NULL) {
-                fprintf (stderr, "glfs_h_creat: error creating %s: from (%p),%s\n",
-                         my_file, dirsrc, strerror (errno));
-                printf ("glfs_h_link(s) tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        /* run tests */
-        /* sym link: /testdir/linkdir/file.txt to ./slnk.txt */
-        ln1 = glfs_h_symlink (fs, parent, my_symlnk, "./file.txt", &sb);
-        if (ln1 == NULL) {
-                fprintf (stderr, "glfs_h_symlink: error creating %s: from (%p),%s\n",
-                         my_symlnk, parent, strerror (errno));
-                printf ("glfs_h_link(s) tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        buf = calloc (1024, sizeof(char));
-        if (buf == NULL) {
-                fprintf (stderr, "Error allocating memory\n");
-                printf ("glfs_h_link(s) tests: FAILED\n");
-                goto out;
-        }
-
-        ret = glfs_h_readlink (fs, ln1, buf, 1024);
-        if (ret <= 0) {
-                fprintf (stderr, "glfs_h_readlink: error reading %s: from (%p),%s\n",
-                         my_symlnk, ln1, strerror (errno));
-                printf ("glfs_h_link(s) tests: FAILED\n");
-                goto out;
-        }
-        if (!(strncmp (buf, my_symlnk, strlen (my_symlnk)))) {
-                fprintf (stderr, "glfs_h_readlink: error mismatch in link name: actual %s: retrieved %s\n",
-                         my_symlnk, buf);
-                printf ("glfs_h_link(s) tests: FAILED\n");
-                goto out;
-        }
-
-        /* link: /testdir/linkdir/file.txt to ./lnk.txt */
-        ret = glfs_h_link (fs, leaf, parent, my_lnk);
-        if (ret != 0) {
-                fprintf (stderr, "glfs_h_link: error creating %s: from (%p),%s\n",
-                         my_lnk, parent, strerror (errno));
-                printf ("glfs_h_link(s) tests: FAILED\n");
-                goto out;
-        }
-        /* TODO: Should write content to a file and read from the link */
-
-        /* link: /testdir/linkdir/dir1/file.txt to ../dir2/slnk.txt */
-        ret = glfs_h_link (fs, dleaf, dirtgt, my_lnk);
-        if (ret != 0) {
-                fprintf (stderr, "glfs_h_link: error creating %s: from (%p),%s\n",
-                         my_lnk, dirtgt, strerror (errno));
-                printf ("glfs_h_link(s) tests: FAILED\n");
-                goto out;
-        }
-        /* TODO: Should write content to a file and read from the link */
-
-        printf ("glfs_h_link(s) tests: PASSED\n");
+    char *my_dir = "linkdir";
+    char *my_file = "file.txt";
+    char *my_symlnk = "slnk.txt";
+    char *my_lnk = "lnk.txt";
+    char *linksrc_dir = "dir1";
+    char *linktgt_dir = "dir2";
+    struct glfs_object *root = NULL, *parent = NULL, *leaf = NULL,
+                       *dirsrc = NULL, *dirtgt = NULL, *dleaf = NULL;
+    struct glfs_object *ln1 = NULL;
+    struct stat sb;
+    int ret;
+    char *buf = NULL;
+
+    printf("glfs_h_link(s) tests: In Progress\n");
+
+    /* Prepare tests */
+    root = glfs_h_lookupat(fs, NULL, full_parent_name, &sb, 0);
+    if (root == NULL) {
+        fprintf(stderr,
+                "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
+                full_parent_name, NULL, strerror(errno));
+        printf("glfs_h_link(s) tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    parent = glfs_h_mkdir(fs, root, my_dir, 0755, &sb);
+    if (parent == NULL) {
+        fprintf(stderr, "glfs_h_mkdir: error creating %s: from (%p),%s\n",
+                my_dir, root, strerror(errno));
+        printf("glfs_h_link(s) tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    leaf = glfs_h_creat(fs, parent, my_file, O_CREAT, 0644, &sb);
+    if (leaf == NULL) {
+        fprintf(stderr, "glfs_h_creat: error creating %s: from (%p),%s\n",
+                my_file, parent, strerror(errno));
+        printf("glfs_h_link(s) tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    dirsrc = glfs_h_mkdir(fs, parent, linksrc_dir, 0755, &sb);
+    if (dirsrc == NULL) {
+        fprintf(stderr, "glfs_h_mkdir: error creating %s: from (%p),%s\n",
+                linksrc_dir, parent, strerror(errno));
+        printf("glfs_h_link(s) tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    dirtgt = glfs_h_mkdir(fs, parent, linktgt_dir, 0755, &sb);
+    if (dirtgt == NULL) {
+        fprintf(stderr, "glfs_h_mkdir: error creating %s: from (%p),%s\n",
+                linktgt_dir, parent, strerror(errno));
+        printf("glfs_h_link(s) tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    dleaf = glfs_h_creat(fs, dirsrc, my_file, O_CREAT, 0644, &sb);
+    if (dleaf == NULL) {
+        fprintf(stderr, "glfs_h_creat: error creating %s: from (%p),%s\n",
+                my_file, dirsrc, strerror(errno));
+        printf("glfs_h_link(s) tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    /* run tests */
+    /* sym link: /testdir/linkdir/file.txt to ./slnk.txt */
+    ln1 = glfs_h_symlink(fs, parent, my_symlnk, "./file.txt", &sb);
+    if (ln1 == NULL) {
+        fprintf(stderr, "glfs_h_symlink: error creating %s: from (%p),%s\n",
+                my_symlnk, parent, strerror(errno));
+        printf("glfs_h_link(s) tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    buf = calloc(1024, sizeof(char));
+    if (buf == NULL) {
+        fprintf(stderr, "Error allocating memory\n");
+        printf("glfs_h_link(s) tests: FAILED\n");
+        goto out;
+    }
+
+    ret = glfs_h_readlink(fs, ln1, buf, 1024);
+    if (ret <= 0) {
+        fprintf(stderr, "glfs_h_readlink: error reading %s: from (%p),%s\n",
+                my_symlnk, ln1, strerror(errno));
+        printf("glfs_h_link(s) tests: FAILED\n");
+        goto out;
+    }
+    if (!(strncmp(buf, my_symlnk, strlen(my_symlnk)))) {
+        fprintf(stderr,
+                "glfs_h_readlink: error mismatch in link name: actual %s: "
+                "retrieved %s\n",
+                my_symlnk, buf);
+        printf("glfs_h_link(s) tests: FAILED\n");
+        goto out;
+    }
+
+    /* link: /testdir/linkdir/file.txt to ./lnk.txt */
+    ret = glfs_h_link(fs, leaf, parent, my_lnk);
+    if (ret != 0) {
+        fprintf(stderr, "glfs_h_link: error creating %s: from (%p),%s\n",
+                my_lnk, parent, strerror(errno));
+        printf("glfs_h_link(s) tests: FAILED\n");
+        goto out;
+    }
+    /* TODO: Should write content to a file and read from the link */
+
+    /* link: /testdir/linkdir/dir1/file.txt to ../dir2/slnk.txt */
+    ret = glfs_h_link(fs, dleaf, dirtgt, my_lnk);
+    if (ret != 0) {
+        fprintf(stderr, "glfs_h_link: error creating %s: from (%p),%s\n",
+                my_lnk, dirtgt, strerror(errno));
+        printf("glfs_h_link(s) tests: FAILED\n");
+        goto out;
+    }
+    /* TODO: Should write content to a file and read from the link */
+
+    printf("glfs_h_link(s) tests: PASSED\n");
 
 out:
-        if (root)
-                glfs_h_close (root);
-        if (parent)
-                glfs_h_close (parent);
-        if (leaf)
-                glfs_h_close (leaf);
-        if (dirsrc)
-                glfs_h_close (dirsrc);
-        if (dirtgt)
-                glfs_h_close (dirtgt);
-        if (dleaf)
-                glfs_h_close (dleaf);
-        if (ln1)
-                glfs_h_close (ln1);
-        if (buf)
-                free (buf);
-
-        return;
+    if (root)
+        glfs_h_close(root);
+    if (parent)
+        glfs_h_close(parent);
+    if (leaf)
+        glfs_h_close(leaf);
+    if (dirsrc)
+        glfs_h_close(dirsrc);
+    if (dirtgt)
+        glfs_h_close(dirtgt);
+    if (dleaf)
+        glfs_h_close(dleaf);
+    if (ln1)
+        glfs_h_close(ln1);
+    if (buf)
+        free(buf);
+
+    return;
 }
 
 void
-test_h_rename (void)
+test_h_rename(void)
 {
-        char               *my_dir = "renamedir";
-        char               *my_file = "file.txt";
-        char               *src_dir = "dir1";
-        char               *tgt_dir = "dir2";
-        struct glfs_object *root = NULL, *parent = NULL, *leaf = NULL,
-                           *dirsrc = NULL, *dirtgt = NULL, *dleaf = NULL;
-        struct stat         sb;
-        int                 ret;
-
-        printf("glfs_h_rename tests: In Progress\n");
-
-        /* Prepare tests */
-        root = glfs_h_lookupat (fs, NULL, full_parent_name, &sb);
-        if (root == NULL) {
-                fprintf (stderr, "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
-                         full_parent_name, NULL, strerror (errno));
-                printf ("glfs_h_rename tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        parent = glfs_h_mkdir (fs, root, my_dir, 0644, &sb);
-        if (parent == NULL) {
-                fprintf (stderr, "glfs_h_mkdir: error creating %s: from (%p),%s\n",
-                         my_dir, root, strerror (errno));
-                printf ("glfs_h_rename tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        leaf = glfs_h_creat (fs, parent, my_file, O_CREAT, 0644, &sb);
-        if (leaf == NULL) {
-                fprintf (stderr, "glfs_h_creat: error creating %s: from (%p),%s\n",
-                         my_file, parent, strerror (errno));
-                printf ("glfs_h_rename tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        dirsrc = glfs_h_mkdir (fs, parent, src_dir, 0644, &sb);
-        if (dirsrc == NULL) {
-                fprintf (stderr, "glfs_h_mkdir: error creating %s: from (%p),%s\n",
-                         src_dir, parent, strerror (errno));
-                printf ("glfs_h_rename tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        dirtgt = glfs_h_mkdir (fs, parent, tgt_dir, 0644, &sb);
-        if (dirtgt == NULL) {
-                fprintf (stderr, "glfs_h_mkdir: error creating %s: from (%p),%s\n",
-                         tgt_dir, parent, strerror (errno));
-                printf ("glfs_h_rename tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        dleaf = glfs_h_creat (fs, dirsrc, my_file, O_CREAT, 0644, &sb);
-        if (dleaf == NULL) {
-                fprintf (stderr, "glfs_h_creat: error creating %s: from (%p),%s\n",
-                         my_file, dirsrc, strerror (errno));
-                printf ("glfs_h_rename tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        /* run tests */
-        /* Rename file.txt -> file1.txt */
-        ret = glfs_h_rename (fs, parent, "file.txt", parent, "file1.txt");
-        if (ret != 0) {
-                fprintf (stderr, "glfs_h_rename: error renaming %s to %s (%s)\n",
-                         "file.txt", "file1.txt", strerror (errno));
-                printf ("glfs_h_rename tests: FAILED\n");
-                goto out;
-        }
-
-        /* rename dir1/file.txt -> file.txt */
-        ret = glfs_h_rename (fs, dirsrc, "file.txt", parent, "file.txt");
-        if (ret != 0) {
-                fprintf (stderr, "glfs_h_rename: error renaming %s/%s to %s (%s)\n",
-                         src_dir, "file.txt", "file.txt", strerror (errno));
-                printf ("glfs_h_rename tests: FAILED\n");
-                goto out;
-        }
-
-        /* rename file1.txt -> file.txt (exists) */
-        ret = glfs_h_rename (fs, parent, "file1.txt", parent, "file.txt");
-        if (ret != 0) {
-                fprintf (stderr, "glfs_h_rename: error renaming %s to %s (%s)\n",
-                         "file.txt", "file.txt", strerror (errno));
-                printf ("glfs_h_rename tests: FAILED\n");
-                goto out;
-        }
-
-        /* rename dir1 -> dir3 */
-        ret = glfs_h_rename (fs, parent, "dir1", parent, "dir3");
-        if (ret != 0) {
-                fprintf (stderr, "glfs_h_rename: error renaming %s to %s (%s)\n",
-                         "dir1", "dir3", strerror (errno));
-                printf ("glfs_h_rename tests: FAILED\n");
-                goto out;
-        }
-
-        /* rename dir2 ->dir3 (exists) */
-        ret = glfs_h_rename (fs, parent, "dir2", parent, "dir3");
-        if (ret != 0) {
-                fprintf (stderr, "glfs_h_rename: error renaming %s to %s (%s)\n",
-                         "dir2", "dir3", strerror (errno));
-                printf ("glfs_h_rename tests: FAILED\n");
-                goto out;
-        }
-
-        /* rename file.txt -> dir3 (fail) */
-        ret = glfs_h_rename (fs, parent, "file.txt", parent, "dir3");
-        if (ret == 0) {
-                fprintf (stderr, "glfs_h_rename: NO error renaming %s to %s (%s)\n",
-                         "file.txt", "dir3", strerror (errno));
-                printf ("glfs_h_rename tests: FAILED\n");
-                goto out;
-        }
-
-        /* rename dir3 -> file.txt (fail) */
-        ret = glfs_h_rename (fs, parent, "dir3", parent, "file.txt");
-        if (ret == 0) {
-                fprintf (stderr, "glfs_h_rename: NO error renaming %s to %s (%s)\n",
-                         "dir3", "file.txt", strerror (errno));
-                printf ("glfs_h_rename tests: FAILED\n");
-                goto out;
-        }
-
-        printf ("glfs_h_rename tests: PASSED\n");
+    char *my_dir = "renamedir";
+    char *my_file = "file.txt";
+    char *src_dir = "dir1";
+    char *tgt_dir = "dir2";
+    struct glfs_object *root = NULL, *parent = NULL, *leaf = NULL,
+                       *dirsrc = NULL, *dirtgt = NULL, *dleaf = NULL;
+    struct stat sb;
+    int ret;
+
+    printf("glfs_h_rename tests: In Progress\n");
+
+    /* Prepare tests */
+    root = glfs_h_lookupat(fs, NULL, full_parent_name, &sb, 0);
+    if (root == NULL) {
+        fprintf(stderr,
+                "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
+                full_parent_name, NULL, strerror(errno));
+        printf("glfs_h_rename tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    parent = glfs_h_mkdir(fs, root, my_dir, 0755, &sb);
+    if (parent == NULL) {
+        fprintf(stderr, "glfs_h_mkdir: error creating %s: from (%p),%s\n",
+                my_dir, root, strerror(errno));
+        printf("glfs_h_rename tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    leaf = glfs_h_creat(fs, parent, my_file, O_CREAT, 0644, &sb);
+    if (leaf == NULL) {
+        fprintf(stderr, "glfs_h_creat: error creating %s: from (%p),%s\n",
+                my_file, parent, strerror(errno));
+        printf("glfs_h_rename tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    dirsrc = glfs_h_mkdir(fs, parent, src_dir, 0755, &sb);
+    if (dirsrc == NULL) {
+        fprintf(stderr, "glfs_h_mkdir: error creating %s: from (%p),%s\n",
+                src_dir, parent, strerror(errno));
+        printf("glfs_h_rename tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    dirtgt = glfs_h_mkdir(fs, parent, tgt_dir, 0755, &sb);
+    if (dirtgt == NULL) {
+        fprintf(stderr, "glfs_h_mkdir: error creating %s: from (%p),%s\n",
+                tgt_dir, parent, strerror(errno));
+        printf("glfs_h_rename tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    dleaf = glfs_h_creat(fs, dirsrc, my_file, O_CREAT, 0644, &sb);
+    if (dleaf == NULL) {
+        fprintf(stderr, "glfs_h_creat: error creating %s: from (%p),%s\n",
+                my_file, dirsrc, strerror(errno));
+        printf("glfs_h_rename tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    /* run tests */
+    /* Rename file.txt -> file1.txt */
+    ret = glfs_h_rename(fs, parent, "file.txt", parent, "file1.txt");
+    if (ret != 0) {
+        fprintf(stderr, "glfs_h_rename: error renaming %s to %s (%s)\n",
+                "file.txt", "file1.txt", strerror(errno));
+        printf("glfs_h_rename tests: FAILED\n");
+        goto out;
+    }
+
+    /* rename dir1/file.txt -> file.txt */
+    ret = glfs_h_rename(fs, dirsrc, "file.txt", parent, "file.txt");
+    if (ret != 0) {
+        fprintf(stderr, "glfs_h_rename: error renaming %s/%s to %s (%s)\n",
+                src_dir, "file.txt", "file.txt", strerror(errno));
+        printf("glfs_h_rename tests: FAILED\n");
+        goto out;
+    }
+
+    /* rename file1.txt -> file.txt (exists) */
+    ret = glfs_h_rename(fs, parent, "file1.txt", parent, "file.txt");
+    if (ret != 0) {
+        fprintf(stderr, "glfs_h_rename: error renaming %s to %s (%s)\n",
+                "file.txt", "file.txt", strerror(errno));
+        printf("glfs_h_rename tests: FAILED\n");
+        goto out;
+    }
+
+    /* rename dir1 -> dir3 */
+    ret = glfs_h_rename(fs, parent, "dir1", parent, "dir3");
+    if (ret != 0) {
+        fprintf(stderr, "glfs_h_rename: error renaming %s to %s (%s)\n", "dir1",
+                "dir3", strerror(errno));
+        printf("glfs_h_rename tests: FAILED\n");
+        goto out;
+    }
+
+    /* rename dir2 ->dir3 (exists) */
+    ret = glfs_h_rename(fs, parent, "dir2", parent, "dir3");
+    if (ret != 0) {
+        fprintf(stderr, "glfs_h_rename: error renaming %s to %s (%s)\n", "dir2",
+                "dir3", strerror(errno));
+        printf("glfs_h_rename tests: FAILED\n");
+        goto out;
+    }
+
+    /* rename file.txt -> dir3 (fail) */
+    ret = glfs_h_rename(fs, parent, "file.txt", parent, "dir3");
+    if (ret == 0) {
+        fprintf(stderr, "glfs_h_rename: NO error renaming %s to %s (%s)\n",
+                "file.txt", "dir3", strerror(errno));
+        printf("glfs_h_rename tests: FAILED\n");
+        goto out;
+    }
+
+    /* rename dir3 -> file.txt (fail) */
+    ret = glfs_h_rename(fs, parent, "dir3", parent, "file.txt");
+    if (ret == 0) {
+        fprintf(stderr, "glfs_h_rename: NO error renaming %s to %s (%s)\n",
+                "dir3", "file.txt", strerror(errno));
+        printf("glfs_h_rename tests: FAILED\n");
+        goto out;
+    }
+
+    printf("glfs_h_rename tests: PASSED\n");
 
 out:
-        if (root)
-                glfs_h_close (root);
-        if (parent)
-                glfs_h_close (parent);
-        if (leaf)
-                glfs_h_close (leaf);
-        if (dirsrc)
-                glfs_h_close (dirsrc);
-        if (dirtgt)
-                glfs_h_close (dirtgt);
-        if (dleaf)
-                glfs_h_close (dleaf);
-
-        return;
+    if (root)
+        glfs_h_close(root);
+    if (parent)
+        glfs_h_close(parent);
+    if (leaf)
+        glfs_h_close(leaf);
+    if (dirsrc)
+        glfs_h_close(dirsrc);
+    if (dirtgt)
+        glfs_h_close(dirtgt);
+    if (dleaf)
+        glfs_h_close(dleaf);
+
+    return;
 }
 
 void
-assimilatetime (struct timespec *ts, struct timespec ts_st,
-                struct timespec ts_ed)
+assimilatetime(struct timespec *ts, struct timespec ts_st,
+               struct timespec ts_ed)
 {
-        if ((ts_ed.tv_nsec - ts_st.tv_nsec) < 0) {
-                ts->tv_sec += ts_ed.tv_sec - ts_st.tv_sec - 1;
-                ts->tv_nsec += 1000000000 + ts_ed.tv_nsec - ts_st.tv_nsec;
-        } else {
-                ts->tv_sec += ts_ed.tv_sec - ts_st.tv_sec;
-                ts->tv_nsec += ts_ed.tv_nsec - ts_st.tv_nsec;
-        }
-
-        if (ts->tv_nsec > 1000000000) {
-                ts->tv_nsec = ts->tv_nsec - 1000000000;
-                ts->tv_sec += 1;
-        }
-
-        return;
+    if ((ts_ed.tv_nsec - ts_st.tv_nsec) < 0) {
+        ts->tv_sec += ts_ed.tv_sec - ts_st.tv_sec - 1;
+        ts->tv_nsec += 1000000000 + ts_ed.tv_nsec - ts_st.tv_nsec;
+    } else {
+        ts->tv_sec += ts_ed.tv_sec - ts_st.tv_sec;
+        ts->tv_nsec += ts_ed.tv_nsec - ts_st.tv_nsec;
+    }
+
+    if (ts->tv_nsec > 1000000000) {
+        ts->tv_nsec = ts->tv_nsec - 1000000000;
+        ts->tv_sec += 1;
+    }
+
+    return;
 }
 
 #define MAX_FILES_CREATE 10
-#define MAXPATHNAME      512
+#define MAXPATHNAME 512
 void
-test_h_performance (void)
+test_h_performance(void)
 {
-        char               *my_dir = "perftest",
-                           *full_dir_path="/testdir/perftest";
-        char               *my_file = "file_", my_file_name[MAXPATHNAME];
-        struct glfs_object *parent = NULL, *leaf = NULL, *dir = NULL;
-        struct stat         sb;
-        int                 ret, i;
-        struct glfs_fd     *fd;
-        struct timespec     c_ts = {0, 0}, c_ts_st, c_ts_ed;
-        struct timespec     o_ts = {0, 0}, o_ts_st, o_ts_ed;
-
-        printf("glfs_h_performance tests: In Progress\n");
-
-        /* Prepare tests */
-        parent = glfs_h_lookupat (fs, NULL, full_parent_name, &sb);
-        if (parent == NULL) {
-                fprintf (stderr, "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
-                         full_parent_name, NULL, strerror (errno));
-                printf ("glfs_h_performance tests: FAILED\n");
-                goto out;
-        }
-
-        dir = glfs_h_mkdir (fs, parent, my_dir, 0644, &sb);
-        if (dir == NULL) {
-                fprintf (stderr, "glfs_h_mkdir: error creating %s: from (%p),%s\n",
-                         my_dir, parent, strerror (errno));
-                printf ("glfs_h_performance tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        /* create performance */
-        ret = clock_gettime (CLOCK_REALTIME, &o_ts_st);
-        if(ret != 0) {
-                fprintf (stderr, "clock_gettime: error %s\n", strerror (errno));
-                printf ("glfs_h_getattrs and setattrs tests: FAILED\n");
-                goto out;
+    char *my_dir = "perftest", *full_dir_path = "/testdir/perftest";
+    char *my_file = "file_", my_file_name[MAXPATHNAME];
+    struct glfs_object *parent = NULL, *leaf = NULL, *dir = NULL;
+    struct stat sb;
+    int ret, i;
+    struct glfs_fd *fd;
+    struct timespec c_ts = {0, 0}, c_ts_st, c_ts_ed;
+    struct timespec o_ts = {0, 0}, o_ts_st, o_ts_ed;
+
+    printf("glfs_h_performance tests: In Progress\n");
+
+    /* Prepare tests */
+    parent = glfs_h_lookupat(fs, NULL, full_parent_name, &sb, 0);
+    if (parent == NULL) {
+        fprintf(stderr,
+                "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
+                full_parent_name, NULL, strerror(errno));
+        printf("glfs_h_performance tests: FAILED\n");
+        goto out;
+    }
+
+    dir = glfs_h_mkdir(fs, parent, my_dir, 0755, &sb);
+    if (dir == NULL) {
+        fprintf(stderr, "glfs_h_mkdir: error creating %s: from (%p),%s\n",
+                my_dir, parent, strerror(errno));
+        printf("glfs_h_performance tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    /* create performance */
+    ret = clock_gettime(CLOCK_REALTIME, &o_ts_st);
+    if (ret != 0) {
+        fprintf(stderr, "clock_gettime: error %s\n", strerror(errno));
+        printf("glfs_h_getattrs and setattrs tests: FAILED\n");
+        goto out;
+    }
+
+    for (i = 0; i < MAX_FILES_CREATE; i++) {
+        sprintf(my_file_name, "%s%d", my_file, i);
+
+        ret = clock_gettime(CLOCK_REALTIME, &c_ts_st);
+        if (ret != 0) {
+            fprintf(stderr, "clock_gettime: error %s\n", strerror(errno));
+            printf("glfs_h_getattrs and setattrs tests: FAILED\n");
+            goto out;
         }
 
-        for (i = 0; i < MAX_FILES_CREATE; i++) {
-                sprintf (my_file_name, "%s%d", my_file, i);
-
-                ret = clock_gettime (CLOCK_REALTIME, &c_ts_st);
-                if(ret != 0) {
-                        fprintf (stderr, "clock_gettime: error %s\n",
-                                 strerror (errno));
-                        printf ("glfs_h_getattrs and setattrs tests: FAILED\n");
-                        goto out;
-                }
-
-                leaf = glfs_h_lookupat (fs, dir, my_file_name, &sb);
-                if (leaf != NULL) {
-                        fprintf (stderr, "glfs_h_lookup: exists %s\n",
-                                 my_file_name);
-                        printf ("glfs_h_performance tests: FAILED\n");
-                        goto out;
-                }
-
-                leaf = glfs_h_creat (fs, dir, my_file_name, O_CREAT, 0644, &sb);
-                if (leaf == NULL) {
-                        fprintf (stderr, "glfs_h_creat: error creating %s: from (%p),%s\n",
-                                 my_file, dir, strerror (errno));
-                        printf ("glfs_h_performance tests: FAILED\n");
-                        goto out;
-                }
-
-                ret = clock_gettime (CLOCK_REALTIME, &c_ts_ed);
-                if(ret != 0) {
-                        fprintf (stderr, "clock_gettime: error %s\n",
-                                 strerror (errno));
-                        printf ("glfs_h_getattrs and setattrs tests: FAILED\n");
-                        goto out;
-                }
-
-                assimilatetime (&c_ts, c_ts_st, c_ts_ed);
-                glfs_h_close (leaf); leaf = NULL;
+        leaf = glfs_h_lookupat(fs, dir, my_file_name, &sb, 0);
+        if (leaf != NULL) {
+            fprintf(stderr, "glfs_h_lookup: exists %s\n", my_file_name);
+            printf("glfs_h_performance tests: FAILED\n");
+            goto out;
         }
 
-        ret = clock_gettime (CLOCK_REALTIME, &o_ts_ed);
-        if(ret != 0) {
-                fprintf (stderr, "clock_gettime: error %s\n", strerror (errno));
-                printf ("glfs_h_getattrs and setattrs tests: FAILED\n");
-                goto out;
+        leaf = glfs_h_creat(fs, dir, my_file_name, O_CREAT, 0644, &sb);
+        if (leaf == NULL) {
+            fprintf(stderr, "glfs_h_creat: error creating %s: from (%p),%s\n",
+                    my_file, dir, strerror(errno));
+            printf("glfs_h_performance tests: FAILED\n");
+            goto out;
         }
 
-        assimilatetime (&o_ts, o_ts_st, o_ts_ed);
-
-        printf ("Creation performance (handle based):\n\t# empty files:%d\n",
-                MAX_FILES_CREATE);
-        printf ("\tOverall time:\n\t\tSecs:%ld\n\t\tnSecs:%ld\n",
-                o_ts.tv_sec, o_ts.tv_nsec);
-        printf ("\tcreate call time time:\n\t\tSecs:%ld\n\t\tnSecs:%ld\n",
-                c_ts.tv_sec, c_ts.tv_nsec);
-
-        /* create using path */
-        c_ts.tv_sec = o_ts.tv_sec = 0;
-        c_ts.tv_nsec = o_ts.tv_nsec = 0;
-
-        sprintf (my_file_name, "%s1", full_dir_path);
-        ret = glfs_mkdir (fs, my_file_name, 0644);
+        ret = clock_gettime(CLOCK_REALTIME, &c_ts_ed);
         if (ret != 0) {
-                fprintf (stderr, "glfs_mkdir: error creating %s: from (%p),%s\n",
-                         my_dir, parent, strerror (errno));
-                printf ("glfs_h_performance tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        ret = clock_gettime (CLOCK_REALTIME, &o_ts_st);
-        if(ret != 0) {
-                fprintf (stderr, "clock_gettime: error %s\n", strerror (errno));
-                printf ("glfs_h_getattrs and setattrs tests: FAILED\n");
-                goto out;
+            fprintf(stderr, "clock_gettime: error %s\n", strerror(errno));
+            printf("glfs_h_getattrs and setattrs tests: FAILED\n");
+            goto out;
+        }
+
+        assimilatetime(&c_ts, c_ts_st, c_ts_ed);
+        glfs_h_close(leaf);
+        leaf = NULL;
+    }
+
+    ret = clock_gettime(CLOCK_REALTIME, &o_ts_ed);
+    if (ret != 0) {
+        fprintf(stderr, "clock_gettime: error %s\n", strerror(errno));
+        printf("glfs_h_getattrs and setattrs tests: FAILED\n");
+        goto out;
+    }
+
+    assimilatetime(&o_ts, o_ts_st, o_ts_ed);
+
+    printf("Creation performance (handle based):\n\t# empty files:%d\n",
+           MAX_FILES_CREATE);
+    printf("\tOverall time:\n\t\tSecs:%ld\n\t\tnSecs:%ld\n", o_ts.tv_sec,
+           o_ts.tv_nsec);
+    printf("\tcreate call time time:\n\t\tSecs:%ld\n\t\tnSecs:%ld\n",
+           c_ts.tv_sec, c_ts.tv_nsec);
+
+    /* create using path */
+    c_ts.tv_sec = o_ts.tv_sec = 0;
+    c_ts.tv_nsec = o_ts.tv_nsec = 0;
+
+    sprintf(my_file_name, "%s1", full_dir_path);
+    ret = glfs_mkdir(fs, my_file_name, 0755);
+    if (ret != 0) {
+        fprintf(stderr, "glfs_mkdir: error creating %s: from (%p),%s\n", my_dir,
+                parent, strerror(errno));
+        printf("glfs_h_performance tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    ret = clock_gettime(CLOCK_REALTIME, &o_ts_st);
+    if (ret != 0) {
+        fprintf(stderr, "clock_gettime: error %s\n", strerror(errno));
+        printf("glfs_h_getattrs and setattrs tests: FAILED\n");
+        goto out;
+    }
+
+    for (i = 0; i < MAX_FILES_CREATE; i++) {
+        sprintf(my_file_name, "%s1/%sn%d", full_dir_path, my_file, i);
+
+        ret = clock_gettime(CLOCK_REALTIME, &c_ts_st);
+        if (ret != 0) {
+            fprintf(stderr, "clock_gettime: error %s\n", strerror(errno));
+            printf("glfs_h_getattrs and setattrs tests: FAILED\n");
+            goto out;
         }
 
-        for (i = 0; i < MAX_FILES_CREATE; i++) {
-                sprintf (my_file_name, "%s1/%sn%d", full_dir_path, my_file, i);
-
-                ret = clock_gettime (CLOCK_REALTIME, &c_ts_st);
-                if(ret != 0) {
-                        fprintf (stderr, "clock_gettime: error %s\n",
-                                 strerror (errno));
-                        printf ("glfs_h_getattrs and setattrs tests: FAILED\n");
-                        goto out;
-                }
-
-                ret = glfs_stat (fs, my_file_name, &sb);
-                if (ret == 0) {
-                        fprintf (stderr, "glfs_stat: exists %s\n",
-                                 my_file_name);
-                        printf ("glfs_h_performance tests: FAILED\n");
-                        goto out;
-                }
-
-                fd = glfs_creat (fs, my_file_name, O_CREAT, 0644);
-                if (fd == NULL) {
-                        fprintf (stderr, "glfs_creat: error creating %s: from (%p),%s\n",
-                                 my_file, dir, strerror (errno));
-                        printf ("glfs_h_performance tests: FAILED\n");
-                        goto out;
-                }
-
-                ret = clock_gettime (CLOCK_REALTIME, &c_ts_ed);
-                if(ret != 0) {
-                        fprintf (stderr, "clock_gettime: error %s\n",
-                                 strerror (errno));
-                        printf ("glfs_h_getattrs and setattrs tests: FAILED\n");
-                        goto out;
-                }
-
-                assimilatetime (&c_ts, c_ts_st, c_ts_ed);
-                glfs_close (fd);
+        ret = glfs_stat(fs, my_file_name, &sb);
+        if (ret == 0) {
+            fprintf(stderr, "glfs_stat: exists %s\n", my_file_name);
+            printf("glfs_h_performance tests: FAILED\n");
+            goto out;
         }
 
-        ret = clock_gettime (CLOCK_REALTIME, &o_ts_ed);
-        if(ret != 0) {
-                fprintf (stderr, "clock_gettime: error %s\n", strerror (errno));
-                printf ("glfs_h_getattrs and setattrs tests: FAILED\n");
-                goto out;
+        fd = glfs_creat(fs, my_file_name, O_CREAT, 0644);
+        if (fd == NULL) {
+            fprintf(stderr, "glfs_creat: error creating %s: from (%p),%s\n",
+                    my_file, dir, strerror(errno));
+            printf("glfs_h_performance tests: FAILED\n");
+            goto out;
         }
 
-        assimilatetime (&o_ts, o_ts_st, o_ts_ed);
-
-        printf ("Creation performance (path based):\n\t# empty files:%d\n",
-                MAX_FILES_CREATE);
-        printf ("\tOverall time:\n\t\tSecs:%ld\n\t\tnSecs:%ld\n",
-                o_ts.tv_sec, o_ts.tv_nsec);
-        printf ("\tcreate call time time:\n\t\tSecs:%ld\n\t\tnSecs:%ld\n",
-                c_ts.tv_sec, c_ts.tv_nsec);
+        ret = clock_gettime(CLOCK_REALTIME, &c_ts_ed);
+        if (ret != 0) {
+            fprintf(stderr, "clock_gettime: error %s\n", strerror(errno));
+            printf("glfs_h_getattrs and setattrs tests: FAILED\n");
+            goto out;
+        }
+
+        assimilatetime(&c_ts, c_ts_st, c_ts_ed);
+        glfs_close(fd);
+    }
+
+    ret = clock_gettime(CLOCK_REALTIME, &o_ts_ed);
+    if (ret != 0) {
+        fprintf(stderr, "clock_gettime: error %s\n", strerror(errno));
+        printf("glfs_h_getattrs and setattrs tests: FAILED\n");
+        goto out;
+    }
+
+    assimilatetime(&o_ts, o_ts_st, o_ts_ed);
+
+    printf("Creation performance (path based):\n\t# empty files:%d\n",
+           MAX_FILES_CREATE);
+    printf("\tOverall time:\n\t\tSecs:%ld\n\t\tnSecs:%ld\n", o_ts.tv_sec,
+           o_ts.tv_nsec);
+    printf("\tcreate call time time:\n\t\tSecs:%ld\n\t\tnSecs:%ld\n",
+           c_ts.tv_sec, c_ts.tv_nsec);
 out:
-        return;
+    return;
 }
 
 int
-test_handleops (int argc, char *argv[])
+test_handleops(int argc, char *argv[])
 {
-        int                 ret = 0;
-        glfs_fd_t          *fd = NULL;
-        struct stat         sb = {0, };
-        struct glfs_object *root = NULL, *parent = NULL, *leaf = NULL,
-                           *tmp = NULL;
-        char                readbuf[32], writebuf[32];
-        unsigned char       leaf_handle[GFAPI_HANDLE_LENGTH];
-
-        char *full_leaf_name = "/testdir/testfile.txt",
-             *leaf_name = "testfile.txt",
-             *relative_leaf_name = "testdir/testfile.txt";
-        char *leaf_name1 = "testfile1.txt";
-        char *full_newparent_name = "/testdir/dir1",
-             *newparent_name = "dir1";
-        char *full_newnod_name = "/testdir/nod1",
-             *newnod_name = "nod1";
-
-        /* Initialize test area */
-        ret = glfs_mkdir (fs, full_parent_name, 0644);
-        if (ret != 0 && errno != EEXIST) {
-                fprintf (stderr, "%s: (%p) %s\n", full_parent_name, fd,
-                        strerror (errno));
-                printf ("Test initialization failed on volume %s\n", argv[1]);
-                goto out;
-        }
-        else if (ret != 0) {
-                printf ("Found test directory %s to be existing\n",
-                        full_parent_name);
-                printf ("Cleanup test directory and restart tests\n");
-                goto out;
-        }
-
-        fd = glfs_creat (fs, full_leaf_name, O_CREAT, 0644);
-        if (fd == NULL) {
-                fprintf (stderr, "%s: (%p) %s\n", full_leaf_name, fd,
-                        strerror (errno));
-                printf ("Test initialization failed on volume %s\n", argv[1]);
-                goto out;
-        }
-        glfs_close (fd);
-
-        printf ("Initialized the test area, within volume %s\n", argv[1]);
-
-        /* Handle based APIs test area */
-
-        /* glfs_lookupat test */
-        printf ("glfs_h_lookupat tests: In Progress\n");
-        /* start at root of the volume */
-        root = glfs_h_lookupat (fs, NULL, "/", &sb);
-        if (root == NULL) {
-                fprintf (stderr, "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
-                         "/", NULL, strerror (errno));
-                printf ("glfs_h_lookupat tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        /* lookup a parent within root */
-        parent = glfs_h_lookupat (fs, root, parent_name, &sb);
-        if (parent == NULL) {
-                fprintf (stderr, "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
-                         parent_name, root, strerror (errno));
-                printf ("glfs_h_lookupat tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        /* lookup a leaf/child within the parent */
-        leaf = glfs_h_lookupat (fs, parent, leaf_name, &sb);
-        if (leaf == NULL) {
-                fprintf (stderr, "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
-                         leaf_name, parent, strerror (errno));
-                printf ("glfs_h_lookupat tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        /* reset */
-        glfs_h_close (root); root = NULL;
-        glfs_h_close (leaf); leaf = NULL;
-        glfs_h_close (parent); parent = NULL;
-
-        /* check absolute paths */
-        root = glfs_h_lookupat (fs, NULL, "/", &sb);
-        if (root == NULL) {
-                fprintf (stderr, "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
-                         "/", NULL, strerror (errno));
-                printf ("glfs_h_lookupat tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        parent = glfs_h_lookupat (fs, NULL, full_parent_name, &sb);
-        if (parent == NULL) {
-                fprintf (stderr, "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
-                         full_parent_name, root, strerror (errno));
-                printf ("glfs_h_lookupat tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        leaf = glfs_h_lookupat (fs, NULL, full_leaf_name, &sb);
-        if (leaf == NULL) {
-                fprintf (stderr, "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
-                         full_leaf_name, parent, strerror (errno));
-                printf ("glfs_h_lookupat tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        /* reset */
-        glfs_h_close (leaf); leaf = NULL;
-
-        /* check multiple component paths */
-        leaf = glfs_h_lookupat (fs, root, relative_leaf_name, &sb);
-        if (leaf == NULL) {
-                fprintf (stderr, "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
-                         relative_leaf_name, parent, strerror (errno));
-                goto out;
-        }
-        peek_stat (&sb);
-
-        /* reset */
-        glfs_h_close (root); root = NULL;
-        glfs_h_close (parent); parent = NULL;
-
-        /* check symlinks in path */
-
-        /* TODO: -ve test cases */
-        /* parent invalid
-        * path invalid
-        * path does not exist after some components
-        * no parent, but relative path
-        * parent and full path? -ve?
-        */
-
-        printf ("glfs_h_lookupat tests: PASSED\n");
-
-        /* glfs_openat test */
-        printf ("glfs_h_open tests: In Progress\n");
-        fd = glfs_h_open (fs, leaf, O_RDWR);
-        if (fd == NULL) {
-                fprintf (stderr, "glfs_h_open: error on open of %s: %s\n",
-                         full_leaf_name, strerror (errno));
-                printf ("glfs_h_open tests: FAILED\n");
-                goto out;
-        }
-
-        /* test read/write based on fd */
-        memcpy (writebuf, "abcdefghijklmnopqrstuvwxyz012345", 32);
-        ret = glfs_write (fd, writebuf, 32, 0);
-
-        glfs_lseek (fd, 0, SEEK_SET);
-
-        ret = glfs_read (fd, readbuf, 32, 0);
-        if (memcmp (readbuf, writebuf, 32)) {
-                printf ("Failed to read what I wrote: %s %s\n", readbuf,
-                        writebuf);
-                glfs_close (fd);
-                printf ("glfs_h_open tests: FAILED\n");
-                goto out;
-        }
-
-        glfs_h_close (leaf); leaf = NULL;
-        glfs_close (fd);
-
-        printf ("glfs_h_open tests: PASSED\n");
-
-        /* Create tests */
-        printf ("glfs_h_creat tests: In Progress\n");
-        parent = glfs_h_lookupat (fs, NULL, full_parent_name, &sb);
-        if (parent == NULL) {
-                fprintf (stderr, "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
-                         full_parent_name, root, strerror (errno));
-                printf ("glfs_h_creat tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        leaf = glfs_h_creat (fs, parent, leaf_name1, O_CREAT, 0644, &sb);
-        if (leaf == NULL) {
-                fprintf (stderr, "glfs_h_creat: error on create of %s: from (%p),%s\n",
-                         leaf_name1, parent, strerror (errno));
-                printf ("glfs_h_creat tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        glfs_h_close (leaf); leaf = NULL;
-
-        leaf = glfs_h_creat (fs, parent, leaf_name1, O_CREAT | O_EXCL, 0644,
-                            &sb);
-        if (leaf != NULL || errno != EEXIST) {
-                fprintf (stderr, "glfs_h_creat: existing file, leaf = (%p), errno = %s\n",
-                        leaf, strerror (errno));
-                printf ("glfs_h_creat tests: FAILED\n");
-                if (leaf != NULL) {
-                        glfs_h_close (leaf); leaf = NULL;
-                }
-        }
-
-        tmp = glfs_h_creat (fs, root, parent_name, O_CREAT, 0644, &sb);
-        if (tmp != NULL || !(errno == EISDIR || errno == EINVAL)) {
-                fprintf (stderr, "glfs_h_creat: dir create, tmp = (%p), errno = %s\n",
-                        leaf, strerror (errno));
-                printf ("glfs_h_creat tests: FAILED\n");
-                if (tmp != NULL) {
-                        glfs_h_close (tmp); tmp = NULL;
-                }
-        }
-
-        /* TODO: Other combinations and -ve cases as applicable */
-        printf ("glfs_h_creat tests: PASSED\n");
-
-        /* extract handle and create from handle test */
-        printf ("glfs_h_extract_handle and glfs_h_create_from_handle tests: In Progress\n");
-        /* TODO: Change the lookup to creat below for a GIFD recovery falure,
-         * that needs to be fixed */
-        leaf = glfs_h_lookupat (fs, parent, leaf_name1, &sb);
-        if (leaf == NULL) {
-                fprintf (stderr, "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
-                         leaf_name1, parent, strerror (errno));
-                printf ("glfs_h_extract_handle tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        ret = glfs_h_extract_handle (leaf, leaf_handle,
-                                             GFAPI_HANDLE_LENGTH);
-        if (ret < 0) {
-                fprintf (stderr, "glfs_h_extract_handle: error extracting handle of %s: %s\n",
-                         full_leaf_name, strerror (errno));
-                printf ("glfs_h_extract_handle tests: FAILED\n");
-                goto out;
-        }
-        peek_handle (leaf_handle);
+    int ret = 0;
+    glfs_fd_t *fd = NULL;
+    struct stat sb = {
+        0,
+    };
+    struct glfs_object *root = NULL, *parent = NULL, *leaf = NULL, *tmp = NULL;
+    char readbuf[32], writebuf[32];
+    unsigned char leaf_handle[GFAPI_HANDLE_LENGTH];
+
+    char *full_leaf_name = "/testdir/testfile.txt", *leaf_name = "testfile.txt",
+         *relative_leaf_name = "testdir/testfile.txt";
+    char *leaf_name1 = "testfile1.txt";
+    char *full_newparent_name = "/testdir/dir1", *newparent_name = "dir1";
+    char *full_newnod_name = "/testdir/nod1", *newnod_name = "nod1";
+
+    /* Initialize test area */
+    ret = glfs_mkdir(fs, full_parent_name, 0755);
+    if (ret != 0 && errno != EEXIST) {
+        fprintf(stderr, "%s: (%p) %s\n", full_parent_name, fd, strerror(errno));
+        printf("Test initialization failed on volume %s\n", argv[1]);
+        goto out;
+    } else if (ret != 0) {
+        printf("Found test directory %s to be existing\n", full_parent_name);
+        printf("Cleanup test directory and restart tests\n");
+        goto out;
+    }
+
+    fd = glfs_creat(fs, full_leaf_name, O_CREAT, 0644);
+    if (fd == NULL) {
+        fprintf(stderr, "%s: (%p) %s\n", full_leaf_name, fd, strerror(errno));
+        printf("Test initialization failed on volume %s\n", argv[1]);
+        goto out;
+    }
+    glfs_close(fd);
+
+    printf("Initialized the test area, within volume %s\n", argv[1]);
+
+    /* Handle based APIs test area */
+
+    /* glfs_lookupat test */
+    printf("glfs_h_lookupat tests: In Progress\n");
+    /* start at root of the volume */
+    root = glfs_h_lookupat(fs, NULL, "/", &sb, 0);
+    if (root == NULL) {
+        fprintf(stderr,
+                "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n", "/",
+                NULL, strerror(errno));
+        printf("glfs_h_lookupat tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    /* lookup a parent within root */
+    parent = glfs_h_lookupat(fs, root, parent_name, &sb, 0);
+    if (parent == NULL) {
+        fprintf(stderr,
+                "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
+                parent_name, root, strerror(errno));
+        printf("glfs_h_lookupat tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    /* lookup a leaf/child within the parent */
+    leaf = glfs_h_lookupat(fs, parent, leaf_name, &sb, 0);
+    if (leaf == NULL) {
+        fprintf(stderr,
+                "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
+                leaf_name, parent, strerror(errno));
+        printf("glfs_h_lookupat tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    /* reset */
+    glfs_h_close(root);
+    root = NULL;
+    glfs_h_close(leaf);
+    leaf = NULL;
+    glfs_h_close(parent);
+    parent = NULL;
+
+    /* check absolute paths */
+    root = glfs_h_lookupat(fs, NULL, "/", &sb, 0);
+    if (root == NULL) {
+        fprintf(stderr,
+                "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n", "/",
+                NULL, strerror(errno));
+        printf("glfs_h_lookupat tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    parent = glfs_h_lookupat(fs, NULL, full_parent_name, &sb, 0);
+    if (parent == NULL) {
+        fprintf(stderr,
+                "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
+                full_parent_name, root, strerror(errno));
+        printf("glfs_h_lookupat tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    leaf = glfs_h_lookupat(fs, NULL, full_leaf_name, &sb, 0);
+    if (leaf == NULL) {
+        fprintf(stderr,
+                "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
+                full_leaf_name, parent, strerror(errno));
+        printf("glfs_h_lookupat tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    /* reset */
+    glfs_h_close(leaf);
+    leaf = NULL;
+
+    /* check multiple component paths */
+    leaf = glfs_h_lookupat(fs, root, relative_leaf_name, &sb, 0);
+    if (leaf == NULL) {
+        fprintf(stderr,
+                "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
+                relative_leaf_name, parent, strerror(errno));
+        goto out;
+    }
+    peek_stat(&sb);
+
+    /* reset */
+    glfs_h_close(root);
+    root = NULL;
+    glfs_h_close(parent);
+    parent = NULL;
+
+    /* check symlinks in path */
+
+    /* TODO: -ve test cases */
+    /* parent invalid
+     * path invalid
+     * path does not exist after some components
+     * no parent, but relative path
+     * parent and full path? -ve?
+     */
+
+    printf("glfs_h_lookupat tests: PASSED\n");
+
+    /* glfs_openat test */
+    printf("glfs_h_open tests: In Progress\n");
+    fd = glfs_h_open(fs, leaf, O_RDWR);
+    if (fd == NULL) {
+        fprintf(stderr, "glfs_h_open: error on open of %s: %s\n",
+                full_leaf_name, strerror(errno));
+        printf("glfs_h_open tests: FAILED\n");
+        goto out;
+    }
+
+    /* test read/write based on fd */
+    memcpy(writebuf, "abcdefghijklmnopqrstuvwxyz012345", 32);
+    ret = glfs_write(fd, writebuf, 32, 0);
+
+    glfs_lseek(fd, 0, SEEK_SET);
+
+    ret = glfs_read(fd, readbuf, 32, 0);
+    if (memcmp(readbuf, writebuf, 32)) {
+        printf("Failed to read what I wrote: %s %s\n", readbuf, writebuf);
+        glfs_close(fd);
+        printf("glfs_h_open tests: FAILED\n");
+        goto out;
+    }
+
+    glfs_h_close(leaf);
+    leaf = NULL;
+    glfs_close(fd);
+
+    printf("glfs_h_open tests: PASSED\n");
+
+    /* Create tests */
+    printf("glfs_h_creat tests: In Progress\n");
+    parent = glfs_h_lookupat(fs, NULL, full_parent_name, &sb, 0);
+    if (parent == NULL) {
+        fprintf(stderr,
+                "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
+                full_parent_name, root, strerror(errno));
+        printf("glfs_h_creat tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    leaf = glfs_h_creat(fs, parent, leaf_name1, O_CREAT, 0644, &sb);
+    if (leaf == NULL) {
+        fprintf(stderr, "glfs_h_creat: error on create of %s: from (%p),%s\n",
+                leaf_name1, parent, strerror(errno));
+        printf("glfs_h_creat tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    glfs_h_close(leaf);
+    leaf = NULL;
+
+    leaf = glfs_h_creat(fs, parent, leaf_name1, O_CREAT | O_EXCL, 0644, &sb);
+    if (leaf != NULL || errno != EEXIST) {
+        fprintf(stderr,
+                "glfs_h_creat: existing file, leaf = (%p), errno = %s\n", leaf,
+                strerror(errno));
+        printf("glfs_h_creat tests: FAILED\n");
+        if (leaf != NULL) {
+            glfs_h_close(leaf);
+            leaf = NULL;
+        }
+    }
+
+    tmp = glfs_h_creat(fs, root, parent_name, O_CREAT, 0644, &sb);
+    if (tmp != NULL || !(errno == EISDIR || errno == EINVAL)) {
+        fprintf(stderr, "glfs_h_creat: dir create, tmp = (%p), errno = %s\n",
+                leaf, strerror(errno));
+        printf("glfs_h_creat tests: FAILED\n");
+        if (tmp != NULL) {
+            glfs_h_close(tmp);
+            tmp = NULL;
+        }
+    }
+
+    /* TODO: Other combinations and -ve cases as applicable */
+    printf("glfs_h_creat tests: PASSED\n");
+
+    /* extract handle and create from handle test */
+    printf(
+        "glfs_h_extract_handle and glfs_h_create_from_handle tests: In "
+        "Progress\n");
+    /* TODO: Change the lookup to create below for a GIFD recovery failure,
+     * that needs to be fixed */
+    leaf = glfs_h_lookupat(fs, parent, leaf_name1, &sb, 0);
+    if (leaf == NULL) {
+        fprintf(stderr,
+                "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
+                leaf_name1, parent, strerror(errno));
+        printf("glfs_h_extract_handle tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    ret = glfs_h_extract_handle(leaf, leaf_handle, GFAPI_HANDLE_LENGTH);
+    if (ret < 0) {
+        fprintf(stderr,
+                "glfs_h_extract_handle: error extracting handle of %s: %s\n",
+                full_leaf_name, strerror(errno));
+        printf("glfs_h_extract_handle tests: FAILED\n");
+        goto out;
+    }
+    peek_handle(leaf_handle);
+
+    glfs_h_close(leaf);
+    leaf = NULL;
+
+    leaf = glfs_h_create_from_handle(fs, leaf_handle, GFAPI_HANDLE_LENGTH, &sb);
+    if (leaf == NULL) {
+        fprintf(
+            stderr,
+            "glfs_h_create_from_handle: error on create of %s: from (%p),%s\n",
+            leaf_name1, leaf_handle, strerror(errno));
+        printf("glfs_h_create_from_handle tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    fd = glfs_h_open(fs, leaf, O_RDWR);
+    if (fd == NULL) {
+        fprintf(stderr, "glfs_h_open: error on open of %s: %s\n",
+                full_leaf_name, strerror(errno));
+        printf("glfs_h_create_from_handle tests: FAILED\n");
+        goto out;
+    }
+
+    /* test read/write based on fd */
+    memcpy(writebuf, "abcdefghijklmnopqrstuvwxyz012345", 32);
+    ret = glfs_write(fd, writebuf, 32, 0);
+
+    glfs_lseek(fd, 0, SEEK_SET);
+
+    ret = glfs_read(fd, readbuf, 32, 0);
+    if (memcmp(readbuf, writebuf, 32)) {
+        printf("Failed to read what I wrote: %s %s\n", writebuf, writebuf);
+        printf("glfs_h_create_from_handle tests: FAILED\n");
+        glfs_close(fd);
+        goto out;
+    }
+
+    glfs_close(fd);
+    glfs_h_close(leaf);
+    leaf = NULL;
+    glfs_h_close(parent);
+    parent = NULL;
+
+    printf(
+        "glfs_h_extract_handle and glfs_h_create_from_handle tests: PASSED\n");
+
+    /* Mkdir tests */
+    printf("glfs_h_mkdir tests: In Progress\n");
+
+    ret = glfs_rmdir(fs, full_newparent_name);
+    if (ret && errno != ENOENT) {
+        fprintf(stderr, "glfs_rmdir: Failed for %s: %s\n", full_newparent_name,
+                strerror(errno));
+        printf("glfs_h_mkdir tests: FAILED\n");
+        goto out;
+    }
+
+    parent = glfs_h_lookupat(fs, NULL, full_parent_name, &sb, 0);
+    if (parent == NULL) {
+        fprintf(stderr,
+                "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
+                full_parent_name, root, strerror(errno));
+        printf("glfs_h_mkdir tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    leaf = glfs_h_mkdir(fs, parent, newparent_name, 0755, &sb);
+    if (leaf == NULL) {
+        fprintf(stderr, "glfs_h_mkdir: error on mkdir of %s: from (%p),%s\n",
+                newparent_name, parent, strerror(errno));
+        printf("glfs_h_mkdir tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    glfs_h_close(leaf);
+    leaf = NULL;
+
+    leaf = glfs_h_mkdir(fs, parent, newparent_name, 0755, &sb);
+    if (leaf != NULL || errno != EEXIST) {
+        fprintf(stderr,
+                "glfs_h_mkdir: existing directory, leaf = (%p), errno = %s\n",
+                leaf, strerror(errno));
+        printf("glfs_h_mkdir tests: FAILED\n");
+        if (leaf != NULL) {
+            glfs_h_close(leaf);
+            leaf = NULL;
+        }
+    }
+
+    glfs_h_close(parent);
+    parent = NULL;
+
+    printf("glfs_h_mkdir tests: PASSED\n");
+
+    /* Mknod tests */
+    printf("glfs_h_mknod tests: In Progress\n");
+    ret = glfs_unlink(fs, full_newnod_name);
+    if (ret && errno != ENOENT) {
+        fprintf(stderr, "glfs_unlink: Failed for %s: %s\n", full_newnod_name,
+                strerror(errno));
+        printf("glfs_h_mknod tests: FAILED\n");
+        goto out;
+    }
+
+    parent = glfs_h_lookupat(fs, NULL, full_parent_name, &sb, 0);
+    if (parent == NULL) {
+        fprintf(stderr,
+                "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
+                full_parent_name, root, strerror(errno));
+        printf("glfs_h_mknod tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    leaf = glfs_h_mknod(fs, parent, newnod_name, S_IFIFO, 0, &sb);
+    if (leaf == NULL) {
+        fprintf(stderr, "glfs_h_mkdir: error on mkdir of %s: from (%p),%s\n",
+                newnod_name, parent, strerror(errno));
+        printf("glfs_h_mknod tests: FAILED\n");
+        goto out;
+    }
+    peek_stat(&sb);
+
+    /* TODO: create op on a FIFO node hangs, need to check and fix
+    tmp = glfs_h_creat (fs, parent, newnod_name, O_CREAT, 0644, &sb);
+    if (tmp != NULL || errno != EINVAL) {
+            fprintf (stderr, "glfs_h_creat: node create, tmp = (%p), errno =
+    %s\n", tmp, strerror (errno)); printf ("glfs_h_creat/mknod tests:
+    FAILED\n"); if (tmp != NULL) { glfs_h_close(tmp); tmp = NULL;
+            }
+    } */
+
+    glfs_h_close(leaf);
+    leaf = NULL;
+
+    leaf = glfs_h_mknod(fs, parent, newnod_name, 0644, 0, &sb);
+    if (leaf != NULL || errno != EEXIST) {
+        fprintf(stderr,
+                "glfs_h_mknod: existing node, leaf = (%p), errno = %s\n", leaf,
+                strerror(errno));
+        printf("glfs_h_mknod tests: FAILED\n");
+        if (leaf != NULL) {
+            glfs_h_close(leaf);
+            leaf = NULL;
+        }
+    }
+
+    glfs_h_close(parent);
+    parent = NULL;
+
+    printf("glfs_h_mknod tests: PASSED\n");
+
+    /* unlink tests */
+    test_h_unlink();
+
+    /* TODO: opendir tests */
+
+    /* getattr tests */
+    test_h_getsetattrs();
+
+    /* TODO: setattr tests */
+
+    /* truncate tests */
+    test_h_truncate();
+
+    /* link tests */
+    test_h_links();
+
+    /* rename tests */
+    test_h_rename();
+
+    /* performance tests */
+    test_h_performance();
+
+    /* END: New APIs test area */
 
-        glfs_h_close (leaf); leaf = NULL;
-
-        leaf = glfs_h_create_from_handle (fs, leaf_handle, GFAPI_HANDLE_LENGTH,
-                                          &sb);
-        if (leaf == NULL) {
-                fprintf (stderr, "glfs_h_create_from_handle: error on create of %s: from (%p),%s\n",
-                         leaf_name1, leaf_handle, strerror (errno));
-                printf ("glfs_h_create_from_handle tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        fd = glfs_h_open (fs, leaf, O_RDWR);
-        if (fd == NULL) {
-                fprintf (stderr, "glfs_h_open: error on open of %s: %s\n",
-                         full_leaf_name, strerror (errno));
-                printf ("glfs_h_create_from_handle tests: FAILED\n");
-                goto out;
-        }
-
-        /* test read/write based on fd */
-        memcpy (writebuf, "abcdefghijklmnopqrstuvwxyz012345", 32);
-        ret = glfs_write (fd, writebuf, 32, 0);
-
-        glfs_lseek (fd, 0, SEEK_SET);
-
-        ret = glfs_read (fd, readbuf, 32, 0);
-        if (memcmp (readbuf, writebuf, 32)) {
-                printf ("Failed to read what I wrote: %s %s\n", writebuf,
-                        writebuf);
-                printf ("glfs_h_create_from_handle tests: FAILED\n");
-                glfs_close (fd);
-                goto out;
-        }
-
-        glfs_close (fd);
-        glfs_h_close (leaf); leaf = NULL;
-        glfs_h_close (parent); parent = NULL;
-
-        printf ("glfs_h_extract_handle and glfs_h_create_from_handle tests: PASSED\n");
-
-        /* Mkdir tests */
-        printf ("glfs_h_mkdir tests: In Progress\n");
+out:
+    /* Cleanup glfs handles */
+    if (root)
+        glfs_h_close(root);
+    if (parent)
+        glfs_h_close(parent);
+    if (leaf)
+        glfs_h_close(leaf);
+
+    return ret;
+}
 
-        ret = glfs_rmdir (fs, full_newparent_name);
-        if (ret && errno != ENOENT) {
-                fprintf (stderr, "glfs_rmdir: Failed for %s: %s\n",
-                         full_newparent_name, strerror (errno));
-                printf ("glfs_h_mkdir tests: FAILED\n");
-                goto out;
-        }
+int
+test_write_apis(glfs_t *fs)
+{
+    /* Add more content here */
+    /* Some apis we can get are */
+    /*
+      0. glfs_set_xlator_option()
+
+      Read/Write combinations:
+      . glfs_{p,}readv/{p,}writev
+      . glfs_pread/pwrite
+
+      tests/basic/gfapi/gfapi-async-calls-test.c
+      . glfs_read_async/write_async
+      . glfs_pread_async/pwrite_async
+      . glfs_readv_async/writev_async
+      . glfs_preadv_async/pwritev_async
+
+      . ftruncate/ftruncate_async
+      . fsync/fsync_async
+      . fdatasync/fdatasync_async
+
+    */
+
+    glfs_fd_t *fd = NULL;
+    char *filename = "/filename2";
+    int flags = O_RDWR;
+    char *buf = "some bytes!";
+    char writestr[TEST_STR_LEN];
+    struct iovec iov = {&writestr, TEST_STR_LEN};
+    int ret, i;
+
+    for (i = 0; i < TEST_STR_LEN; i++)
+        writestr[i] = 0x11;
+
+    fd = glfs_open(fs, filename, flags);
+    if (!fd)
+        fprintf(stderr, "open(%s): (%p) %s\n", filename, fd, strerror(errno));
+
+    ret = glfs_writev(fd, &iov, 1, flags);
+    if (ret < 0) {
+        fprintf(stderr, "writev(%s): %d (%s)\n", filename, ret,
+                strerror(errno));
+    }
+
+    ret = glfs_pwrite(fd, buf, 10, 4, flags, NULL, NULL);
+    if (ret < 0) {
+        fprintf(stderr, "pwrite(%s): %d (%s)\n", filename, ret,
+                strerror(errno));
+    }
+
+    ret = glfs_pwritev(fd, &iov, 1, 4, flags);
+    if (ret < 0) {
+        fprintf(stderr, "pwritev(%s): %d (%s)\n", filename, ret,
+                strerror(errno));
+    }
+
+    return 0;
+}
 
-        parent = glfs_h_lookupat (fs, NULL, full_parent_name, &sb);
-        if (parent == NULL) {
-                fprintf (stderr, "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
-                         full_parent_name, root, strerror (errno));
-                printf ("glfs_h_mkdir tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
+int
+test_metadata_ops(glfs_t *fs, glfs_t *fs2)
+{
+    glfs_fd_t *fd = NULL;
+    glfs_fd_t *fd2 = NULL;
+    struct stat sb = {
+        0,
+    };
+    struct glfs_stat gsb = {
+        0,
+    };
+    struct statvfs sfs;
+    char readbuf[32];
+    char writebuf[32];
 
-        leaf = glfs_h_mkdir (fs, parent, newparent_name, 0644, &sb);
-        if (leaf == NULL) {
-                fprintf (stderr, "glfs_h_mkdir: error on mkdir of %s: from (%p),%s\n",
-                         newparent_name, parent, strerror (errno));
-                printf ("glfs_h_mkdir tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        glfs_h_close (leaf); leaf = NULL;
-
-        leaf = glfs_h_mkdir (fs, parent, newparent_name, 0644, &sb);
-        if (leaf != NULL || errno != EEXIST) {
-                fprintf (stderr, "glfs_h_mkdir: existing directory, leaf = (%p), errno = %s\n",
-                         leaf, strerror (errno));
-                printf ("glfs_h_mkdir tests: FAILED\n");
-                if (leaf != NULL) {
-                        glfs_h_close (leaf); leaf = NULL;
-                }
-        }
+    char *filename = "/filename2";
+    int ret;
 
-        glfs_h_close (parent); parent = NULL;
+    ret = glfs_lstat(fs, filename, &sb);
+    fprintf(stderr, "lstat(%s): (%d) %s\n", filename, ret, strerror(errno));
 
-        printf ("glfs_h_mkdir tests: PASSED\n");
+    fd = glfs_creat(fs, filename, O_RDWR, 0644);
+    fprintf(stderr, "creat(%s): (%p) %s\n", filename, fd, strerror(errno));
 
-        /* Mknod tests */
-        printf ("glfs_h_mknod tests: In Progress\n");
-        ret = glfs_unlink (fs, full_newnod_name);
-        if (ret && errno != ENOENT) {
-                fprintf (stderr, "glfs_unlink: Failed for %s: %s\n",
-                         full_newnod_name, strerror (errno));
-                printf ("glfs_h_mknod tests: FAILED\n");
-                goto out;
-        }
+    fd2 = glfs_open(fs2, filename, O_RDWR);
+    fprintf(stderr, "open(%s): (%p) %s\n", filename, fd, strerror(errno));
 
-        parent = glfs_h_lookupat (fs, NULL, full_parent_name, &sb);
-        if (parent == NULL) {
-                fprintf (stderr, "glfs_h_lookupat: error on lookup of %s: from (%p),%s\n",
-                         full_parent_name, root, strerror (errno));
-                printf ("glfs_h_mknod tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
+    glfs_lseek(fd2, 0, SEEK_SET);
 
-        leaf = glfs_h_mknod (fs, parent, newnod_name, S_IFIFO, 0, &sb);
-        if (leaf == NULL) {
-                fprintf (stderr, "glfs_h_mkdir: error on mkdir of %s: from (%p),%s\n",
-                         newnod_name, parent, strerror (errno));
-                printf ("glfs_h_mknod tests: FAILED\n");
-                goto out;
-        }
-        peek_stat (&sb);
-
-        /* TODO: creat op on a FIFO node hangs, need to check and fix
-        tmp = glfs_h_creat (fs, parent, newnod_name, O_CREAT, 0644, &sb);
-        if (tmp != NULL || errno != EINVAL) {
-                fprintf (stderr, "glfs_h_creat: node create, tmp = (%p), errno = %s\n",
-                        tmp, strerror (errno));
-                printf ("glfs_h_creat/mknod tests: FAILED\n");
-                if (tmp != NULL) {
-                        glfs_h_close(tmp); tmp = NULL;
-                }
-        } */
-
-        glfs_h_close (leaf); leaf = NULL;
-
-        leaf = glfs_h_mknod (fs, parent, newnod_name, 0644, 0, &sb);
-        if (leaf != NULL || errno != EEXIST) {
-                fprintf (stderr, "glfs_h_mknod: existing node, leaf = (%p), errno = %s\n",
-                         leaf, strerror (errno));
-                printf ("glfs_h_mknod tests: FAILED\n");
-                if (leaf != NULL) {
-                        glfs_h_close (leaf); leaf = NULL;
-                }
-        }
+    ret = glfs_read(fd2, readbuf, 32, 0);
 
-        glfs_h_close (parent); parent = NULL;
+    printf("read %d, %s", ret, readbuf);
 
-        printf ("glfs_h_mknod tests: PASSED\n");
+    /* get stat */
+    ret = glfs_fstat(fd2, &sb);
 
-        /* unlink tests */
-        test_h_unlink ();
+    ret = glfs_access(fs, filename, R_OK);
 
-        /* TODO: opendir tests */
+    /* set stat */
+    /* TODO: got some errors, need to fix */
+    /* ret = glfs_fsetattr(fd2, &gsb); */
 
-        /* getattr tests */
-        test_h_getsetattrs ();
+    glfs_close(fd);
+    glfs_close(fd2);
 
-        /* TODO: setattr tests */
+    filename = "/filename3";
+    ret = glfs_mknod(fs, filename, S_IFIFO, 0);
+    fprintf(stderr, "%s: (%d) %s\n", filename, ret, strerror(errno));
 
-        /* truncate tests */
-        test_h_truncate();
+    ret = glfs_lstat(fs, filename, &sb);
+    fprintf(stderr, "%s: (%d) %s\n", filename, ret, strerror(errno));
 
-        /* link tests */
-        test_h_links ();
+    ret = glfs_rename(fs, filename, "/filename4");
+    fprintf(stderr, "rename(%s): (%d) %s\n", filename, ret, strerror(errno));
 
-        /* rename tests */
-        test_h_rename ();
+    ret = glfs_unlink(fs, "/filename4");
+    fprintf(stderr, "unlink(%s): (%d) %s\n", "/filename4", ret,
+            strerror(errno));
 
-        /* performance tests */
-        test_h_performance ();
+    filename = "/dirname2";
+    ret = glfs_mkdir(fs, filename, 0);
+    fprintf(stderr, "%s: (%d) %s\n", filename, ret, strerror(errno));
 
-        /* END: New APIs test area */
+    ret = glfs_lstat(fs, filename, &sb);
+    fprintf(stderr, "lstat(%s): (%d) %s\n", filename, ret, strerror(errno));
 
-out:
-        /* Cleanup glfs handles */
-        if (root)
-                glfs_h_close (root);
-        if (parent)
-                glfs_h_close (parent);
-        if (leaf)
-                glfs_h_close (leaf);
-
-        return ret;
+    ret = glfs_rmdir(fs, filename);
+    fprintf(stderr, "rmdir(%s): (%d) %s\n", filename, ret, strerror(errno));
 }
-
 int
-main (int argc, char *argv[])
+main(int argc, char *argv[])
 {
-        glfs_t    *fs2 = NULL;
-        int        ret = 0;
-        glfs_fd_t *fd = NULL;
-        glfs_fd_t *fd2 = NULL;
-        struct stat sb = {0, };
-        char       readbuf[32];
-        char       writebuf[32];
-
-        char      *filename = "/filename2";
-
-        if (argc != 3) {
-                printf ("Expect following args\n\t%s <volname> <hostname>\n", argv[0]);
-                return -1;
-        }
-
-        fs = glfs_new (argv[1]);
-        if (!fs) {
-                fprintf (stderr, "glfs_new: returned NULL\n");
-                return 1;
-        }
-
-//      ret = glfs_set_volfile (fs, "/tmp/posix.vol");
-
-        ret = glfs_set_volfile_server (fs, "tcp", argv[2], 24007);
-
-//      ret = glfs_set_volfile_server (fs, "unix", "/tmp/gluster.sock", 0);
-
-        ret = glfs_set_logging (fs, "/dev/stderr", 7);
+    glfs_t *fs2 = NULL;
+    int ret = 0;
+    glfs_fd_t *fd = NULL;
+    glfs_fd_t *fd2 = NULL;
+    struct stat sb = {
+        0,
+    };
+    struct glfs_stat gsb = {
+        0,
+    };
+    struct statvfs sfs;
+    char readbuf[32];
+    char writebuf[32];
 
-        ret = glfs_init (fs);
+    char *filename = "/filename2";
 
-        fprintf (stderr, "glfs_init: returned %d\n", ret);
+    if (argc != 3) {
+        printf("Expect following args\n\t%s <volname> <hostname>\n", argv[0]);
+        return -1;
+    }
 
-        sleep (2);
+    fs = glfs_new(argv[1]);
+    if (!fs) {
+        fprintf(stderr, "glfs_new: returned NULL\n");
+        return 1;
+    }
 
-        fs2 = glfs_new (argv[1]);
-        if (!fs2) {
-                fprintf (stderr, "glfs_new: returned NULL\n");
-                return 1;
-        }
-
-
-//      ret = glfs_set_volfile (fs2, "/tmp/posix.vol");
-
-        ret = glfs_set_volfile_server (fs2, "tcp", argv[2], 24007);
+    //      ret = glfs_set_volfile (fs, "/tmp/posix.vol");
 
-        ret = glfs_set_logging (fs2, "/dev/stderr", 7);
+    ret = glfs_set_volfile_server(fs, "tcp", argv[2], 24007);
 
-        ret = glfs_init (fs2);
+    //      ret = glfs_set_volfile_server (fs, "unix", "/tmp/gluster.sock", 0);
 
-        fprintf (stderr, "glfs_init: returned %d\n", ret);
+    ret = glfs_set_logging(fs, "/dev/stderr", 7);
 
-        ret = glfs_lstat (fs, filename, &sb);
-        fprintf (stderr, "%s: (%d) %s\n", filename, ret, strerror (errno));
+    ret = glfs_init(fs);
 
-        fd = glfs_creat (fs, filename, O_RDWR, 0644);
-        fprintf (stderr, "%s: (%p) %s\n", filename, fd, strerror (errno));
+    fprintf(stderr, "glfs_init: returned %d\n", ret);
 
-        fd2 = glfs_open (fs2, filename, O_RDWR);
-        fprintf (stderr, "%s: (%p) %s\n", filename, fd, strerror (errno));
+    if (ret)
+        goto out;
 
-        sprintf (writebuf, "hi there\n");
-        ret = glfs_write (fd, writebuf, 32, 0);
+    sleep(2);
 
-        glfs_lseek (fd2, 0, SEEK_SET);
+    fs2 = glfs_new(argv[1]);
+    if (!fs2) {
+        fprintf(stderr, "glfs_new: returned NULL\n");
+        return 1;
+    }
 
-        ret = glfs_read (fd2, readbuf, 32, 0);
+    //      ret = glfs_set_volfile (fs2, "/tmp/posix.vol");
 
-        printf ("read %d, %s", ret, readbuf);
+    ret = glfs_set_volfile_server(fs2, "tcp", argv[2], 24007);
 
-        glfs_close (fd);
-        glfs_close (fd2);
+    ret = glfs_set_logging(fs2, "/dev/stderr", 7);
 
-        filename = "/filename3";
-        ret = glfs_mknod (fs, filename, S_IFIFO, 0);
-        fprintf (stderr, "%s: (%d) %s\n", filename, ret, strerror (errno));
+    ret = glfs_init(fs2);
 
-        ret = glfs_lstat (fs, filename, &sb);
-        fprintf (stderr, "%s: (%d) %s\n", filename, ret, strerror (errno));
+    fprintf(stderr, "glfs_init: returned %d\n", ret);
 
+    test_metadata_ops(fs, fs2);
 
-        ret = glfs_rename (fs, filename, "/filename4");
-        fprintf (stderr, "rename(%s): (%d) %s\n", filename, ret,
-                 strerror (errno));
+    test_dirops(fs);
 
-        ret = glfs_unlink (fs, "/filename4");
-        fprintf (stderr, "unlink(%s): (%d) %s\n", "/filename4", ret,
-                 strerror (errno));
+    test_xattr(fs);
 
-        filename = "/dirname2";
-        ret = glfs_mkdir (fs, filename, 0);
-        fprintf (stderr, "%s: (%d) %s\n", filename, ret, strerror (errno));
+    test_chdir(fs);
 
-        ret = glfs_lstat (fs, filename, &sb);
-        fprintf (stderr, "lstat(%s): (%d) %s\n", filename, ret, strerror (errno));
+    test_handleops(argc, argv);
+    // done
 
-        ret = glfs_rmdir (fs, filename);
-        fprintf (stderr, "rmdir(%s): (%d) %s\n", filename, ret, strerror (errno));
+    /* Test some extra apis */
+    test_write_apis(fs);
 
-        test_dirops (fs);
+    glfs_statvfs(fs, "/", &sfs);
 
-        test_xattr (fs);
+    glfs_fini(fs);
+    glfs_fini(fs2);
 
-        test_chdir (fs);
-
-        test_handleops (argc, argv);
-        // done
-
-        glfs_fini (fs);
-        glfs_fini (fs2);
-
-        return ret;
+    ret = 0;
+out:
+    return ret;
 }
diff --git a/api/src/Makefile.am b/api/src/Makefile.am
index 0a2101fe549..7f9a7d17b35 100644
--- a/api/src/Makefile.am
+++ b/api/src/Makefile.am
@@ -1,22 +1,27 @@
 lib_LTLIBRARIES = libgfapi.la
-noinst_HEADERS = glfs-mem-types.h glfs-internal.h
+noinst_HEADERS = glfs-mem-types.h glfs-internal.h gfapi-messages.h
 libgfapi_HEADERS = glfs.h glfs-handles.h
 libgfapidir = $(includedir)/glusterfs/api
 
+EXTRA_DIST = gfapi.map gfapi.aliases
+
 libgfapi_la_SOURCES = glfs.c glfs-mgmt.c glfs-fops.c glfs-resolve.c \
 	glfs-handleops.c
 libgfapi_la_LIBADD = $(top_builddir)/libglusterfs/src/libglusterfs.la \
 	$(top_builddir)/rpc/rpc-lib/src/libgfrpc.la \
-	$(top_builddir)/rpc/xdr/src/libgfxdr.la \
-	$(GF_LDADD)
+	$(top_builddir)/rpc/xdr/src/libgfxdr.la
 
-libgfapi_la_CPPFLAGS = $(GF_CPPFLAGS) -D__USE_FILE_OFFSET64 \
-	-I$(top_srcdir)/libglusterfs/src \
-	-I$(top_srcdir)/rpc/rpc-lib/src \
-	-I$(top_srcdir)/rpc/xdr/src
+libgfapi_la_LDFLAGS = -version-info $(GFAPI_LT_VERSION) $(GF_LDFLAGS) \
+	$(GFAPI_EXTRA_LDFLAGS) $(ACL_LIBS)
 
-libgfapi_la_LDFLAGS = -version-info $(GFAPI_LT_VERSION) \
-	-Wl,--version-script=gfapi.map
+libgfapi_la_CPPFLAGS = $(GF_CPPFLAGS) -I$(top_srcdir)/libglusterfs/src \
+        -I$(top_srcdir)/rpc/rpc-lib/src \
+        -I$(top_srcdir)/rpc/xdr/src \
+        -I$(top_builddir)/rpc/xdr/src \
+        -DDATADIR=\"$(localstatedir)\" \
+        -D__USE_FILE_OFFSET64 -D__USE_LARGEFILE64
+
+AM_CFLAGS = -Wall $(GF_CFLAGS)
 
 xlator_LTLIBRARIES = api.la
 xlatordir = $(libdir)/glusterfs/$(PACKAGE_VERSION)/xlator/mount
@@ -27,12 +32,12 @@ $(install_xlatorLTLIBRARIES): install-libLTLIBRARIES
 
 api_la_SOURCES = glfs-master.c
 api_la_DEPENDENCIES = libgfapi.la
-api_la_LDFLAGS = -module -avoid-version
+api_la_CPPFLAGS = $(GF_CPPFLAGS) -I$(top_srcdir)/libglusterfs/src \
+	-I$(top_srcdir)/rpc/xdr/src \
+	-I$(top_builddir)/rpc/xdr/src
+api_la_LDFLAGS = -module $(GF_XLATOR_DEFAULT_LDFLAGS)
+#api_la_LDFLAGS = -module $(GF_XLATOR_DEFAULT_LDFLAGS) $(GF_LDFLAGS)
 api_la_LIBADD = $(top_builddir)/libglusterfs/src/libglusterfs.la \
 	$(top_builddir)/rpc/rpc-lib/src/libgfrpc.la \
 	$(top_builddir)/rpc/xdr/src/libgfxdr.la \
 	$(top_builddir)/api/src/libgfapi.la
-
-AM_CPPFLAGS = $(GF_CPPFLAGS) -I$(top_srcdir)/libglusterfs/src
-
-AM_CFLAGS = -Wall $(GF_CFLAGS)
diff --git a/api/src/README.Symbol_Versions b/api/src/README.Symbol_Versions
new file mode 100644
index 00000000000..b6ec95f9311
--- /dev/null
+++ b/api/src/README.Symbol_Versions
@@ -0,0 +1,3 @@
+
+See ../../doc/developer-guide/gfapi-symbol-versions.md
+
diff --git a/api/src/gfapi-messages.h b/api/src/gfapi-messages.h
new file mode 100644
index 00000000000..b9223940416
--- /dev/null
+++ b/api/src/gfapi-messages.h
@@ -0,0 +1,147 @@
+/*
+ *   Copyright (c) 2015-2018 Red Hat, Inc. <http://www.redhat.com>
+ *   This file is part of GlusterFS.
+ *
+ *   This file is licensed to you under your choice of the GNU Lesser
+ *   General Public License, version 3 or any later version (LGPLv3 or
+ *   later), or the GNU General Public License, version 2 (GPLv2), in all
+ *   cases as published by the Free Software Foundation.
+ *   */
+
+#ifndef _GFAPI_MESSAGES_H__
+#define _GFAPI_MESSAGES_H__
+
+#include <glusterfs/glfs-message-id.h>
+
+/* To add new message IDs, append new identifiers at the end of the list.
+ *
+ * Never remove a message ID. If it's not used anymore, you can rename it or
+ * leave it as it is, but not delete it. This is to prevent reutilization of
+ * IDs by other messages.
+ *
+ * The component name must match one of the entries defined in
+ * glfs-message-id.h.
+ */
+
+GLFS_MSGID(API, API_MSG_MEM_ACCT_INIT_FAILED, API_MSG_MASTER_XLATOR_INIT_FAILED,
+           API_MSG_GFAPI_XLATOR_INIT_FAILED, API_MSG_VOLFILE_OPEN_FAILED,
+           API_MSG_VOL_SPEC_FILE_ERROR, API_MSG_GLFS_FSOBJ_NULL,
+           API_MSG_INVALID_ENTRY, API_MSG_FSMUTEX_LOCK_FAILED,
+           API_MSG_COND_WAIT_FAILED, API_MSG_FSMUTEX_UNLOCK_FAILED,
+           API_MSG_INODE_REFRESH_FAILED, API_MSG_GRAPH_CONSTRUCT_FAILED,
+           API_MSG_API_XLATOR_ERROR, API_MSG_XDR_PAYLOAD_FAILED,
+           API_MSG_GET_VOLINFO_CBK_FAILED, API_MSG_FETCH_VOLUUID_FAILED,
+           API_MSG_INSUFF_SIZE, API_MSG_FRAME_CREAT_FAILED,
+           API_MSG_DICT_SET_FAILED, API_MSG_XDR_DECODE_FAILED,
+           API_MSG_GET_VOLFILE_FAILED, API_MSG_WRONG_OPVERSION,
+           API_MSG_DICT_SERIALIZE_FAILED, API_MSG_REMOTE_HOST_CONN_FAILED,
+           API_MSG_VOLFILE_SERVER_EXHAUST, API_MSG_CREATE_RPC_CLIENT_FAILED,
+           API_MSG_REG_NOTIFY_FUNC_FAILED, API_MSG_REG_CBK_FUNC_FAILED,
+           API_MSG_GET_CWD_FAILED, API_MSG_FGETXATTR_FAILED,
+           API_MSG_LOCKINFO_KEY_MISSING, API_MSG_FSETXATTR_FAILED,
+           API_MSG_FSYNC_FAILED, API_MSG_FDCREATE_FAILED,
+           API_MSG_INODE_PATH_FAILED, API_MSG_SYNCOP_OPEN_FAILED,
+           API_MSG_LOCK_MIGRATE_FAILED, API_MSG_OPENFD_SKIPPED,
+           API_MSG_FIRST_LOOKUP_GRAPH_FAILED, API_MSG_CWD_GRAPH_REF_FAILED,
+           API_MSG_SWITCHED_GRAPH, API_MSG_XDR_RESPONSE_DECODE_FAILED,
+           API_MSG_VOLFILE_INFO, API_MSG_VOLFILE_CONNECTING, API_MSG_NEW_GRAPH,
+           API_MSG_ALLOC_FAILED, API_MSG_CREATE_HANDLE_FAILED,
+           API_MSG_INODE_LINK_FAILED, API_MSG_STATEDUMP_FAILED,
+           API_MSG_XREADDIRP_R_FAILED, API_MSG_LOCK_INSERT_MERGE_FAILED,
+           API_MSG_SETTING_LOCK_TYPE_FAILED, API_MSG_INODE_FIND_FAILED,
+           API_MSG_FDCTX_SET_FAILED, API_MSG_UPCALL_SYNCOP_FAILED,
+           API_MSG_INVALID_ARG, API_MSG_UPCALL_EVENT_NULL_RECEIVED,
+           API_MSG_FLAGS_HANDLE, API_MSG_FDCREATE_FAILED_ON_GRAPH,
+           API_MSG_TRANS_RDMA_DEP, API_MSG_TRANS_NOT_SUPPORTED,
+           API_MSG_FS_NOT_INIT, API_MSG_INVALID_SYSRQ,
+           API_MSG_DECODE_XDR_FAILED, API_MSG_NULL, API_MSG_CALL_NOT_SUCCESSFUL,
+           API_MSG_CALL_NOT_VALID, API_MSG_UNABLE_TO_DEL,
+           API_MSG_REMOTE_HOST_DISCONN, API_MSG_HANDLE_NOT_SET);
+
+#define API_MSG_ALLOC_FAILED_STR "Upcall allocation failed"
+#define API_MSG_LOCK_INSERT_MERGE_FAILED_STR                                   \
+    "Lock insertion and splitting/merging failed"
+#define API_MSG_SETTING_LOCK_TYPE_FAILED_STR "Setting lock type failed"
+
+#define API_MSG_INVALID_ARG_STR "Invalid"
+#define API_MSG_INVALID_ENTRY_STR "Upcall entry validation failed"
+#define API_MSG_INODE_FIND_FAILED_STR "Unable to find inode entry"
+#define API_MSG_CREATE_HANDLE_FAILED_STR "handle creation failed"
+#define API_MSG_UPCALL_EVENT_NULL_RECEIVED_STR                                 \
+    "Upcall_EVENT_NULL received. Skipping it"
+#define API_MSG_UPCALL_SYNCOP_FAILED_STR "Synctask for upcall failed"
+#define API_MSG_FDCREATE_FAILED_STR "Allocating anonymous fd failed"
+#define API_MSG_XREADDIRP_R_FAILED_STR "glfs_x_readdirp_r failed"
+#define API_MSG_FDCTX_SET_FAILED_STR "Setting fd ctx failed"
+#define API_MSG_FLAGS_HANDLE_STR "arg not set. Flags handled are"
+#define API_MSG_INODE_REFRESH_FAILED_STR "inode refresh failed"
+#define API_MSG_INODE_LINK_FAILED_STR "inode linking failed"
+#define API_MSG_GET_CWD_FAILED_STR "Failed to get cwd"
+#define API_MSG_FGETXATTR_FAILED_STR "fgetxattr failed"
+#define API_MSG_LOCKINFO_KEY_MISSING_STR "missing lockinfo key"
+#define API_MSG_FSYNC_FAILED_STR "fsync() failed"
+#define API_MSG_FDCREATE_FAILED_ON_GRAPH_STR "fd_create failed on graph"
+#define API_MSG_INODE_PATH_FAILED_STR "inode_path failed"
+#define API_MSG_SYNCOP_OPEN_FAILED_STR "syncop_open failed"
+#define API_MSG_LOCK_MIGRATE_FAILED_STR "lock migration failed on graph"
+#define API_MSG_OPENFD_SKIPPED_STR "skipping openfd in graph"
+#define API_MSG_FIRST_LOOKUP_GRAPH_FAILED_STR "first lookup on graph failed"
+#define API_MSG_CWD_GRAPH_REF_FAILED_STR "cwd refresh of graph failed"
+#define API_MSG_SWITCHED_GRAPH_STR "switched to graph"
+#define API_MSG_FSETXATTR_FAILED_STR "fsetxattr failed"
+#define API_MSG_MEM_ACCT_INIT_FAILED_STR "Memory accounting init failed"
+#define API_MSG_MASTER_XLATOR_INIT_FAILED_STR                                  \
+    "master xlator for initialization failed"
+#define API_MSG_GFAPI_XLATOR_INIT_FAILED_STR                                   \
+    "failed to initialize gfapi translator"
+#define API_MSG_VOLFILE_OPEN_FAILED_STR "volume file open failed"
+#define API_MSG_VOL_SPEC_FILE_ERROR_STR "Cannot reach volume specification file"
+#define API_MSG_TRANS_RDMA_DEP_STR                                             \
+    "transport RDMA is deprecated, falling back to tcp"
+#define API_MSG_TRANS_NOT_SUPPORTED_STR                                        \
+    "transport is not supported, possible values tcp|unix"
+#define API_MSG_GLFS_FSOBJ_NULL_STR "fs is NULL"
+#define API_MSG_FS_NOT_INIT_STR "fs is not properly initialized"
+#define API_MSG_FSMUTEX_LOCK_FAILED_STR                                        \
+    "pthread lock on glfs mutex, returned error"
+#define API_MSG_FSMUTEX_UNLOCK_FAILED_STR                                      \
+    "pthread unlock on glfs mutex, returned error"
+#define API_MSG_COND_WAIT_FAILED_STR "cond wait failed"
+#define API_MSG_INVALID_SYSRQ_STR "not a valid sysrq"
+#define API_MSG_GRAPH_CONSTRUCT_FAILED_STR "failed to construct the graph"
+#define API_MSG_API_XLATOR_ERROR_STR                                           \
+    "api master xlator cannot be specified in volume file"
+#define API_MSG_STATEDUMP_FAILED_STR "statedump failed"
+#define API_MSG_DECODE_XDR_FAILED_STR                                          \
+    "Failed to decode xdr response for GF_CBK_STATEDUMP"
+#define API_MSG_NULL_STR "NULL"
+#define API_MSG_XDR_PAYLOAD_FAILED_STR "failed to create XDR payload"
+#define API_MSG_CALL_NOT_SUCCESSFUL_STR                                        \
+    "GET_VOLUME_INFO RPC call is not successful"
+#define API_MSG_XDR_RESPONSE_DECODE_FAILED_STR                                 \
+    "Failed to decode xdr response for GET_VOLUME_INFO"
+#define API_MSG_CALL_NOT_VALID_STR                                             \
+    "Response received for GET_VOLUME_INFO RPC is not valid"
+#define API_MSG_GET_VOLINFO_CBK_FAILED_STR                                     \
+    "In GET_VOLUME_INFO cbk, received error"
+#define API_MSG_FETCH_VOLUUID_FAILED_STR "Unable to fetch volume UUID"
+#define API_MSG_INSUFF_SIZE_STR "Insufficient size passed"
+#define API_MSG_FRAME_CREAT_FAILED_STR "failed to create the frame"
+#define API_MSG_DICT_SET_FAILED_STR "failed to set"
+#define API_MSG_XDR_DECODE_FAILED_STR "XDR decoding error"
+#define API_MSG_GET_VOLFILE_FAILED_STR "failed to get the volume file"
+#define API_MSG_VOLFILE_INFO_STR "No change in volfile, continuing"
+#define API_MSG_UNABLE_TO_DEL_STR "unable to delete file"
+#define API_MSG_WRONG_OPVERSION_STR                                            \
+    "Server is operating at an op-version which is not supported"
+#define API_MSG_DICT_SERIALIZE_FAILED_STR "Failed to serialize dictionary"
+#define API_MSG_REMOTE_HOST_CONN_FAILED_STR "Failed to connect to remote-host"
+#define API_MSG_REMOTE_HOST_DISCONN_STR "disconnected from remote-host"
+#define API_MSG_VOLFILE_SERVER_EXHAUST_STR "Exhausted all volfile servers"
+#define API_MSG_VOLFILE_CONNECTING_STR "connecting to next volfile server"
+#define API_MSG_CREATE_RPC_CLIENT_FAILED_STR "failed to create rpc clnt"
+#define API_MSG_REG_NOTIFY_FUNC_FAILED_STR "failed to register notify function"
+#define API_MSG_REG_CBK_FUNC_FAILED_STR "failed to register callback function"
+#define API_MSG_NEW_GRAPH_STR "New graph coming up"
+#define API_MSG_HANDLE_NOT_SET_STR "handle not set. Flags handled for xstat are"
+#endif /* !_GFAPI_MESSAGES_H__ */
diff --git a/api/src/gfapi.aliases b/api/src/gfapi.aliases
new file mode 100644
index 00000000000..bc639e6b99f
--- /dev/null
+++ b/api/src/gfapi.aliases
@@ -0,0 +1,201 @@
+
+_priv_glfs_loc_touchup _glfs_loc_touchup$GFAPI_PRIVATE_3.4.0
+_priv_glfs_active_subvol _glfs_active_subvol$GFAPI_PRIVATE_3.4.0
+_priv_glfs_subvol_done _glfs_subvol_done$GFAPI_PRIVATE_3.4.0
+_priv_glfs_init_done _glfs_init_done$GFAPI_PRIVATE_3.4.0
+_priv_glfs_resolve_at _glfs_resolve_at$GFAPI_PRIVATE_3.4.0
+
+_pub_glfs_new _glfs_new$GFAPI_3.4.0
+_pub_glfs_set_volfile _glfs_set_volfile$GFAPI_3.4.0
+_pub_glfs_set_volfile_server _glfs_set_volfile_server$GFAPI_3.4.0
+_pub_glfs_set_logging _glfs_set_logging$GFAPI_3.4.0
+_pub_glfs_init _glfs_init$GFAPI_3.4.0
+_pub_glfs_fini _glfs_fini$GFAPI_3.4.0
+_pub_glfs_open _glfs_open$GFAPI_3.4.0
+_pub_glfs_creat _glfs_creat$GFAPI_3.4.0
+_pub_glfs_close _glfs_close$GFAPI_3.4.0
+_pub_glfs_from_glfd _glfs_from_glfd$GFAPI_3.4.0
+_pub_glfs_set_xlator_option _glfs_set_xlator_option$GFAPI_3.4.0
+_pub_glfs_read _glfs_read$GFAPI_3.4.0
+_pub_glfs_write _glfs_write$GFAPI_3.4.0
+_pub_glfs_readv _glfs_readv$GFAPI_3.4.0
+_pub_glfs_writev _glfs_writev$GFAPI_3.4.0
+_pub_glfs_pread34 _glfs_pread$GFAPI_3.4.0
+_pub_glfs_pwrite34 _glfs_pwrite$GFAPI_3.4.0
+_pub_glfs_pread_async34 _glfs_pread_async$GFAPI_3.4.0
+_pub_glfs_pwrite_async34 _glfs_pwrite_async$GFAPI_3.4.0
+_pub_glfs_preadv _glfs_preadv$GFAPI_3.4.0
+_pub_glfs_pwritev _glfs_pwritev$GFAPI_3.4.0
+_pub_glfs_lseek _glfs_lseek$GFAPI_3.4.0
+_pub_glfs_ftruncate34 _glfs_ftruncate$GFAPI_3.4.0
+_pub_glfs_ftruncate_async34 _glfs_ftruncate_async$GFAPI_3.4.0
+_pub_glfs_lstat _glfs_lstat$GFAPI_3.4.0
+_pub_glfs_stat _glfs_stat$GFAPI_3.4.0
+_pub_glfs_fstat _glfs_fstat$GFAPI_3.4.0
+_pub_glfs_fsync34 _glfs_fsync$GFAPI_3.4.0
+_pub_glfs_fsync_async34 _glfs_fsync_async$GFAPI_3.4.0
+_pub_glfs_fdatasync34 _glfs_fdatasync$GFAPI_3.4.0
+_pub_glfs_fdatasync_async34 _glfs_fdatasync_async$GFAPI_3.4.0
+_pub_glfs_access _glfs_access$GFAPI_3.4.0
+_pub_glfs_symlink _glfs_symlink$GFAPI_3.4.0
+_pub_glfs_readlink _glfs_readlink$GFAPI_3.4.0
+_pub_glfs_mknod _glfs_mknod$GFAPI_3.4.0
+_pub_glfs_mkdir _glfs_mkdir$GFAPI_3.4.0
+_pub_glfs_unlink _glfs_unlink$GFAPI_3.4.0
+_pub_glfs_rmdir _glfs_rmdir$GFAPI_3.4.0
+_pub_glfs_rename _glfs_rename$GFAPI_3.4.0
+_pub_glfs_link _glfs_link$GFAPI_3.4.0
+_pub_glfs_opendir _glfs_opendir$GFAPI_3.4.0
+_pub_glfs_readdir_r _glfs_readdir_r$GFAPI_3.4.0
+_pub_glfs_readdirplus_r _glfs_readdirplus_r$GFAPI_3.4.0
+_pub_glfs_telldir _glfs_telldir$GFAPI_3.4.0
+_pub_glfs_seekdir _glfs_seekdir$GFAPI_3.4.0
+_pub_glfs_closedir _glfs_closedir$GFAPI_3.4.0
+_pub_glfs_statvfs _glfs_statvfs$GFAPI_3.4.0
+_pub_glfs_chmod _glfs_chmod$GFAPI_3.4.0
+_pub_glfs_fchmod _glfs_fchmod$GFAPI_3.4.0
+_pub_glfs_chown _glfs_chown$GFAPI_3.4.0
+_pub_glfs_lchown _glfs_lchown$GFAPI_3.4.0
+_pub_glfs_fchown _glfs_fchown$GFAPI_3.4.0
+_pub_glfs_utimens _glfs_utimens$GFAPI_3.4.0
+_pub_glfs_lutimens _glfs_lutimens$GFAPI_3.4.0
+_pub_glfs_futimens _glfs_futimens$GFAPI_3.4.0
+_pub_glfs_getxattr _glfs_getxattr$GFAPI_3.4.0
+_pub_glfs_lgetxattr _glfs_lgetxattr$GFAPI_3.4.0
+_pub_glfs_fgetxattr _glfs_fgetxattr$GFAPI_3.4.0
+_pub_glfs_listxattr _glfs_listxattr$GFAPI_3.4.0
+_pub_glfs_llistxattr _glfs_llistxattr$GFAPI_3.4.0
+_pub_glfs_flistxattr _glfs_flistxattr$GFAPI_3.4.0
+_pub_glfs_setxattr _glfs_setxattr$GFAPI_3.4.0
+_pub_glfs_lsetxattr _glfs_lsetxattr$GFAPI_3.4.0
+_pub_glfs_fsetxattr _glfs_fsetxattr$GFAPI_3.4.0
+_pub_glfs_removexattr _glfs_removexattr$GFAPI_3.4.0
+_pub_glfs_lremovexattr _glfs_lremovexattr$GFAPI_3.4.0
+_pub_glfs_fremovexattr _glfs_fremovexattr$GFAPI_3.4.0
+_pub_glfs_getcwd _glfs_getcwd$GFAPI_3.4.0
+_pub_glfs_chdir _glfs_chdir$GFAPI_3.4.0
+_pub_glfs_fchdir _glfs_fchdir$GFAPI_3.4.0
+_pub_glfs_realpath34 _glfs_realpath$GFAPI_3.4.0
+_pub_glfs_posix_lock _glfs_posix_lock$GFAPI_3.4.0
+_pub_glfs_dup _glfs_dup$GFAPI_3.4.0
+
+_pub_glfs_setfsuid _glfs_setfsuid$GFAPI_3.4.2
+_pub_glfs_setfsgid _glfs_setfsgid$GFAPI_3.4.2
+_pub_glfs_setfsgroups _glfs_setfsgroups$GFAPI_3.4.2
+_pub_glfs_h_lookupat34 _glfs_h_lookupat$GFAPI_3.4.2
+_pub_glfs_h_creat _glfs_h_creat$GFAPI_3.4.2
+_pub_glfs_h_mkdir _glfs_h_mkdir$GFAPI_3.4.2
+_pub_glfs_h_mknod _glfs_h_mknod$GFAPI_3.4.2
+_pub_glfs_h_symlink _glfs_h_symlink$GFAPI_3.4.2
+_pub_glfs_h_unlink _glfs_h_unlink$GFAPI_3.4.2
+_pub_glfs_h_close _glfs_h_close$GFAPI_3.4.2
+_pub_glfs_h_truncate _glfs_h_truncate$GFAPI_3.4.2
+_pub_glfs_h_stat _glfs_h_stat$GFAPI_3.4.2
+_pub_glfs_h_getattrs _glfs_h_getattrs$GFAPI_3.4.2
+_pub_glfs_h_setattrs _glfs_h_setattrs$GFAPI_3.4.2
+_pub_glfs_h_readlink _glfs_h_readlink$GFAPI_3.4.2
+_pub_glfs_h_link _glfs_h_link$GFAPI_3.4.2
+_pub_glfs_h_rename _glfs_h_rename$GFAPI_3.4.2
+_pub_glfs_h_extract_handle _glfs_h_extract_handle$GFAPI_3.4.2
+_pub_glfs_h_create_from_handle _glfs_h_create_from_handle$GFAPI_3.4.2
+_pub_glfs_h_opendir _glfs_h_opendir$GFAPI_3.4.2
+_pub_glfs_h_open _glfs_h_open$GFAPI_3.4.2
+
+_pub_glfs_get_volumeid _glfs_get_volumeid$GFAPI_3.5.0
+_pub_glfs_readdir _glfs_readdir$GFAPI_3.5.0
+_pub_glfs_readdirplus _glfs_readdirplus$GFAPI_3.5.0
+_pub_glfs_fallocate _glfs_fallocate$GFAPI_3.5.0
+_pub_glfs_discard _glfs_discard$GFAPI_3.5.0
+_pub_glfs_zerofill _glfs_zerofill$GFAPI_3.5.0
+_pub_glfs_caller_specific_init _glfs_caller_specific_init$GFAPI_3.5.0
+_pub_glfs_h_setxattrs _glfs_h_setxattrs$GFAPI_3.5.0
+
+_pub_glfs_unset_volfile_server _glfs_unset_volfile_server$GFAPI_3.5.1
+_pub_glfs_h_getxattrs _glfs_h_getxattrs$GFAPI_3.5.1
+_pub_glfs_h_removexattrs _glfs_h_removexattrs$GFAPI_3.5.1
+
+_pub_glfs_get_volfile _glfs_get_volfile$GFAPI_3.6.0
+_pub_glfs_h_access _glfs_h_access$GFAPI_3.6.0
+
+_pub_glfs_h_poll_upcall370 _glfs_h_poll_upcall$GFAPI_3.7.0
+_pub_glfs_h_acl_set _glfs_h_acl_set$GFAPI_3.7.0
+_pub_glfs_h_acl_get _glfs_h_acl_get$GFAPI_3.7.0
+_pub_glfs_h_statfs _glfs_h_statfs$GFAPI_3.7.0
+_pub_glfs_h_anonymous_read _glfs_h_anonymous_read$GFAPI_3.7.0
+_pub_glfs_h_anonymous_write _glfs_h_anonymous_write$GFAPI_3.7.0
+
+_priv_glfs_free_from_ctx _glfs_free_from_ctx$GFAPI_PRIVATE_3.7.0
+_priv_glfs_new_from_ctx _glfs_new_from_ctx$GFAPI_PRIVATE_3.7.0
+_priv_glfs_resolve _glfs_resolve$GFAPI_PRIVATE_3.7.0
+_priv_glfs_process_upcall_event _glfs_process_upcall_event$GFAPI_PRIVATE_3.7.0
+
+_pub_glfs_h_lookupat _glfs_h_lookupat$GFAPI_3.7.4
+
+_pub_glfs_truncate _glfs_truncate$GFAPI_3.7.15
+
+_pub_glfs_free _glfs_free$GFAPI_3.7.16
+_pub_glfs_h_poll_upcall _glfs_h_poll_upcall$GFAPI_3.7.16
+_pub_glfs_upcall_get_fs _glfs_upcall_get_fs$GFAPI_3.7.16
+_pub_glfs_upcall_get_reason _glfs_upcall_get_reason$GFAPI_3.7.16
+_pub_glfs_upcall_get_event _glfs_upcall_get_event$GFAPI_3.7.16
+_pub_glfs_upcall_inode_get_object _glfs_upcall_inode_get_object$GFAPI_3.7.16
+_pub_glfs_upcall_inode_get_flags _glfs_upcall_inode_get_flags$GFAPI_3.7.16
+_pub_glfs_upcall_inode_get_stat _glfs_upcall_inode_get_stat$GFAPI_3.7.16
+_pub_glfs_upcall_inode_get_expire _glfs_upcall_inode_get_expire$GFAPI_3.7.16
+_pub_glfs_upcall_inode_get_pobject _glfs_upcall_inode_get_pobject$GFAPI_3.7.16
+_pub_glfs_upcall_inode_get_pstat _glfs_upcall_inode_get_pstat$GFAPI_3.7.16
+_pub_glfs_upcall_inode_get_oldpobject _glfs_upcall_inode_get_oldpobject$GFAPI_3.7.16
+_pub_glfs_upcall_inode_get_oldpstat _glfs_upcall_inode_get_oldpstat$GFAPI_3.7.16
+
+_pub_glfs_realpath _glfs_realpath$GFAPI_3.7.17
+
+_pub_glfs_sysrq _glfs_sysrq$GFAPI_3.10.0
+
+_pub_glfs_fd_set_lkowner _glfs_fd_set_lkowner$GFAPI_3.10.7
+
+_pub_glfs_xreaddirplus_r _glfs_xreaddirplus_r$GFAPI_3.11.0
+_pub_glfs_xreaddirplus_r_get_stat _glfs_xreaddirplus_r_get_stat$GFAPI_3.11.0
+_pub_glfs_xreaddirplus_r_get_object _glfs_xreaddirplus_r_get_object$GFAPI_3.11.0
+_pub_glfs_object_copy _glfs_object_copy$GFAPI_3.11.0
+
+_priv_glfs_ipc _glfs_ipc$GFAPI_3.12.0
+
+_pub_glfs_upcall_register _glfs_upcall_register$GFAPI_3.13.0
+_pub_glfs_upcall_unregister _glfs_upcall_unregister$GFAPI_3.13.0
+
+_pub_glfs_setfsleaseid _glfs_setfsleaseid$GFAPI_4.0.0
+_pub_glfs_file_lock _glfs_file_lock$GFAPI_4.0.0
+_pub_glfs_lease _glfs_lease$GFAPI_4.0.0
+_pub_glfs_h_lease _glfs_h_lease$GFAPI_4.0.0
+_pub_glfs_upcall_lease_get_object _glfs_upcall_lease_get_object$GFAPI_4.1.6
+_pub_glfs_upcall_lease_get_lease_type _glfs_upcall_lease_get_lease_type$GFAPI_4.1.6
+
+_priv_glfs_statx _glfs_statx$GFAPI_6.0
+_priv_glfs_iatt_from_statx _glfs_iatt_from_statx$GFAPI_6.0
+_priv_glfs_setfspid _glfs_setfspid$GFAPI_6.1
+
+_pub_glfs_read_async _glfs_read_async$GFAPI_6.0
+_pub_glfs_write_async _glfs_write_async$GFAPI_6.0
+_pub_glfs_readv_async _glfs_readv_async$GFAPI_6.0
+_pub_glfs_writev_async _glfs_writev_async$GFAPI_6.0
+_pub_glfs_pread _glfs_pread$GFAPI_6.0
+_pub_glfs_pwrite _glfs_pwrite$GFAPI_6.0
+_pub_glfs_pread_async _glfs_pread_async$GFAPI_6.0
+_pub_glfs_pwrite_async _glfs_pwrite_async$GFAPI_6.0
+_pub_glfs_preadv_async _glfs_preadv_async$GFAPI_6.0
+_pub_glfs_pwritev_async _glfs_pwritev_async$GFAPI_6.0
+_pub_glfs_fsync _glfs_fsync$GFAPI_6.0
+_pub_glfs_fsync_async _glfs_fsync_async$GFAPI_6.0
+_pub_glfs_fdatasync _glfs_fdatasync$GFAPI_6.0
+_pub_glfs_fdatasync_async _glfs_fdatasync_async$GFAPI_6.0
+_pub_glfs_ftruncate _glfs_ftruncate$GFAPI_6.0
+_pub_glfs_ftruncate_async _glfs_ftruncate_async$GFAPI_6.0
+_pub_glfs_discard_async _glfs_discard_async$GFAPI_6.0
+_pub_glfs_zerofill_async _glfs_zerofill_async$GFAPI_6.0
+_pub_glfs_copy_file_range _glfs_copy_file_range$GFAPI_6.0
+_pub_glfs_fsetattr _glfs_fsetattr$GFAPI_6.0
+_pub_glfs_setattr _glfs_setattr$GFAPI_6.0
+
+_pub_glfs_set_statedump_path _glfs_set_statedump_path@GFAPI_7.0
+
+_pub_glfs_h_creat_open _glfs_h_creat_open@GFAPI_6.6
diff --git a/api/src/gfapi.map.in b/api/src/gfapi.map
index 7fc95971f1b..228ac47c084 100644
--- a/api/src/gfapi.map.in
+++ b/api/src/gfapi.map
@@ -39,7 +39,6 @@ GFAPI_3.4.0 {
 		glfs_preadv_async;
 		glfs_pwritev_async;
 		glfs_lseek;
-		glfs_truncate;
 		glfs_ftruncate;
 		glfs_ftruncate_async;
 		glfs_lstat;
@@ -98,7 +97,6 @@ GFAPI_3.4.2 {
 		glfs_setfsuid;
 		glfs_setfsgid;
 		glfs_setfsgroups;
-		glfs_h_lookupat;
 		glfs_h_creat;
 		glfs_h_mkdir;
 		glfs_h_mknod;
@@ -116,6 +114,7 @@ GFAPI_3.4.2 {
 		glfs_h_create_from_handle;
 		glfs_h_opendir;
 		glfs_h_open;
+		glfs_h_lookupat;
 } GFAPI_3.4.0;
 
 GFAPI_3.5.0 {
@@ -145,9 +144,140 @@ GFAPI_3.6.0 {
 		glfs_h_access;
 } GFAPI_3.5.1;
 
+GFAPI_3.7.0 {
+	global:
+		glfs_h_poll_upcall;
+		glfs_h_acl_set;
+		glfs_h_acl_get;
+		glfs_h_statfs;
+		glfs_h_anonymous_read;
+		glfs_h_anonymous_write;
+} GFAPI_3.6.0;
+
 GFAPI_PRIVATE_3.7.0 {
 	global:
 		glfs_free_from_ctx;
 		glfs_new_from_ctx;
-} GFAPI_3.6.0;
+		glfs_resolve;
+		glfs_process_upcall_event;
+} GFAPI_3.7.0;
+
+GFAPI_3.7.4 {
+	global:
+		glfs_h_lookupat;
+} GFAPI_PRIVATE_3.7.0;
+
+GFAPI_3.7.15 {
+	global:
+		glfs_truncate;
+} GFAPI_3.7.4;
+
+GFAPI_3.7.16 {
+	global:
+		glfs_free;
+		glfs_upcall_get_fs;
+		glfs_upcall_get_reason;
+		glfs_upcall_get_event;
+		glfs_upcall_inode_get_object;
+		glfs_upcall_inode_get_flags;
+		glfs_upcall_inode_get_stat;
+		glfs_upcall_inode_get_expire;
+		glfs_upcall_inode_get_pobject;
+		glfs_upcall_inode_get_pstat;
+		glfs_upcall_inode_get_oldpobject;
+		glfs_upcall_inode_get_oldpstat;
+		glfs_h_poll_upcall;
+} GFAPI_3.7.15;
+
+GFAPI_3.7.17 {
+	global:
+		glfs_realpath;
+} GFAPI_3.7.16;
+
+GFAPI_3.10.0 {
+	global:
+		glfs_sysrq;
+} GFAPI_3.7.17;
+
+GFAPI_3.10.7 {
+	global:
+		glfs_fd_set_lkowner;
+} GFAPI_3.10.0;
+
+GFAPI_3.11.0 {
+		glfs_xreaddirplus_r;
+		glfs_xreaddirplus_r_get_stat;
+		glfs_xreaddirplus_r_get_object;
+		glfs_object_copy;
+} GFAPI_3.10.7;
+
+GFAPI_PRIVATE_3.12.0 {
+	global:
+		glfs_ipc;
+} GFAPI_3.11.0;
+
+GFAPI_3.13.0 {
+	global:
+		glfs_upcall_register;
+		glfs_upcall_unregister;
+} GFAPI_PRIVATE_3.12.0;
+
+GFAPI_4.0.0 {
+	global:
+		glfs_setfsleaseid;
+		glfs_file_lock;
+		glfs_lease;
+		glfs_h_lease;
+} GFAPI_3.13.0;
 
+GFAPI_4.1.6 {
+	global:
+		glfs_upcall_lease_get_object;
+		glfs_upcall_lease_get_lease_type;
+} GFAPI_4.0.0;
+
+GFAPI_PRIVATE_6.0 {
+	global:
+		glfs_statx;
+		glfs_iatt_from_statx;
+} GFAPI_4.1.6;
+
+GFAPI_6.0 {
+	global:
+		glfs_read_async;
+		glfs_write_async;
+		glfs_readv_async;
+		glfs_writev_async;
+		glfs_pread;
+		glfs_pwrite;
+		glfs_pread_async;
+		glfs_pwrite_async;
+		glfs_preadv_async;
+		glfs_pwritev_async;
+		glfs_fsync;
+		glfs_fsync_async;
+		glfs_fdatasync;
+		glfs_fdatasync_async;
+		glfs_ftruncate;
+		glfs_ftruncate_async;
+		glfs_discard_async;
+		glfs_zerofill_async;
+		glfs_copy_file_range;
+		glfs_setattr;
+		glfs_fsetattr;
+} GFAPI_PRIVATE_6.0;
+
+GFAPI_PRIVATE_6.1 {
+	global:
+		glfs_setfspid;
+} GFAPI_6.0;
+
+GFAPI_6.6 {
+	global:
+		glfs_h_creat_open;
+} GFAPI_PRIVATE_6.1;
+
+GFAPI_7.0 {
+	global:
+		glfs_set_statedump_path;
+} GFAPI_6.6;
diff --git a/api/src/glfs-fops.c b/api/src/glfs-fops.c
index b93a077fcdf..6aa3c5602d1 100644
--- a/api/src/glfs-fops.c
+++ b/api/src/glfs-fops.c
@@ -1,6 +1,6 @@
 
 /*
-  Copyright (c) 2012 Red Hat, Inc. <http://www.redhat.com>
+  Copyright (c) 2012-2018 Red Hat, Inc. <http://www.redhat.com>
   This file is part of GlusterFS.
 
   This file is licensed to you under your choice of the GNU Lesser
@@ -9,13 +9,22 @@
   cases as published by the Free Software Foundation.
 */
 
+/* for SEEK_HOLE and SEEK_DATA */
+#ifndef _GNU_SOURCE
+#define _GNU_SOURCE
+#endif
+
+#include <unistd.h>
 
 #include "glfs-internal.h"
 #include "glfs-mem-types.h"
-#include "syncop.h"
+#include <glusterfs/syncop.h>
 #include "glfs.h"
-#include "compat-errno.h"
+#include "gfapi-messages.h"
+#include <glusterfs/compat-errno.h>
 #include <limits.h>
+#include "glusterfs3.h"
+#include <glusterfs/iatt.h>
 
 #ifdef NAME_MAX
 #define GF_NAME_MAX NAME_MAX
@@ -23,3475 +32,6414 @@
 #define GF_NAME_MAX 255
 #endif
 
+struct upcall_syncop_args {
+    struct glfs *fs;
+    struct gf_upcall upcall_data;
+};
+
 #define READDIRBUF_SIZE (sizeof(struct dirent) + GF_NAME_MAX + 1)
 
+typedef void (*glfs_io_cbk34)(glfs_fd_t *fd, ssize_t ret, void *data);
+
+/*
+ * This function will mark glfd for deletion and decrement its refcount.
+ */
 int
-glfs_loc_link (loc_t *loc, struct iatt *iatt)
+glfs_mark_glfd_for_deletion(struct glfs_fd *glfd)
 {
-	int ret = -1;
-	inode_t *linked_inode = NULL;
+    LOCK(&glfd->lock);
+    {
+        glfd->state = GLFD_CLOSE;
+    }
+    UNLOCK(&glfd->lock);
 
-	if (!loc->inode) {
-		errno = EINVAL;
-		return -1;
-	}
+    GF_REF_PUT(glfd);
 
-	linked_inode = inode_link (loc->inode, loc->parent, loc->name, iatt);
-	if (linked_inode) {
-		inode_lookup (linked_inode);
-		inode_unref (linked_inode);
-		ret = 0;
-	} else {
-		ret = -1;
-		errno = ENOMEM;
-	}
-
-	return ret;
+    return 0;
 }
 
+/* This function is useful for all async fops. There is chance that glfd is
+ * closed before async fop is completed. When glfd is closed we change the
+ * state to GLFD_CLOSE.
+ *
+ * This function will return _gf_true if the glfd is still valid else return
+ * _gf_false.
+ */
+gf_boolean_t
+glfs_is_glfd_still_valid(struct glfs_fd *glfd)
+{
+    gf_boolean_t ret = _gf_false;
+
+    LOCK(&glfd->lock);
+    {
+        if (glfd->state != GLFD_CLOSE)
+            ret = _gf_true;
+    }
+    UNLOCK(&glfd->lock);
+
+    return ret;
+}
 
 void
-glfs_iatt_to_stat (struct glfs *fs, struct iatt *iatt, struct stat *stat)
+glfd_set_state_bind(struct glfs_fd *glfd)
 {
-	iatt_to_stat (iatt, stat);
-	stat->st_dev = fs->dev_id;
-}
+    LOCK(&glfd->lock);
+    {
+        glfd->state = GLFD_OPEN;
+    }
+    UNLOCK(&glfd->lock);
 
+    fd_bind(glfd->fd);
+    glfs_fd_bind(glfd);
 
+    return;
+}
+
+/*
+ * This routine is called when an upcall event of type
+ * 'GF_UPCALL_CACHE_INVALIDATION' is received.
+ * It makes a copy of the contents of the upcall cache-invalidation
+ * data received into an entry which is stored in the upcall list
+ * maintained by gfapi.
+ */
 int
-glfs_loc_unlink (loc_t *loc)
+glfs_get_upcall_cache_invalidation(struct gf_upcall *to_up_data,
+                                   struct gf_upcall *from_up_data)
 {
-	inode_unlink (loc->inode, loc->parent, loc->name);
+    struct gf_upcall_cache_invalidation *ca_data = NULL;
+    struct gf_upcall_cache_invalidation *f_ca_data = NULL;
+    int ret = -1;
 
-	return 0;
-}
+    GF_VALIDATE_OR_GOTO(THIS->name, to_up_data, out);
+    GF_VALIDATE_OR_GOTO(THIS->name, from_up_data, out);
 
+    f_ca_data = from_up_data->data;
+    GF_VALIDATE_OR_GOTO(THIS->name, f_ca_data, out);
 
-struct glfs_fd *
-pub_glfs_open (struct glfs *fs, const char *path, int flags)
-{
-	int              ret = -1;
-	struct glfs_fd  *glfd = NULL;
-	xlator_t        *subvol = NULL;
-	loc_t            loc = {0, };
-	struct iatt      iatt = {0, };
-	int              reval = 0;
+    ca_data = GF_CALLOC(1, sizeof(*ca_data), glfs_mt_upcall_entry_t);
 
-	__glfs_entry_fs (fs);
+    if (!ca_data) {
+        gf_smsg(THIS->name, GF_LOG_ERROR, errno, API_MSG_ALLOC_FAILED, "entry",
+                NULL);
+        goto out;
+    }
 
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    to_up_data->data = ca_data;
 
-	glfd = glfs_fd_new (fs);
-	if (!glfd)
-		goto out;
+    ca_data->flags = f_ca_data->flags;
+    ca_data->expire_time_attr = f_ca_data->expire_time_attr;
+    ca_data->stat = f_ca_data->stat;
+    ca_data->p_stat = f_ca_data->p_stat;
+    ca_data->oldp_stat = f_ca_data->oldp_stat;
 
-retry:
-	ret = glfs_resolve (fs, subvol, path, &loc, &iatt, reval);
+    ret = 0;
+out:
+    return ret;
+}
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+int
+glfs_get_upcall_lease(struct gf_upcall *to_up_data,
+                      struct gf_upcall *from_up_data)
+{
+    struct gf_upcall_recall_lease *ca_data = NULL;
+    struct gf_upcall_recall_lease *f_ca_data = NULL;
+    int ret = -1;
 
-	if (ret)
-		goto out;
+    GF_VALIDATE_OR_GOTO(THIS->name, to_up_data, out);
+    GF_VALIDATE_OR_GOTO(THIS->name, from_up_data, out);
 
-	if (IA_ISDIR (iatt.ia_type)) {
-		ret = -1;
-		errno = EISDIR;
-		goto out;
-	}
+    f_ca_data = from_up_data->data;
+    GF_VALIDATE_OR_GOTO(THIS->name, f_ca_data, out);
 
-	if (!IA_ISREG (iatt.ia_type)) {
-		ret = -1;
-		errno = EINVAL;
-		goto out;
-	}
+    ca_data = GF_CALLOC(1, sizeof(*ca_data), glfs_mt_upcall_entry_t);
 
-	if (glfd->fd) {
-		/* Retry. Safe to touch glfd->fd as we
-		   still have not glfs_fd_bind() yet.
-		*/
-		fd_unref (glfd->fd);
-		glfd->fd = NULL;
-	}
+    if (!ca_data) {
+        gf_smsg(THIS->name, GF_LOG_ERROR, errno, API_MSG_ALLOC_FAILED, "entry",
+                NULL);
+        goto out;
+    }
 
-	glfd->fd = fd_create (loc.inode, getpid());
-	if (!glfd->fd) {
-		ret = -1;
-		errno = ENOMEM;
-		goto out;
-	}
-        glfd->fd->flags = flags;
+    to_up_data->data = ca_data;
 
-	ret = syncop_open (subvol, &loc, flags, glfd->fd);
-        DECODE_SYNCOP_ERR (ret);
+    ca_data->lease_type = f_ca_data->lease_type;
+    gf_uuid_copy(ca_data->tid, f_ca_data->tid);
+    ca_data->dict = f_ca_data->dict;
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    ret = 0;
 out:
-	loc_wipe (&loc);
-
-	if (ret && glfd) {
-		glfs_fd_destroy (glfd);
-		glfd = NULL;
-	} else if (glfd) {
-		fd_bind (glfd->fd);
-		glfs_fd_bind (glfd);
-	}
+    return ret;
+}
+int
+glfs_loc_link(loc_t *loc, struct iatt *iatt)
+{
+    int ret = -1;
+    inode_t *old_inode = NULL;
+    uint64_t ctx_value = LOOKUP_NOT_NEEDED;
+
+    if (!loc->inode) {
+        errno = EINVAL;
+        return -1;
+    }
+
+    old_inode = loc->inode;
+
+    /* If the inode already exists in the cache, the inode
+     * returned here points to the existing one. We need
+     * to update loc.inode accordingly.
+     */
+    loc->inode = inode_link(loc->inode, loc->parent, loc->name, iatt);
+    if (loc->inode) {
+        inode_ctx_set(loc->inode, THIS, &ctx_value);
+        inode_lookup(loc->inode);
+        inode_unref(old_inode);
+        ret = 0;
+    } else {
+        ret = -1;
+    }
+
+    return ret;
+}
 
-	priv_glfs_subvol_done (fs, subvol);
+void
+glfs_iatt_to_stat(struct glfs *fs, struct iatt *iatt, struct stat *stat)
+{
+    iatt_to_stat(iatt, stat);
+    stat->st_dev = fs->dev_id;
+}
 
-	return glfd;
+void
+glfs_iatt_to_statx(struct glfs *fs, const struct iatt *iatt,
+                   struct glfs_stat *statx)
+{
+    statx->glfs_st_mask = 0;
+
+    statx->glfs_st_mode = 0;
+    if (IATT_TYPE_VALID(iatt->ia_flags)) {
+        statx->glfs_st_mode |= st_mode_type_from_ia(iatt->ia_type);
+        statx->glfs_st_mask |= GLFS_STAT_TYPE;
+    }
+
+    if (IATT_MODE_VALID(iatt->ia_flags)) {
+        statx->glfs_st_mode |= st_mode_prot_from_ia(iatt->ia_prot);
+        statx->glfs_st_mask |= GLFS_STAT_MODE;
+    }
+
+    if (IATT_NLINK_VALID(iatt->ia_flags)) {
+        statx->glfs_st_nlink = iatt->ia_nlink;
+        statx->glfs_st_mask |= GLFS_STAT_NLINK;
+    }
+
+    if (IATT_UID_VALID(iatt->ia_flags)) {
+        statx->glfs_st_uid = iatt->ia_uid;
+        statx->glfs_st_mask |= GLFS_STAT_UID;
+    }
+
+    if (IATT_GID_VALID(iatt->ia_flags)) {
+        statx->glfs_st_gid = iatt->ia_gid;
+        statx->glfs_st_mask |= GLFS_STAT_GID;
+    }
+
+    if (IATT_ATIME_VALID(iatt->ia_flags)) {
+        statx->glfs_st_atime.tv_sec = iatt->ia_atime;
+        statx->glfs_st_atime.tv_nsec = iatt->ia_atime_nsec;
+        statx->glfs_st_mask |= GLFS_STAT_ATIME;
+    }
+
+    if (IATT_MTIME_VALID(iatt->ia_flags)) {
+        statx->glfs_st_mtime.tv_sec = iatt->ia_mtime;
+        statx->glfs_st_mtime.tv_nsec = iatt->ia_mtime_nsec;
+        statx->glfs_st_mask |= GLFS_STAT_MTIME;
+    }
+
+    if (IATT_CTIME_VALID(iatt->ia_flags)) {
+        statx->glfs_st_ctime.tv_sec = iatt->ia_ctime;
+        statx->glfs_st_ctime.tv_nsec = iatt->ia_ctime_nsec;
+        statx->glfs_st_mask |= GLFS_STAT_CTIME;
+    }
+
+    if (IATT_BTIME_VALID(iatt->ia_flags)) {
+        statx->glfs_st_btime.tv_sec = iatt->ia_btime;
+        statx->glfs_st_btime.tv_nsec = iatt->ia_btime_nsec;
+        statx->glfs_st_mask |= GLFS_STAT_BTIME;
+    }
+
+    if (IATT_INO_VALID(iatt->ia_flags)) {
+        statx->glfs_st_ino = iatt->ia_ino;
+        statx->glfs_st_mask |= GLFS_STAT_INO;
+    }
+
+    if (IATT_SIZE_VALID(iatt->ia_flags)) {
+        statx->glfs_st_size = iatt->ia_size;
+        statx->glfs_st_mask |= GLFS_STAT_SIZE;
+    }
+
+    if (IATT_BLOCKS_VALID(iatt->ia_flags)) {
+        statx->glfs_st_blocks = iatt->ia_blocks;
+        statx->glfs_st_mask |= GLFS_STAT_BLOCKS;
+    }
+
+    /* unconditionally present, encode as is */
+    statx->glfs_st_blksize = iatt->ia_blksize;
+    statx->glfs_st_rdev_major = ia_major(iatt->ia_rdev);
+    statx->glfs_st_rdev_minor = ia_minor(iatt->ia_rdev);
+    statx->glfs_st_dev_major = ia_major(fs->dev_id);
+    statx->glfs_st_dev_minor = ia_minor(fs->dev_id);
+
+    /* At present we do not read any localFS attributes and pass them along,
+     * so setting this to 0. As we start supporting file attributes we can
+     * populate the same here as well */
+    statx->glfs_st_attributes = 0;
+    statx->glfs_st_attributes_mask = 0;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_open, 3.4.0);
+GFAPI_SYMVER_PRIVATE_DEFAULT(glfs_iatt_from_statx, 6.0)
+void
+priv_glfs_iatt_from_statx(struct iatt *iatt, const struct glfs_stat *statx)
+{
+    /* Most code in xlators are not checking validity flags before accessing
+    the items. Hence zero everything before setting valid items */
+    memset(iatt, 0, sizeof(struct iatt));
+
+    if (GLFS_STAT_TYPE_VALID(statx->glfs_st_mask)) {
+        iatt->ia_type = ia_type_from_st_mode(statx->glfs_st_mode);
+        iatt->ia_flags |= IATT_TYPE;
+    }
+
+    if (GLFS_STAT_MODE_VALID(statx->glfs_st_mask)) {
+        iatt->ia_prot = ia_prot_from_st_mode(statx->glfs_st_mode);
+        iatt->ia_flags |= IATT_MODE;
+    }
+
+    if (GLFS_STAT_NLINK_VALID(statx->glfs_st_mask)) {
+        iatt->ia_nlink = statx->glfs_st_nlink;
+        iatt->ia_flags |= IATT_NLINK;
+    }
+
+    if (GLFS_STAT_UID_VALID(statx->glfs_st_mask)) {
+        iatt->ia_uid = statx->glfs_st_uid;
+        iatt->ia_flags |= IATT_UID;
+    }
+
+    if (GLFS_STAT_GID_VALID(statx->glfs_st_mask)) {
+        iatt->ia_gid = statx->glfs_st_gid;
+        iatt->ia_flags |= IATT_GID;
+    }
+
+    if (GLFS_STAT_ATIME_VALID(statx->glfs_st_mask)) {
+        iatt->ia_atime = statx->glfs_st_atime.tv_sec;
+        iatt->ia_atime_nsec = statx->glfs_st_atime.tv_nsec;
+        iatt->ia_flags |= IATT_ATIME;
+    }
+
+    if (GLFS_STAT_MTIME_VALID(statx->glfs_st_mask)) {
+        iatt->ia_mtime = statx->glfs_st_mtime.tv_sec;
+        iatt->ia_mtime_nsec = statx->glfs_st_mtime.tv_nsec;
+        iatt->ia_flags |= IATT_MTIME;
+    }
+
+    if (GLFS_STAT_CTIME_VALID(statx->glfs_st_mask)) {
+        iatt->ia_ctime = statx->glfs_st_ctime.tv_sec;
+        iatt->ia_ctime_nsec = statx->glfs_st_ctime.tv_nsec;
+        iatt->ia_flags |= IATT_CTIME;
+    }
+
+    if (GLFS_STAT_BTIME_VALID(statx->glfs_st_mask)) {
+        iatt->ia_btime = statx->glfs_st_btime.tv_sec;
+        iatt->ia_btime_nsec = statx->glfs_st_btime.tv_nsec;
+        iatt->ia_flags |= IATT_BTIME;
+    }
+
+    if (GLFS_STAT_INO_VALID(statx->glfs_st_mask)) {
+        iatt->ia_ino = statx->glfs_st_ino;
+        iatt->ia_flags |= IATT_INO;
+    }
+
+    if (GLFS_STAT_SIZE_VALID(statx->glfs_st_mask)) {
+        iatt->ia_size = statx->glfs_st_size;
+        iatt->ia_flags |= IATT_SIZE;
+    }
+
+    if (GLFS_STAT_BLOCKS_VALID(statx->glfs_st_mask)) {
+        iatt->ia_blocks = statx->glfs_st_blocks;
+        iatt->ia_flags |= IATT_BLOCKS;
+    }
+
+    /* unconditionally present, encode as is */
+    iatt->ia_blksize = statx->glfs_st_blksize;
+    iatt->ia_rdev = makedev(statx->glfs_st_rdev_major,
+                            statx->glfs_st_rdev_minor);
+    iatt->ia_dev = makedev(statx->glfs_st_dev_major, statx->glfs_st_dev_minor);
+    iatt->ia_attributes = statx->glfs_st_attributes;
+    iatt->ia_attributes_mask = statx->glfs_st_attributes_mask;
+}
 
+void
+glfsflags_from_gfapiflags(struct glfs_stat *stat, int *glvalid)
+{
+    *glvalid = 0;
+    if (stat->glfs_st_mask & GLFS_STAT_MODE) {
+        *glvalid |= GF_SET_ATTR_MODE;
+    }
+
+    if (stat->glfs_st_mask & GLFS_STAT_SIZE) {
+        *glvalid |= GF_SET_ATTR_SIZE;
+    }
+
+    if (stat->glfs_st_mask & GLFS_STAT_UID) {
+        *glvalid |= GF_SET_ATTR_UID;
+    }
+
+    if (stat->glfs_st_mask & GLFS_STAT_GID) {
+        *glvalid |= GF_SET_ATTR_GID;
+    }
+
+    if (stat->glfs_st_mask & GLFS_STAT_ATIME) {
+        *glvalid |= GF_SET_ATTR_ATIME;
+    }
+
+    if (stat->glfs_st_mask & GLFS_STAT_MTIME) {
+        *glvalid |= GF_SET_ATTR_MTIME;
+    }
+}
 
 int
-pub_glfs_close (struct glfs_fd *glfd)
+glfs_loc_unlink(loc_t *loc)
 {
-	xlator_t  *subvol = NULL;
-	int        ret = -1;
-	fd_t      *fd = NULL;
-	struct glfs *fs = NULL;
+    inode_unlink(loc->inode, loc->parent, loc->name);
 
-	__glfs_entry_fd (glfd);
+    /* since glfs_h_* objects hold a reference to inode
+     * it is safe to keep lookup count to '0' */
+    if (!inode_has_dentry(loc->inode))
+        inode_forget(loc->inode, 0);
 
-	subvol = priv_glfs_active_subvol (glfd->fs);
-        if (!subvol) {
-                ret = -1;
-                errno = EIO;
-                goto out;
-        }
+    return 0;
+}
 
-	fd = glfs_resolve_fd (glfd->fs, subvol, glfd);
-	if (!fd) {
-		ret = -1;
-		errno = EBADFD;
-		goto out;
-	}
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_open, 3.4.0)
+struct glfs_fd *
+pub_glfs_open(struct glfs *fs, const char *path, int flags)
+{
+    int ret = -1;
+    struct glfs_fd *glfd = NULL;
+    xlator_t *subvol = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    int reval = 0;
+    dict_t *fop_attr = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    glfd = glfs_fd_new(fs);
+    if (!glfd)
+        goto out;
 
-	ret = syncop_flush (subvol, fd);
-        DECODE_SYNCOP_ERR (ret);
-out:
-	fs = glfd->fs;
-	glfs_fd_destroy (glfd);
+retry:
+    ret = glfs_resolve(fs, subvol, path, &loc, &iatt, reval);
 
-	if (fd)
-		fd_unref (fd);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	priv_glfs_subvol_done (fs, subvol);
+    if (ret)
+        goto out;
 
-	return ret;
-}
+    if (IA_ISDIR(iatt.ia_type)) {
+        ret = -1;
+        errno = EISDIR;
+        goto out;
+    }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_close, 3.4.0);
+    if (!IA_ISREG(iatt.ia_type)) {
+        ret = -1;
+        errno = EINVAL;
+        goto out;
+    }
 
+    if (glfd->fd) {
+        /* Retry. Safe to touch glfd->fd as we
+           still have not glfs_fd_bind() yet.
+        */
+        fd_unref(glfd->fd);
+        glfd->fd = NULL;
+    }
+
+    glfd->fd = fd_create(loc.inode, getpid());
+    if (!glfd->fd) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+    glfd->fd->flags = flags;
+
+    ret = get_fop_attr_thrd_key(&fop_attr);
+    if (ret)
+        gf_msg_debug("gfapi", 0, "Getting leaseid from thread failed");
+
+    ret = syncop_open(subvol, &loc, flags, glfd->fd, fop_attr, NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
+out:
+    loc_wipe(&loc);
 
-int
-pub_glfs_lstat (struct glfs *fs, const char *path, struct stat *stat)
-{
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	loc_t            loc = {0, };
-	struct iatt      iatt = {0, };
-	int              reval = 0;
+    if (fop_attr)
+        dict_unref(fop_attr);
 
-	__glfs_entry_fs (fs);
+    if (ret && glfd) {
+        GF_REF_PUT(glfd);
+        glfd = NULL;
+    } else if (glfd) {
+        glfd_set_state_bind(glfd);
+    }
 
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
-retry:
-	ret = glfs_lresolve (fs, subvol, path, &loc, &iatt, reval);
+    glfs_subvol_done(fs, subvol);
+
+    __GLFS_EXIT_FS;
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+invalid_fs:
+    return glfd;
+}
 
-	if (ret == 0 && stat)
-		glfs_iatt_to_stat (fs, &iatt, stat);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_close, 3.4.0)
+int
+pub_glfs_close(struct glfs_fd *glfd)
+{
+    xlator_t *subvol = NULL;
+    int ret = -1;
+    fd_t *fd = NULL;
+    struct glfs *fs = NULL;
+    dict_t *fop_attr = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        ret = -1;
+        errno = EBADFD;
+        goto out;
+    }
+
+    if (glfd->lk_owner.len != 0) {
+        ret = syncopctx_setfslkowner(&glfd->lk_owner);
+        if (ret)
+            goto out;
+    }
+    ret = get_fop_attr_thrd_key(&fop_attr);
+    if (ret)
+        gf_msg_debug("gfapi", 0, "Getting leaseid from thread failed");
+
+    ret = syncop_flush(subvol, fd, fop_attr, NULL);
+    DECODE_SYNCOP_ERR(ret);
 out:
-	loc_wipe (&loc);
+    fs = glfd->fs;
 
-	priv_glfs_subvol_done (fs, subvol);
+    if (fd)
+        fd_unref(fd);
+    if (fop_attr)
+        dict_unref(fop_attr);
 
-	return ret;
-}
+    glfs_mark_glfd_for_deletion(glfd);
+    glfs_subvol_done(fs, subvol);
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_lstat, 3.4.0);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_lstat, 3.4.0)
 int
-pub_glfs_stat (struct glfs *fs, const char *path, struct stat *stat)
+pub_glfs_lstat(struct glfs *fs, const char *path, struct stat *stat)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	loc_t            loc = {0, };
-	struct iatt      iatt = {0, };
-	int              reval = 0;
-
-	__glfs_entry_fs (fs);
-
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    int reval = 0;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
 retry:
-	ret = glfs_resolve (fs, subvol, path, &loc, &iatt, reval);
+    ret = glfs_lresolve(fs, subvol, path, &loc, &iatt, reval);
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	if (ret == 0 && stat)
-		glfs_iatt_to_stat (fs, &iatt, stat);
+    if (ret == 0 && stat)
+        glfs_iatt_to_stat(fs, &iatt, stat);
 out:
-	loc_wipe (&loc);
-
-	priv_glfs_subvol_done (fs, subvol);
+    loc_wipe(&loc);
 
-	return ret;
-}
+    glfs_subvol_done(fs, subvol);
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_stat, 3.4.0);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_stat, 3.4.0)
 int
-pub_glfs_fstat (struct glfs_fd *glfd, struct stat *stat)
+pub_glfs_stat(struct glfs *fs, const char *path, struct stat *stat)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	struct iatt      iatt = {0, };
-	fd_t            *fd = NULL;
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    int reval = 0;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+retry:
+    ret = glfs_resolve(fs, subvol, path, &loc, &iatt, reval);
 
-	__glfs_entry_fd (glfd);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	subvol = priv_glfs_active_subvol (glfd->fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    if (ret == 0 && stat)
+        glfs_iatt_to_stat(fs, &iatt, stat);
+out:
+    loc_wipe(&loc);
 
-	fd = glfs_resolve_fd (glfd->fs, subvol, glfd);
-	if (!fd) {
-		ret = -1;
-		errno = EBADFD;
-		goto out;
-	}
+    glfs_subvol_done(fs, subvol);
 
-	ret = syncop_fstat (subvol, fd, &iatt);
-        DECODE_SYNCOP_ERR (ret);
+    __GLFS_EXIT_FS;
 
-	if (ret == 0 && stat)
-		glfs_iatt_to_stat (glfd->fs, &iatt, stat);
-out:
-	if (fd)
-		fd_unref (fd);
+invalid_fs:
+    return ret;
+}
 
-	priv_glfs_subvol_done (glfd->fs, subvol);
+GFAPI_SYMVER_PRIVATE_DEFAULT(glfs_statx, 6.0)
+int
+priv_glfs_statx(struct glfs *fs, const char *path, const unsigned int mask,
+                struct glfs_stat *statxbuf)
+{
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    int reval = 0;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    if (path == NULL) {
+        ret = -1;
+        errno = EINVAL;
+        goto out;
+    }
+
+    if (mask & ~GLFS_STAT_ALL) {
+        ret = -1;
+        errno = EINVAL;
+        goto out;
+    }
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
 
-	return ret;
-}
+retry:
+    ret = glfs_resolve(fs, subvol, path, &loc, &iatt, reval);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fstat, 3.4.0);
+    if (ret == 0 && statxbuf)
+        glfs_iatt_to_statx(fs, &iatt, statxbuf);
+out:
+    loc_wipe(&loc);
 
+    glfs_subvol_done(fs, subvol);
 
-struct glfs_fd *
-pub_glfs_creat (struct glfs *fs, const char *path, int flags, mode_t mode)
-{
-	int              ret = -1;
-	struct glfs_fd  *glfd = NULL;
-	xlator_t        *subvol = NULL;
-	loc_t            loc = {0, };
-	struct iatt      iatt = {0, };
-	uuid_t           gfid;
-	dict_t          *xattr_req = NULL;
-	int              reval = 0;
-
-	__glfs_entry_fs (fs);
-
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
-
-	xattr_req = dict_new ();
-	if (!xattr_req) {
-		ret = -1;
-		errno = ENOMEM;
-		goto out;
-	}
-
-	uuid_generate (gfid);
-	ret = dict_set_static_bin (xattr_req, "gfid-req", gfid, 16);
-	if (ret) {
-		ret = -1;
-		errno = ENOMEM;
-		goto out;
-	}
-
-	glfd = glfs_fd_new (fs);
-	if (!glfd)
-		goto out;
-
-	/* This must be glfs_resolve() and NOT glfs_lresolve().
-	   That is because open("name", O_CREAT) where "name"
-	   is a danging symlink must create the dangling
-	   destinataion.
-	*/
-retry:
-	ret = glfs_resolve (fs, subvol, path, &loc, &iatt, reval);
-
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
-
-	if (ret == -1 && errno != ENOENT)
-		/* Any other type of error is fatal */
-		goto out;
-
-	if (ret == -1 && errno == ENOENT && !loc.parent)
-		/* The parent directory or an ancestor even
-		   higher does not exist
-		*/
-		goto out;
-
-	if (loc.inode) {
-		if (flags & O_EXCL) {
-			ret = -1;
-			errno = EEXIST;
-			goto out;
-		}
-
-		if (IA_ISDIR (iatt.ia_type)) {
-			ret = -1;
-			errno = EISDIR;
-			goto out;
-		}
-
-		if (!IA_ISREG (iatt.ia_type)) {
-			ret = -1;
-			errno = EINVAL;
-			goto out;
-		}
-	}
-
-	if (ret == -1 && errno == ENOENT) {
-		loc.inode = inode_new (loc.parent->table);
-		if (!loc.inode) {
-			ret = -1;
-			errno = ENOMEM;
-			goto out;
-		}
-	}
-
-	if (glfd->fd) {
-		/* Retry. Safe to touch glfd->fd as we
-		   still have not glfs_fd_bind() yet.
-		*/
-		fd_unref (glfd->fd);
-		glfd->fd = NULL;
-	}
-
-	glfd->fd = fd_create (loc.inode, getpid());
-	if (!glfd->fd) {
-		ret = -1;
-		errno = ENOMEM;
-		goto out;
-	}
-        glfd->fd->flags = flags;
-
-	if (ret == 0) {
-		ret = syncop_open (subvol, &loc, flags, glfd->fd);
-                DECODE_SYNCOP_ERR (ret);
-	} else {
-		ret = syncop_create (subvol, &loc, flags, mode, glfd->fd,
-				     xattr_req, &iatt);
-                DECODE_SYNCOP_ERR (ret);
-	}
-
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
-
-	if (ret == 0)
-		ret = glfs_loc_link (&loc, &iatt);
-out:
-	loc_wipe (&loc);
-
-	if (xattr_req)
-		dict_unref (xattr_req);
-
-	if (ret && glfd) {
-		glfs_fd_destroy (glfd);
-		glfd = NULL;
-	} else if (glfd) {
-		fd_bind (glfd->fd);
-		glfs_fd_bind (glfd);
-	}
-
-	priv_glfs_subvol_done (fs, subvol);
-
-	return glfd;
-}
-
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_creat, 3.4.0);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
-off_t
-pub_glfs_lseek (struct glfs_fd *glfd, off_t offset, int whence)
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fstat, 3.4.0)
+int
+pub_glfs_fstat(struct glfs_fd *glfd, struct stat *stat)
 {
-	struct stat sb = {0, };
-	int         ret = -1;
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    struct iatt iatt = {
+        0,
+    };
+    fd_t *fd = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    GF_REF_GET(glfd);
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        ret = -1;
+        errno = EBADFD;
+        goto out;
+    }
+
+    ret = syncop_fstat(subvol, fd, &iatt, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    if (ret == 0 && stat)
+        glfs_iatt_to_stat(glfd->fs, &iatt, stat);
+out:
+    if (fd)
+        fd_unref(fd);
+    if (glfd)
+        GF_REF_PUT(glfd);
 
-	__glfs_entry_fd (glfd);
+    glfs_subvol_done(glfd->fs, subvol);
 
-	switch (whence) {
-	case SEEK_SET:
-		glfd->offset = offset;
-		break;
-	case SEEK_CUR:
-		glfd->offset += offset;
-		break;
-	case SEEK_END:
-		ret = pub_glfs_fstat (glfd, &sb);
-		if (ret) {
-			/* seek cannot fail :O */
-			break;
-		}
-		glfd->offset = sb.st_size + offset;
-		break;
-	}
+    __GLFS_EXIT_FS;
 
-	return glfd->offset;
+invalid_fs:
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_lseek, 3.4.0);
-
-
-ssize_t
-pub_glfs_preadv (struct glfs_fd *glfd, const struct iovec *iovec, int iovcnt,
-                 off_t offset, int flags)
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_creat, 3.4.0)
+struct glfs_fd *
+pub_glfs_creat(struct glfs *fs, const char *path, int flags, mode_t mode)
 {
-	xlator_t       *subvol = NULL;
-	ssize_t         ret = -1;
-	ssize_t         size = -1;
-	struct iovec   *iov = NULL;
-	int             cnt = 0;
-	struct iobref  *iobref = NULL;
-	fd_t           *fd = NULL;
-
-	__glfs_entry_fd (glfd);
+    int ret = -1;
+    struct glfs_fd *glfd = NULL;
+    xlator_t *subvol = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    uuid_t gfid;
+    dict_t *xattr_req = NULL;
+    int reval = 0;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    xattr_req = dict_new();
+    if (!xattr_req) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    gf_uuid_generate(gfid);
+    ret = dict_set_gfuuid(xattr_req, "gfid-req", gfid, true);
+    if (ret) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    glfd = glfs_fd_new(fs);
+    if (!glfd)
+        goto out;
+
+    /* This must be glfs_resolve() and NOT glfs_lresolve().
+       That is because open("name", O_CREAT) where "name"
+       is a danging symlink must create the dangling
+       destination.
+    */
+retry:
+    ret = glfs_resolve(fs, subvol, path, &loc, &iatt, reval);
 
-	subvol = priv_glfs_active_subvol (glfd->fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	fd = glfs_resolve_fd (glfd->fs, subvol, glfd);
-	if (!fd) {
-		ret = -1;
-		errno = EBADFD;
-		goto out;
-	}
+    if (ret == -1 && errno != ENOENT)
+        /* Any other type of error is fatal */
+        goto out;
 
-	size = iov_length (iovec, iovcnt);
+    if (ret == -1 && errno == ENOENT && !loc.parent)
+        /* The parent directory or an ancestor even
+           higher does not exist
+        */
+        goto out;
 
-	ret = syncop_readv (subvol, fd, size, offset, 0, &iov, &cnt, &iobref);
-        DECODE_SYNCOP_ERR (ret);
-	if (ret <= 0)
-		goto out;
+    if (loc.inode) {
+        if (flags & O_EXCL) {
+            ret = -1;
+            errno = EEXIST;
+            goto out;
+        }
 
-	size = iov_copy (iovec, iovcnt, iov, cnt); /* FIXME!!! */
+        if (IA_ISDIR(iatt.ia_type)) {
+            ret = -1;
+            errno = EISDIR;
+            goto out;
+        }
 
-	glfd->offset = (offset + size);
+        if (!IA_ISREG(iatt.ia_type)) {
+            ret = -1;
+            errno = EINVAL;
+            goto out;
+        }
+    }
+
+    if (ret == -1 && errno == ENOENT) {
+        loc.inode = inode_new(loc.parent->table);
+        if (!loc.inode) {
+            ret = -1;
+            errno = ENOMEM;
+            goto out;
+        }
+    }
 
-	ret = size;
+    if (glfd->fd) {
+        /* Retry. Safe to touch glfd->fd as we
+           still have not glfs_fd_bind() yet.
+        */
+        fd_unref(glfd->fd);
+        glfd->fd = NULL;
+    }
+
+    glfd->fd = fd_create(loc.inode, getpid());
+    if (!glfd->fd) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+    glfd->fd->flags = flags;
+
+    if (get_fop_attr_thrd_key(&xattr_req))
+        gf_msg_debug("gfapi", 0, "Getting leaseid from thread failed");
+    if (ret == 0) {
+        ret = syncop_open(subvol, &loc, flags, glfd->fd, xattr_req, NULL);
+        DECODE_SYNCOP_ERR(ret);
+    } else {
+        ret = syncop_create(subvol, &loc, flags, mode, glfd->fd, &iatt,
+                            xattr_req, NULL);
+        DECODE_SYNCOP_ERR(ret);
+    }
+
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
+
+    if (ret == 0)
+        ret = glfs_loc_link(&loc, &iatt);
 out:
-        if (iov)
-                GF_FREE (iov);
-        if (iobref)
-                iobref_unref (iobref);
+    loc_wipe(&loc);
+
+    if (xattr_req)
+        dict_unref(xattr_req);
+
+    if (ret && glfd) {
+        GF_REF_PUT(glfd);
+        glfd = NULL;
+    } else if (glfd) {
+        glfd_set_state_bind(glfd);
+    }
 
-	if (fd)
-		fd_unref (fd);
+    glfs_subvol_done(fs, subvol);
 
-	priv_glfs_subvol_done (glfd->fs, subvol);
+    __GLFS_EXIT_FS;
 
-	return ret;
+invalid_fs:
+    return glfd;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_preadv, 3.4.0);
+#ifdef HAVE_SEEK_HOLE
+static int
+glfs_seek(struct glfs_fd *glfd, off_t offset, int whence)
+{
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    fd_t *fd = NULL;
+    gf_seek_what_t what = 0;
+    off_t off = -1;
+
+    switch (whence) {
+        case SEEK_DATA:
+            what = GF_SEEK_DATA;
+            break;
+        case SEEK_HOLE:
+            what = GF_SEEK_HOLE;
+            break;
+        default:
+            /* other SEEK_* do not make sense, all operations get an offset
+             * and the position in the fd is not tracked */
+            errno = EINVAL;
+            goto out;
+    }
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        errno = EBADFD;
+        goto done;
+    }
+
+    ret = syncop_seek(subvol, fd, offset, what, NULL, &off);
+    DECODE_SYNCOP_ERR(ret);
+
+    if (ret != -1)
+        glfd->offset = off;
+
+done:
+    if (fd)
+        fd_unref(fd);
+
+    glfs_subvol_done(glfd->fs, subvol);
 
+out:
+    return ret;
+}
+#endif
 
-ssize_t
-pub_glfs_read (struct glfs_fd *glfd, void *buf, size_t count, int flags)
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_lseek, 3.4.0)
+off_t
+pub_glfs_lseek(struct glfs_fd *glfd, off_t offset, int whence)
 {
-	struct iovec iov = {0, };
-	ssize_t      ret = 0;
+    struct stat sb = {
+        0,
+    };
+    int ret = -1;
+    off_t off = -1;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    GF_REF_GET(glfd);
+
+    switch (whence) {
+        case SEEK_SET:
+            glfd->offset = offset;
+            ret = 0;
+            break;
+        case SEEK_CUR:
+            glfd->offset += offset;
+            ret = 0;
+            break;
+        case SEEK_END:
+            ret = pub_glfs_fstat(glfd, &sb);
+            if (ret) {
+                /* seek cannot fail :O */
+                break;
+            }
+            glfd->offset = sb.st_size + offset;
+            break;
+#ifdef HAVE_SEEK_HOLE
+        case SEEK_DATA:
+        case SEEK_HOLE:
+            ret = glfs_seek(glfd, offset, whence);
+            break;
+#endif
+        default:
+            errno = EINVAL;
+    }
 
-	iov.iov_base = buf;
-	iov.iov_len = count;
+    if (glfd)
+        GF_REF_PUT(glfd);
 
-	ret = pub_glfs_preadv (glfd, &iov, 1, glfd->offset, flags);
+    __GLFS_EXIT_FS;
 
-	return ret;
-}
+    if (ret != -1)
+        off = glfd->offset;
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_read, 3.4.0);
+    return off;
 
+invalid_fs:
+    return -1;
+}
 
-ssize_t
-pub_glfs_pread (struct glfs_fd *glfd, void *buf, size_t count, off_t offset,
-                int flags)
+static ssize_t
+glfs_preadv_common(struct glfs_fd *glfd, const struct iovec *iovec, int iovcnt,
+                   off_t offset, int flags, struct glfs_stat *poststat)
 {
-	struct iovec iov = {0, };
-	ssize_t      ret = 0;
-
-	iov.iov_base = buf;
-	iov.iov_len = count;
+    xlator_t *subvol = NULL;
+    ssize_t ret = -1;
+    ssize_t size = -1;
+    struct iovec *iov = NULL;
+    int cnt = 0;
+    struct iobref *iobref = NULL;
+    fd_t *fd = NULL;
+    struct iatt iatt = {
+        0,
+    };
+    dict_t *fop_attr = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    GF_REF_GET(glfd);
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        ret = -1;
+        errno = EBADFD;
+        goto out;
+    }
+
+    size = iov_length(iovec, iovcnt);
+
+    ret = get_fop_attr_thrd_key(&fop_attr);
+    if (ret)
+        gf_msg_debug("gfapi", 0, "Getting leaseid from thread failed");
+
+    ret = syncop_readv(subvol, fd, size, offset, 0, &iov, &cnt, &iobref, &iatt,
+                       fop_attr, NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    if (ret >= 0 && poststat)
+        glfs_iatt_to_statx(glfd->fs, &iatt, poststat);
+
+    if (ret <= 0)
+        goto out;
+
+    size = iov_copy(iovec, iovcnt, iov, cnt); /* FIXME!!! */
+
+    glfd->offset = (offset + size);
+
+    ret = size;
+out:
+    if (iov)
+        GF_FREE(iov);
+    if (iobref)
+        iobref_unref(iobref);
 
-	ret = pub_glfs_preadv (glfd, &iov, 1, offset, flags);
+    if (fd)
+        fd_unref(fd);
+    if (glfd)
+        GF_REF_PUT(glfd);
+    if (fop_attr)
+        dict_unref(fop_attr);
 
-	return ret;
-}
+    glfs_subvol_done(glfd->fs, subvol);
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_pread, 3.4.0);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_preadv, 3.4.0)
 ssize_t
-pub_glfs_readv (struct glfs_fd *glfd, const struct iovec *iov, int count,
-                int flags)
+pub_glfs_preadv(struct glfs_fd *glfd, const struct iovec *iovec, int iovcnt,
+                off_t offset, int flags)
 {
-	ssize_t      ret = 0;
-
-	ret = pub_glfs_preadv (glfd, iov, count, glfd->offset, flags);
-
-	return ret;
+    return glfs_preadv_common(glfd, iovec, iovcnt, offset, flags, NULL);
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_readv, 3.4.0);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_read, 3.4.0)
+ssize_t
+pub_glfs_read(struct glfs_fd *glfd, void *buf, size_t count, int flags)
+{
+    struct iovec iov = {
+        0,
+    };
+    ssize_t ret = 0;
 
+    if (glfd == NULL) {
+        errno = EBADF;
+        return -1;
+    }
 
-struct glfs_io {
-	struct glfs_fd      *glfd;
-	int                  op;
-	off_t                offset;
-	struct iovec        *iov;
-	int                  count;
-	int                  flags;
-	glfs_io_cbk          fn;
-	void                *data;
-};
+    iov.iov_base = buf;
+    iov.iov_len = count;
 
+    ret = pub_glfs_preadv(glfd, &iov, 1, glfd->offset, flags);
 
-static int
-glfs_io_async_cbk (int ret, call_frame_t *frame, void *data)
+    return ret;
+}
+
+GFAPI_SYMVER_PUBLIC(glfs_pread34, glfs_pread, 3.4.0)
+ssize_t
+pub_glfs_pread34(struct glfs_fd *glfd, void *buf, size_t count, off_t offset,
+                 int flags)
 {
-	struct glfs_io  *gio = data;
+    struct iovec iov = {
+        0,
+    };
+    ssize_t ret = 0;
 
-	gio->fn (gio->glfd, ret, gio->data);
+    iov.iov_base = buf;
+    iov.iov_len = count;
 
-	GF_FREE (gio->iov);
-	GF_FREE (gio);
+    ret = pub_glfs_preadv(glfd, &iov, 1, offset, flags);
 
-	return 0;
+    return ret;
 }
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_pread, 6.0)
 ssize_t
-pub_glfs_pwritev (struct glfs_fd *, const struct iovec *, int, off_t, int);
+pub_glfs_pread(struct glfs_fd *glfd, void *buf, size_t count, off_t offset,
+               int flags, struct glfs_stat *poststat)
+{
+    struct iovec iov = {
+        0,
+    };
+    ssize_t ret = 0;
 
-int
-pub_glfs_ftruncate (struct glfs_fd *, off_t);
+    iov.iov_base = buf;
+    iov.iov_len = count;
 
-int
-pub_glfs_fdatasync (struct glfs_fd *);
+    ret = glfs_preadv_common(glfd, &iov, 1, offset, flags, poststat);
 
-int
-pub_glfs_fsync (struct glfs_fd *glfd);
+    return ret;
+}
 
-int
-pub_glfs_discard (struct glfs_fd *, off_t, size_t);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_readv, 3.4.0)
+ssize_t
+pub_glfs_readv(struct glfs_fd *glfd, const struct iovec *iov, int count,
+               int flags)
+{
+    ssize_t ret = 0;
 
-int
-pub_glfs_zerofill (struct glfs_fd *, off_t, off_t);
+    if (glfd == NULL) {
+        errno = EBADF;
+        return -1;
+    }
 
-static int
-glfs_io_async_task (void *data)
-{
-	struct glfs_io *gio = data;
-	ssize_t         ret = 0;
-
-	switch (gio->op) {
-	case GF_FOP_WRITE:
-		ret = pub_glfs_pwritev (gio->glfd, gio->iov, gio->count,
-				    gio->offset, gio->flags);
-		break;
-	case GF_FOP_FTRUNCATE:
-		ret = pub_glfs_ftruncate (gio->glfd, gio->offset);
-		break;
-	case GF_FOP_FSYNC:
-		if (gio->flags)
-			ret = pub_glfs_fdatasync (gio->glfd);
-		else
-			ret = pub_glfs_fsync (gio->glfd);
-		break;
-	case GF_FOP_DISCARD:
-		ret = pub_glfs_discard (gio->glfd, gio->offset, gio->count);
-		break;
-        case GF_FOP_ZEROFILL:
-                ret = pub_glfs_zerofill(gio->glfd, gio->offset, gio->count);
-                break;
-	}
+    ret = pub_glfs_preadv(glfd, iov, count, glfd->offset, flags);
 
-	return (int) ret;
+    return ret;
 }
 
+struct glfs_io {
+    struct glfs_fd *glfd;
+    int op;
+    off_t offset;
+    struct iovec *iov;
+    int count;
+    int flags;
+    gf_boolean_t oldcb;
+    union {
+        glfs_io_cbk34 fn34;
+        glfs_io_cbk fn;
+    };
+    void *data;
+};
 
-int
-glfs_preadv_async_cbk (call_frame_t *frame, void *cookie, xlator_t *this,
-		       int op_ret, int op_errno, struct iovec *iovec,
-		       int count, struct iatt *stbuf, struct iobref *iobref,
-		       dict_t *xdata)
+static int
+glfs_io_async_cbk(int op_ret, int op_errno, call_frame_t *frame, void *cookie,
+                  struct iovec *iovec, int count, struct iatt *prebuf,
+                  struct iatt *postbuf)
 {
-	struct glfs_io *gio = NULL;
-	xlator_t       *subvol = NULL;
-	struct glfs    *fs = NULL;
-	struct glfs_fd *glfd = NULL;
-
-
-	gio = frame->local;
-	frame->local = NULL;
-	subvol = cookie;
-	glfd = gio->glfd;
-	fs = glfd->fs;
-
-	if (op_ret <= 0)
-		goto out;
+    struct glfs_io *gio = NULL;
+    xlator_t *subvol = NULL;
+    struct glfs *fs = NULL;
+    struct glfs_fd *glfd = NULL;
+    int ret = -1;
+    struct glfs_stat prestat = {}, *prestatp = NULL;
+    struct glfs_stat poststat = {}, *poststatp = NULL;
+
+    GF_VALIDATE_OR_GOTO("gfapi", frame, inval);
+    GF_VALIDATE_OR_GOTO("gfapi", cookie, inval);
+
+    gio = frame->local;
+    frame->local = NULL;
+    subvol = cookie;
+    glfd = gio->glfd;
+    fs = glfd->fs;
+
+    if (!glfs_is_glfd_still_valid(glfd))
+        goto err;
+
+    if (op_ret <= 0) {
+        goto out;
+    } else if (gio->op == GF_FOP_READ) {
+        if (!iovec) {
+            op_ret = -1;
+            op_errno = EINVAL;
+            goto out;
+        }
 
-	op_ret = iov_copy (gio->iov, gio->count, iovec, count);
+        op_ret = iov_copy(gio->iov, gio->count, iovec, count);
+        glfd->offset = gio->offset + op_ret;
+    } else if (gio->op == GF_FOP_WRITE) {
+        glfd->offset = gio->offset + gio->iov->iov_len;
+    }
 
-	glfd->offset = gio->offset + op_ret;
 out:
-	errno = op_errno;
-	gio->fn (gio->glfd, op_ret, gio->data);
+    errno = op_errno;
+    if (gio->oldcb) {
+        gio->fn34(gio->glfd, op_ret, gio->data);
+    } else {
+        if (prebuf) {
+            prestatp = &prestat;
+            glfs_iatt_to_statx(fs, prebuf, prestatp);
+        }
 
-	GF_FREE (gio->iov);
-	GF_FREE (gio);
-	STACK_DESTROY (frame->root);
-	priv_glfs_subvol_done (fs, subvol);
+        if (postbuf) {
+            poststatp = &poststat;
+            glfs_iatt_to_statx(fs, postbuf, poststatp);
+        }
 
-	return 0;
+        gio->fn(gio->glfd, op_ret, prestatp, poststatp, gio->data);
+    }
+err:
+    fd_unref(glfd->fd);
+    /* Since the async operation is complete
+     * release the ref taken during the start
+     * of async operation
+     */
+    GF_REF_PUT(glfd);
+
+    GF_FREE(gio->iov);
+    GF_FREE(gio);
+    STACK_DESTROY(frame->root);
+    glfs_subvol_done(fs, subvol);
+
+    ret = 0;
+inval:
+    return ret;
 }
 
+static int
+glfs_preadv_async_cbk(call_frame_t *frame, void *cookie, xlator_t *this,
+                      int op_ret, int op_errno, struct iovec *iovec, int count,
+                      struct iatt *stbuf, struct iobref *iobref, dict_t *xdata)
+{
+    glfs_io_async_cbk(op_ret, op_errno, frame, cookie, iovec, count, NULL,
+                      stbuf);
 
-int
-pub_glfs_preadv_async (struct glfs_fd *glfd, const struct iovec *iovec,
-                       int count, off_t offset, int flags, glfs_io_cbk fn,
-                       void *data)
+    return 0;
+}
+
+static int
+glfs_preadv_async_common(struct glfs_fd *glfd, const struct iovec *iovec,
+                         int count, off_t offset, int flags, gf_boolean_t oldcb,
+                         glfs_io_cbk fn, void *data)
 {
-	struct glfs_io *gio = NULL;
-	int             ret = 0;
-	call_frame_t   *frame = NULL;
-	xlator_t       *subvol = NULL;
-	glfs_t         *fs = NULL;
-	fd_t           *fd = NULL;
-
-	__glfs_entry_fd (glfd);
-
-	subvol = priv_glfs_active_subvol (glfd->fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
-
-	fd = glfs_resolve_fd (glfd->fs, subvol, glfd);
-	if (!fd) {
-		ret = -1;
-		errno = EBADFD;
-		goto out;
-	}
-
-	fs = glfd->fs;
-
-	frame = syncop_create_frame (THIS);
-	if (!frame) {
-		ret = -1;
-		errno = ENOMEM;
-		goto out;
-	}
-
-	gio = GF_CALLOC (1, sizeof (*gio), glfs_mt_glfs_io_t);
-	if (!gio) {
-		ret = -1;
-		errno = ENOMEM;
-		goto out;
-	}
-
-	gio->iov = iov_dup (iovec, count);
-	if (!gio->iov) {
-		ret = -1;
-		errno = ENOMEM;
-		goto out;
-	}
-
-	gio->op     = GF_FOP_READ;
-	gio->glfd   = glfd;
-	gio->count  = count;
-	gio->offset = offset;
-	gio->flags  = flags;
-	gio->fn     = fn;
-	gio->data   = data;
-
-	frame->local = gio;
-
-	STACK_WIND_COOKIE (frame, glfs_preadv_async_cbk, subvol, subvol,
-			   subvol->fops->readv, fd, iov_length (iovec, count),
-			   offset, flags, NULL);
+    struct glfs_io *gio = NULL;
+    int ret = 0;
+    call_frame_t *frame = NULL;
+    xlator_t *subvol = NULL;
+    struct glfs *fs = NULL;
+    fd_t *fd = NULL;
+    dict_t *fop_attr = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    GF_REF_GET(glfd);
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        ret = -1;
+        errno = EBADFD;
+        goto out;
+    }
+
+    fs = glfd->fs;
+
+    frame = syncop_create_frame(THIS);
+    if (!frame) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    gio = GF_CALLOC(1, sizeof(*gio), glfs_mt_glfs_io_t);
+    if (!gio) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    gio->iov = iov_dup(iovec, count);
+    if (!gio->iov) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    gio->op = GF_FOP_READ;
+    gio->glfd = glfd;
+    gio->count = count;
+    gio->offset = offset;
+    gio->flags = flags;
+    gio->oldcb = oldcb;
+    gio->fn = fn;
+    gio->data = data;
+
+    frame->local = gio;
+
+    ret = get_fop_attr_thrd_key(&fop_attr);
+    if (ret)
+        gf_msg_debug("gfapi", 0, "Getting leaseid from thread failed");
+
+    STACK_WIND_COOKIE(frame, glfs_preadv_async_cbk, subvol, subvol,
+                      subvol->fops->readv, fd, iov_length(iovec, count), offset,
+                      flags, fop_attr);
 
 out:
-        if (ret) {
-                if (gio) {
-                        GF_FREE (gio->iov);
-                        GF_FREE (gio);
-                }
-                if (frame) {
-                        STACK_DESTROY (frame->root);
-                }
-		priv_glfs_subvol_done (fs, subvol);
-	}
+    if (ret) {
+        if (fd)
+            fd_unref(fd);
+        if (glfd)
+            GF_REF_PUT(glfd);
+        if (gio) {
+            GF_FREE(gio->iov);
+            GF_FREE(gio);
+        }
+        if (frame) {
+            STACK_DESTROY(frame->root);
+        }
+        glfs_subvol_done(fs, subvol);
+    }
+    if (fop_attr)
+        dict_unref(fop_attr);
 
-	if (fd)
-		fd_unref (fd);
+    __GLFS_EXIT_FS;
 
-	return ret;
-}
+    return ret;
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_preadv_async, 3.4.0);
+invalid_fs:
+    return -1;
+}
 
+GFAPI_SYMVER_PUBLIC(glfs_preadv_async34, glfs_preadv_async, 3.4.0)
+int
+pub_glfs_preadv_async34(struct glfs_fd *glfd, const struct iovec *iovec,
+                        int count, off_t offset, int flags, glfs_io_cbk34 fn,
+                        void *data)
+{
+    return glfs_preadv_async_common(glfd, iovec, count, offset, flags, _gf_true,
+                                    (void *)fn, data);
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_preadv_async, 6.0)
 int
-pub_glfs_read_async (struct glfs_fd *glfd, void *buf, size_t count, int flags,
-                     glfs_io_cbk fn, void *data)
+pub_glfs_preadv_async(struct glfs_fd *glfd, const struct iovec *iovec,
+                      int count, off_t offset, int flags, glfs_io_cbk fn,
+                      void *data)
 {
-	struct iovec iov = {0, };
-	ssize_t      ret = 0;
+    return glfs_preadv_async_common(glfd, iovec, count, offset, flags,
+                                    _gf_false, fn, data);
+}
 
-	iov.iov_base = buf;
-	iov.iov_len = count;
+GFAPI_SYMVER_PUBLIC(glfs_read_async34, glfs_read_async, 3.4.0)
+int
+pub_glfs_read_async34(struct glfs_fd *glfd, void *buf, size_t count, int flags,
+                      glfs_io_cbk34 fn, void *data)
+{
+    struct iovec iov = {
+        0,
+    };
+    ssize_t ret = 0;
 
-	ret = pub_glfs_preadv_async (glfd, &iov, 1, glfd->offset, flags, fn, data);
+    if (glfd == NULL) {
+        errno = EBADF;
+        return -1;
+    }
 
-	return ret;
-}
+    iov.iov_base = buf;
+    iov.iov_len = count;
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_read_async, 3.4.0);
+    ret = glfs_preadv_async_common(glfd, &iov, 1, glfd->offset, flags, _gf_true,
+                                   (void *)fn, data);
 
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_read_async, 6.0)
 int
-pub_glfs_pread_async (struct glfs_fd *glfd, void *buf, size_t count,
-                      off_t offset, int flags, glfs_io_cbk fn, void *data)
+pub_glfs_read_async(struct glfs_fd *glfd, void *buf, size_t count, int flags,
+                    glfs_io_cbk fn, void *data)
 {
-	struct iovec iov = {0, };
-	ssize_t      ret = 0;
+    struct iovec iov = {
+        0,
+    };
+    ssize_t ret = 0;
 
-	iov.iov_base = buf;
-	iov.iov_len = count;
+    if (glfd == NULL) {
+        errno = EBADF;
+        return -1;
+    }
 
-	ret = pub_glfs_preadv_async (glfd, &iov, 1, offset, flags, fn, data);
+    iov.iov_base = buf;
+    iov.iov_len = count;
 
-	return ret;
-}
-
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_pread_async, 3.4.0);
+    ret = glfs_preadv_async_common(glfd, &iov, 1, glfd->offset, flags,
+                                   _gf_false, fn, data);
 
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC(glfs_pread_async34, glfs_pread_async, 3.4.0)
 int
-pub_glfs_readv_async (struct glfs_fd *glfd, const struct iovec *iov, int count,
-                      int flags, glfs_io_cbk fn, void *data)
+pub_glfs_pread_async34(struct glfs_fd *glfd, void *buf, size_t count,
+                       off_t offset, int flags, glfs_io_cbk34 fn, void *data)
 {
-	ssize_t      ret = 0;
+    struct iovec iov = {
+        0,
+    };
+    ssize_t ret = 0;
 
-	ret = pub_glfs_preadv_async (glfd, iov, count, glfd->offset, flags,
-				  fn, data);
-	return ret;
-}
+    iov.iov_base = buf;
+    iov.iov_len = count;
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_readv_async, 3.4.0);
+    ret = glfs_preadv_async_common(glfd, &iov, 1, offset, flags, _gf_true,
+                                   (void *)fn, data);
 
+    return ret;
+}
 
-ssize_t
-pub_glfs_pwritev (struct glfs_fd *glfd, const struct iovec *iovec, int iovcnt,
-                  off_t offset, int flags)
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_pread_async, 6.0)
+int
+pub_glfs_pread_async(struct glfs_fd *glfd, void *buf, size_t count,
+                     off_t offset, int flags, glfs_io_cbk fn, void *data)
 {
-	xlator_t       *subvol = NULL;
-	int             ret = -1;
-	size_t          size = -1;
-	struct iobref  *iobref = NULL;
-	struct iobuf   *iobuf = NULL;
-	struct iovec    iov = {0, };
-	fd_t           *fd = NULL;
+    struct iovec iov = {
+        0,
+    };
+    ssize_t ret = 0;
 
-	__glfs_entry_fd (glfd);
+    iov.iov_base = buf;
+    iov.iov_len = count;
 
-	subvol = priv_glfs_active_subvol (glfd->fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    ret = glfs_preadv_async_common(glfd, &iov, 1, offset, flags, _gf_false, fn,
+                                   data);
 
-	fd = glfs_resolve_fd (glfd->fs, subvol, glfd);
-	if (!fd) {
-		ret = -1;
-		errno = EBADFD;
-		goto out;
-	}
+    return ret;
+}
 
-	size = iov_length (iovec, iovcnt);
+GFAPI_SYMVER_PUBLIC(glfs_readv_async34, glfs_readv_async, 3.4.0)
+int
+pub_glfs_readv_async34(struct glfs_fd *glfd, const struct iovec *iov, int count,
+                       int flags, glfs_io_cbk34 fn, void *data)
+{
+    ssize_t ret = 0;
 
-	iobuf = iobuf_get2 (subvol->ctx->iobuf_pool, size);
-	if (!iobuf) {
-		ret = -1;
-		errno = ENOMEM;
-		goto out;
-	}
+    if (glfd == NULL) {
+        errno = EBADF;
+        return -1;
+    }
 
-	iobref = iobref_new ();
-	if (!iobref) {
-		iobuf_unref (iobuf);
-		errno = ENOMEM;
-		ret = -1;
-		goto out;
-	}
+    ret = glfs_preadv_async_common(glfd, iov, count, glfd->offset, flags,
+                                   _gf_true, (void *)fn, data);
+    return ret;
+}
 
-	ret = iobref_add (iobref, iobuf);
-	if (ret) {
-		iobuf_unref (iobuf);
-		iobref_unref (iobref);
-		errno = ENOMEM;
-		ret = -1;
-		goto out;
-	}
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_readv_async, 6.0)
+int
+pub_glfs_readv_async(struct glfs_fd *glfd, const struct iovec *iov, int count,
+                     int flags, glfs_io_cbk fn, void *data)
+{
+    ssize_t ret = 0;
 
-	iov_unload (iobuf_ptr (iobuf), iovec, iovcnt);  /* FIXME!!! */
+    if (glfd == NULL) {
+        errno = EBADF;
+        return -1;
+    }
 
-	iov.iov_base = iobuf_ptr (iobuf);
-	iov.iov_len = size;
+    ret = glfs_preadv_async_common(glfd, iov, count, glfd->offset, flags,
+                                   _gf_false, fn, data);
+    return ret;
+}
 
-	ret = syncop_writev (subvol, fd, &iov, 1, offset, iobref, flags);
-        DECODE_SYNCOP_ERR (ret);
+static ssize_t
+glfs_pwritev_common(struct glfs_fd *glfd, const struct iovec *iovec, int iovcnt,
+                    off_t offset, int flags, struct glfs_stat *prestat,
+                    struct glfs_stat *poststat)
+{
+    xlator_t *subvol = NULL;
+    int ret = -1;
+    struct iobref *iobref = NULL;
+    struct iobuf *iobuf = NULL;
+    struct iovec iov = {
+        0,
+    };
+    fd_t *fd = NULL;
+    struct iatt preiatt =
+                    {
+                        0,
+                    },
+                postiatt = {
+                    0,
+                };
+    dict_t *fop_attr = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    GF_REF_GET(glfd);
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        ret = -1;
+        errno = EBADFD;
+        goto out;
+    }
+
+    ret = iobuf_copy(subvol->ctx->iobuf_pool, iovec, iovcnt, &iobref, &iobuf,
+                     &iov);
+    if (ret)
+        goto out;
+
+    ret = get_fop_attr_thrd_key(&fop_attr);
+    if (ret)
+        gf_msg_debug("gfapi", 0, "Getting leaseid from thread failed");
+
+    ret = syncop_writev(subvol, fd, &iov, 1, offset, iobref, flags, &preiatt,
+                        &postiatt, fop_attr, NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    if (ret >= 0) {
+        if (prestat)
+            glfs_iatt_to_statx(glfd->fs, &preiatt, prestat);
+        if (poststat)
+            glfs_iatt_to_statx(glfd->fs, &postiatt, poststat);
+    }
+
+    if (ret <= 0)
+        goto out;
+
+    glfd->offset = (offset + iov.iov_len);
+out:
+    if (iobuf)
+        iobuf_unref(iobuf);
+    if (iobref)
+        iobref_unref(iobref);
+    if (fd)
+        fd_unref(fd);
+    if (glfd)
+        GF_REF_PUT(glfd);
+    if (fop_attr)
+        dict_unref(fop_attr);
+
+    glfs_subvol_done(glfd->fs, subvol);
+
+    __GLFS_EXIT_FS;
+
+invalid_fs:
+    return ret;
+}
 
-	iobuf_unref (iobuf);
-	iobref_unref (iobref);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_copy_file_range, 6.0)
+ssize_t
+pub_glfs_copy_file_range(struct glfs_fd *glfd_in, off64_t *off_in,
+                         struct glfs_fd *glfd_out, off64_t *off_out, size_t len,
+                         unsigned int flags, struct glfs_stat *statbuf,
+                         struct glfs_stat *prestat, struct glfs_stat *poststat)
+{
+    xlator_t *subvol = NULL;
+    int ret = -1;
+    fd_t *fd_in = NULL;
+    fd_t *fd_out = NULL;
+    struct iatt preiatt =
+                    {
+                        0,
+                    },
+                iattbuf =
+                    {
+                        0,
+                    },
+                postiatt = {
+                    0,
+                };
+    dict_t *fop_attr = NULL;
+    off64_t pos_in;
+    off64_t pos_out;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd_in, invalid_fs);
+    __GLFS_ENTRY_VALIDATE_FD(glfd_out, invalid_fs);
+
+    GF_REF_GET(glfd_in);
+    GF_REF_GET(glfd_out);
+
+    if (glfd_in->fs != glfd_out->fs) {
+        ret = -1;
+        errno = EXDEV;
+        goto out;
+    }
+
+    subvol = glfs_active_subvol(glfd_in->fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    fd_in = glfs_resolve_fd(glfd_in->fs, subvol, glfd_in);
+    if (!fd_in) {
+        ret = -1;
+        errno = EBADFD;
+        goto out;
+    }
+
+    fd_out = glfs_resolve_fd(glfd_out->fs, subvol, glfd_out);
+    if (!fd_out) {
+        ret = -1;
+        errno = EBADFD;
+        goto out;
+    }
+
+    /*
+     * This is based on how the vfs layer in the kernel handles
+     * copy_file_range call. Upon receiving it follows the
+     * below method to consider the offset.
+     * if (off_in != NULL)
+     *    use the value off_in to perform the op
+     * else if off_in == NULL
+     *    use the current file offset position to perform the op
+     *
+     * For gfapi, glfd->offset is used. For a freshly opened
+     * fd, the offset is set to 0.
+     */
+    if (off_in)
+        pos_in = *off_in;
+    else
+        pos_in = glfd_in->offset;
+
+    if (off_out)
+        pos_out = *off_out;
+    else
+        pos_out = glfd_out->offset;
+
+    ret = get_fop_attr_thrd_key(&fop_attr);
+    if (ret)
+        gf_msg_debug("gfapi", 0, "Getting leaseid from thread failed");
+
+    ret = syncop_copy_file_range(subvol, fd_in, pos_in, fd_out, pos_out, len,
+                                 flags, &iattbuf, &preiatt, &postiatt, fop_attr,
+                                 NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    if (ret >= 0) {
+        pos_in += ret;
+        pos_out += ret;
+
+        if (off_in)
+            *off_in = pos_in;
+        if (off_out)
+            *off_out = pos_out;
+
+        if (statbuf)
+            glfs_iatt_to_statx(glfd_in->fs, &iattbuf, statbuf);
+        if (prestat)
+            glfs_iatt_to_statx(glfd_in->fs, &preiatt, prestat);
+        if (poststat)
+            glfs_iatt_to_statx(glfd_in->fs, &postiatt, poststat);
+    }
+
+    if (ret <= 0)
+        goto out;
+
+    /*
+     * If *off_in is NULL, then there is no offset info that can
+     * obtained from the input argument. Hence follow below method.
+     *  If *off_in is NULL, then
+     *     glfd->offset = offset + ret;
+     * else
+     *     do nothing.
+     *
+     * According to the man page of copy_file_range, if off_in is
+     * NULL, then the offset of the source file is advanced by
+     * the return value of the fop. The same applies to off_out as
+     * well. Otherwise, if *off_in is not NULL, then the offset
+     * is not advanced by the filesystem. The entity which sends
+     * the copy_file_range call is supposed to advance the offset
+     * value in its buffer (pointed to by *off_in or *off_out)
+     * by the return value of copy_file_range.
+     */
+    if (!off_in)
+        glfd_in->offset += ret;
+
+    if (!off_out)
+        glfd_out->offset += ret;
 
-	if (ret <= 0)
-		goto out;
+out:
+    if (fd_in)
+        fd_unref(fd_in);
+    if (fd_out)
+        fd_unref(fd_out);
+    if (glfd_in)
+        GF_REF_PUT(glfd_in);
+    if (glfd_out)
+        GF_REF_PUT(glfd_out);
+    if (fop_attr)
+        dict_unref(fop_attr);
+
+    glfs_subvol_done(glfd_in->fs, subvol);
+
+    __GLFS_EXIT_FS;
+
+invalid_fs:
+    return ret;
+}
 
-	glfd->offset = (offset + size);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_pwritev, 3.4.0)
+ssize_t
+pub_glfs_pwritev(struct glfs_fd *glfd, const struct iovec *iovec, int iovcnt,
+                 off_t offset, int flags)
+{
+    return glfs_pwritev_common(glfd, iovec, iovcnt, offset, flags, NULL, NULL);
+}
 
-out:
-	if (fd)
-		fd_unref (fd);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_write, 3.4.0)
+ssize_t
+pub_glfs_write(struct glfs_fd *glfd, const void *buf, size_t count, int flags)
+{
+    struct iovec iov = {
+        0,
+    };
+    ssize_t ret = 0;
 
-	priv_glfs_subvol_done (glfd->fs, subvol);
+    if (glfd == NULL) {
+        errno = EBADF;
+        return -1;
+    }
 
-	return ret;
-}
+    iov.iov_base = (void *)buf;
+    iov.iov_len = count;
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_pwritev, 3.4.0);
+    ret = pub_glfs_pwritev(glfd, &iov, 1, glfd->offset, flags);
 
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_writev, 3.4.0)
 ssize_t
-pub_glfs_write (struct glfs_fd *glfd, const void *buf, size_t count, int flags)
+pub_glfs_writev(struct glfs_fd *glfd, const struct iovec *iov, int count,
+                int flags)
 {
-	struct iovec iov = {0, };
-	ssize_t      ret = 0;
+    ssize_t ret = 0;
 
-	iov.iov_base = (void *) buf;
-	iov.iov_len = count;
+    if (glfd == NULL) {
+        errno = EBADF;
+        return -1;
+    }
 
-	ret = pub_glfs_pwritev (glfd, &iov, 1, glfd->offset, flags);
+    ret = pub_glfs_pwritev(glfd, iov, count, glfd->offset, flags);
 
-	return ret;
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_write, 3.4.0);
-
-
+GFAPI_SYMVER_PUBLIC(glfs_pwrite34, glfs_pwrite, 3.4.0)
 ssize_t
-pub_glfs_writev (struct glfs_fd *glfd, const struct iovec *iov, int count,
-                 int flags)
+pub_glfs_pwrite34(struct glfs_fd *glfd, const void *buf, size_t count,
+                  off_t offset, int flags)
 {
-	ssize_t      ret = 0;
-
-	ret = pub_glfs_pwritev (glfd, iov, count, glfd->offset, flags);
+    struct iovec iov = {
+        0,
+    };
+    ssize_t ret = 0;
 
-	return ret;
-}
+    iov.iov_base = (void *)buf;
+    iov.iov_len = count;
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_writev, 3.4.0);
+    ret = pub_glfs_pwritev(glfd, &iov, 1, offset, flags);
 
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_pwrite, 6.0)
 ssize_t
-pub_glfs_pwrite (struct glfs_fd *glfd, const void *buf, size_t count,
-                 off_t offset, int flags)
+pub_glfs_pwrite(struct glfs_fd *glfd, const void *buf, size_t count,
+                off_t offset, int flags, struct glfs_stat *prestat,
+                struct glfs_stat *poststat)
 {
-	struct iovec iov = {0, };
-	ssize_t      ret = 0;
+    struct iovec iov = {
+        0,
+    };
+    ssize_t ret = 0;
 
-	iov.iov_base = (void *) buf;
-	iov.iov_len = count;
+    iov.iov_base = (void *)buf;
+    iov.iov_len = count;
 
-	ret = pub_glfs_pwritev (glfd, &iov, 1, offset, flags);
+    ret = glfs_pwritev_common(glfd, &iov, 1, offset, flags, prestat, poststat);
 
-	return ret;
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_pwrite, 3.4.0);
+extern glfs_t *
+pub_glfs_from_glfd(glfs_fd_t *);
 
+static int
+glfs_pwritev_async_cbk(call_frame_t *frame, void *cookie, xlator_t *this,
+                       int op_ret, int op_errno, struct iatt *prebuf,
+                       struct iatt *postbuf, dict_t *xdata)
+{
+    glfs_io_async_cbk(op_ret, op_errno, frame, cookie, NULL, 0, prebuf,
+                      postbuf);
+
+    return 0;
+}
 
-extern glfs_t *pub_glfs_from_glfd (glfs_fd_t *);
+static int
+glfs_pwritev_async_common(struct glfs_fd *glfd, const struct iovec *iovec,
+                          int count, off_t offset, int flags,
+                          gf_boolean_t oldcb, glfs_io_cbk fn, void *data)
+{
+    struct glfs_io *gio = NULL;
+    int ret = -1;
+    call_frame_t *frame = NULL;
+    xlator_t *subvol = NULL;
+    fd_t *fd = NULL;
+    struct iobref *iobref = NULL;
+    struct iobuf *iobuf = NULL;
+    dict_t *fop_attr = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    /* Need to take explicit ref so that the fd
+     * is not destroyed before the fop is complete
+     */
+    GF_REF_GET(glfd);
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        errno = EBADFD;
+        goto out;
+    }
+
+    gio = GF_CALLOC(1, sizeof(*gio), glfs_mt_glfs_io_t);
+    if (!gio) {
+        errno = ENOMEM;
+        goto out;
+    }
+
+    gio->op = GF_FOP_WRITE;
+    gio->glfd = glfd;
+    gio->offset = offset;
+    gio->flags = flags;
+    gio->oldcb = oldcb;
+    gio->fn = fn;
+    gio->data = data;
+    gio->count = 1;
+    gio->iov = GF_CALLOC(gio->count, sizeof(*(gio->iov)), gf_common_mt_iovec);
+    if (!gio->iov) {
+        errno = ENOMEM;
+        goto out;
+    }
+
+    ret = iobuf_copy(subvol->ctx->iobuf_pool, iovec, count, &iobref, &iobuf,
+                     gio->iov);
+    if (ret)
+        goto out;
+
+    frame = syncop_create_frame(THIS);
+    if (!frame) {
+        errno = ENOMEM;
+        ret = -1;
+        goto out;
+    }
+
+    frame->local = gio;
+
+    ret = get_fop_attr_thrd_key(&fop_attr);
+    if (ret)
+        gf_msg_debug("gfapi", 0, "Getting leaseid from thread failed");
+
+    STACK_WIND_COOKIE(frame, glfs_pwritev_async_cbk, subvol, subvol,
+                      subvol->fops->writev, fd, gio->iov, gio->count, offset,
+                      flags, iobref, fop_attr);
+
+    ret = 0;
+out:
+    if (ret) {
+        if (fd)
+            fd_unref(fd);
+        if (glfd)
+            GF_REF_PUT(glfd);
+        GF_FREE(gio);
+        /*
+         * If there is any error condition check after the frame
+         * creation, we have to destroy the frame root.
+         */
+        glfs_subvol_done(glfd->fs, subvol);
+    }
+    if (fop_attr)
+        dict_unref(fop_attr);
+
+    if (iobuf)
+        iobuf_unref(iobuf);
+    if (iobref)
+        iobref_unref(iobref);
+
+    __GLFS_EXIT_FS;
+
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC(glfs_pwritev_async34, glfs_pwritev_async, 3.4.0)
 int
-pub_glfs_pwritev_async (struct glfs_fd *glfd, const struct iovec *iovec,
-                        int count, off_t offset, int flags, glfs_io_cbk fn,
-                        void *data)
+pub_glfs_pwritev_async34(struct glfs_fd *glfd, const struct iovec *iovec,
+                         int count, off_t offset, int flags, glfs_io_cbk34 fn,
+                         void *data)
 {
-	struct glfs_io *gio = NULL;
-	int             ret = 0;
+    return glfs_pwritev_async_common(glfd, iovec, count, offset, flags,
+                                     _gf_true, (void *)fn, data);
+}
 
-	gio = GF_CALLOC (1, sizeof (*gio), glfs_mt_glfs_io_t);
-	if (!gio) {
-		errno = ENOMEM;
-		return -1;
-	}
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_pwritev_async, 6.0)
+int
+pub_glfs_pwritev_async(struct glfs_fd *glfd, const struct iovec *iovec,
+                       int count, off_t offset, int flags, glfs_io_cbk fn,
+                       void *data)
+{
+    return glfs_pwritev_async_common(glfd, iovec, count, offset, flags,
+                                     _gf_false, fn, data);
+}
 
-	gio->iov = iov_dup (iovec, count);
-	if (!gio->iov) {
-		GF_FREE (gio);
-		errno = ENOMEM;
-		return -1;
-	}
+GFAPI_SYMVER_PUBLIC(glfs_write_async34, glfs_write_async, 3.4.0)
+int
+pub_glfs_write_async34(struct glfs_fd *glfd, const void *buf, size_t count,
+                       int flags, glfs_io_cbk34 fn, void *data)
+{
+    struct iovec iov = {
+        0,
+    };
+    ssize_t ret = 0;
 
-	gio->op     = GF_FOP_WRITE;
-	gio->glfd   = glfd;
-	gio->count  = count;
-	gio->offset = offset;
-	gio->flags  = flags;
-	gio->fn     = fn;
-	gio->data   = data;
+    if (glfd == NULL) {
+        errno = EBADF;
+        return -1;
+    }
 
-	ret = synctask_new (pub_glfs_from_glfd (glfd)->ctx->env,
-			    glfs_io_async_task, glfs_io_async_cbk,
-			    NULL, gio);
+    iov.iov_base = (void *)buf;
+    iov.iov_len = count;
 
-	if (ret) {
-		GF_FREE (gio->iov);
-		GF_FREE (gio);
-	}
+    ret = glfs_pwritev_async_common(glfd, &iov, 1, glfd->offset, flags,
+                                    _gf_true, (void *)fn, data);
 
-	return ret;
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_pwritev_async, 3.4.0);
-
-
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_write_async, 6.0)
 int
-pub_glfs_write_async (struct glfs_fd *glfd, const void *buf, size_t count,
-                      int flags, glfs_io_cbk fn, void *data)
+pub_glfs_write_async(struct glfs_fd *glfd, const void *buf, size_t count,
+                     int flags, glfs_io_cbk fn, void *data)
 {
-	struct iovec iov = {0, };
-	ssize_t      ret = 0;
+    struct iovec iov = {
+        0,
+    };
+    ssize_t ret = 0;
 
-	iov.iov_base = (void *) buf;
-	iov.iov_len = count;
+    if (glfd == NULL) {
+        errno = EBADF;
+        return -1;
+    }
 
-	ret = pub_glfs_pwritev_async (glfd, &iov, 1, glfd->offset, flags, fn, data);
-
-	return ret;
-}
+    iov.iov_base = (void *)buf;
+    iov.iov_len = count;
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_write_async, 3.4.0);
+    ret = glfs_pwritev_async_common(glfd, &iov, 1, glfd->offset, flags,
+                                    _gf_false, fn, data);
 
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC(glfs_pwrite_async34, glfs_pwrite_async, 3.4.0)
 int
-pub_glfs_pwrite_async (struct glfs_fd *glfd, const void *buf, int count,
-                       off_t offset, int flags, glfs_io_cbk fn, void *data)
+pub_glfs_pwrite_async34(struct glfs_fd *glfd, const void *buf, int count,
+                        off_t offset, int flags, glfs_io_cbk34 fn, void *data)
 {
-	struct iovec iov = {0, };
-	ssize_t      ret = 0;
+    struct iovec iov = {
+        0,
+    };
+    ssize_t ret = 0;
 
-	iov.iov_base = (void *) buf;
-	iov.iov_len = count;
+    iov.iov_base = (void *)buf;
+    iov.iov_len = count;
 
-	ret = pub_glfs_pwritev_async (glfd, &iov, 1, offset, flags, fn, data);
+    ret = glfs_pwritev_async_common(glfd, &iov, 1, offset, flags, _gf_true,
+                                    (void *)fn, data);
 
-	return ret;
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_pwrite_async, 3.4.0);
-
-
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_pwrite_async, 6.0)
 int
-pub_glfs_writev_async (struct glfs_fd *glfd, const struct iovec *iov, int count,
-                       int flags, glfs_io_cbk fn, void *data)
+pub_glfs_pwrite_async(struct glfs_fd *glfd, const void *buf, int count,
+                      off_t offset, int flags, glfs_io_cbk fn, void *data)
 {
-	ssize_t      ret = 0;
+    struct iovec iov = {
+        0,
+    };
+    ssize_t ret = 0;
 
-	ret = pub_glfs_pwritev_async (glfd, iov, count, glfd->offset, flags,
-				   fn, data);
-	return ret;
-}
+    iov.iov_base = (void *)buf;
+    iov.iov_len = count;
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_writev_async, 3.4.0);
+    ret = glfs_pwritev_async_common(glfd, &iov, 1, offset, flags, _gf_false, fn,
+                                    data);
 
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC(glfs_writev_async34, glfs_writev_async, 3.4.0)
 int
-pub_glfs_fsync (struct glfs_fd *glfd)
+pub_glfs_writev_async34(struct glfs_fd *glfd, const struct iovec *iov,
+                        int count, int flags, glfs_io_cbk34 fn, void *data)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	fd_t            *fd = NULL;
+    ssize_t ret = 0;
 
-	__glfs_entry_fd (glfd);
+    if (glfd == NULL) {
+        errno = EBADF;
+        return -1;
+    }
 
-	subvol = priv_glfs_active_subvol (glfd->fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
-
-	fd = glfs_resolve_fd (glfd->fs, subvol, glfd);
-	if (!fd) {
-		ret = -1;
-		errno = EBADFD;
-		goto out;
-	}
+    ret = glfs_pwritev_async_common(glfd, iov, count, glfd->offset, flags,
+                                    _gf_true, (void *)fn, data);
+    return ret;
+}
 
-	ret = syncop_fsync (subvol, fd, 0);
-        DECODE_SYNCOP_ERR (ret);
-out:
-	if (fd)
-		fd_unref (fd);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_writev_async, 6.0)
+int
+pub_glfs_writev_async(struct glfs_fd *glfd, const struct iovec *iov, int count,
+                      int flags, glfs_io_cbk fn, void *data)
+{
+    ssize_t ret = 0;
 
-	priv_glfs_subvol_done (glfd->fs, subvol);
+    if (glfd == NULL) {
+        errno = EBADF;
+        return -1;
+    }
 
-	return ret;
+    ret = glfs_pwritev_async_common(glfd, iov, count, glfd->offset, flags,
+                                    _gf_false, fn, data);
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fsync, 3.4.0);
-
-
 static int
-glfs_fsync_async_common (struct glfs_fd *glfd, glfs_io_cbk fn, void *data,
-			 int dataonly)
+glfs_fsync_common(struct glfs_fd *glfd, struct glfs_stat *prestat,
+                  struct glfs_stat *poststat)
 {
-	struct glfs_io *gio = NULL;
-	int             ret = 0;
-
-	gio = GF_CALLOC (1, sizeof (*gio), glfs_mt_glfs_io_t);
-	if (!gio) {
-		errno = ENOMEM;
-		return -1;
-	}
-
-	gio->op     = GF_FOP_FSYNC;
-	gio->glfd   = glfd;
-	gio->flags  = dataonly;
-	gio->fn     = fn;
-	gio->data   = data;
-
-	ret = synctask_new (pub_glfs_from_glfd (glfd)->ctx->env,
-			    glfs_io_async_task, glfs_io_async_cbk,
-			    NULL, gio);
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    fd_t *fd = NULL;
+    struct iatt preiatt =
+                    {
+                        0,
+                    },
+                postiatt = {
+                    0,
+                };
+    dict_t *fop_attr = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    GF_REF_GET(glfd);
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        ret = -1;
+        errno = EBADFD;
+        goto out;
+    }
+
+    ret = get_fop_attr_thrd_key(&fop_attr);
+    if (ret)
+        gf_msg_debug("gfapi", 0, "Getting leaseid from thread failed");
+
+    ret = syncop_fsync(subvol, fd, 0, &preiatt, &postiatt, fop_attr, NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    if (ret >= 0) {
+        if (prestat)
+            glfs_iatt_to_statx(glfd->fs, &preiatt, prestat);
+        if (poststat)
+            glfs_iatt_to_statx(glfd->fs, &postiatt, poststat);
+    }
+out:
+    if (fd)
+        fd_unref(fd);
+    if (glfd)
+        GF_REF_PUT(glfd);
+    if (fop_attr)
+        dict_unref(fop_attr);
 
-	if (ret) {
-		GF_FREE (gio->iov);
-		GF_FREE (gio);
-	}
+    glfs_subvol_done(glfd->fs, subvol);
 
-	return ret;
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
 }
 
-
+GFAPI_SYMVER_PUBLIC(glfs_fsync34, glfs_fsync, 3.4.0)
 int
-pub_glfs_fsync_async (struct glfs_fd *glfd, glfs_io_cbk fn, void *data)
+pub_glfs_fsync34(struct glfs_fd *glfd)
 {
-	return glfs_fsync_async_common (glfd, fn, data, 0);
+    return glfs_fsync_common(glfd, NULL, NULL);
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fsync_async, 3.4.0);
-
-
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fsync, 6.0)
 int
-pub_glfs_fdatasync (struct glfs_fd *glfd)
+pub_glfs_fsync(struct glfs_fd *glfd, struct glfs_stat *prestat,
+               struct glfs_stat *poststat)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	fd_t            *fd = NULL;
+    return glfs_fsync_common(glfd, prestat, poststat);
+}
 
-	__glfs_entry_fd (glfd);
+static int
+glfs_fsync_async_cbk(call_frame_t *frame, void *cookie, xlator_t *this,
+                     int32_t op_ret, int32_t op_errno, struct iatt *prebuf,
+                     struct iatt *postbuf, dict_t *xdata)
+{
+    glfs_io_async_cbk(op_ret, op_errno, frame, cookie, NULL, 0, prebuf,
+                      postbuf);
 
-	subvol = priv_glfs_active_subvol (glfd->fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    return 0;
+}
 
-	fd = glfs_resolve_fd (glfd->fs, subvol, glfd);
-	if (!fd) {
-		ret = -1;
-		errno = EBADFD;
-		goto out;
-	}
+static int
+glfs_fsync_async_common(struct glfs_fd *glfd, gf_boolean_t oldcb,
+                        glfs_io_cbk fn, void *data, int dataonly)
+{
+    struct glfs_io *gio = NULL;
+    int ret = 0;
+    call_frame_t *frame = NULL;
+    xlator_t *subvol = NULL;
+    fd_t *fd = NULL;
+
+    /* Need to take explicit ref so that the fd
+     * is not destroyed before the fop is complete
+     */
+    GF_REF_GET(glfd);
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        ret = -1;
+        errno = EBADFD;
+        goto out;
+    }
+
+    frame = syncop_create_frame(THIS);
+    if (!frame) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    gio = GF_CALLOC(1, sizeof(*gio), glfs_mt_glfs_io_t);
+    if (!gio) {
+        errno = ENOMEM;
+        ret = -1;
+        goto out;
+    }
+
+    gio->op = GF_FOP_FSYNC;
+    gio->glfd = glfd;
+    gio->flags = dataonly;
+    gio->oldcb = oldcb;
+    gio->fn = fn;
+    gio->data = data;
+
+    frame->local = gio;
+
+    STACK_WIND_COOKIE(frame, glfs_fsync_async_cbk, subvol, subvol,
+                      subvol->fops->fsync, fd, dataonly, NULL);
 
-	ret = syncop_fsync (subvol, fd, 1);
-        DECODE_SYNCOP_ERR (ret);
 out:
-	if (fd)
-		fd_unref (fd);
-
-	priv_glfs_subvol_done (glfd->fs, subvol);
-
-	return ret;
+    if (ret) {
+        if (fd)
+            fd_unref(fd);
+        GF_REF_PUT(glfd);
+        GF_FREE(gio);
+        if (frame)
+            STACK_DESTROY(frame->root);
+        glfs_subvol_done(glfd->fs, subvol);
+    }
+
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fdatasync, 3.4.0);
-
-
+GFAPI_SYMVER_PUBLIC(glfs_fsync_async34, glfs_fsync_async, 3.4.0)
 int
-pub_glfs_fdatasync_async (struct glfs_fd *glfd, glfs_io_cbk fn, void *data)
+pub_glfs_fsync_async34(struct glfs_fd *glfd, glfs_io_cbk34 fn, void *data)
 {
-	return glfs_fsync_async_common (glfd, fn, data, 1);
-}
+    int ret = -1;
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fdatasync_async, 3.4.0);
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
 
+    ret = glfs_fsync_async_common(glfd, _gf_true, (void *)fn, data, 0);
+
+    __GLFS_EXIT_FS;
+
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fsync_async, 6.0)
 int
-pub_glfs_ftruncate (struct glfs_fd *glfd, off_t offset)
+pub_glfs_fsync_async(struct glfs_fd *glfd, glfs_io_cbk fn, void *data)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	fd_t            *fd = NULL;
+    int ret = -1;
 
-	__glfs_entry_fd (glfd);
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
 
-	subvol = priv_glfs_active_subvol (glfd->fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    ret = glfs_fsync_async_common(glfd, _gf_false, fn, data, 0);
 
-	fd = glfs_resolve_fd (glfd->fs, subvol, glfd);
-	if (!fd) {
-		ret = -1;
-		errno = EBADFD;
-		goto out;
-	}
+    __GLFS_EXIT_FS;
 
-	ret = syncop_ftruncate (subvol, fd, offset);
-        DECODE_SYNCOP_ERR (ret);
+invalid_fs:
+    return ret;
+}
+
+static int
+glfs_fdatasync_common(struct glfs_fd *glfd, struct glfs_stat *prestat,
+                      struct glfs_stat *poststat)
+{
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    fd_t *fd = NULL;
+    struct iatt preiatt =
+                    {
+                        0,
+                    },
+                postiatt = {
+                    0,
+                };
+    dict_t *fop_attr = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    GF_REF_GET(glfd);
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        ret = -1;
+        errno = EBADFD;
+        goto out;
+    }
+
+    ret = get_fop_attr_thrd_key(&fop_attr);
+    if (ret)
+        gf_msg_debug("gfapi", 0, "Getting leaseid from thread failed");
+
+    ret = syncop_fsync(subvol, fd, 1, &preiatt, &postiatt, fop_attr, NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    if (ret >= 0) {
+        if (prestat)
+            glfs_iatt_to_statx(glfd->fs, &preiatt, prestat);
+        if (poststat)
+            glfs_iatt_to_statx(glfd->fs, &postiatt, poststat);
+    }
 out:
-	if (fd)
-		fd_unref (fd);
+    if (fd)
+        fd_unref(fd);
+    if (glfd)
+        GF_REF_PUT(glfd);
+    if (fop_attr)
+        dict_unref(fop_attr);
 
-	priv_glfs_subvol_done (glfd->fs, subvol);
+    glfs_subvol_done(glfd->fs, subvol);
 
-	return ret;
-}
+    __GLFS_EXIT_FS;
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_ftruncate, 3.4.0);
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC(glfs_fdatasync34, glfs_fdatasync, 3.4.0)
+int
+pub_glfs_fdatasync34(struct glfs_fd *glfd)
+{
+    return glfs_fdatasync_common(glfd, NULL, NULL);
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fdatasync, 6.0)
 int
-pub_glfs_ftruncate_async (struct glfs_fd *glfd, off_t offset, glfs_io_cbk fn,
-                          void *data)
+pub_glfs_fdatasync(struct glfs_fd *glfd, struct glfs_stat *prestat,
+                   struct glfs_stat *poststat)
 {
-	struct glfs_io *gio = NULL;
-	int             ret = 0;
+    return glfs_fdatasync_common(glfd, prestat, poststat);
+}
 
-	gio = GF_CALLOC (1, sizeof (*gio), glfs_mt_glfs_io_t);
-	if (!gio) {
-		errno = ENOMEM;
-		return -1;
-	}
+GFAPI_SYMVER_PUBLIC(glfs_fdatasync_async34, glfs_fdatasync_async, 3.4.0)
+int
+pub_glfs_fdatasync_async34(struct glfs_fd *glfd, glfs_io_cbk34 fn, void *data)
+{
+    int ret = -1;
 
-	gio->op     = GF_FOP_FTRUNCATE;
-	gio->glfd   = glfd;
-	gio->offset = offset;
-	gio->fn     = fn;
-	gio->data   = data;
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
 
-	ret = synctask_new (pub_glfs_from_glfd (glfd)->ctx->env,
-			    glfs_io_async_task, glfs_io_async_cbk,
-			    NULL, gio);
+    ret = glfs_fsync_async_common(glfd, _gf_true, (void *)fn, data, 1);
 
-	if (ret) {
-		GF_FREE (gio->iov);
-		GF_FREE (gio);
-	}
+    __GLFS_EXIT_FS;
 
-	return ret;
+invalid_fs:
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_ftruncate_async, 3.4.0);
-
-
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fdatasync_async, 6.0)
 int
-pub_glfs_access (struct glfs *fs, const char *path, int mode)
+pub_glfs_fdatasync_async(struct glfs_fd *glfd, glfs_io_cbk fn, void *data)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	loc_t            loc = {0, };
-	struct iatt      iatt = {0, };
-	int              reval = 0;
+    int ret = -1;
 
-	__glfs_entry_fs (fs);
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
 
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
-retry:
-	ret = glfs_resolve (fs, subvol, path, &loc, &iatt, reval);
-
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    ret = glfs_fsync_async_common(glfd, _gf_false, fn, data, 1);
 
-	if (ret)
-		goto out;
+    __GLFS_EXIT_FS;
 
-	ret = syncop_access (subvol, &loc, mode);
-        DECODE_SYNCOP_ERR (ret);
+invalid_fs:
+    return ret;
+}
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+static int
+glfs_ftruncate_common(struct glfs_fd *glfd, off_t offset,
+                      struct glfs_stat *prestat, struct glfs_stat *poststat)
+{
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    fd_t *fd = NULL;
+    struct iatt preiatt =
+                    {
+                        0,
+                    },
+                postiatt = {
+                    0,
+                };
+    dict_t *fop_attr = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    GF_REF_GET(glfd);
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        ret = -1;
+        errno = EBADFD;
+        goto out;
+    }
+
+    ret = get_fop_attr_thrd_key(&fop_attr);
+    if (ret)
+        gf_msg_debug("gfapi", 0, "Getting leaseid from thread failed");
+
+    ret = syncop_ftruncate(subvol, fd, offset, &preiatt, &postiatt, fop_attr,
+                           NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    if (ret >= 0) {
+        if (prestat)
+            glfs_iatt_to_statx(glfd->fs, &preiatt, prestat);
+        if (poststat)
+            glfs_iatt_to_statx(glfd->fs, &postiatt, poststat);
+    }
 out:
-	loc_wipe (&loc);
-
-	priv_glfs_subvol_done (fs, subvol);
+    if (fd)
+        fd_unref(fd);
+    if (glfd)
+        GF_REF_PUT(glfd);
+    if (fop_attr)
+        dict_unref(fop_attr);
 
-	return ret;
-}
+    glfs_subvol_done(glfd->fs, subvol);
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_access, 3.4.0);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC(glfs_ftruncate34, glfs_ftruncate, 3.4.0)
 int
-pub_glfs_symlink (struct glfs *fs, const char *data, const char *path)
+pub_glfs_ftruncate34(struct glfs_fd *glfd, off_t offset)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	loc_t            loc = {0, };
-	struct iatt      iatt = {0, };
-	uuid_t           gfid;
-	dict_t          *xattr_req = NULL;
-	int              reval = 0;
-
-	__glfs_entry_fs (fs);
-
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    return glfs_ftruncate_common(glfd, offset, NULL, NULL);
+}
 
-	xattr_req = dict_new ();
-	if (!xattr_req) {
-		ret = -1;
-		errno = ENOMEM;
-		goto out;
-	}
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_ftruncate, 6.0)
+int
+pub_glfs_ftruncate(struct glfs_fd *glfd, off_t offset,
+                   struct glfs_stat *prestat, struct glfs_stat *poststat)
+{
+    return glfs_ftruncate_common(glfd, offset, prestat, poststat);
+}
 
-	uuid_generate (gfid);
-	ret = dict_set_static_bin (xattr_req, "gfid-req", gfid, 16);
-	if (ret) {
-		ret = -1;
-		errno = ENOMEM;
-		goto out;
-	}
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_truncate, 3.7.15)
+int
+pub_glfs_truncate(struct glfs *fs, const char *path, off_t length)
+{
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    int reval = 0;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
 retry:
-	ret = glfs_lresolve (fs, subvol, path, &loc, &iatt, reval);
-
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
-
-	if (loc.inode) {
-		errno = EEXIST;
-		ret = -1;
-		goto out;
-	}
+    ret = glfs_resolve(fs, subvol, path, &loc, &iatt, reval);
 
-	if (ret == -1 && errno != ENOENT)
-		/* Any other type of error is fatal */
-		goto out;
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	if (ret == -1 && errno == ENOENT && !loc.parent)
-		/* The parent directory or an ancestor even
-		   higher does not exist
-		*/
-		goto out;
+    if (ret)
+        goto out;
 
-	/* ret == -1 && errno == ENOENT */
-	loc.inode = inode_new (loc.parent->table);
-	if (!loc.inode) {
-		ret = -1;
-		errno = ENOMEM;
-		goto out;
-	}
+    ret = syncop_truncate(subvol, &loc, length, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
 
-	ret = syncop_symlink (subvol, &loc, data, xattr_req, &iatt);
-        DECODE_SYNCOP_ERR (ret);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
+out:
+    loc_wipe(&loc);
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    glfs_subvol_done(fs, subvol);
 
-	if (ret == 0)
-		ret = glfs_loc_link (&loc, &iatt);
-out:
-	loc_wipe (&loc);
+    __GLFS_EXIT_FS;
 
-	if (xattr_req)
-		dict_unref (xattr_req);
+invalid_fs:
+    return ret;
+}
 
-	priv_glfs_subvol_done (fs, subvol);
+static int
+glfs_ftruncate_async_cbk(call_frame_t *frame, void *cookie, xlator_t *this,
+                         int32_t op_ret, int32_t op_errno, struct iatt *prebuf,
+                         struct iatt *postbuf, dict_t *xdata)
+{
+    glfs_io_async_cbk(op_ret, op_errno, frame, cookie, NULL, 0, prebuf,
+                      postbuf);
 
-	return ret;
+    return 0;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_symlink, 3.4.0);
+static int
+glfs_ftruncate_async_common(struct glfs_fd *glfd, off_t offset,
+                            gf_boolean_t oldcb, glfs_io_cbk fn, void *data)
+{
+    struct glfs_io *gio = NULL;
+    int ret = -1;
+    call_frame_t *frame = NULL;
+    xlator_t *subvol = NULL;
+    fd_t *fd = NULL;
+    dict_t *fop_attr = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    /* Need to take explicit ref so that the fd
+     * is not destroyed before the fop is complete
+     */
+    GF_REF_GET(glfd);
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        errno = EBADFD;
+        goto out;
+    }
+
+    frame = syncop_create_frame(THIS);
+    if (!frame) {
+        errno = ENOMEM;
+        goto out;
+    }
+
+    gio = GF_CALLOC(1, sizeof(*gio), glfs_mt_glfs_io_t);
+    if (!gio) {
+        errno = ENOMEM;
+        goto out;
+    }
+
+    gio->op = GF_FOP_FTRUNCATE;
+    gio->glfd = glfd;
+    gio->offset = offset;
+    gio->oldcb = oldcb;
+    gio->fn = fn;
+    gio->data = data;
+
+    frame->local = gio;
+
+    ret = get_fop_attr_thrd_key(&fop_attr);
+    if (ret)
+        gf_msg_debug("gfapi", 0, "Getting leaseid from thread failed");
+
+    STACK_WIND_COOKIE(frame, glfs_ftruncate_async_cbk, subvol, subvol,
+                      subvol->fops->ftruncate, fd, offset, fop_attr);
+
+    ret = 0;
 
+out:
+    if (ret) {
+        if (fd)
+            fd_unref(fd);
+        if (glfd)
+            GF_REF_PUT(glfd);
+        GF_FREE(gio);
+        if (frame)
+            STACK_DESTROY(frame->root);
+        glfs_subvol_done(glfd->fs, subvol);
+    }
+    if (fop_attr)
+        dict_unref(fop_attr);
+
+    __GLFS_EXIT_FS;
+
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC(glfs_ftruncate_async34, glfs_ftruncate_async, 3.4.0)
 int
-pub_glfs_readlink (struct glfs *fs, const char *path, char *buf, size_t bufsiz)
+pub_glfs_ftruncate_async34(struct glfs_fd *glfd, off_t offset, glfs_io_cbk34 fn,
+                           void *data)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	loc_t            loc = {0, };
-	struct iatt      iatt = {0, };
-	int              reval = 0;
-	char            *linkval = NULL;
+    return glfs_ftruncate_async_common(glfd, offset, _gf_true, (void *)fn,
+                                       data);
+}
 
-	__glfs_entry_fs (fs);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_ftruncate_async, 6.0)
+int
+pub_glfs_ftruncate_async(struct glfs_fd *glfd, off_t offset, glfs_io_cbk fn,
+                         void *data)
+{
+    return glfs_ftruncate_async_common(glfd, offset, _gf_false, fn, data);
+}
 
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_access, 3.4.0)
+int
+pub_glfs_access(struct glfs *fs, const char *path, int mode)
+{
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    int reval = 0;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
 retry:
-	ret = glfs_lresolve (fs, subvol, path, &loc, &iatt, reval);
-
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    ret = glfs_resolve(fs, subvol, path, &loc, &iatt, reval);
 
-	if (ret)
-		goto out;
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	if (iatt.ia_type != IA_IFLNK) {
-		ret = -1;
-		errno = EINVAL;
-		goto out;
-	}
+    if (ret)
+        goto out;
 
-	ret = syncop_readlink (subvol, &loc, &linkval, bufsiz);
-        DECODE_SYNCOP_ERR (ret);
-	if (ret > 0) {
-		memcpy (buf, linkval, ret);
-		GF_FREE (linkval);
-	}
+    ret = syncop_access(subvol, &loc, mode, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 out:
-	loc_wipe (&loc);
-
-	priv_glfs_subvol_done (fs, subvol);
+    loc_wipe(&loc);
 
-	return ret;
-}
+    glfs_subvol_done(fs, subvol);
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_readlink, 3.4.0);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_symlink, 3.4.0)
 int
-pub_glfs_mknod (struct glfs *fs, const char *path, mode_t mode, dev_t dev)
+pub_glfs_symlink(struct glfs *fs, const char *data, const char *path)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	loc_t            loc = {0, };
-	struct iatt      iatt = {0, };
-	uuid_t           gfid;
-	dict_t          *xattr_req = NULL;
-	int              reval = 0;
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    uuid_t gfid;
+    dict_t *xattr_req = NULL;
+    int reval = 0;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    xattr_req = dict_new();
+    if (!xattr_req) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    gf_uuid_generate(gfid);
+    ret = dict_set_gfuuid(xattr_req, "gfid-req", gfid, true);
+    if (ret) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+retry:
+    ret = glfs_lresolve(fs, subvol, path, &loc, &iatt, reval);
 
-	__glfs_entry_fs (fs);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    if (loc.inode) {
+        errno = EEXIST;
+        ret = -1;
+        goto out;
+    }
 
-	xattr_req = dict_new ();
-	if (!xattr_req) {
-		ret = -1;
-		errno = ENOMEM;
-		goto out;
-	}
+    if (ret == -1 && errno != ENOENT)
+        /* Any other type of error is fatal */
+        goto out;
 
-	uuid_generate (gfid);
-	ret = dict_set_static_bin (xattr_req, "gfid-req", gfid, 16);
-	if (ret) {
-		ret = -1;
-		errno = ENOMEM;
-		goto out;
-	}
-retry:
-	ret = glfs_lresolve (fs, subvol, path, &loc, &iatt, reval);
+    if (ret == -1 && errno == ENOENT && !loc.parent)
+        /* The parent directory or an ancestor even
+           higher does not exist
+        */
+        goto out;
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    /* ret == -1 && errno == ENOENT */
+    loc.inode = inode_new(loc.parent->table);
+    if (!loc.inode) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
 
-	if (loc.inode) {
-		errno = EEXIST;
-		ret = -1;
-		goto out;
-	}
+    ret = syncop_symlink(subvol, &loc, data, &iatt, xattr_req, NULL);
+    DECODE_SYNCOP_ERR(ret);
 
-	if (ret == -1 && errno != ENOENT)
-		/* Any other type of error is fatal */
-		goto out;
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	if (ret == -1 && errno == ENOENT && !loc.parent)
-		/* The parent directory or an ancestor even
-		   higher does not exist
-		*/
-		goto out;
+    if (ret == 0)
+        ret = glfs_loc_link(&loc, &iatt);
+out:
+    loc_wipe(&loc);
 
-	/* ret == -1 && errno == ENOENT */
-	loc.inode = inode_new (loc.parent->table);
-	if (!loc.inode) {
-		ret = -1;
-		errno = ENOMEM;
-		goto out;
-	}
+    if (xattr_req)
+        dict_unref(xattr_req);
 
-	ret = syncop_mknod (subvol, &loc, mode, dev, xattr_req, &iatt);
-        DECODE_SYNCOP_ERR (ret);
+    glfs_subvol_done(fs, subvol);
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    __GLFS_EXIT_FS;
 
-	if (ret == 0)
-		ret = glfs_loc_link (&loc, &iatt);
-out:
-	loc_wipe (&loc);
+invalid_fs:
+    return ret;
+}
 
-	if (xattr_req)
-		dict_unref (xattr_req);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_readlink, 3.4.0)
+int
+pub_glfs_readlink(struct glfs *fs, const char *path, char *buf, size_t bufsiz)
+{
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    int reval = 0;
+    char *linkval = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+retry:
+    ret = glfs_lresolve(fs, subvol, path, &loc, &iatt, reval);
 
-	priv_glfs_subvol_done (fs, subvol);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	return ret;
-}
+    if (ret)
+        goto out;
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_mknod, 3.4.0);
+    if (iatt.ia_type != IA_IFLNK) {
+        ret = -1;
+        errno = EINVAL;
+        goto out;
+    }
 
+    ret = syncop_readlink(subvol, &loc, &linkval, bufsiz, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+    if (ret > 0) {
+        memcpy(buf, linkval, ret);
+        GF_FREE(linkval);
+    }
 
-int
-pub_glfs_mkdir (struct glfs *fs, const char *path, mode_t mode)
-{
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	loc_t            loc = {0, };
-	struct iatt      iatt = {0, };
-	uuid_t           gfid;
-	dict_t          *xattr_req = NULL;
-	int              reval = 0;
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
+out:
+    loc_wipe(&loc);
 
-	__glfs_entry_fs (fs);
+    glfs_subvol_done(fs, subvol);
 
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    __GLFS_EXIT_FS;
 
-	xattr_req = dict_new ();
-	if (!xattr_req) {
-		ret = -1;
-		errno = ENOMEM;
-		goto out;
-	}
+invalid_fs:
+    return ret;
+}
 
-	uuid_generate (gfid);
-	ret = dict_set_static_bin (xattr_req, "gfid-req", gfid, 16);
-	if (ret) {
-		ret = -1;
-		errno = ENOMEM;
-		goto out;
-	}
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_mknod, 3.4.0)
+int
+pub_glfs_mknod(struct glfs *fs, const char *path, mode_t mode, dev_t dev)
+{
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    uuid_t gfid;
+    dict_t *xattr_req = NULL;
+    int reval = 0;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    xattr_req = dict_new();
+    if (!xattr_req) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    gf_uuid_generate(gfid);
+    ret = dict_set_gfuuid(xattr_req, "gfid-req", gfid, true);
+    if (ret) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
 retry:
-	ret = glfs_lresolve (fs, subvol, path, &loc, &iatt, reval);
+    ret = glfs_lresolve(fs, subvol, path, &loc, &iatt, reval);
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	if (loc.inode) {
-		errno = EEXIST;
-		ret = -1;
-		goto out;
-	}
+    if (loc.inode) {
+        errno = EEXIST;
+        ret = -1;
+        goto out;
+    }
 
-	if (ret == -1 && errno != ENOENT)
-		/* Any other type of error is fatal */
-		goto out;
+    if (ret == -1 && errno != ENOENT)
+        /* Any other type of error is fatal */
+        goto out;
 
-	if (ret == -1 && errno == ENOENT && !loc.parent)
-		/* The parent directory or an ancestor even
-		   higher does not exist
-		*/
-		goto out;
+    if (ret == -1 && errno == ENOENT && !loc.parent)
+        /* The parent directory or an ancestor even
+           higher does not exist
+        */
+        goto out;
 
-	/* ret == -1 && errno == ENOENT */
-	loc.inode = inode_new (loc.parent->table);
-	if (!loc.inode) {
-		ret = -1;
-		errno = ENOMEM;
-		goto out;
-	}
+    /* ret == -1 && errno == ENOENT */
+    loc.inode = inode_new(loc.parent->table);
+    if (!loc.inode) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
 
-	ret = syncop_mkdir (subvol, &loc, mode, xattr_req, &iatt);
-        DECODE_SYNCOP_ERR (ret);
+    ret = syncop_mknod(subvol, &loc, mode, dev, &iatt, xattr_req, NULL);
+    DECODE_SYNCOP_ERR(ret);
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	if (ret == 0)
-		ret = glfs_loc_link (&loc, &iatt);
+    if (ret == 0)
+        ret = glfs_loc_link(&loc, &iatt);
 out:
-	loc_wipe (&loc);
-
-	if (xattr_req)
-		dict_unref (xattr_req);
+    loc_wipe(&loc);
 
-	priv_glfs_subvol_done (fs, subvol);
+    if (xattr_req)
+        dict_unref(xattr_req);
 
-	return ret;
-}
+    glfs_subvol_done(fs, subvol);
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_mkdir, 3.4.0);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_mkdir, 3.4.0)
 int
-pub_glfs_unlink (struct glfs *fs, const char *path)
+pub_glfs_mkdir(struct glfs *fs, const char *path, mode_t mode)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	loc_t            loc = {0, };
-	struct iatt      iatt = {0, };
-	int              reval = 0;
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    uuid_t gfid;
+    dict_t *xattr_req = NULL;
+    int reval = 0;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    xattr_req = dict_new();
+    if (!xattr_req) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    gf_uuid_generate(gfid);
+    ret = dict_set_gfuuid(xattr_req, "gfid-req", gfid, true);
+    if (ret) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+retry:
+    ret = glfs_lresolve(fs, subvol, path, &loc, &iatt, reval);
 
-	__glfs_entry_fs (fs);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
-retry:
-	ret = glfs_lresolve (fs, subvol, path, &loc, &iatt, reval);
+    if (loc.inode) {
+        errno = EEXIST;
+        ret = -1;
+        goto out;
+    }
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    if (ret == -1 && errno != ENOENT)
+        /* Any other type of error is fatal */
+        goto out;
 
-	if (ret)
-		goto out;
+    if (ret == -1 && errno == ENOENT && !loc.parent)
+        /* The parent directory or an ancestor even
+           higher does not exist
+        */
+        goto out;
 
-	if (iatt.ia_type == IA_IFDIR) {
-		ret = -1;
-		errno = EISDIR;
-		goto out;
-	}
+    /* ret == -1 && errno == ENOENT */
+    loc.inode = inode_new(loc.parent->table);
+    if (!loc.inode) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
 
-	ret = syncop_unlink (subvol, &loc);
-        DECODE_SYNCOP_ERR (ret);
+    ret = syncop_mkdir(subvol, &loc, mode, &iatt, xattr_req, NULL);
+    DECODE_SYNCOP_ERR(ret);
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	if (ret == 0)
-		ret = glfs_loc_unlink (&loc);
+    if (ret == 0)
+        ret = glfs_loc_link(&loc, &iatt);
 out:
-	loc_wipe (&loc);
+    loc_wipe(&loc);
 
-	priv_glfs_subvol_done (fs, subvol);
+    if (xattr_req)
+        dict_unref(xattr_req);
 
-	return ret;
-}
+    glfs_subvol_done(fs, subvol);
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_unlink, 3.4.0);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_unlink, 3.4.0)
 int
-pub_glfs_rmdir (struct glfs *fs, const char *path)
+pub_glfs_unlink(struct glfs *fs, const char *path)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	loc_t            loc = {0, };
-	struct iatt      iatt = {0, };
-	int              reval = 0;
-
-	__glfs_entry_fs (fs);
-
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    int reval = 0;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
 retry:
-	ret = glfs_lresolve (fs, subvol, path, &loc, &iatt, reval);
+    ret = glfs_lresolve(fs, subvol, path, &loc, &iatt, reval);
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	if (ret)
-		goto out;
+    if (ret)
+        goto out;
 
-	if (iatt.ia_type != IA_IFDIR) {
-		ret = -1;
-		errno = ENOTDIR;
-		goto out;
-	}
+    if (iatt.ia_type == IA_IFDIR) {
+        ret = -1;
+        errno = EISDIR;
+        goto out;
+    }
 
-	ret = syncop_rmdir (subvol, &loc, 0);
-        DECODE_SYNCOP_ERR (ret);
+    /* TODO: Add leaseid */
+    ret = syncop_unlink(subvol, &loc, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	if (ret == 0)
-		ret = glfs_loc_unlink (&loc);
+    if (ret == 0)
+        ret = glfs_loc_unlink(&loc);
 out:
-	loc_wipe (&loc);
+    loc_wipe(&loc);
 
-	priv_glfs_subvol_done (fs, subvol);
+    glfs_subvol_done(fs, subvol);
 
-	return ret;
-}
-
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_rmdir, 3.4.0);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_rmdir, 3.4.0)
 int
-pub_glfs_rename (struct glfs *fs, const char *oldpath, const char *newpath)
+pub_glfs_rmdir(struct glfs *fs, const char *path)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	loc_t            oldloc = {0, };
-	loc_t            newloc = {0, };
-	struct iatt      oldiatt = {0, };
-	struct iatt      newiatt = {0, };
-	int              reval = 0;
-
-	__glfs_entry_fs (fs);
-
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    int reval = 0;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
 retry:
-	ret = glfs_lresolve (fs, subvol, oldpath, &oldloc, &oldiatt, reval);
+    ret = glfs_lresolve(fs, subvol, path, &loc, &iatt, reval);
 
-	ESTALE_RETRY (ret, errno, reval, &oldloc, retry);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	if (ret)
-		goto out;
-retrynew:
-	ret = glfs_lresolve (fs, subvol, newpath, &newloc, &newiatt, reval);
-
-	ESTALE_RETRY (ret, errno, reval, &newloc, retrynew);
-
-	if (ret && errno != ENOENT && newloc.parent)
-		goto out;
-
-	if (newiatt.ia_type != IA_INVAL) {
-                if ((oldiatt.ia_type == IA_IFDIR) !=
-                    (newiatt.ia_type == IA_IFDIR)) {
-                       /* Either both old and new must be dirs,
-                        * or both must be non-dirs. Else, fail.
-                        */
-                       ret = -1;
-                       errno = EISDIR;
-                       goto out;
-                }
-        }
+    if (ret)
+        goto out;
 
-	/* TODO: check if new or old is a prefix of the other, and fail EINVAL */
+    if (iatt.ia_type != IA_IFDIR) {
+        ret = -1;
+        errno = ENOTDIR;
+        goto out;
+    }
 
-	ret = syncop_rename (subvol, &oldloc, &newloc);
-        DECODE_SYNCOP_ERR (ret);
+    ret = syncop_rmdir(subvol, &loc, 0, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
 
-	if (ret == -1 && errno == ESTALE) {
-		if (reval < DEFAULT_REVAL_COUNT) {
-			reval++;
-			loc_wipe (&oldloc);
-			loc_wipe (&newloc);
-			goto retry;
-		}
-	}
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	if (ret == 0)
-		inode_rename (oldloc.parent->table, oldloc.parent, oldloc.name,
-			      newloc.parent, newloc.name, oldloc.inode,
-			      &oldiatt);
+    if (ret == 0)
+        ret = glfs_loc_unlink(&loc);
 out:
-	loc_wipe (&oldloc);
-	loc_wipe (&newloc);
-
-	priv_glfs_subvol_done (fs, subvol);
+    loc_wipe(&loc);
 
-	return ret;
-}
+    glfs_subvol_done(fs, subvol);
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_rename, 3.4.0);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_rename, 3.4.0)
 int
-pub_glfs_link (struct glfs *fs, const char *oldpath, const char *newpath)
+pub_glfs_rename(struct glfs *fs, const char *oldpath, const char *newpath)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	loc_t            oldloc = {0, };
-	loc_t            newloc = {0, };
-	struct iatt      oldiatt = {0, };
-	struct iatt      newiatt = {0, };
-	int              reval = 0;
-
-	__glfs_entry_fs (fs);
-
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    loc_t oldloc = {
+        0,
+    };
+    loc_t newloc = {
+        0,
+    };
+    struct iatt oldiatt = {
+        0,
+    };
+    struct iatt newiatt = {
+        0,
+    };
+    int reval = 0;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
 retry:
-	ret = glfs_lresolve (fs, subvol, oldpath, &oldloc, &oldiatt, reval);
+    ret = glfs_lresolve(fs, subvol, oldpath, &oldloc, &oldiatt, reval);
 
-	ESTALE_RETRY (ret, errno, reval, &oldloc, retry);
+    ESTALE_RETRY(ret, errno, reval, &oldloc, retry);
 
-	if (ret)
-		goto out;
+    if (ret)
+        goto out;
 retrynew:
-	ret = glfs_lresolve (fs, subvol, newpath, &newloc, &newiatt, reval);
+    ret = glfs_lresolve(fs, subvol, newpath, &newloc, &newiatt, reval);
 
-	ESTALE_RETRY (ret, errno, reval, &newloc, retrynew);
+    ESTALE_RETRY(ret, errno, reval, &newloc, retrynew);
 
-	if (ret == 0) {
-		ret = -1;
-		errno = EEXIST;
-		goto out;
-	}
+    if (ret && errno != ENOENT && newloc.parent)
+        goto out;
 
-	if (oldiatt.ia_type == IA_IFDIR) {
-		ret = -1;
-		errno = EISDIR;
-		goto out;
-	}
+    if (newiatt.ia_type != IA_INVAL) {
+        if ((oldiatt.ia_type == IA_IFDIR) != (newiatt.ia_type == IA_IFDIR)) {
+            /* Either both old and new must be dirs,
+             * or both must be non-dirs. Else, fail.
+             */
+            ret = -1;
+            errno = EISDIR;
+            goto out;
+        }
+    }
 
-        /* Filling the inode of the hard link to be same as that of the
-           original file
-        */
-	if (newloc.inode) {
-		inode_unref (newloc.inode);
-		newloc.inode = NULL;
-	}
-        newloc.inode = inode_ref (oldloc.inode);
+    /* TODO: - check if new or old is a prefix of the other, and fail EINVAL
+     *       - Add leaseid */
 
-	ret = syncop_link (subvol, &oldloc, &newloc);
-        DECODE_SYNCOP_ERR (ret);
+    ret = syncop_rename(subvol, &oldloc, &newloc, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
 
-	if (ret == -1 && errno == ESTALE) {
-		loc_wipe (&oldloc);
-		loc_wipe (&newloc);
-		if (reval--)
-			goto retry;
-	}
+    if (ret == -1 && errno == ESTALE) {
+        if (reval < DEFAULT_REVAL_COUNT) {
+            reval++;
+            loc_wipe(&oldloc);
+            loc_wipe(&newloc);
+            goto retry;
+        }
+    }
+
+    if (ret == 0) {
+        inode_rename(oldloc.parent->table, oldloc.parent, oldloc.name,
+                     newloc.parent, newloc.name, oldloc.inode, &oldiatt);
 
-	if (ret == 0)
-		ret = glfs_loc_link (&newloc, &oldiatt);
+        if (newloc.inode && !inode_has_dentry(newloc.inode))
+            inode_forget(newloc.inode, 0);
+    }
 out:
-	loc_wipe (&oldloc);
-	loc_wipe (&newloc);
+    loc_wipe(&oldloc);
+    loc_wipe(&newloc);
+
+    glfs_subvol_done(fs, subvol);
 
-	priv_glfs_subvol_done (fs, subvol);
+    __GLFS_EXIT_FS;
 
-	return ret;
+invalid_fs:
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_link, 3.4.0);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_link, 3.4.0)
+int
+pub_glfs_link(struct glfs *fs, const char *oldpath, const char *newpath)
+{
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    loc_t oldloc = {
+        0,
+    };
+    loc_t newloc = {
+        0,
+    };
+    struct iatt oldiatt = {
+        0,
+    };
+    struct iatt newiatt = {
+        0,
+    };
+    int reval = 0;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+retry:
+    ret = glfs_lresolve(fs, subvol, oldpath, &oldloc, &oldiatt, reval);
 
+    ESTALE_RETRY(ret, errno, reval, &oldloc, retry);
 
-struct glfs_fd *
-pub_glfs_opendir (struct glfs *fs, const char *path)
-{
-	int              ret = -1;
-	struct glfs_fd  *glfd = NULL;
-	xlator_t        *subvol = NULL;
-	loc_t            loc = {0, };
-	struct iatt      iatt = {0, };
-	int              reval = 0;
+    if (ret)
+        goto out;
+retrynew:
+    ret = glfs_lresolve(fs, subvol, newpath, &newloc, &newiatt, reval);
+
+    ESTALE_RETRY(ret, errno, reval, &newloc, retrynew);
+
+    if (ret == 0) {
+        ret = -1;
+        errno = EEXIST;
+        goto out;
+    }
+
+    if (oldiatt.ia_type == IA_IFDIR) {
+        ret = -1;
+        errno = EISDIR;
+        goto out;
+    }
+
+    /* Filling the inode of the hard link to be same as that of the
+       original file
+    */
+    if (newloc.inode) {
+        inode_unref(newloc.inode);
+        newloc.inode = NULL;
+    }
+    newloc.inode = inode_ref(oldloc.inode);
+
+    ret = syncop_link(subvol, &oldloc, &newloc, &newiatt, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    if (ret == -1 && errno == ESTALE) {
+        loc_wipe(&oldloc);
+        loc_wipe(&newloc);
+        if (reval--)
+            goto retry;
+    }
+
+    if (ret == 0)
+        ret = glfs_loc_link(&newloc, &newiatt);
+out:
+    loc_wipe(&oldloc);
+    loc_wipe(&newloc);
 
-	__glfs_entry_fs (fs);
+    glfs_subvol_done(fs, subvol);
 
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    __GLFS_EXIT_FS;
 
-	glfd = glfs_fd_new (fs);
-	if (!glfd)
-		goto out;
+invalid_fs:
+    return ret;
+}
 
-	INIT_LIST_HEAD (&glfd->entries);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_opendir, 3.4.0)
+struct glfs_fd *
+pub_glfs_opendir(struct glfs *fs, const char *path)
+{
+    int ret = -1;
+    struct glfs_fd *glfd = NULL;
+    xlator_t *subvol = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    int reval = 0;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    glfd = glfs_fd_new(fs);
+    if (!glfd)
+        goto out;
+
+    INIT_LIST_HEAD(&glfd->entries);
 retry:
-	ret = glfs_resolve (fs, subvol, path, &loc, &iatt, reval);
+    ret = glfs_resolve(fs, subvol, path, &loc, &iatt, reval);
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	if (ret)
-		goto out;
+    if (ret)
+        goto out;
 
-	if (!IA_ISDIR (iatt.ia_type)) {
-		ret = -1;
-		errno = ENOTDIR;
-		goto out;
-	}
+    if (!IA_ISDIR(iatt.ia_type)) {
+        ret = -1;
+        errno = ENOTDIR;
+        goto out;
+    }
 
-	if (glfd->fd) {
-		/* Retry. Safe to touch glfd->fd as we
-		   still have not glfs_fd_bind() yet.
-		*/
-		fd_unref (glfd->fd);
-		glfd->fd = NULL;
-	}
+    if (glfd->fd) {
+        /* Retry. Safe to touch glfd->fd as we
+           still have not glfs_fd_bind() yet.
+        */
+        fd_unref(glfd->fd);
+        glfd->fd = NULL;
+    }
 
-	glfd->fd = fd_create (loc.inode, getpid());
-	if (!glfd->fd) {
-		ret = -1;
-		errno = ENOMEM;
-		goto out;
-	}
+    glfd->fd = fd_create(loc.inode, getpid());
+    if (!glfd->fd) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
 
-	ret = syncop_opendir (subvol, &loc, glfd->fd);
-        DECODE_SYNCOP_ERR (ret);
+    ret = syncop_opendir(subvol, &loc, glfd->fd, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 out:
-	loc_wipe (&loc);
+    loc_wipe(&loc);
 
-	if (ret && glfd) {
-		glfs_fd_destroy (glfd);
-		glfd = NULL;
-	} else if (glfd) {
-		fd_bind (glfd->fd);
-		glfs_fd_bind (glfd);
-	}
+    if (ret && glfd) {
+        GF_REF_PUT(glfd);
+        glfd = NULL;
+    } else if (glfd) {
+        glfd_set_state_bind(glfd);
+    }
 
-	priv_glfs_subvol_done (fs, subvol);
+    glfs_subvol_done(fs, subvol);
 
-	return glfd;
-}
-
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_opendir, 3.4.0);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return glfd;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_closedir, 3.4.0)
 int
-pub_glfs_closedir (struct glfs_fd *glfd)
+pub_glfs_closedir(struct glfs_fd *glfd)
 {
-	__glfs_entry_fd (glfd);
+    int ret = -1;
 
-	gf_dirent_free (list_entry (&glfd->entries, gf_dirent_t, list));
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
 
-	glfs_fd_destroy (glfd);
+    gf_dirent_free(list_entry(&glfd->entries, gf_dirent_t, list));
 
-	return 0;
-}
+    glfs_mark_glfd_for_deletion(glfd);
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_closedir, 3.4.0);
+    __GLFS_EXIT_FS;
 
+    ret = 0;
 
-long
-pub_glfs_telldir (struct glfs_fd *fd)
-{
-	return fd->offset;
+invalid_fs:
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_telldir, 3.4.0);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_telldir, 3.4.0)
+long
+pub_glfs_telldir(struct glfs_fd *fd)
+{
+    if (fd == NULL) {
+        errno = EBADF;
+        return -1;
+    }
 
+    return fd->offset;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_seekdir, 3.4.0)
 void
-pub_glfs_seekdir (struct glfs_fd *fd, long offset)
+pub_glfs_seekdir(struct glfs_fd *fd, long offset)
 {
-	gf_dirent_t *entry = NULL;
-	gf_dirent_t *tmp = NULL;
+    gf_dirent_t *entry = NULL;
+    gf_dirent_t *tmp = NULL;
 
-	if (fd->offset == offset)
-		return;
+    if (fd == NULL) {
+        errno = EBADF;
+        return;
+    }
 
-	fd->offset = offset;
-	fd->next = NULL;
+    if (fd->offset == offset)
+        return;
 
-	list_for_each_entry_safe (entry, tmp, &fd->entries, list) {
-		if (entry->d_off != offset)
-			continue;
+    fd->offset = offset;
+    fd->next = NULL;
 
-		if (&tmp->list != &fd->entries) {
-			/* found! */
-			fd->next = tmp;
-			return;
-		}
-	}
-	/* could not find entry at requested offset in the cache.
-	   next readdir_r() will result in glfd_entry_refresh()
-	*/
-}
-
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_seekdir, 3.4.0);
+    list_for_each_entry_safe(entry, tmp, &fd->entries, list)
+    {
+        if (entry->d_off != offset)
+            continue;
 
+        if (&tmp->list != &fd->entries) {
+            /* found! */
+            fd->next = tmp;
+            return;
+        }
+    }
+    /* could not find entry at requested offset in the cache.
+       next readdir_r() will result in glfd_entry_refresh()
+    */
+}
 
-int
-pub_glfs_discard_async (struct glfs_fd *glfd, off_t offset, size_t len,
-                        glfs_io_cbk fn, void *data)
+static int
+glfs_discard_async_cbk(call_frame_t *frame, void *cookie, xlator_t *this,
+                       int32_t op_ret, int32_t op_errno,
+                       struct iatt *preop_stbuf, struct iatt *postop_stbuf,
+                       dict_t *xdata)
 {
-	struct glfs_io *gio = NULL;
-	int             ret = 0;
-
-	gio = GF_CALLOC (1, sizeof (*gio), glfs_mt_glfs_io_t);
-	if (!gio) {
-		errno = ENOMEM;
-		return -1;
-	}
-
-	gio->op     = GF_FOP_DISCARD;
-	gio->glfd   = glfd;
-	gio->offset = offset;
-	gio->count  = len;
-	gio->fn     = fn;
-	gio->data   = data;
+    glfs_io_async_cbk(op_ret, op_errno, frame, cookie, NULL, 0, preop_stbuf,
+                      postop_stbuf);
 
-	ret = synctask_new (pub_glfs_from_glfd (glfd)->ctx->env,
-			    glfs_io_async_task, glfs_io_async_cbk,
-			    NULL, gio);
-
-	if (ret) {
-		GF_FREE (gio->iov);
-		GF_FREE (gio);
-	}
-
-	return ret;
+    return 0;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_discard_async, 3.5.0);
-
+static int
+glfs_discard_async_common(struct glfs_fd *glfd, off_t offset, size_t len,
+                          gf_boolean_t oldcb, glfs_io_cbk fn, void *data)
+{
+    struct glfs_io *gio = NULL;
+    int ret = -1;
+    call_frame_t *frame = NULL;
+    xlator_t *subvol = NULL;
+    fd_t *fd = NULL;
+    dict_t *fop_attr = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    /* Need to take explicit ref so that the fd
+     * is not destroyed before the fop is complete
+     */
+    GF_REF_GET(glfd);
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        errno = EBADFD;
+        goto out;
+    }
+
+    frame = syncop_create_frame(THIS);
+    if (!frame) {
+        errno = ENOMEM;
+        goto out;
+    }
+
+    gio = GF_CALLOC(1, sizeof(*gio), glfs_mt_glfs_io_t);
+    if (!gio) {
+        errno = ENOMEM;
+        goto out;
+    }
+
+    gio->op = GF_FOP_DISCARD;
+    gio->glfd = glfd;
+    gio->offset = offset;
+    gio->count = len;
+    gio->oldcb = oldcb;
+    gio->fn = fn;
+    gio->data = data;
+
+    frame->local = gio;
+    ret = get_fop_attr_thrd_key(&fop_attr);
+    if (ret)
+        gf_msg_debug("gfapi", 0, "Getting leaseid from thread failed");
+
+    STACK_WIND_COOKIE(frame, glfs_discard_async_cbk, subvol, subvol,
+                      subvol->fops->discard, fd, offset, len, fop_attr);
+
+    ret = 0;
+out:
+    if (ret) {
+        if (fd)
+            fd_unref(fd);
+        if (glfd)
+            GF_REF_PUT(glfd);
+        GF_FREE(gio);
+        if (frame)
+            STACK_DESTROY(frame->root);
+        glfs_subvol_done(glfd->fs, subvol);
+    }
+
+    __GLFS_EXIT_FS;
+
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC(glfs_discard_async35, glfs_discard_async, 3.5.0)
 int
-pub_glfs_zerofill_async (struct glfs_fd *glfd, off_t offset, off_t len,
-                         glfs_io_cbk fn, void *data)
+pub_glfs_discard_async35(struct glfs_fd *glfd, off_t offset, size_t len,
+                         glfs_io_cbk34 fn, void *data)
 {
-        struct glfs_io *gio  = NULL;
-        int             ret  = 0;
-
-        gio = GF_CALLOC (1, sizeof (*gio), glfs_mt_glfs_io_t);
-        if (!gio) {
-                errno = ENOMEM;
-                return -1;
-        }
+    return glfs_discard_async_common(glfd, offset, len, _gf_true, (void *)fn,
+                                     data);
+}
 
-        gio->op     = GF_FOP_ZEROFILL;
-        gio->glfd   = glfd;
-        gio->offset = offset;
-        gio->count  = len;
-        gio->fn     = fn;
-        gio->data   = data;
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_discard_async, 6.0)
+int
+pub_glfs_discard_async(struct glfs_fd *glfd, off_t offset, size_t len,
+                       glfs_io_cbk fn, void *data)
+{
+    return glfs_discard_async_common(glfd, offset, len, _gf_false, fn, data);
+}
 
-        ret = synctask_new (pub_glfs_from_glfd (glfd)->ctx->env,
-                            glfs_io_async_task, glfs_io_async_cbk,
-                            NULL, gio);
+static int
+glfs_zerofill_async_cbk(call_frame_t *frame, void *cookie, xlator_t *this,
+                        int32_t op_ret, int32_t op_errno,
+                        struct iatt *preop_stbuf, struct iatt *postop_stbuf,
+                        dict_t *xdata)
+{
+    glfs_io_async_cbk(op_ret, op_errno, frame, cookie, NULL, 0, preop_stbuf,
+                      postop_stbuf);
 
-        if (ret) {
-                GF_FREE (gio->iov);
-                GF_FREE (gio);
-        }
+    return 0;
+}
 
-        return ret;
+static int
+glfs_zerofill_async_common(struct glfs_fd *glfd, off_t offset, off_t len,
+                           gf_boolean_t oldcb, glfs_io_cbk fn, void *data)
+{
+    struct glfs_io *gio = NULL;
+    int ret = -1;
+    call_frame_t *frame = NULL;
+    xlator_t *subvol = NULL;
+    fd_t *fd = NULL;
+    dict_t *fop_attr = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    /* Need to take explicit ref so that the fd
+     * is not destroyed before the fop is complete
+     */
+    GF_REF_GET(glfd);
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        errno = EBADFD;
+        goto out;
+    }
+
+    frame = syncop_create_frame(THIS);
+    if (!frame) {
+        errno = ENOMEM;
+        goto out;
+    }
+
+    gio = GF_CALLOC(1, sizeof(*gio), glfs_mt_glfs_io_t);
+    if (!gio) {
+        errno = ENOMEM;
+        goto out;
+    }
+
+    gio->op = GF_FOP_ZEROFILL;
+    gio->glfd = glfd;
+    gio->offset = offset;
+    gio->count = len;
+    gio->oldcb = oldcb;
+    gio->fn = fn;
+    gio->data = data;
+
+    frame->local = gio;
+
+    ret = get_fop_attr_thrd_key(&fop_attr);
+    if (ret)
+        gf_msg_debug("gfapi", 0, "Getting leaseid from thread failed");
+
+    STACK_WIND_COOKIE(frame, glfs_zerofill_async_cbk, subvol, subvol,
+                      subvol->fops->zerofill, fd, offset, len, fop_attr);
+    ret = 0;
+out:
+    if (ret) {
+        if (fd)
+            fd_unref(fd);
+        if (glfd)
+            GF_REF_PUT(glfd);
+        GF_FREE(gio);
+        if (frame)
+            STACK_DESTROY(frame->root);
+        glfs_subvol_done(glfd->fs, subvol);
+    }
+    if (fop_attr)
+        dict_unref(fop_attr);
+
+    __GLFS_EXIT_FS;
+
+invalid_fs:
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_zerofill_async, 3.5.0);
+GFAPI_SYMVER_PUBLIC(glfs_zerofill_async35, glfs_zerofill_async, 3.5.0)
+int
+pub_glfs_zerofill_async35(struct glfs_fd *glfd, off_t offset, off_t len,
+                          glfs_io_cbk34 fn, void *data)
+{
+    return glfs_zerofill_async_common(glfd, offset, len, _gf_true, (void *)fn,
+                                      data);
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_zerofill_async, 6.0)
+int
+pub_glfs_zerofill_async(struct glfs_fd *glfd, off_t offset, off_t len,
+                        glfs_io_cbk fn, void *data)
+{
+    return glfs_zerofill_async_common(glfd, offset, len, _gf_false, fn, data);
+}
 
 void
-gf_dirent_to_dirent (gf_dirent_t *gf_dirent, struct dirent *dirent)
+gf_dirent_to_dirent(gf_dirent_t *gf_dirent, struct dirent *dirent)
 {
-	dirent->d_ino = gf_dirent->d_ino;
+    dirent->d_ino = gf_dirent->d_ino;
 
 #ifdef _DIRENT_HAVE_D_OFF
-	dirent->d_off = gf_dirent->d_off;
+    dirent->d_off = gf_dirent->d_off;
 #endif
 
 #ifdef _DIRENT_HAVE_D_TYPE
-	dirent->d_type = gf_dirent->d_type;
+    dirent->d_type = gf_dirent->d_type;
 #endif
 
 #ifdef _DIRENT_HAVE_D_NAMLEN
-	dirent->d_namlen = strlen (gf_dirent->d_name);
+    dirent->d_namlen = strlen(gf_dirent->d_name);
 #endif
 
-	strncpy (dirent->d_name, gf_dirent->d_name, GF_NAME_MAX + 1);
+    snprintf(dirent->d_name, NAME_MAX + 1, "%s", gf_dirent->d_name);
 }
 
-
 int
-glfd_entry_refresh (struct glfs_fd *glfd, int plus)
+glfd_entry_refresh(struct glfs_fd *glfd, int plus)
 {
-	xlator_t        *subvol = NULL;
-	gf_dirent_t      entries;
-	gf_dirent_t      old;
-	int              ret = -1;
-	fd_t            *fd = NULL;
-
-	subvol = priv_glfs_active_subvol (glfd->fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
-
-	fd = glfs_resolve_fd (glfd->fs, subvol, glfd);
-	if (!fd) {
-		ret = -1;
-		errno = EBADFD;
-		goto out;
-	}
-
-	if (fd->inode->ia_type != IA_IFDIR) {
-		ret = -1;
-		errno = EBADF;
-		goto out;
-	}
-
-	INIT_LIST_HEAD (&entries.list);
-	INIT_LIST_HEAD (&old.list);
+    xlator_t *subvol = NULL;
+    gf_dirent_t entries;
+    gf_dirent_t old;
+    gf_dirent_t *entry = NULL;
+    int ret = -1;
+    fd_t *fd = NULL;
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        ret = -1;
+        errno = EBADFD;
+        goto out;
+    }
+
+    if (fd->inode->ia_type != IA_IFDIR) {
+        ret = -1;
+        errno = EBADF;
+        goto out;
+    }
+
+    INIT_LIST_HEAD(&entries.list);
+    INIT_LIST_HEAD(&old.list);
+
+    if (plus)
+        ret = syncop_readdirp(subvol, fd, 131072, glfd->offset, &entries, NULL,
+                              NULL);
+    else
+        ret = syncop_readdir(subvol, fd, 131072, glfd->offset, &entries, NULL,
+                             NULL);
+    DECODE_SYNCOP_ERR(ret);
+    if (ret >= 0) {
+        if (plus) {
+            list_for_each_entry(entry, &entries.list, list)
+            {
+                if ((!entry->inode && (!IA_ISDIR(entry->d_stat.ia_type))) ||
+                    ((entry->d_stat.ia_ctime == 0) &&
+                     strcmp(entry->d_name, ".") &&
+                     strcmp(entry->d_name, ".."))) {
+                    /* entry->inode for directories will be
+                     * always set to null to force a lookup
+                     * on the dentry. Hence to not degrade
+                     * readdir performance, we skip lookups
+                     * for directory entries. Also we will have
+                     * proper stat if directory present on
+                     * hashed subvolume.
+                     *
+                     * In addition, if the stat is invalid, force
+                     * lookup to fetch proper stat.
+                     */
+                    gf_fill_iatt_for_dirent(entry, fd->inode, subvol);
+                }
+            }
 
-	if (plus)
-		ret = syncop_readdirp (subvol, fd, 131072, glfd->offset,
-				       NULL, &entries);
-	else
-		ret = syncop_readdir (subvol, fd, 131072, glfd->offset,
-				      &entries);
-        DECODE_SYNCOP_ERR (ret);
-	if (ret >= 0) {
-		if (plus)
-			gf_link_inodes_from_dirent (THIS, fd->inode, &entries);
+            gf_link_inodes_from_dirent(THIS, fd->inode, &entries);
+        }
 
-		list_splice_init (&glfd->entries, &old.list);
-		list_splice_init (&entries.list, &glfd->entries);
+        list_splice_init(&glfd->entries, &old.list);
+        list_splice_init(&entries.list, &glfd->entries);
 
-		/* spurious errno is dangerous for glfd_entry_next() */
-		errno = 0;
-	}
+        /* spurious errno is dangerous for glfd_entry_next() */
+        errno = 0;
+    }
 
-	if (ret > 0)
-		glfd->next = list_entry (glfd->entries.next, gf_dirent_t, list);
+    if (ret > 0)
+        glfd->next = list_entry(glfd->entries.next, gf_dirent_t, list);
 
-	gf_dirent_free (&old);
+    gf_dirent_free(&old);
 out:
-	if (fd)
-		fd_unref (fd);
+    if (fd)
+        fd_unref(fd);
 
-	priv_glfs_subvol_done (glfd->fs, subvol);
+    glfs_subvol_done(glfd->fs, subvol);
 
-	return ret;
+    return ret;
 }
 
-
 gf_dirent_t *
-glfd_entry_next (struct glfs_fd *glfd, int plus)
+glfd_entry_next(struct glfs_fd *glfd, int plus)
 {
-	gf_dirent_t     *entry = NULL;
-	int              ret = -1;
+    gf_dirent_t *entry = NULL;
+    int ret = -1;
 
-	if (!glfd->offset || !glfd->next) {
-		ret = glfd_entry_refresh (glfd, plus);
-		if (ret < 0)
-			return NULL;
-	}
+    if (!glfd->offset || !glfd->next) {
+        ret = glfd_entry_refresh(glfd, plus);
+        if (ret < 0)
+            return NULL;
+    }
 
-	entry = glfd->next;
-	if (!entry)
-		return NULL;
+    entry = glfd->next;
+    if (!entry)
+        return NULL;
 
-	if (&entry->next->list == &glfd->entries)
-		glfd->next = NULL;
-	else
-		glfd->next = entry->next;
+    if (&entry->next->list == &glfd->entries)
+        glfd->next = NULL;
+    else
+        glfd->next = entry->next;
 
-	glfd->offset = entry->d_off;
+    glfd->offset = entry->d_off;
 
-	return entry;
+    return entry;
 }
 
-
-static struct dirent *
-glfs_readdirbuf_get (struct glfs_fd *glfd)
+struct dirent *
+glfs_readdirbuf_get(struct glfs_fd *glfd)
 {
-        struct dirent *buf = NULL;
-
-        LOCK (&glfd->fd->lock);
-        {
-                buf = glfd->readdirbuf;
-                if (buf) {
-                        memset (buf, 0, READDIRBUF_SIZE);
-                        goto unlock;
-                }
-
-                buf = GF_CALLOC (1, READDIRBUF_SIZE, glfs_mt_readdirbuf_t);
-                if (!buf) {
-                        errno = ENOMEM;
-                        goto unlock;
-                }
+    struct dirent *buf = NULL;
+
+    LOCK(&glfd->fd->lock);
+    {
+        buf = glfd->readdirbuf;
+        if (buf) {
+            memset(buf, 0, READDIRBUF_SIZE);
+            goto unlock;
+        }
 
-                glfd->readdirbuf = buf;
+        buf = GF_CALLOC(1, READDIRBUF_SIZE, glfs_mt_readdirbuf_t);
+        if (!buf) {
+            errno = ENOMEM;
+            goto unlock;
         }
+
+        glfd->readdirbuf = buf;
+    }
 unlock:
-        UNLOCK (&glfd->fd->lock);
+    UNLOCK(&glfd->fd->lock);
 
-        return buf;
+    return buf;
 }
 
-
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_readdirplus_r, 3.4.0)
 int
-pub_glfs_readdirplus_r (struct glfs_fd *glfd, struct stat *stat,
-                        struct dirent *ext, struct dirent **res)
+pub_glfs_readdirplus_r(struct glfs_fd *glfd, struct stat *stat,
+                       struct dirent *ext, struct dirent **res)
 {
-	int              ret = 0;
-	gf_dirent_t     *entry = NULL;
-	struct dirent   *buf = NULL;
+    int ret = 0;
+    gf_dirent_t *entry = NULL;
+    struct dirent *buf = NULL;
 
-	__glfs_entry_fd (glfd);
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
 
-	errno = 0;
+    GF_REF_GET(glfd);
 
-	if (ext)
-		buf = ext;
-	else
-		buf = glfs_readdirbuf_get (glfd);
+    errno = 0;
 
-	if (!buf) {
-		errno = ENOMEM;
-		return -1;
-	}
+    if (ext)
+        buf = ext;
+    else
+        buf = glfs_readdirbuf_get(glfd);
 
-	entry = glfd_entry_next (glfd, !!stat);
-	if (errno)
-		ret = -1;
+    if (!buf) {
+        errno = ENOMEM;
+        ret = -1;
+        goto out;
+    }
 
-	if (res) {
-		if (entry)
-			*res = buf;
-		else
-			*res = NULL;
-	}
+    entry = glfd_entry_next(glfd, !!stat);
+    if (errno)
+        ret = -1;
 
-	if (entry) {
-		gf_dirent_to_dirent (entry, buf);
-		if (stat)
-			glfs_iatt_to_stat (glfd->fs, &entry->d_stat, stat);
-	}
+    if (res) {
+        if (entry)
+            *res = buf;
+        else
+            *res = NULL;
+    }
 
-	return ret;
-}
+    if (entry) {
+        gf_dirent_to_dirent(entry, buf);
+        if (stat)
+            glfs_iatt_to_stat(glfd->fs, &entry->d_stat, stat);
+    }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_readdirplus_r, 3.4.0);
+out:
+    if (glfd)
+        GF_REF_PUT(glfd);
+
+    __GLFS_EXIT_FS;
+
+    return ret;
 
+invalid_fs:
+    return -1;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_readdir_r, 3.4.0)
 int
-pub_glfs_readdir_r (struct glfs_fd *glfd, struct dirent *buf,
-                    struct dirent **res)
+pub_glfs_readdir_r(struct glfs_fd *glfd, struct dirent *buf,
+                   struct dirent **res)
 {
-	return pub_glfs_readdirplus_r (glfd, 0, buf, res);
+    return pub_glfs_readdirplus_r(glfd, 0, buf, res);
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_readdir_r, 3.4.0);
-
-
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_readdirplus, 3.5.0)
 struct dirent *
-pub_glfs_readdirplus (struct glfs_fd *glfd, struct stat *stat)
+pub_glfs_readdirplus(struct glfs_fd *glfd, struct stat *stat)
 {
-        struct dirent *res = NULL;
-        int ret = -1;
+    struct dirent *res = NULL;
+    int ret = -1;
 
-        ret = pub_glfs_readdirplus_r (glfd, stat, NULL, &res);
-        if (ret)
-                return NULL;
+    ret = pub_glfs_readdirplus_r(glfd, stat, NULL, &res);
+    if (ret)
+        return NULL;
 
-        return res;
+    return res;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_readdirplus, 3.5.0);
-
-
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_readdir, 3.5.0)
 struct dirent *
-pub_glfs_readdir (struct glfs_fd *glfd)
+pub_glfs_readdir(struct glfs_fd *glfd)
 {
-        return pub_glfs_readdirplus (glfd, NULL);
+    return pub_glfs_readdirplus(glfd, NULL);
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_readdir, 3.5.0);
-
-
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_statvfs, 3.4.0)
 int
-pub_glfs_statvfs (struct glfs *fs, const char *path, struct statvfs *buf)
+pub_glfs_statvfs(struct glfs *fs, const char *path, struct statvfs *buf)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	loc_t            loc = {0, };
-	struct iatt      iatt = {0, };
-	int              reval = 0;
-
-	__glfs_entry_fs (fs);
-
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    int reval = 0;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
 retry:
-	ret = glfs_resolve (fs, subvol, path, &loc, &iatt, reval);
+    ret = glfs_resolve(fs, subvol, path, &loc, &iatt, reval);
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	if (ret)
-		goto out;
+    if (ret)
+        goto out;
 
-	ret = syncop_statfs (subvol, &loc, NULL, buf, NULL);
-        DECODE_SYNCOP_ERR (ret);
+    ret = syncop_statfs(subvol, &loc, buf, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 out:
-	loc_wipe (&loc);
-
-	priv_glfs_subvol_done (fs, subvol);
+    loc_wipe(&loc);
 
-	return ret;
-}
+    glfs_subvol_done(fs, subvol);
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_statvfs, 3.4.0);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_setattr, 6.0)
 int
-glfs_setattr (struct glfs *fs, const char *path, struct iatt *iatt,
-	      int valid, int follow)
+pub_glfs_setattr(struct glfs *fs, const char *path, struct glfs_stat *stat,
+                 int follow)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	loc_t            loc = {0, };
-	struct iatt      riatt = {0, };
-	int              reval = 0;
-
-	__glfs_entry_fs (fs);
-
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    int ret = -1;
+    int glvalid;
+    xlator_t *subvol = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt riatt = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    int reval = 0;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    GF_VALIDATE_OR_GOTO("glfs_setattr", stat, out);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
 retry:
-	if (follow)
-		ret = glfs_resolve (fs, subvol, path, &loc, &riatt, reval);
-	else
-		ret = glfs_lresolve (fs, subvol, path, &loc, &riatt, reval);
+    if (follow)
+        ret = glfs_resolve(fs, subvol, path, &loc, &riatt, reval);
+    else
+        ret = glfs_lresolve(fs, subvol, path, &loc, &riatt, reval);
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	if (ret)
-		goto out;
+    if (ret)
+        goto out;
 
-	ret = syncop_setattr (subvol, &loc, iatt, valid, 0, 0);
-        DECODE_SYNCOP_ERR (ret);
+    glfs_iatt_from_statx(&iatt, stat);
+    glfsflags_from_gfapiflags(stat, &glvalid);
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    /* TODO : Add leaseid */
+    ret = syncop_setattr(subvol, &loc, &iatt, glvalid, 0, 0, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 out:
-	loc_wipe (&loc);
+    loc_wipe(&loc);
 
-	priv_glfs_subvol_done (fs, subvol);
+    glfs_subvol_done(fs, subvol);
 
-	return ret;
-}
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fsetattr, 6.0)
 int
-glfs_fsetattr (struct glfs_fd *glfd, struct iatt *iatt, int valid)
+pub_glfs_fsetattr(struct glfs_fd *glfd, struct glfs_stat *stat)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	fd_t            *fd = NULL;
-
-	__glfs_entry_fd (glfd);
-
-	subvol = priv_glfs_active_subvol (glfd->fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
-
-	fd = glfs_resolve_fd (glfd->fs, subvol, glfd);
-	if (!fd) {
-		ret = -1;
-		errno = EBADFD;
-		goto out;
-	}
-
-	ret = syncop_fsetattr (subvol, fd, iatt, valid, 0, 0);
-        DECODE_SYNCOP_ERR (ret);
+    int ret = -1;
+    int glvalid;
+    struct iatt iatt = {
+        0,
+    };
+    xlator_t *subvol = NULL;
+    fd_t *fd = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    GF_REF_GET(glfd);
+
+    GF_VALIDATE_OR_GOTO("glfs_fsetattr", stat, out);
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        ret = -1;
+        errno = EBADFD;
+        goto out;
+    }
+
+    glfs_iatt_from_statx(&iatt, stat);
+    glfsflags_from_gfapiflags(stat, &glvalid);
+
+    /* TODO : Add leaseid */
+    ret = syncop_fsetattr(subvol, fd, &iatt, glvalid, 0, 0, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
 out:
-	if (fd)
-		fd_unref (fd);
+    if (fd)
+        fd_unref(fd);
+    if (glfd)
+        GF_REF_PUT(glfd);
 
-	priv_glfs_subvol_done (glfd->fs, subvol);
+    glfs_subvol_done(glfd->fs, subvol);
 
-	return ret;
-}
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_chmod, 3.4.0)
 int
-pub_glfs_chmod (struct glfs *fs, const char *path, mode_t mode)
+pub_glfs_chmod(struct glfs *fs, const char *path, mode_t mode)
 {
-	int              ret = -1;
-	struct iatt      iatt = {0, };
-	int              valid = 0;
+    int ret = -1;
+    struct glfs_stat stat = {
+        0,
+    };
 
-	iatt.ia_prot = ia_prot_from_st_mode (mode);
-	valid = GF_SET_ATTR_MODE;
+    stat.glfs_st_mode = mode;
+    stat.glfs_st_mask = GLFS_STAT_MODE;
 
-	ret = glfs_setattr (fs, path, &iatt, valid, 1);
+    ret = glfs_setattr(fs, path, &stat, 1);
 
-	return ret;
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_chmod, 3.4.0);
-
-
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fchmod, 3.4.0)
 int
-pub_glfs_fchmod (struct glfs_fd *glfd, mode_t mode)
+pub_glfs_fchmod(struct glfs_fd *glfd, mode_t mode)
 {
-	int              ret = -1;
-	struct iatt      iatt = {0, };
-	int              valid = 0;
+    int ret = -1;
+    struct glfs_stat stat = {
+        0,
+    };
 
-	iatt.ia_prot = ia_prot_from_st_mode (mode);
-	valid = GF_SET_ATTR_MODE;
+    stat.glfs_st_mode = mode;
+    stat.glfs_st_mask = GLFS_STAT_MODE;
 
-	ret = glfs_fsetattr (glfd, &iatt, valid);
+    ret = glfs_fsetattr(glfd, &stat);
 
-	return ret;
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fchmod, 3.4.0);
-
-
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_chown, 3.4.0)
 int
-pub_glfs_chown (struct glfs *fs, const char *path, uid_t uid, gid_t gid)
+pub_glfs_chown(struct glfs *fs, const char *path, uid_t uid, gid_t gid)
 {
-	int              ret = 0;
-	int              valid = 0;
-	struct iatt      iatt = {0, };
+    int ret = 0;
+    struct glfs_stat stat = {
+        0,
+    };
 
-        if (uid != (uid_t) -1) {
-                iatt.ia_uid = uid;
-                valid = GF_SET_ATTR_UID;
-        }
+    if (uid != (uid_t)-1) {
+        stat.glfs_st_uid = uid;
+        stat.glfs_st_mask = GLFS_STAT_UID;
+    }
 
-        if (gid != (uid_t) -1) {
-                iatt.ia_gid = gid;
-                valid = valid | GF_SET_ATTR_GID;
-        }
+    if (gid != (uid_t)-1) {
+        stat.glfs_st_gid = gid;
+        stat.glfs_st_mask = stat.glfs_st_mask | GLFS_STAT_GID;
+    }
 
-        if (valid)
-	        ret = glfs_setattr (fs, path, &iatt, valid, 1);
+    if (stat.glfs_st_mask)
+        ret = glfs_setattr(fs, path, &stat, 1);
 
-	return ret;
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_chown, 3.4.0);
-
-
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_lchown, 3.4.0)
 int
-pub_glfs_lchown (struct glfs *fs, const char *path, uid_t uid, gid_t gid)
+pub_glfs_lchown(struct glfs *fs, const char *path, uid_t uid, gid_t gid)
 {
-	int              ret = 0;
-	int              valid = 0;
-	struct iatt      iatt = {0, };
+    int ret = 0;
+    struct glfs_stat stat = {
+        0,
+    };
 
-        if (uid != (uid_t) -1) {
-	        iatt.ia_uid = uid;
-	        valid = GF_SET_ATTR_UID;
-        }
+    if (uid != (uid_t)-1) {
+        stat.glfs_st_uid = uid;
+        stat.glfs_st_mask = GLFS_STAT_UID;
+    }
 
-        if (gid != (uid_t) -1) {
-                iatt.ia_gid = gid;
-                valid = valid | GF_SET_ATTR_GID;
-        }
+    if (gid != (uid_t)-1) {
+        stat.glfs_st_gid = gid;
+        stat.glfs_st_mask = stat.glfs_st_mask | GLFS_STAT_GID;
+    }
 
-        if (valid)
-	        ret = glfs_setattr (fs, path, &iatt, valid, 0);
+    if (stat.glfs_st_mask)
+        ret = glfs_setattr(fs, path, &stat, 0);
 
-	return ret;
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_lchown, 3.4.0);
-
-
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fchown, 3.4.0)
 int
-pub_glfs_fchown (struct glfs_fd *glfd, uid_t uid, gid_t gid)
+pub_glfs_fchown(struct glfs_fd *glfd, uid_t uid, gid_t gid)
 {
-	int              ret = 0;
-	int              valid = 0;
-	struct iatt      iatt = {0, };
+    int ret = 0;
+    struct glfs_stat stat = {
+        0,
+    };
 
-        if (uid != (uid_t) -1) {
-                iatt.ia_uid = uid;
-                valid = GF_SET_ATTR_UID;
-        }
+    if (uid != (uid_t)-1) {
+        stat.glfs_st_uid = uid;
+        stat.glfs_st_mask = GLFS_STAT_UID;
+    }
 
-        if (gid != (uid_t) -1) {
-                iatt.ia_gid = gid;
-                valid = valid | GF_SET_ATTR_GID;
-        }
+    if (gid != (uid_t)-1) {
+        stat.glfs_st_gid = gid;
+        stat.glfs_st_mask = stat.glfs_st_mask | GLFS_STAT_GID;
+    }
 
-        if (valid)
-	        ret = glfs_fsetattr (glfd, &iatt, valid);
+    if (stat.glfs_st_mask)
+        ret = glfs_fsetattr(glfd, &stat);
 
-	return ret;
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fchown, 3.4.0);
-
-
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_utimens, 3.4.0)
 int
-pub_glfs_utimens (struct glfs *fs, const char *path, struct timespec times[2])
+pub_glfs_utimens(struct glfs *fs, const char *path,
+                 const struct timespec times[2])
 {
-	int              ret = -1;
-	int              valid = 0;
-	struct iatt      iatt = {0, };
+    int ret = -1;
+    struct glfs_stat stat = {
+        0,
+    };
 
-	iatt.ia_atime = times[0].tv_sec;
-	iatt.ia_atime_nsec = times[0].tv_nsec;
-	iatt.ia_mtime = times[1].tv_sec;
-	iatt.ia_mtime_nsec = times[1].tv_nsec;
+    stat.glfs_st_atime = times[0];
+    stat.glfs_st_mtime = times[1];
 
-	valid = GF_SET_ATTR_ATIME|GF_SET_ATTR_MTIME;
+    stat.glfs_st_mask = GLFS_STAT_ATIME | GLFS_STAT_MTIME;
 
-	ret = glfs_setattr (fs, path, &iatt, valid, 1);
+    ret = glfs_setattr(fs, path, &stat, 1);
 
-	return ret;
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_utimens, 3.4.0);
-
-
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_lutimens, 3.4.0)
 int
-pub_glfs_lutimens (struct glfs *fs, const char *path, struct timespec times[2])
+pub_glfs_lutimens(struct glfs *fs, const char *path,
+                  const struct timespec times[2])
 {
-	int              ret = -1;
-	int              valid = 0;
-	struct iatt      iatt = {0, };
+    int ret = -1;
+    struct glfs_stat stat = {
+        0,
+    };
 
-	iatt.ia_atime = times[0].tv_sec;
-	iatt.ia_atime_nsec = times[0].tv_nsec;
-	iatt.ia_mtime = times[1].tv_sec;
-	iatt.ia_mtime_nsec = times[1].tv_nsec;
+    stat.glfs_st_atime = times[0];
+    stat.glfs_st_mtime = times[1];
 
-	valid = GF_SET_ATTR_ATIME|GF_SET_ATTR_MTIME;
+    stat.glfs_st_mask = GLFS_STAT_ATIME | GLFS_STAT_MTIME;
 
-	ret = glfs_setattr (fs, path, &iatt, valid, 0);
+    ret = glfs_setattr(fs, path, &stat, 0);
 
-	return ret;
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_lutimens, 3.4.0);
-
-
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_futimens, 3.4.0)
 int
-pub_glfs_futimens (struct glfs_fd *glfd, struct timespec times[2])
+pub_glfs_futimens(struct glfs_fd *glfd, const struct timespec times[2])
 {
-	int              ret = -1;
-	int              valid = 0;
-	struct iatt      iatt = {0, };
+    int ret = -1;
+    struct glfs_stat stat = {
+        0,
+    };
 
-	iatt.ia_atime = times[0].tv_sec;
-	iatt.ia_atime_nsec = times[0].tv_nsec;
-	iatt.ia_mtime = times[1].tv_sec;
-	iatt.ia_mtime_nsec = times[1].tv_nsec;
+    stat.glfs_st_atime = times[0];
+    stat.glfs_st_mtime = times[1];
 
-	valid = GF_SET_ATTR_ATIME|GF_SET_ATTR_MTIME;
+    stat.glfs_st_mask = GLFS_STAT_ATIME | GLFS_STAT_MTIME;
 
-	ret = glfs_fsetattr (glfd, &iatt, valid);
+    ret = glfs_fsetattr(glfd, &stat);
 
-	return ret;
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_futimens, 3.4.0);
-
-
 int
-glfs_getxattr_process (void *value, size_t size, dict_t *xattr,
-		       const char *name)
+glfs_getxattr_process(void *value, size_t size, dict_t *xattr, const char *name)
 {
-	data_t *data = NULL;
-	int     ret = -1;
-
-	data = dict_get (xattr, (char *)name);
-	if (!data) {
-		errno = ENODATA;
-		ret = -1;
-		goto out;
-	}
-
-	ret = data->len;
-	if (!value || !size)
-		goto out;
-
-	if (size < ret) {
-		ret = -1;
-		errno = ERANGE;
-		goto out;
-	}
-
-	memcpy (value, data->data, ret);
+    data_t *data = NULL;
+    int ret = -1;
+
+    data = dict_get(xattr, (char *)name);
+    if (!data) {
+        errno = ENODATA;
+        ret = -1;
+        goto out;
+    }
+
+    ret = data->len;
+    if (!value || !size)
+        goto out;
+
+    if (size < ret) {
+        ret = -1;
+        errno = ERANGE;
+        goto out;
+    }
+
+    memcpy(value, data->data, ret);
 out:
-	if (xattr)
-		dict_unref (xattr);
-	return ret;
+    return ret;
 }
 
-
 ssize_t
-glfs_getxattr_common (struct glfs *fs, const char *path, const char *name,
-		      void *value, size_t size, int follow)
-{
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	loc_t            loc = {0, };
-	struct iatt      iatt = {0, };
-	dict_t          *xattr = NULL;
-	int              reval = 0;
-
-	__glfs_entry_fs (fs);
-
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+glfs_getxattr_common(struct glfs *fs, const char *path, const char *name,
+                     void *value, size_t size, int follow)
+{
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    dict_t *xattr = NULL;
+    int reval = 0;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    if (!name || *name == '\0') {
+        ret = -1;
+        errno = EINVAL;
+        goto out;
+    }
+
+    if (strlen(name) > GF_XATTR_NAME_MAX) {
+        ret = -1;
+        errno = ENAMETOOLONG;
+        goto out;
+    }
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
 retry:
-	if (follow)
-		ret = glfs_resolve (fs, subvol, path, &loc, &iatt, reval);
-	else
-		ret = glfs_lresolve (fs, subvol, path, &loc, &iatt, reval);
+    if (follow)
+        ret = glfs_resolve(fs, subvol, path, &loc, &iatt, reval);
+    else
+        ret = glfs_lresolve(fs, subvol, path, &loc, &iatt, reval);
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	if (ret)
-		goto out;
+    if (ret)
+        goto out;
 
-	ret = syncop_getxattr (subvol, &loc, &xattr, name);
-        DECODE_SYNCOP_ERR (ret);
+    ret = syncop_getxattr(subvol, &loc, &xattr, name, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	if (ret)
-		goto out;
+    if (ret)
+        goto out;
 
-	ret = glfs_getxattr_process (value, size, xattr, name);
+    ret = glfs_getxattr_process(value, size, xattr, name);
 out:
-	loc_wipe (&loc);
+    loc_wipe(&loc);
 
-	priv_glfs_subvol_done (fs, subvol);
+    if (xattr)
+        dict_unref(xattr);
 
-	return ret;
-}
+    glfs_subvol_done(fs, subvol);
+
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_getxattr, 3.4.0)
 ssize_t
-pub_glfs_getxattr (struct glfs *fs, const char *path, const char *name,
-                   void *value, size_t size)
+pub_glfs_getxattr(struct glfs *fs, const char *path, const char *name,
+                  void *value, size_t size)
 {
-	return glfs_getxattr_common (fs, path, name, value, size, 1);
+    return glfs_getxattr_common(fs, path, name, value, size, 1);
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_getxattr, 3.4.0);
-
-
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_lgetxattr, 3.4.0)
 ssize_t
-pub_glfs_lgetxattr (struct glfs *fs, const char *path, const char *name,
-                    void *value, size_t size)
+pub_glfs_lgetxattr(struct glfs *fs, const char *path, const char *name,
+                   void *value, size_t size)
 {
-	return glfs_getxattr_common (fs, path, name, value, size, 0);
+    return glfs_getxattr_common(fs, path, name, value, size, 0);
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_lgetxattr, 3.4.0);
-
-
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fgetxattr, 3.4.0)
 ssize_t
-pub_glfs_fgetxattr (struct glfs_fd *glfd, const char *name, void *value,
-                    size_t size)
+pub_glfs_fgetxattr(struct glfs_fd *glfd, const char *name, void *value,
+                   size_t size)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	dict_t          *xattr = NULL;
-	fd_t            *fd = NULL;
-
-	__glfs_entry_fd (glfd);
-
-	subvol = priv_glfs_active_subvol (glfd->fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
-
-	fd = glfs_resolve_fd (glfd->fs, subvol, glfd);
-	if (!fd) {
-		ret = -1;
-		errno = EBADFD;
-		goto out;
-	}
-
-	ret = syncop_fgetxattr (subvol, fd, &xattr, name);
-        DECODE_SYNCOP_ERR (ret);
-	if (ret)
-		goto out;
-
-	ret = glfs_getxattr_process (value, size, xattr, name);
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    dict_t *xattr = NULL;
+    fd_t *fd = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    GF_REF_GET(glfd);
+
+    if (!name || *name == '\0') {
+        ret = -1;
+        errno = EINVAL;
+        goto out;
+    }
+
+    if (strlen(name) > GF_XATTR_NAME_MAX) {
+        ret = -1;
+        errno = ENAMETOOLONG;
+        goto out;
+    }
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        ret = -1;
+        errno = EBADFD;
+        goto out;
+    }
+
+    ret = syncop_fgetxattr(subvol, fd, &xattr, name, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+    if (ret)
+        goto out;
+
+    ret = glfs_getxattr_process(value, size, xattr, name);
 out:
-	if (fd)
-		fd_unref (fd);
+    if (fd)
+        fd_unref(fd);
+    if (glfd)
+        GF_REF_PUT(glfd);
+    if (xattr)
+        dict_unref(xattr);
 
-	priv_glfs_subvol_done (glfd->fs, subvol);
+    glfs_subvol_done(glfd->fs, subvol);
 
-	return ret;
-}
-
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fgetxattr, 3.4.0);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
 int
-glfs_listxattr_process (void *value, size_t size, dict_t *xattr)
+glfs_listxattr_process(void *value, size_t size, dict_t *xattr)
 {
-	int ret = -1;
+    int ret = -1;
 
-	if (!xattr)
-		goto out;
+    if (!xattr)
+        goto out;
 
-	ret = dict_keys_join (NULL, 0, xattr, NULL);
+    ret = dict_keys_join(NULL, 0, xattr, NULL);
 
-        if (!value || !size)
-                goto out;
+    if (!value || !size)
+        goto out;
 
-	if (size < ret) {
-		ret = -1;
-		errno = ERANGE;
-	} else {
-		dict_keys_join (value, size, xattr, NULL);
-	}
+    if (size < ret) {
+        ret = -1;
+        errno = ERANGE;
+    } else {
+        dict_keys_join(value, size, xattr, NULL);
+    }
 
 out:
-        if (xattr)
-                dict_unref (xattr);
-
-	return ret;
+    return ret;
 }
 
-
 ssize_t
-glfs_listxattr_common (struct glfs *fs, const char *path, void *value,
-		       size_t size, int follow)
+glfs_listxattr_common(struct glfs *fs, const char *path, void *value,
+                      size_t size, int follow)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	loc_t            loc = {0, };
-	struct iatt      iatt = {0, };
-	dict_t          *xattr = NULL;
-	int              reval = 0;
-
-	__glfs_entry_fs (fs);
-
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    dict_t *xattr = NULL;
+    int reval = 0;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
 
 retry:
-	if (follow)
-		ret = glfs_resolve (fs, subvol, path, &loc, &iatt, reval);
-	else
-		ret = glfs_lresolve (fs, subvol, path, &loc, &iatt, reval);
+    if (follow)
+        ret = glfs_resolve(fs, subvol, path, &loc, &iatt, reval);
+    else
+        ret = glfs_lresolve(fs, subvol, path, &loc, &iatt, reval);
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	if (ret)
-		goto out;
+    if (ret)
+        goto out;
 
-	ret = syncop_getxattr (subvol, &loc, &xattr, NULL);
-        DECODE_SYNCOP_ERR (ret);
+    ret = syncop_getxattr(subvol, &loc, &xattr, NULL, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	if (ret)
-		goto out;
+    if (ret)
+        goto out;
 
-	ret = glfs_listxattr_process (value, size, xattr);
+    ret = glfs_listxattr_process(value, size, xattr);
 out:
-	loc_wipe (&loc);
+    loc_wipe(&loc);
 
-	priv_glfs_subvol_done (fs, subvol);
+    if (xattr)
+        dict_unref(xattr);
 
-	return ret;
-}
+    glfs_subvol_done(fs, subvol);
 
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
+
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_listxattr, 3.4.0)
 ssize_t
-pub_glfs_listxattr (struct glfs *fs, const char *path, void *value, size_t size)
+pub_glfs_listxattr(struct glfs *fs, const char *path, void *value, size_t size)
 {
-	return glfs_listxattr_common (fs, path, value, size, 1);
+    return glfs_listxattr_common(fs, path, value, size, 1);
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_listxattr, 3.4.0);
-
-
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_llistxattr, 3.4.0)
 ssize_t
-pub_glfs_llistxattr (struct glfs *fs, const char *path, void *value, size_t size)
+pub_glfs_llistxattr(struct glfs *fs, const char *path, void *value, size_t size)
 {
-	return glfs_listxattr_common (fs, path, value, size, 0);
+    return glfs_listxattr_common(fs, path, value, size, 0);
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_llistxattr, 3.4.0);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_flistxattr, 3.4.0)
+ssize_t
+pub_glfs_flistxattr(struct glfs_fd *glfd, void *value, size_t size)
+{
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    dict_t *xattr = NULL;
+    fd_t *fd = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    GF_REF_GET(glfd);
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        ret = -1;
+        errno = EBADFD;
+        goto out;
+    }
+
+    ret = syncop_fgetxattr(subvol, fd, &xattr, NULL, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+    if (ret)
+        goto out;
+
+    ret = glfs_listxattr_process(value, size, xattr);
+out:
+    if (fd)
+        fd_unref(fd);
+    if (glfd)
+        GF_REF_PUT(glfd);
+    if (xattr)
+        dict_unref(xattr);
+
+    glfs_subvol_done(glfd->fs, subvol);
 
+    __GLFS_EXIT_FS;
 
-ssize_t
-pub_glfs_flistxattr (struct glfs_fd *glfd, void *value, size_t size)
+invalid_fs:
+    return ret;
+}
+
+int
+glfs_setxattr_common(struct glfs *fs, const char *path, const char *name,
+                     const void *value, size_t size, int flags, int follow)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	dict_t          *xattr = NULL;
-	fd_t            *fd = NULL;
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    dict_t *xattr = NULL;
+    int reval = 0;
+    void *value_cp = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    if (!name || *name == '\0') {
+        ret = -1;
+        errno = EINVAL;
+        goto out;
+    }
+
+    if (strlen(name) > GF_XATTR_NAME_MAX) {
+        ret = -1;
+        errno = ENAMETOOLONG;
+        goto out;
+    }
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+retry:
+    if (follow)
+        ret = glfs_resolve(fs, subvol, path, &loc, &iatt, reval);
+    else
+        ret = glfs_lresolve(fs, subvol, path, &loc, &iatt, reval);
+
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	__glfs_entry_fd (glfd);
+    if (ret)
+        goto out;
 
-	subvol = priv_glfs_active_subvol (glfd->fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    value_cp = gf_memdup(value, size);
+    GF_CHECK_ALLOC_AND_LOG(subvol->name, value_cp, ret,
+                           "Failed to"
+                           " duplicate setxattr value",
+                           out);
 
-	fd = glfs_resolve_fd (glfd->fs, subvol, glfd);
-	if (!fd) {
-		ret = -1;
-		errno = EBADFD;
-		goto out;
-	}
+    xattr = dict_for_key_value(name, value_cp, size, _gf_false);
+    if (!xattr) {
+        GF_FREE(value_cp);
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
 
-	ret = syncop_fgetxattr (subvol, fd, &xattr, NULL);
-        DECODE_SYNCOP_ERR (ret);
-	if (ret)
-		goto out;
+    ret = syncop_setxattr(subvol, &loc, xattr, flags, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
 
-	ret = glfs_listxattr_process (value, size, xattr);
 out:
-	if (fd)
-		fd_unref (fd);
+    loc_wipe(&loc);
+    if (xattr)
+        dict_unref(xattr);
 
-	priv_glfs_subvol_done (glfd->fs, subvol);
+    glfs_subvol_done(fs, subvol);
 
-	return ret;
+    __GLFS_EXIT_FS;
+
+invalid_fs:
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_flistxattr, 3.4.0);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_setxattr, 3.4.0)
+int
+pub_glfs_setxattr(struct glfs *fs, const char *path, const char *name,
+                  const void *value, size_t size, int flags)
+{
+    return glfs_setxattr_common(fs, path, name, value, size, flags, 1);
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_lsetxattr, 3.4.0)
+int
+pub_glfs_lsetxattr(struct glfs *fs, const char *path, const char *name,
+                   const void *value, size_t size, int flags)
+{
+    return glfs_setxattr_common(fs, path, name, value, size, flags, 0);
+}
 
-dict_t *
-dict_for_key_value (const char *name, const char *value, size_t size)
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fsetxattr, 3.4.0)
+int
+pub_glfs_fsetxattr(struct glfs_fd *glfd, const char *name, const void *value,
+                   size_t size, int flags)
 {
-	dict_t *xattr = NULL;
-	int     ret = 0;
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    dict_t *xattr = NULL;
+    fd_t *fd = NULL;
+    void *value_cp = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    GF_REF_GET(glfd);
+
+    if (!name || *name == '\0') {
+        ret = -1;
+        errno = EINVAL;
+        goto out;
+    }
+
+    if (strlen(name) > GF_XATTR_NAME_MAX) {
+        ret = -1;
+        errno = ENAMETOOLONG;
+        goto out;
+    }
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        ret = -1;
+        errno = EBADFD;
+        goto out;
+    }
+
+    value_cp = gf_memdup(value, size);
+    GF_CHECK_ALLOC_AND_LOG(subvol->name, value_cp, ret,
+                           "Failed to"
+                           " duplicate setxattr value",
+                           out);
+
+    xattr = dict_for_key_value(name, value_cp, size, _gf_false);
+    if (!xattr) {
+        GF_FREE(value_cp);
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    ret = syncop_fsetxattr(subvol, fd, xattr, flags, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+out:
+    if (xattr)
+        dict_unref(xattr);
 
-	xattr = dict_new ();
-	if (!xattr)
-		return NULL;
+    if (fd)
+        fd_unref(fd);
+    if (glfd)
+        GF_REF_PUT(glfd);
 
-	ret = dict_set_static_bin (xattr, (char *)name, (void *)value, size);
-	if (ret) {
-		dict_destroy (xattr);
-		xattr = NULL;
-	}
+    glfs_subvol_done(glfd->fs, subvol);
 
-	return xattr;
-}
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
 int
-glfs_setxattr_common (struct glfs *fs, const char *path, const char *name,
-		      const void *value, size_t size, int flags, int follow)
+glfs_removexattr_common(struct glfs *fs, const char *path, const char *name,
+                        int follow)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	loc_t            loc = {0, };
-	struct iatt      iatt = {0, };
-	dict_t          *xattr = NULL;
-	int              reval = 0;
-
-	__glfs_entry_fs (fs);
-
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    int reval = 0;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
 retry:
-	if (follow)
-		ret = glfs_resolve (fs, subvol, path, &loc, &iatt, reval);
-	else
-		ret = glfs_lresolve (fs, subvol, path, &loc, &iatt, reval);
-
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    if (follow)
+        ret = glfs_resolve(fs, subvol, path, &loc, &iatt, reval);
+    else
+        ret = glfs_lresolve(fs, subvol, path, &loc, &iatt, reval);
 
-	if (ret)
-		goto out;
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	xattr = dict_for_key_value (name, value, size);
-	if (!xattr) {
-		ret = -1;
-		errno = ENOMEM;
-		goto out;
-	}
+    if (ret)
+        goto out;
 
-	ret = syncop_setxattr (subvol, &loc, xattr, flags);
-        DECODE_SYNCOP_ERR (ret);
+    ret = syncop_removexattr(subvol, &loc, name, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
 out:
-	loc_wipe (&loc);
-	if (xattr)
-		dict_unref (xattr);
+    loc_wipe(&loc);
 
-	priv_glfs_subvol_done (fs, subvol);
+    glfs_subvol_done(fs, subvol);
 
-	return ret;
+    __GLFS_EXIT_FS;
+
+invalid_fs:
+    return ret;
 }
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_removexattr, 3.4.0)
+int
+pub_glfs_removexattr(struct glfs *fs, const char *path, const char *name)
+{
+    return glfs_removexattr_common(fs, path, name, 1);
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_lremovexattr, 3.4.0)
 int
-pub_glfs_setxattr (struct glfs *fs, const char *path, const char *name,
-                   const void *value, size_t size, int flags)
+pub_glfs_lremovexattr(struct glfs *fs, const char *path, const char *name)
 {
-	return glfs_setxattr_common (fs, path, name, value, size, flags, 1);
+    return glfs_removexattr_common(fs, path, name, 0);
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_setxattr, 3.4.0);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fremovexattr, 3.4.0)
+int
+pub_glfs_fremovexattr(struct glfs_fd *glfd, const char *name)
+{
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    fd_t *fd = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    GF_REF_GET(glfd);
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        ret = -1;
+        errno = EBADFD;
+        goto out;
+    }
+
+    ret = syncop_fremovexattr(subvol, fd, name, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+out:
+    if (fd)
+        fd_unref(fd);
+    if (glfd)
+        GF_REF_PUT(glfd);
 
+    glfs_subvol_done(glfd->fs, subvol);
 
+    __GLFS_EXIT_FS;
+
+invalid_fs:
+    return ret;
+}
+
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fallocate, 3.5.0)
 int
-pub_glfs_lsetxattr (struct glfs *fs, const char *path, const char *name,
-                    const void *value, size_t size, int flags)
+pub_glfs_fallocate(struct glfs_fd *glfd, int keep_size, off_t offset,
+                   size_t len)
 {
-	return glfs_setxattr_common (fs, path, name, value, size, flags, 0);
-}
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    fd_t *fd = NULL;
+    dict_t *fop_attr = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    GF_REF_GET(glfd);
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        ret = -1;
+        errno = EBADFD;
+        goto out;
+    }
+
+    ret = get_fop_attr_thrd_key(&fop_attr);
+    if (ret)
+        gf_msg_debug("gfapi", 0, "Getting leaseid from thread failed");
+
+    ret = syncop_fallocate(subvol, fd, keep_size, offset, len, fop_attr, NULL);
+    DECODE_SYNCOP_ERR(ret);
+out:
+    if (fd)
+        fd_unref(fd);
+    if (glfd)
+        GF_REF_PUT(glfd);
+    if (fop_attr)
+        dict_unref(fop_attr);
+
+    glfs_subvol_done(glfd->fs, subvol);
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_lsetxattr, 3.4.0);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_discard, 3.5.0)
 int
-pub_glfs_fsetxattr (struct glfs_fd *glfd, const char *name, const void *value,
-                    size_t size, int flags)
+pub_glfs_discard(struct glfs_fd *glfd, off_t offset, size_t len)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	dict_t          *xattr = NULL;
-	fd_t            *fd = NULL;
-
-	__glfs_entry_fd (glfd);
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    fd_t *fd = NULL;
+    dict_t *fop_attr = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    GF_REF_GET(glfd);
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        ret = -1;
+        errno = EBADFD;
+        goto out;
+    }
+
+    ret = get_fop_attr_thrd_key(&fop_attr);
+    if (ret)
+        gf_msg_debug("gfapi", 0, "Getting leaseid from thread failed");
+
+    ret = syncop_discard(subvol, fd, offset, len, fop_attr, NULL);
+    DECODE_SYNCOP_ERR(ret);
+out:
+    if (fd)
+        fd_unref(fd);
+    if (glfd)
+        GF_REF_PUT(glfd);
+    if (fop_attr)
+        dict_unref(fop_attr);
 
-	subvol = priv_glfs_active_subvol (glfd->fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    glfs_subvol_done(glfd->fs, subvol);
 
-	fd = glfs_resolve_fd (glfd->fs, subvol, glfd);
-	if (!fd) {
-		ret = -1;
-		errno = EBADFD;
-		goto out;
-	}
+    __GLFS_EXIT_FS;
 
-	xattr = dict_for_key_value (name, value, size);
-	if (!xattr) {
-		ret = -1;
-		errno = ENOMEM;
-		goto out;
-	}
+invalid_fs:
+    return ret;
+}
 
-	ret = syncop_fsetxattr (subvol, fd, xattr, flags);
-        DECODE_SYNCOP_ERR (ret);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_zerofill, 3.5.0)
+int
+pub_glfs_zerofill(struct glfs_fd *glfd, off_t offset, off_t len)
+{
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    fd_t *fd = NULL;
+    dict_t *fop_attr = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    GF_REF_GET(glfd);
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        errno = EBADFD;
+        goto out;
+    }
+
+    ret = get_fop_attr_thrd_key(&fop_attr);
+    if (ret)
+        gf_msg_debug("gfapi", 0, "Getting leaseid from thread failed");
+
+    ret = syncop_zerofill(subvol, fd, offset, len, fop_attr, NULL);
+    DECODE_SYNCOP_ERR(ret);
 out:
-	if (xattr)
-		dict_unref (xattr);
+    if (fd)
+        fd_unref(fd);
+    if (glfd)
+        GF_REF_PUT(glfd);
+    if (fop_attr)
+        dict_unref(fop_attr);
 
-	if (fd)
-		fd_unref (fd);
+    glfs_subvol_done(glfd->fs, subvol);
 
-	priv_glfs_subvol_done (glfd->fs, subvol);
+    __GLFS_EXIT_FS;
 
-	return ret;
+invalid_fs:
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fsetxattr, 3.4.0);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_chdir, 3.4.0)
+int
+pub_glfs_chdir(struct glfs *fs, const char *path)
+{
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    int reval = 0;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+retry:
+    ret = glfs_resolve(fs, subvol, path, &loc, &iatt, reval);
+
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
+
+    if (ret)
+        goto out;
+
+    if (!IA_ISDIR(iatt.ia_type)) {
+        ret = -1;
+        errno = ENOTDIR;
+        goto out;
+    }
 
+    glfs_cwd_set(fs, loc.inode);
+
+out:
+    loc_wipe(&loc);
+
+    glfs_subvol_done(fs, subvol);
+
+    __GLFS_EXIT_FS;
+
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fchdir, 3.4.0)
 int
-glfs_removexattr_common (struct glfs *fs, const char *path, const char *name,
-			 int follow)
+pub_glfs_fchdir(struct glfs_fd *glfd)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	loc_t            loc = {0, };
-	struct iatt      iatt = {0, };
-	int              reval = 0;
+    int ret = -1;
+    inode_t *inode = NULL;
+    xlator_t *subvol = NULL;
+    fd_t *fd = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    GF_REF_GET(glfd);
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        ret = -1;
+        errno = EBADFD;
+        goto out;
+    }
+
+    inode = fd->inode;
+
+    if (!IA_ISDIR(inode->ia_type)) {
+        ret = -1;
+        errno = ENOTDIR;
+        goto out;
+    }
+
+    glfs_cwd_set(glfd->fs, inode);
+    ret = 0;
+out:
+    if (fd)
+        fd_unref(fd);
+    if (glfd)
+        GF_REF_PUT(glfd);
 
-	__glfs_entry_fs (fs);
+    glfs_subvol_done(glfd->fs, subvol);
 
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
-retry:
-	if (follow)
-		ret = glfs_resolve (fs, subvol, path, &loc, &iatt, reval);
-	else
-		ret = glfs_lresolve (fs, subvol, path, &loc, &iatt, reval);
+    __GLFS_EXIT_FS;
+
+invalid_fs:
+    return ret;
+}
+
+static gf_boolean_t warn_realpath = _gf_true; /* log once */
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+static char *
+glfs_realpath_common(struct glfs *fs, const char *path, char *resolved_path,
+                     gf_boolean_t warn_deprecated)
+{
+    int ret = -1;
+    char *retpath = NULL;
+    char *allocpath = NULL;
+    xlator_t *subvol = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    int reval = 0;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    if (resolved_path)
+        retpath = resolved_path;
+    else if (warn_deprecated) {
+        retpath = allocpath = malloc(PATH_MAX + 1);
+        if (warn_realpath) {
+            warn_realpath = _gf_false;
+            gf_log(THIS->name, GF_LOG_WARNING,
+                   "this application "
+                   "is compiled against an old version of "
+                   "libgfapi, it should use glfs_free() to "
+                   "release the path returned by "
+                   "glfs_realpath()");
+        }
+    } else {
+        retpath = allocpath = GLFS_CALLOC(1, PATH_MAX + 1, NULL,
+                                          glfs_mt_realpath_t);
+    }
+
+    if (!retpath) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+retry:
+    ret = glfs_resolve(fs, subvol, path, &loc, &iatt, reval);
 
-	if (ret)
-		goto out;
+    ESTALE_RETRY(ret, errno, reval, &loc, retry);
 
-	ret = syncop_removexattr (subvol, &loc, name, 0);
-        DECODE_SYNCOP_ERR (ret);
+    if (ret)
+        goto out;
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    if (loc.path) {
+        snprintf(retpath, PATH_MAX + 1, "%s", loc.path);
+    }
 
 out:
-	loc_wipe (&loc);
+    loc_wipe(&loc);
+
+    if (ret == -1) {
+        if (warn_deprecated && allocpath)
+            free(allocpath);
+        else if (allocpath)
+            GLFS_FREE(allocpath);
+        retpath = NULL;
+    }
 
-	priv_glfs_subvol_done (fs, subvol);
+    glfs_subvol_done(fs, subvol);
 
-	return ret;
+    __GLFS_EXIT_FS;
+
+invalid_fs:
+    return retpath;
 }
 
+GFAPI_SYMVER_PUBLIC(glfs_realpath34, glfs_realpath, 3.4.0)
+char *
+pub_glfs_realpath34(struct glfs *fs, const char *path, char *resolved_path)
+{
+    return glfs_realpath_common(fs, path, resolved_path, _gf_true);
+}
 
-int
-pub_glfs_removexattr (struct glfs *fs, const char *path, const char *name)
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_realpath, 3.7.17)
+char *
+pub_glfs_realpath(struct glfs *fs, const char *path, char *resolved_path)
 {
-	return glfs_removexattr_common (fs, path, name, 1);
+    return glfs_realpath_common(fs, path, resolved_path, _gf_false);
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_removexattr, 3.4.0);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_getcwd, 3.4.0)
+char *
+pub_glfs_getcwd(struct glfs *fs, char *buf, size_t n)
+{
+    int ret = -1;
+    inode_t *inode = NULL;
+    char *path = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    if (!buf || n < 2) {
+        ret = -1;
+        errno = EINVAL;
+        goto out;
+    }
+
+    inode = glfs_cwd_get(fs);
+
+    if (!inode) {
+        strncpy(buf, "/", n);
+        ret = 0;
+        goto out;
+    }
+
+    ret = inode_path(inode, 0, &path);
+    if (n <= ret) {
+        ret = -1;
+        errno = ERANGE;
+        goto out;
+    }
+
+    strncpy(buf, path, n);
+    ret = 0;
+out:
+    GF_FREE(path);
 
+    if (inode)
+        inode_unref(inode);
 
-int
-pub_glfs_lremovexattr (struct glfs *fs, const char *path, const char *name)
-{
-	return glfs_removexattr_common (fs, path, name, 0);
+    __GLFS_EXIT_FS;
+
+invalid_fs:
+    if (ret < 0)
+        return NULL;
+
+    return buf;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_lremovexattr, 3.4.0);
+static void
+gf_flock_to_flock(struct gf_flock *gf_flock, struct flock *flock)
+{
+    flock->l_type = gf_flock->l_type;
+    flock->l_whence = gf_flock->l_whence;
+    flock->l_start = gf_flock->l_start;
+    flock->l_len = gf_flock->l_len;
+    flock->l_pid = gf_flock->l_pid;
+}
 
+static void
+gf_flock_from_flock(struct gf_flock *gf_flock, struct flock *flock)
+{
+    gf_flock->l_type = flock->l_type;
+    gf_flock->l_whence = flock->l_whence;
+    gf_flock->l_start = flock->l_start;
+    gf_flock->l_len = flock->l_len;
+    gf_flock->l_pid = flock->l_pid;
+}
 
-int
-pub_glfs_fremovexattr (struct glfs_fd *glfd, const char *name)
+static int
+glfs_lock_common(struct glfs_fd *glfd, int cmd, struct flock *flock,
+                 dict_t *xdata)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	fd_t            *fd = NULL;
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    struct gf_flock gf_flock = {
+        0,
+    };
+    struct gf_flock saved_flock = {
+        0,
+    };
+    fd_t *fd = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    if (!flock) {
+        errno = EINVAL;
+        goto out;
+    }
+
+    GF_REF_GET(glfd);
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        ret = -1;
+        errno = EBADFD;
+        goto out;
+    }
+
+    /* Generate glusterfs flock structure from client flock
+     * structure to be processed by server */
+    gf_flock_from_flock(&gf_flock, flock);
+
+    /* Keep another copy of flock for split/merge of locks
+     * at client side */
+    gf_flock_from_flock(&saved_flock, flock);
+
+    if (glfd->lk_owner.len != 0) {
+        ret = syncopctx_setfslkowner(&glfd->lk_owner);
+
+        if (ret)
+            goto out;
+    }
 
-	__glfs_entry_fd (glfd);
+    ret = get_fop_attr_thrd_key(&xdata);
+    if (ret)
+        gf_msg_debug("gfapi", 0, "Getting leaseid from thread failed");
 
-	subvol = priv_glfs_active_subvol (glfd->fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    ret = syncop_lk(subvol, fd, cmd, &gf_flock, xdata, NULL);
+    DECODE_SYNCOP_ERR(ret);
 
-	fd = glfs_resolve_fd (glfd->fs, subvol, glfd);
-	if (!fd) {
-		ret = -1;
-		errno = EBADFD;
-		goto out;
-	}
+    /* Convert back from gf_flock to flock as expected by application */
+    gf_flock_to_flock(&gf_flock, flock);
+
+    if (ret == 0 && (cmd == F_SETLK || cmd == F_SETLKW)) {
+        ret = fd_lk_insert_and_merge(fd, cmd, &saved_flock);
+        if (ret) {
+            gf_smsg(THIS->name, GF_LOG_ERROR, 0,
+                    API_MSG_LOCK_INSERT_MERGE_FAILED, "gfid=%s",
+                    uuid_utoa(fd->inode->gfid), NULL);
+            ret = 0;
+        }
+    }
 
-	ret = syncop_fremovexattr (subvol, fd, name, 0);
-        DECODE_SYNCOP_ERR (ret);
 out:
-	if (fd)
-		fd_unref (fd);
+    if (fd)
+        fd_unref(fd);
+    if (glfd)
+        GF_REF_PUT(glfd);
 
-	priv_glfs_subvol_done (glfd->fs, subvol);
+    glfs_subvol_done(glfd->fs, subvol);
 
-	return ret;
+    __GLFS_EXIT_FS;
+
+invalid_fs:
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fremovexattr, 3.4.0);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_file_lock, 4.0.0)
+int
+pub_glfs_file_lock(struct glfs_fd *glfd, int cmd, struct flock *flock,
+                   glfs_lock_mode_t lk_mode)
+{
+    int ret = -1;
+    dict_t *xdata_in = NULL;
+
+    if (lk_mode == GLFS_LK_MANDATORY) {
+        /* Create a new dictionary */
+        xdata_in = dict_new();
+        if (xdata_in == NULL) {
+            ret = -1;
+            errno = ENOMEM;
+            goto out;
+        }
+
+        /* Set GF_LK_MANDATORY internally within dictionary to map
+         * GLFS_LK_MANDATORY */
+        ret = dict_set_uint32(xdata_in, GF_LOCK_MODE, GF_LK_MANDATORY);
+        if (ret) {
+            gf_smsg(THIS->name, GF_LOG_ERROR, 0,
+                    API_MSG_SETTING_LOCK_TYPE_FAILED, NULL);
+            ret = -1;
+            errno = ENOMEM;
+            goto out;
+        }
+    }
+
+    ret = glfs_lock_common(glfd, cmd, flock, xdata_in);
+out:
+    if (xdata_in)
+        dict_unref(xdata_in);
 
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_posix_lock, 3.4.0)
 int
-pub_glfs_fallocate (struct glfs_fd *glfd, int keep_size, off_t offset, size_t len)
+pub_glfs_posix_lock(struct glfs_fd *glfd, int cmd, struct flock *flock)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	fd_t		*fd = NULL;
+    return glfs_lock_common(glfd, cmd, flock, NULL);
+}
 
-	__glfs_entry_fd (glfd);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fd_set_lkowner, 3.10.7)
+int
+pub_glfs_fd_set_lkowner(struct glfs_fd *glfd, void *data, int len)
+{
+    int ret = -1;
 
-	subvol = priv_glfs_active_subvol (glfd->fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
 
-	fd = glfs_resolve_fd (glfd->fs, subvol, glfd);
-	if (!fd) {
-		ret = -1;
-		errno = EBADFD;
-		goto out;
-	}
+    if (!GF_REF_GET(glfd)) {
+        goto invalid_fs;
+    }
 
-	ret = syncop_fallocate (subvol, fd, keep_size, offset, len);
-        DECODE_SYNCOP_ERR (ret);
-out:
-	if (fd)
-		fd_unref(fd);
+    GF_VALIDATE_OR_GOTO(THIS->name, data, out);
 
-	priv_glfs_subvol_done (glfd->fs, subvol);
+    if ((len <= 0) || (len > GFAPI_MAX_LOCK_OWNER_LEN)) {
+        errno = EINVAL;
+        gf_smsg(THIS->name, GF_LOG_ERROR, errno, API_MSG_INVALID_ARG,
+                "lk_owner len=%d", len, NULL);
+        goto out;
+    }
 
-	return ret;
-}
+    glfd->lk_owner.len = len;
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fallocate, 3.5.0);
+    memcpy(glfd->lk_owner.data, data, len);
 
+    ret = 0;
+out:
+    if (glfd)
+        GF_REF_PUT(glfd);
 
-int
-pub_glfs_discard (struct glfs_fd *glfd, off_t offset, size_t len)
+    __GLFS_EXIT_FS;
+
+invalid_fs:
+    return ret;
+}
+
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_dup, 3.4.0)
+struct glfs_fd *
+pub_glfs_dup(struct glfs_fd *glfd)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	fd_t		*fd = NULL;
+    xlator_t *subvol = NULL;
+    fd_t *fd = NULL;
+    struct glfs_fd *dupfd = NULL;
+    struct glfs *fs = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    GF_REF_GET(glfd);
+
+    fs = glfd->fs;
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(fs, subvol, glfd);
+    if (!fd) {
+        errno = EBADFD;
+        goto out;
+    }
+
+    dupfd = glfs_fd_new(fs);
+    if (!dupfd) {
+        errno = ENOMEM;
+        goto out;
+    }
+
+    dupfd->fd = fd_ref(fd);
+    dupfd->state = glfd->state;
+out:
+    if (fd)
+        fd_unref(fd);
+    if (dupfd)
+        glfs_fd_bind(dupfd);
+    if (glfd)
+        GF_REF_PUT(glfd);
 
-	__glfs_entry_fd (glfd);
+    glfs_subvol_done(fs, subvol);
 
-	subvol = priv_glfs_active_subvol (glfd->fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    __GLFS_EXIT_FS;
 
-	fd = glfs_resolve_fd (glfd->fs, subvol, glfd);
-	if (!fd) {
-		ret = -1;
-		errno = EBADFD;
-		goto out;
-	}
+invalid_fs:
+    return dupfd;
+}
+
+static void
+glfs_enqueue_upcall_data(struct glfs *fs, struct gf_upcall *upcall_data)
+{
+    int ret = -1;
+    upcall_entry *u_list = NULL;
+
+    if (!fs || !upcall_data)
+        goto out;
+
+    u_list = GF_CALLOC(1, sizeof(*u_list), glfs_mt_upcall_entry_t);
+
+    if (!u_list) {
+        gf_smsg(THIS->name, GF_LOG_ERROR, ENOMEM, API_MSG_ALLOC_FAILED, "entry",
+                NULL);
+        goto out;
+    }
+
+    INIT_LIST_HEAD(&u_list->upcall_list);
+
+    gf_uuid_copy(u_list->upcall_data.gfid, upcall_data->gfid);
+    u_list->upcall_data.event_type = upcall_data->event_type;
+
+    switch (upcall_data->event_type) {
+        case GF_UPCALL_CACHE_INVALIDATION:
+            ret = glfs_get_upcall_cache_invalidation(&u_list->upcall_data,
+                                                     upcall_data);
+            break;
+        case GF_UPCALL_RECALL_LEASE:
+            ret = glfs_get_upcall_lease(&u_list->upcall_data, upcall_data);
+            break;
+        default:
+            break;
+    }
+
+    if (ret) {
+        gf_smsg(THIS->name, GF_LOG_ERROR, errno, API_MSG_INVALID_ENTRY, NULL);
+        goto out;
+    }
+
+    pthread_mutex_lock(&fs->upcall_list_mutex);
+    {
+        list_add_tail(&u_list->upcall_list, &fs->upcall_list);
+    }
+    pthread_mutex_unlock(&fs->upcall_list_mutex);
+
+    ret = 0;
 
-	ret = syncop_discard (subvol, fd, offset, len);
-        DECODE_SYNCOP_ERR (ret);
 out:
-	if (fd)
-		fd_unref(fd);
+    if (ret && u_list) {
+        GF_FREE(u_list->upcall_data.data);
+        GF_FREE(u_list);
+    }
+}
 
-	priv_glfs_subvol_done (glfd->fs, subvol);
+static void
+glfs_free_upcall_lease(void *to_free)
+{
+    struct glfs_upcall_lease *arg = to_free;
 
-	return ret;
-}
+    if (!arg)
+        return;
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_discard, 3.5.0);
+    if (arg->object)
+        glfs_h_close(arg->object);
 
+    GF_FREE(arg);
+}
 
 int
-pub_glfs_zerofill (struct glfs_fd *glfd, off_t offset, off_t len)
+glfs_recall_lease_fd(struct glfs *fs, struct gf_upcall *up_data)
 {
-        int               ret             = -1;
-        xlator_t         *subvol          = NULL;
-        fd_t             *fd              = NULL;
-
-        __glfs_entry_fd (glfd);
-
-        subvol = priv_glfs_active_subvol (glfd->fs);
-        if (!subvol) {
-                errno = EIO;
-                goto out;
+    struct gf_upcall_recall_lease *recall_lease = NULL;
+    xlator_t *subvol = NULL;
+    int ret = 0;
+    inode_t *inode = NULL;
+    struct glfs_fd *glfd = NULL;
+    struct glfs_fd *tmp = NULL;
+    struct list_head glfd_list;
+    fd_t *fd = NULL;
+    uint64_t value = 0;
+    struct glfs_lease lease = {
+        0,
+    };
+
+    GF_VALIDATE_OR_GOTO("gfapi", up_data, out);
+    GF_VALIDATE_OR_GOTO("gfapi", fs, out);
+
+    recall_lease = up_data->data;
+    GF_VALIDATE_OR_GOTO("gfapi", recall_lease, out);
+
+    INIT_LIST_HEAD(&glfd_list);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    gf_msg_debug(THIS->name, 0, "Recall lease received for gfid:%s",
+                 uuid_utoa(up_data->gfid));
+
+    inode = inode_find(subvol->itable, up_data->gfid);
+    if (!inode) {
+        ret = -1;
+        gf_smsg(THIS->name, GF_LOG_ERROR, errno, API_MSG_INODE_FIND_FAILED,
+                "gfid=%s", uuid_utoa(up_data->gfid), "graph_id=%d",
+                subvol->graph->id, NULL);
+        goto out;
+    }
+
+    LOCK(&inode->lock);
+    {
+        list_for_each_entry(fd, &inode->fd_list, inode_list)
+        {
+            ret = fd_ctx_get(fd, subvol, &value);
+            glfd = (struct glfs_fd *)(uintptr_t)value;
+            if (glfd) {
+                gf_msg_trace(THIS->name, 0, "glfd (%p) has held lease", glfd);
+                GF_REF_GET(glfd);
+                list_add_tail(&glfd->list, &glfd_list);
+            }
         }
+    }
+    UNLOCK(&inode->lock);
 
-        fd = glfs_resolve_fd (glfd->fs, subvol, glfd);
-        if (!fd) {
-                errno = EBADFD;
-                goto out;
+    if (!list_empty(&glfd_list)) {
+        list_for_each_entry_safe(glfd, tmp, &glfd_list, list)
+        {
+            LOCK(&glfd->lock);
+            {
+                if (glfd->state != GLFD_CLOSE) {
+                    gf_msg_trace(THIS->name, 0,
+                                 "glfd (%p) has held lease, "
+                                 "calling recall cbk",
+                                 glfd);
+                    glfd->cbk(lease, glfd->cookie);
+                }
+            }
+            UNLOCK(&glfd->lock);
+
+            list_del_init(&glfd->list);
+            GF_REF_PUT(glfd);
         }
+    }
 
-        ret = syncop_zerofill (subvol, fd, offset, len);
-        DECODE_SYNCOP_ERR (ret);
 out:
-        if (fd)
-                fd_unref(fd);
+    return ret;
+}
 
-        priv_glfs_subvol_done (glfd->fs, subvol);
+static int
+glfs_recall_lease_upcall(struct glfs *fs, struct glfs_upcall *up_arg,
+                         struct gf_upcall *up_data)
+{
+    struct gf_upcall_recall_lease *recall_lease = NULL;
+    struct glfs_object *object = NULL;
+    xlator_t *subvol = NULL;
+    int ret = -1;
+    struct glfs_upcall_lease *up_lease_arg = NULL;
+
+    GF_VALIDATE_OR_GOTO("gfapi", up_data, out);
+    GF_VALIDATE_OR_GOTO("gfapi", fs, out);
+
+    recall_lease = up_data->data;
+    GF_VALIDATE_OR_GOTO("gfapi", recall_lease, out);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        errno = EIO;
+        goto out;
+    }
+
+    gf_msg_debug(THIS->name, 0, "Recall lease received for gfid:%s",
+                 uuid_utoa(up_data->gfid));
+
+    object = glfs_h_find_handle(fs, up_data->gfid, GFAPI_HANDLE_LENGTH);
+    if (!object) {
+        /* The reason handle creation will fail is because we
+         * couldn't find the inode in the gfapi inode table.
+         *
+         * But since application would have taken inode_ref, the
+         * only case when this can happen is when it has closed
+         * the handle and hence will no more be interested in
+         * the upcall for this particular gfid.
+         */
+        gf_smsg(THIS->name, GF_LOG_DEBUG, errno, API_MSG_CREATE_HANDLE_FAILED,
+                "gfid=%s", uuid_utoa(up_data->gfid), NULL);
+        errno = ESTALE;
+        goto out;
+    }
+
+    up_lease_arg = GF_CALLOC(1, sizeof(struct glfs_upcall_lease),
+                             glfs_mt_upcall_inode_t);
+    up_lease_arg->object = object;
+
+    GF_VALIDATE_OR_GOTO("glfs_recall_lease", up_lease_arg, out);
+
+    up_lease_arg->lease_type = recall_lease->lease_type;
+
+    up_arg->reason = GLFS_UPCALL_RECALL_LEASE;
+    up_arg->event = up_lease_arg;
+    up_arg->free_event = glfs_free_upcall_lease;
+
+    ret = 0;
 
-        return ret;
+out:
+    if (ret) {
+        /* Close p_object and oldp_object as well if being referenced.*/
+        if (object)
+            glfs_h_close(object);
+
+        /* Set reason to prevent applications from using ->event */
+        up_arg->reason = GF_UPCALL_EVENT_NULL;
+    }
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_zerofill, 3.5.0);
-
+static int
+upcall_syncop_args_free(struct upcall_syncop_args *args)
+{
+    dict_t *dict = NULL;
+    struct gf_upcall *upcall_data = NULL;
+
+    if (args) {
+        upcall_data = &args->upcall_data;
+        switch (upcall_data->event_type) {
+            case GF_UPCALL_CACHE_INVALIDATION:
+                dict = ((struct gf_upcall_cache_invalidation *)(upcall_data
+                                                                    ->data))
+                           ->dict;
+                break;
+            case GF_UPCALL_RECALL_LEASE:
+                dict = ((struct gf_upcall_recall_lease *)(upcall_data->data))
+                           ->dict;
+                break;
+        }
+        if (dict)
+            dict_unref(dict);
+
+        GF_FREE(upcall_data->client_uid);
+        GF_FREE(upcall_data->data);
+    }
+    GF_FREE(args);
+    return 0;
+}
 
-int
-pub_glfs_chdir (struct glfs *fs, const char *path)
+static int
+glfs_upcall_syncop_cbk(int ret, call_frame_t *frame, void *opaque)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	loc_t            loc = {0, };
-	struct iatt      iatt = {0, };
-	int              reval = 0;
+    struct upcall_syncop_args *args = opaque;
 
-	__glfs_entry_fs (fs);
+    (void)upcall_syncop_args_free(args);
 
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
-retry:
-	ret = glfs_resolve (fs, subvol, path, &loc, &iatt, reval);
+    return 0;
+}
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+static int
+glfs_cbk_upcall_syncop(void *opaque)
+{
+    struct upcall_syncop_args *args = opaque;
+    struct gf_upcall *upcall_data = NULL;
+    struct glfs_upcall *up_arg = NULL;
+    struct glfs *fs;
+    int ret = -1;
+
+    fs = args->fs;
+    upcall_data = &args->upcall_data;
+
+    if (!upcall_data) {
+        goto out;
+    }
+
+    up_arg = GLFS_CALLOC(1, sizeof(struct gf_upcall), glfs_release_upcall,
+                         glfs_mt_upcall_entry_t);
+    if (!up_arg) {
+        gf_smsg(THIS->name, GF_LOG_ERROR, ENOMEM, API_MSG_ALLOC_FAILED, "entry",
+                NULL);
+        goto out;
+    }
+
+    switch (upcall_data->event_type) {
+        case GF_UPCALL_CACHE_INVALIDATION:
+            ret = glfs_h_poll_cache_invalidation(fs, up_arg, upcall_data);
+            break;
+        case GF_UPCALL_RECALL_LEASE:
+            ret = glfs_recall_lease_upcall(fs, up_arg, upcall_data);
+            break;
+        default:
+            errno = EINVAL;
+    }
+
+    /* It could so happen that the file which got
+     * upcall notification may have got deleted by
+     * the same client. In such cases up_arg->reason
+     * is set to GLFS_UPCALL_EVENT_NULL. No need to
+     * send upcall then
+     */
+    if (up_arg->reason == GLFS_UPCALL_EVENT_NULL) {
+        gf_smsg(THIS->name, GF_LOG_DEBUG, errno,
+                API_MSG_UPCALL_EVENT_NULL_RECEIVED, NULL);
+        ret = 0;
+        GLFS_FREE(up_arg);
+        goto out;
+    } else if (ret) {
+        gf_smsg(THIS->name, GF_LOG_ERROR, errno, API_MSG_INVALID_ENTRY, NULL);
+        GLFS_FREE(up_arg);
+        goto out;
+    }
+
+    if (fs->up_cbk && up_arg)
+        (fs->up_cbk)(up_arg, fs->up_data);
+
+    /* application takes care of calling glfs_free on up_arg post
+     * their processing */
 
-	if (ret)
-		goto out;
+out:
+    return ret;
+}
 
-	if (!IA_ISDIR (iatt.ia_type)) {
-		ret = -1;
-		errno = ENOTDIR;
-		goto out;
-	}
+static struct gf_upcall_cache_invalidation *
+gf_copy_cache_invalidation(struct gf_upcall_cache_invalidation *src)
+{
+    struct gf_upcall_cache_invalidation *dst = NULL;
 
-	glfs_cwd_set (fs, loc.inode);
+    if (!src)
+        goto out;
 
-out:
-	loc_wipe (&loc);
+    dst = GF_CALLOC(1, sizeof(struct gf_upcall_cache_invalidation),
+                    glfs_mt_upcall_entry_t);
 
-	priv_glfs_subvol_done (fs, subvol);
+    if (!dst) {
+        gf_smsg(THIS->name, GF_LOG_ERROR, ENOMEM, API_MSG_ALLOC_FAILED, "entry",
+                NULL);
+        goto out;
+    }
 
-	return ret;
-}
+    dst->flags = src->flags;
+    dst->expire_time_attr = src->expire_time_attr;
+    dst->stat = src->stat;
+    dst->p_stat = src->p_stat;
+    dst->oldp_stat = src->oldp_stat;
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_chdir, 3.4.0);
+    if (src->dict)
+        dst->dict = dict_copy_with_ref(src->dict, NULL);
 
+    return dst;
+out:
+    return NULL;
+}
 
-int
-pub_glfs_fchdir (struct glfs_fd *glfd)
+static struct gf_upcall_recall_lease *
+gf_copy_recall_lease(struct gf_upcall_recall_lease *src)
 {
-	int       ret = -1;
-	inode_t  *inode = NULL;
-	xlator_t *subvol = NULL;
-	fd_t     *fd = NULL;
+    struct gf_upcall_recall_lease *dst = NULL;
 
-	__glfs_entry_fd (glfd);
+    if (!src)
+        goto out;
 
-	subvol = priv_glfs_active_subvol (glfd->fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    dst = GF_CALLOC(1, sizeof(struct gf_upcall_recall_lease),
+                    glfs_mt_upcall_entry_t);
 
-	fd = glfs_resolve_fd (glfd->fs, subvol, glfd);
-	if (!fd) {
-		ret = -1;
-		errno = EBADFD;
-		goto out;
-	}
+    if (!dst) {
+        gf_smsg(THIS->name, GF_LOG_ERROR, ENOMEM, API_MSG_ALLOC_FAILED, "entry",
+                NULL);
+        goto out;
+    }
 
-	inode = fd->inode;
+    dst->lease_type = src->lease_type;
+    memcpy(dst->tid, src->tid, 16);
 
-	if (!IA_ISDIR (inode->ia_type)) {
-		ret = -1;
-		errno = ENOTDIR;
-		goto out;
-	}
+    if (src->dict)
+        dst->dict = dict_copy_with_ref(src->dict, NULL);
 
-	glfs_cwd_set (glfd->fs, inode);
-	ret = 0;
+    return dst;
 out:
-	if (fd)
-		fd_unref (fd);
+    return NULL;
+}
 
-	priv_glfs_subvol_done (glfd->fs, subvol);
+static struct upcall_syncop_args *
+upcall_syncop_args_init(struct glfs *fs, struct gf_upcall *upcall_data)
+{
+    struct upcall_syncop_args *args = NULL;
+    int ret = -1;
+    struct gf_upcall *t_data = NULL;
+
+    if (!fs || !upcall_data)
+        goto out;
+
+    args = GF_CALLOC(1, sizeof(struct upcall_syncop_args),
+                     glfs_mt_upcall_entry_t);
+    if (!args) {
+        gf_smsg(THIS->name, GF_LOG_ERROR, ENOMEM, API_MSG_ALLOC_FAILED,
+                "syncop args", NULL);
+        goto out;
+    }
+
+    /* Note: we are not taking any ref on fs here.
+     * Ideally applications have to unregister for upcall events
+     * or stop polling for upcall events before performing
+     * glfs_fini. And as for outstanding synctasks created, we wait
+     * for all syncenv threads to finish tasks before cleaning up the
+     * fs->ctx. Hence it seems safe to process these callback
+     * notification without taking any lock/ref.
+     */
+    args->fs = fs;
+    t_data = &(args->upcall_data);
+    t_data->client_uid = gf_strdup(upcall_data->client_uid);
+
+    gf_uuid_copy(t_data->gfid, upcall_data->gfid);
+    t_data->event_type = upcall_data->event_type;
+
+    switch (t_data->event_type) {
+        case GF_UPCALL_CACHE_INVALIDATION:
+            t_data->data = gf_copy_cache_invalidation(
+                (struct gf_upcall_cache_invalidation *)upcall_data->data);
+            break;
+        case GF_UPCALL_RECALL_LEASE:
+            t_data->data = gf_copy_recall_lease(
+                (struct gf_upcall_recall_lease *)upcall_data->data);
+            break;
+    }
+
+    if (!t_data->data)
+        goto out;
+
+    return args;
+out:
+    if (ret) {
+        if (args) {
+            GF_FREE(args->upcall_data.client_uid);
+            GF_FREE(args);
+        }
+    }
 
-	return ret;
+    return NULL;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fchdir, 3.4.0);
+static void
+glfs_cbk_upcall_data(struct glfs *fs, struct gf_upcall *upcall_data)
+{
+    struct upcall_syncop_args *args = NULL;
+    int ret = -1;
 
+    if (!fs || !upcall_data)
+        goto out;
 
-char *
-pub_glfs_realpath (struct glfs *fs, const char *path, char *resolved_path)
-{
-	int              ret = -1;
-	char            *retpath = NULL;
-	char            *allocpath = NULL;
-	xlator_t        *subvol = NULL;
-	loc_t            loc = {0, };
-	struct iatt      iatt = {0, };
-	int              reval = 0;
-
-	__glfs_entry_fs (fs);
-
-	if (resolved_path)
-		retpath = resolved_path;
-	else
-		retpath = allocpath = malloc (PATH_MAX + 1);
-
-	if (!retpath) {
-		ret = -1;
-		errno = ENOMEM;
-		goto out;
-	}
-
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
-retry:
-	ret = glfs_resolve (fs, subvol, path, &loc, &iatt, reval);
+    if (!(fs->upcall_events & upcall_data->event_type)) {
+        /* ignore events which application hasn't registered*/
+        goto out;
+    }
 
-	ESTALE_RETRY (ret, errno, reval, &loc, retry);
+    args = upcall_syncop_args_init(fs, upcall_data);
 
-	if (ret)
-		goto out;
+    if (!args)
+        goto out;
 
-	if (loc.path) {
-		strncpy (retpath, loc.path, PATH_MAX);
-		retpath[PATH_MAX] = 0;
-	}
+    ret = synctask_new(THIS->ctx->env, glfs_cbk_upcall_syncop,
+                       glfs_upcall_syncop_cbk, NULL, args);
+    /* should we retry incase of failure? */
+    if (ret) {
+        gf_smsg(THIS->name, GF_LOG_ERROR, errno, API_MSG_UPCALL_SYNCOP_FAILED,
+                "event_type=%d", upcall_data->event_type, "gfid=%s",
+                (char *)(upcall_data->gfid), NULL);
+        upcall_syncop_args_free(args);
+    }
 
 out:
-	loc_wipe (&loc);
-
-	if (ret == -1) {
-		if (allocpath)
-			free (allocpath);
-		retpath = NULL;
-	}
+    return;
+}
 
-	priv_glfs_subvol_done (fs, subvol);
+/*
+ * This routine is called in case of any notification received
+ * from the server. All the upcall events are queued up in a list
+ * to be read by the applications.
+ *
+ * In case if the application registers a cbk function, that shall
+ * be called by this routine in case of any event received.
+ * The cbk fn is responsible for notifying the
+ * applications the way it desires for each event queued (for eg.,
+ * can raise a signal or broadcast a cond variable etc.)
+ *
+ * Otherwise all the upcall events are queued up in a list
+ * to be read/polled by the applications.
+ */
+GFAPI_SYMVER_PRIVATE_DEFAULT(glfs_process_upcall_event, 3.7.0)
+void
+priv_glfs_process_upcall_event(struct glfs *fs, void *data)
+{
+    glusterfs_ctx_t *ctx = NULL;
+    struct gf_upcall *upcall_data = NULL;
 
-	return retpath;
-}
+    DECLARE_OLD_THIS;
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_realpath, 3.4.0);
+    gf_msg_debug(THIS->name, 0, "Upcall gfapi callback is called");
 
+    __GLFS_ENTRY_VALIDATE_FS(fs, err);
 
-char *
-pub_glfs_getcwd (struct glfs *fs, char *buf, size_t n)
-{
-	int              ret = -1;
-	inode_t         *inode = NULL;
-	char            *path = NULL;
+    if (!data)
+        goto out;
 
-	__glfs_entry_fs (fs);
+    /* Unlike in I/O path, "glfs_fini" would not have freed
+     * 'fs' by the time we take lock as it waits for all epoll
+     * threads to exit including this
+     */
+    pthread_mutex_lock(&fs->mutex);
+    {
+        ctx = fs->ctx;
 
-	if (!buf || n < 2) {
-		ret = -1;
-		errno = EINVAL;
-		goto out;
-	}
+        /* if we're not interested in upcalls (anymore), skip them */
+        if (ctx->cleanup_started || !fs->cache_upcalls) {
+            pthread_mutex_unlock(&fs->mutex);
+            goto out;
+        }
 
-	inode = glfs_cwd_get (fs);
+        fs->pin_refcnt++;
+    }
+    pthread_mutex_unlock(&fs->mutex);
+
+    upcall_data = (struct gf_upcall *)data;
+
+    gf_msg_trace(THIS->name, 0, "Upcall gfapi gfid = %s",
+                 (char *)(upcall_data->gfid));
+
+    /* *
+     * TODO: RECALL LEASE for each glfd
+     *
+     * In case of RECALL_LEASE, we could associate separate
+     * cbk function for each glfd either by
+     * - extending pub_glfs_lease to accept new args (recall_cbk_fn, cookie)
+     * - or by defining new API "glfs_register_recall_cbk_fn (glfd,
+     * recall_cbk_fn, cookie) . In such cases, flag it and instead of calling
+     * below upcall functions, define a new one to go through the glfd list and
+     * invoke each of theirs recall_cbk_fn.
+     * */
+
+    if (fs->up_cbk) { /* upcall cbk registered */
+        (void)glfs_cbk_upcall_data(fs, upcall_data);
+    } else {
+        (void)glfs_enqueue_upcall_data(fs, upcall_data);
+    }
+
+    pthread_mutex_lock(&fs->mutex);
+    {
+        fs->pin_refcnt--;
+    }
+    pthread_mutex_unlock(&fs->mutex);
 
-	if (!inode) {
-		strncpy (buf, "/", n);
-		ret = 0;
-		goto out;
-	}
+out:
+    __GLFS_EXIT_FS;
+err:
+    return;
+}
 
-	ret = inode_path (inode, 0, &path);
-	if (n <= ret) {
-		ret = -1;
-		errno = ERANGE;
-		goto out;
-	}
+ssize_t
+glfs_anonymous_pwritev(struct glfs *fs, struct glfs_object *object,
+                       const struct iovec *iovec, int iovcnt, off_t offset,
+                       int flags)
+{
+    xlator_t *subvol = NULL;
+    struct iobref *iobref = NULL;
+    struct iobuf *iobuf = NULL;
+    struct iovec iov = {
+        0,
+    };
+    inode_t *inode = NULL;
+    fd_t *fd = NULL;
+    int ret = -1;
+    size_t size = -1;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    inode = glfs_resolve_inode(fs, subvol, object);
+    if (!inode) {
+        ret = -1;
+        errno = ESTALE;
+        goto out;
+    }
+
+    fd = fd_anonymous(inode);
+    if (!fd) {
+        ret = -1;
+        gf_smsg("gfapi", GF_LOG_ERROR, ENOMEM, API_MSG_FDCREATE_FAILED, NULL);
+        errno = ENOMEM;
+        goto out;
+    }
+
+    size = iov_length(iovec, iovcnt);
+
+    iobuf = iobuf_get2(subvol->ctx->iobuf_pool, size);
+    if (!iobuf) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    iobref = iobref_new();
+    if (!iobref) {
+        iobuf_unref(iobuf);
+        errno = ENOMEM;
+        ret = -1;
+        goto out;
+    }
+
+    ret = iobref_add(iobref, iobuf);
+    if (ret) {
+        iobuf_unref(iobuf);
+        iobref_unref(iobref);
+        errno = ENOMEM;
+        ret = -1;
+        goto out;
+    }
+
+    iov_unload(iobuf_ptr(iobuf), iovec, iovcnt);
+
+    iov.iov_base = iobuf_ptr(iobuf);
+    iov.iov_len = size;
+
+    /* TODO : set leaseid */
+    ret = syncop_writev(subvol, fd, &iov, 1, offset, iobref, flags, NULL, NULL,
+                        NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    iobuf_unref(iobuf);
+    iobref_unref(iobref);
+
+    if (ret <= 0)
+        goto out;
 
-	strncpy (buf, path, n);
-	ret = 0;
 out:
-	GF_FREE (path);
 
-	if (inode)
-		inode_unref (inode);
+    if (fd)
+        fd_unref(fd);
 
-	if (ret < 0)
-		return NULL;
+    if (inode)
+        inode_unref(inode);
 
-	return buf;
-}
+    glfs_subvol_done(fs, subvol);
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_getcwd, 3.4.0);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
-static void
-gf_flock_to_flock (struct gf_flock *gf_flock, struct flock *flock)
+ssize_t
+glfs_anonymous_preadv(struct glfs *fs, struct glfs_object *object,
+                      const struct iovec *iovec, int iovcnt, off_t offset,
+                      int flags)
 {
-	flock->l_type   = gf_flock->l_type;
-	flock->l_whence = gf_flock->l_whence;
-	flock->l_start  = gf_flock->l_start;
-	flock->l_len    = gf_flock->l_len;
-	flock->l_pid    = gf_flock->l_pid;
-}
+    xlator_t *subvol = NULL;
+    struct iovec *iov = NULL;
+    struct iobref *iobref = NULL;
+    inode_t *inode = NULL;
+    fd_t *fd = NULL;
+    int cnt = 0;
+    ssize_t ret = -1;
+    ssize_t size = -1;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    inode = glfs_resolve_inode(fs, subvol, object);
+    if (!inode) {
+        ret = -1;
+        errno = ESTALE;
+        goto out;
+    }
+
+    fd = fd_anonymous(inode);
+    if (!fd) {
+        ret = -1;
+        gf_smsg("gfapi", GF_LOG_ERROR, ENOMEM, API_MSG_FDCREATE_FAILED, NULL);
+        errno = ENOMEM;
+        goto out;
+    }
+
+    size = iov_length(iovec, iovcnt);
+
+    /* TODO : set leaseid */
+    ret = syncop_readv(subvol, fd, size, offset, flags, &iov, &cnt, &iobref,
+                       NULL, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+    if (ret <= 0)
+        goto out;
+
+    size = iov_copy(iovec, iovcnt, iov, cnt);
+
+    ret = size;
+out:
+    if (iov)
+        GF_FREE(iov);
+    if (iobref)
+        iobref_unref(iobref);
+    if (fd)
+        fd_unref(fd);
+
+    if (inode)
+        inode_unref(inode);
+
+    glfs_subvol_done(fs, subvol);
 
+    __GLFS_EXIT_FS;
+
+invalid_fs:
+    return ret;
+}
 
 static void
-gf_flock_from_flock (struct gf_flock *gf_flock, struct flock *flock)
+glfs_release_xreaddirp_stat(void *ptr)
 {
-	gf_flock->l_type   = flock->l_type;
-	gf_flock->l_whence = flock->l_whence;
-	gf_flock->l_start  = flock->l_start;
-	gf_flock->l_len    = flock->l_len;
-	gf_flock->l_pid    = flock->l_pid;
-}
+    struct glfs_xreaddirp_stat *to_free = ptr;
 
+    if (to_free->object)
+        glfs_h_close(to_free->object);
+}
 
+/*
+ * Given glfd of a directory, this function does readdirp and returns
+ * xstat along with dirents.
+ */
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_xreaddirplus_r, 3.11.0)
 int
-pub_glfs_posix_lock (struct glfs_fd *glfd, int cmd, struct flock *flock)
+pub_glfs_xreaddirplus_r(struct glfs_fd *glfd, uint32_t flags,
+                        struct glfs_xreaddirp_stat **xstat_p,
+                        struct dirent *ext, struct dirent **res)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	struct gf_flock  gf_flock = {0, };
-	struct gf_flock  saved_flock = {0, };
-	fd_t            *fd = NULL;
+    int ret = -1;
+    gf_dirent_t *entry = NULL;
+    struct dirent *buf = NULL;
+    struct glfs_xreaddirp_stat *xstat = NULL;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    GF_REF_GET(glfd);
+
+    GF_VALIDATE_OR_GOTO(THIS->name, xstat_p, out);
+    GF_VALIDATE_OR_GOTO(THIS->name, res, out);
+
+    errno = 0;
+
+    if (ext)
+        buf = ext;
+    else
+        buf = glfs_readdirbuf_get(glfd);
+
+    if (!buf)
+        goto out;
+
+    xstat = GLFS_CALLOC(1, sizeof(struct glfs_xreaddirp_stat),
+                        glfs_release_xreaddirp_stat, glfs_mt_xreaddirp_stat_t);
+
+    if (!xstat)
+        goto out;
+
+    /* this is readdirplus operation */
+    entry = glfd_entry_next(glfd, 1);
+
+    /* XXX: Ideally when we reach EOD, errno should have been
+     * set to ENOENT. But that doesn't seem to be the case.
+     *
+     * The only way to confirm if its EOD at this point is that
+     * errno == 0 and entry == NULL
+     */
+    if (errno)
+        goto out;
+
+    if (!entry) {
+        /* reached EOD, ret = 0  */
+        ret = 0;
+        *res = NULL;
+        *xstat_p = NULL;
+
+        /* free xstat as applications shall not be using it */
+        GLFS_FREE(xstat);
+
+        goto out;
+    }
+
+    *res = buf;
+    gf_dirent_to_dirent(entry, buf);
+
+    if (flags & GFAPI_XREADDIRP_STAT) {
+        glfs_iatt_to_stat(glfd->fs, &entry->d_stat, &xstat->st);
+        xstat->flags_handled |= GFAPI_XREADDIRP_STAT;
+    }
+
+    if ((flags & GFAPI_XREADDIRP_HANDLE) &&
+        /* skip . and .. */
+        strcmp(buf->d_name, ".") && strcmp(buf->d_name, "..")) {
+        /* Now create object.
+         * We can use "glfs_h_find_handle" as well as inodes would have
+         * already got linked as part of 'gf_link_inodes_from_dirent' */
+        xstat->object = glfs_h_create_from_handle(
+            glfd->fs, entry->d_stat.ia_gfid, GFAPI_HANDLE_LENGTH, NULL);
+
+        if (xstat->object) { /* success */
+            /* note: xstat->object->inode->ref is taken
+             * This shall be unref'ed when application does
+             * glfs_free(xstat) */
+            xstat->flags_handled |= GFAPI_XREADDIRP_HANDLE;
+        }
+    }
 
-	__glfs_entry_fd (glfd);
+    ret = xstat->flags_handled;
+    *xstat_p = xstat;
 
-	subvol = priv_glfs_active_subvol (glfd->fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+    gf_msg_debug(THIS->name, 0,
+                 "xreaddirp- requested_flags (%x) , processed_flags (%x)",
+                 flags, xstat->flags_handled);
 
-	fd = glfs_resolve_fd (glfd->fs, subvol, glfd);
-	if (!fd) {
-		ret = -1;
-		errno = EBADFD;
-		goto out;
-	}
+out:
+    GF_REF_PUT(glfd);
 
-	gf_flock_from_flock (&gf_flock, flock);
-	gf_flock_from_flock (&saved_flock, flock);
-	ret = syncop_lk (subvol, fd, cmd, &gf_flock);
-        DECODE_SYNCOP_ERR (ret);
-	gf_flock_to_flock (&gf_flock, flock);
+    if (ret < 0) {
+        gf_smsg(THIS->name, GF_LOG_WARNING, errno, API_MSG_XREADDIRP_R_FAILED,
+                "reason=%s", strerror(errno), NULL);
 
-	if (ret == 0 && (cmd == F_SETLK || cmd == F_SETLKW))
-		fd_lk_insert_and_merge (fd, cmd, &saved_flock);
-out:
-	if (fd)
-		fd_unref (fd);
+        if (xstat)
+            GLFS_FREE(xstat);
+    }
 
-	priv_glfs_subvol_done (glfd->fs, subvol);
+    __GLFS_EXIT_FS;
 
-	return ret;
+    return ret;
+
+invalid_fs:
+    return -1;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_posix_lock, 3.4.0);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_xreaddirplus_get_stat, 3.11.0)
+struct stat *
+pub_glfs_xreaddirplus_get_stat(struct glfs_xreaddirp_stat *xstat)
+{
+    GF_VALIDATE_OR_GOTO("glfs_xreaddirplus_get_stat", xstat, out);
 
+    if (!xstat->flags_handled & GFAPI_XREADDIRP_STAT)
+        gf_smsg(THIS->name, GF_LOG_ERROR, errno, API_MSG_FLAGS_HANDLE,
+                "GFAPI_XREADDIRP_STAT"
+                "xstat=%p",
+                xstat, "handles=%x", xstat->flags_handled, NULL);
+    return &xstat->st;
 
-struct glfs_fd *
-pub_glfs_dup (struct glfs_fd *glfd)
+out:
+    return NULL;
+}
+
+void
+gf_lease_to_glfs_lease(struct gf_lease *gf_lease, struct glfs_lease *lease)
 {
-	xlator_t  *subvol = NULL;
-	fd_t      *fd = NULL;
-	glfs_fd_t *dupfd = NULL;
-	struct glfs *fs = NULL;
+    u_int lease_type = gf_lease->lease_type;
+    lease->cmd = gf_lease->cmd;
+    lease->lease_type = lease_type;
+    memcpy(lease->lease_id, gf_lease->lease_id, LEASE_ID_SIZE);
+}
 
-	__glfs_entry_fd (glfd);
+void
+glfs_lease_to_gf_lease(struct glfs_lease *lease, struct gf_lease *gf_lease)
+{
+    u_int lease_type = lease->lease_type;
+    gf_lease->cmd = lease->cmd;
+    gf_lease->lease_type = lease_type;
+    memcpy(gf_lease->lease_id, lease->lease_id, LEASE_ID_SIZE);
+}
 
-	fs = glfd->fs;
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		errno = EIO;
-		goto out;
-	}
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_lease, 4.0.0)
+int
+pub_glfs_lease(struct glfs_fd *glfd, struct glfs_lease *lease,
+               glfs_recall_cbk fn, void *data)
+{
+    int ret = -1;
+    loc_t loc = {
+        0,
+    };
+    xlator_t *subvol = NULL;
+    fd_t *fd = NULL;
+    struct gf_lease gf_lease = {
+        0,
+    };
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FD(glfd, invalid_fs);
+
+    GF_REF_GET(glfd);
+
+    if (!is_valid_lease_id(lease->lease_id)) {
+        ret = -1;
+        errno = EINVAL;
+        goto out;
+    }
+
+    subvol = glfs_active_subvol(glfd->fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    fd = glfs_resolve_fd(glfd->fs, subvol, glfd);
+    if (!fd) {
+        ret = -1;
+        errno = EBADFD;
+        goto out;
+    }
+
+    switch (lease->lease_type) {
+        case GLFS_RD_LEASE:
+            if ((fd->flags != O_RDONLY) && !(fd->flags & O_RDWR)) {
+                ret = -1;
+                errno = EINVAL;
+                goto out;
+            }
+            break;
+        case GLFS_RW_LEASE:
+            if (!((fd->flags & O_WRONLY) || (fd->flags & O_RDWR))) {
+                ret = -1;
+                errno = EINVAL;
+                goto out;
+            }
+            break;
+        default:
+            if (lease->cmd != GLFS_GET_LEASE) {
+                ret = -1;
+                errno = EINVAL;
+                goto out;
+            }
+            break;
+    }
+
+    /* populate loc */
+    GLFS_LOC_FILL_INODE(fd->inode, loc, out);
+
+    glfs_lease_to_gf_lease(lease, &gf_lease);
 
-	fd = glfs_resolve_fd (fs, subvol, glfd);
-	if (!fd) {
-		errno = EBADFD;
-		goto out;
-	}
+    ret = syncop_lease(subvol, &loc, &gf_lease, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
 
-	dupfd = glfs_fd_new (fs);
-	if (!dupfd) {
-		errno = ENOMEM;
-		goto out;
-	}
+    gf_lease_to_glfs_lease(&gf_lease, lease);
+
+    /* TODO: Add leases for client replay
+    if (ret == 0 && (cmd == F_SETLK || cmd == F_SETLKW))
+            fd_lk_insert_and_merge (fd, cmd, &saved_flock);
+    */
+    if (ret == 0) {
+        ret = fd_ctx_set(glfd->fd, subvol, (uint64_t)(long)glfd);
+        if (ret) {
+            gf_smsg(subvol->name, GF_LOG_ERROR, ENOMEM,
+                    API_MSG_FDCTX_SET_FAILED, "fd=%p", glfd->fd, NULL);
+            goto out;
+        }
+        glfd->cbk = fn;
+        glfd->cookie = data;
+    }
 
-	dupfd->fd = fd_ref (fd);
 out:
-	if (fd)
-		fd_unref (fd);
-	if (dupfd)
-		glfs_fd_bind (dupfd);
 
-	priv_glfs_subvol_done (fs, subvol);
+    if (glfd)
+        GF_REF_PUT(glfd);
 
-	return dupfd;
-}
+    if (subvol)
+        glfs_subvol_done(glfd->fs, subvol);
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_dup, 3.4.0);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
diff --git a/api/src/glfs-handleops.c b/api/src/glfs-handleops.c
index dadbda3675f..53c2ee896f9 100644
--- a/api/src/glfs-handleops.c
+++ b/api/src/glfs-handleops.c
@@ -1,5 +1,5 @@
 /*
- *  Copyright (c) 2013 Red Hat, Inc. <http://www.redhat.com>
+ *  Copyright (c) 2013-2018 Red Hat, Inc. <http://www.redhat.com>
  *  This file is part of GlusterFS.
  *
  *  This file is licensed to you under your choice of the GNU Lesser
@@ -8,1592 +8,2648 @@
  *  cases as published by the Free Software Foundation.
  */
 
-
 #include "glfs-internal.h"
 #include "glfs-mem-types.h"
-#include "syncop.h"
+#include <glusterfs/syncop.h>
 #include "glfs.h"
 #include "glfs-handles.h"
+#include "gfapi-messages.h"
 
 int
-glfs_listxattr_process (void *value, size_t size, dict_t *xattr);
+glfs_listxattr_process(void *value, size_t size, dict_t *xattr);
 
-static void
-glfs_iatt_from_stat (struct stat *stat, int valid, struct iatt *iatt,
-                     int *glvalid)
+void
+glfs_iatt_from_stat(struct stat *stat, int valid, struct iatt *iatt,
+                    int *glvalid)
 {
-        /* validate in args */
-        if ((stat == NULL) || (iatt == NULL) || (glvalid == NULL)) {
-                errno = EINVAL;
-                return;
-        }
-
-        *glvalid = 0;
-
-        if (valid & GFAPI_SET_ATTR_MODE) {
-                iatt->ia_prot = ia_prot_from_st_mode (stat->st_mode);
-                *glvalid |= GF_SET_ATTR_MODE;
-        }
-
-        if (valid & GFAPI_SET_ATTR_UID) {
-                iatt->ia_uid = stat->st_uid;
-                *glvalid |= GF_SET_ATTR_UID;
-        }
-
-        if (valid & GFAPI_SET_ATTR_GID) {
-                iatt->ia_gid = stat->st_gid;
-                *glvalid |= GF_SET_ATTR_GID;
-        }
-
-        if (valid & GFAPI_SET_ATTR_ATIME) {
-                iatt->ia_atime = stat->st_atime;
-                iatt->ia_atime_nsec = ST_ATIM_NSEC (stat);
-                *glvalid |= GF_SET_ATTR_ATIME;
-        }
-
-        if (valid & GFAPI_SET_ATTR_MTIME) {
-                iatt->ia_mtime = stat->st_mtime;
-                iatt->ia_mtime_nsec = ST_MTIM_NSEC (stat);
-                *glvalid |= GF_SET_ATTR_MTIME;
-        }
-
+    /* validate in args */
+    if ((stat == NULL) || (iatt == NULL) || (glvalid == NULL)) {
+        errno = EINVAL;
         return;
+    }
+
+    *glvalid = 0;
+
+    if (valid & GFAPI_SET_ATTR_MODE) {
+        iatt->ia_prot = ia_prot_from_st_mode(stat->st_mode);
+        *glvalid |= GF_SET_ATTR_MODE;
+    }
+
+    if (valid & GFAPI_SET_ATTR_UID) {
+        iatt->ia_uid = stat->st_uid;
+        *glvalid |= GF_SET_ATTR_UID;
+    }
+
+    if (valid & GFAPI_SET_ATTR_GID) {
+        iatt->ia_gid = stat->st_gid;
+        *glvalid |= GF_SET_ATTR_GID;
+    }
+
+    if (valid & GFAPI_SET_ATTR_ATIME) {
+        iatt->ia_atime = stat->st_atime;
+        iatt->ia_atime_nsec = ST_ATIM_NSEC(stat);
+        *glvalid |= GF_SET_ATTR_ATIME;
+    }
+
+    if (valid & GFAPI_SET_ATTR_MTIME) {
+        iatt->ia_mtime = stat->st_mtime;
+        iatt->ia_mtime_nsec = ST_MTIM_NSEC(stat);
+        *glvalid |= GF_SET_ATTR_MTIME;
+    }
+
+    return;
 }
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_lookupat, 3.7.4)
 struct glfs_object *
-pub_glfs_h_lookupat (struct glfs *fs, struct glfs_object *parent,
-                     const char *path, struct stat *stat)
+pub_glfs_h_lookupat(struct glfs *fs, struct glfs_object *parent,
+                    const char *path, struct stat *stat, int follow)
 {
-        int                      ret = 0;
-        xlator_t                *subvol = NULL;
-        inode_t                 *inode = NULL;
-        struct iatt              iatt = {0, };
-        struct glfs_object      *object = NULL;
-        loc_t                    loc = {0, };
-
-        /* validate in args */
-        if ((fs == NULL) || (path == NULL)) {
-                errno = EINVAL;
-                return NULL;
-        }
-
-        __glfs_entry_fs (fs);
-
-        /* get the active volume */
-        subvol = priv_glfs_active_subvol (fs);
-        if (!subvol) {
-                errno = EIO;
-                goto out;
+    int ret = 0;
+    xlator_t *subvol = NULL;
+    inode_t *inode = NULL;
+    struct iatt iatt = {
+        0,
+    };
+    struct glfs_object *object = NULL;
+    loc_t loc = {
+        0,
+    };
+
+    DECLARE_OLD_THIS;
+
+    /* validate in args */
+    if (path == NULL) {
+        errno = EINVAL;
+        return NULL;
+    }
+
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        errno = EIO;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    if (parent) {
+        inode = glfs_resolve_inode(fs, subvol, parent);
+        if (!inode) {
+            errno = ESTALE;
+            goto out;
         }
+    }
 
-        /* get/refresh the in arg objects inode in correlation to the xlator */
-        if (parent) {
-                inode = glfs_resolve_inode (fs, subvol, parent);
-                if (!inode) {
-                        errno = ESTALE;
-                        goto out;
-                }
-        }
+    /* fop/op */
+    ret = glfs_resolve_at(fs, subvol, inode, path, &loc, &iatt, follow, 0);
 
-        /* fop/op */
-        ret = priv_glfs_resolve_at (fs, subvol, inode, path, &loc, &iatt,
-                                    0 /*TODO: links? */, 0);
+    /* populate out args */
+    if (!ret) {
+        if (stat)
+            glfs_iatt_to_stat(fs, &iatt, stat);
 
-        /* populate out args */
-        if (!ret) {
-                if (stat)
-                        glfs_iatt_to_stat (fs, &iatt, stat);
-
-                ret = glfs_create_object (&loc, &object);
-        }
+        ret = glfs_create_object(&loc, &object);
+    }
 
 out:
-        loc_wipe (&loc);
+    loc_wipe(&loc);
 
-        if (inode)
-                inode_unref (inode);
+    if (inode)
+        inode_unref(inode);
 
-        priv_glfs_subvol_done (fs, subvol);
+    glfs_subvol_done(fs, subvol);
 
-        return object;
-}
+    __GLFS_EXIT_FS;
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_lookupat, 3.4.2);
+invalid_fs:
+    return object;
+}
 
+GFAPI_SYMVER_PUBLIC(glfs_h_lookupat34, glfs_h_lookupat, 3.4.2)
+struct glfs_object *
+pub_glfs_h_lookupat34(struct glfs *fs, struct glfs_object *parent,
+                      const char *path, struct stat *stat)
+{
+    return pub_glfs_h_lookupat(fs, parent, path, stat, 0);
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_statfs, 3.7.0)
 int
-pub_glfs_h_stat (struct glfs *fs, struct glfs_object *object, struct stat *stat)
+pub_glfs_h_statfs(struct glfs *fs, struct glfs_object *object,
+                  struct statvfs *statvfs)
 {
-        int              ret = -1;
-        xlator_t        *subvol = NULL;
-        inode_t         *inode = NULL;
-        loc_t            loc = {0, };
-        struct iatt      iatt = {0, };
-
-        /* validate in args */
-        if ((fs == NULL) || (object == NULL)) {
-                errno = EINVAL;
-                return -1;
-        }
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    inode_t *inode = NULL;
+    loc_t loc = {
+        0,
+    };
+
+    DECLARE_OLD_THIS;
+
+    /* validate in args */
+    if ((fs == NULL) || (object == NULL || statvfs == NULL)) {
+        errno = EINVAL;
+        return -1;
+    }
+
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    inode = glfs_resolve_inode(fs, subvol, object);
+    if (!inode) {
+        errno = ESTALE;
+        goto out;
+    }
+
+    /* populate loc */
+    GLFS_LOC_FILL_INODE(inode, loc, out);
+
+    /* fop/op */
+    ret = syncop_statfs(subvol, &loc, statvfs, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    loc_wipe(&loc);
 
-        __glfs_entry_fs (fs);
-
-        /* get the active volume */
-        subvol = priv_glfs_active_subvol (fs);
-        if (!subvol) {
-                ret = -1;
-                errno = EIO;
-                goto out;
-        }
+out:
+    if (inode)
+        inode_unref(inode);
 
-        /* get/refresh the in arg objects inode in correlation to the xlator */
-        inode = glfs_resolve_inode (fs, subvol, object);
-        if (!inode) {
-                errno = ESTALE;
-                goto out;
-        }
+    glfs_subvol_done(fs, subvol);
 
-        /* populate loc */
-        GLFS_LOC_FILL_INODE (inode, loc, out);
+    __GLFS_EXIT_FS;
 
-        /* fop/op */
-        ret = syncop_stat (subvol, &loc, &iatt);
-        DECODE_SYNCOP_ERR (ret);
+invalid_fs:
+    return ret;
+}
 
-        /* populate out args */
-        if (!ret && stat) {
-                glfs_iatt_to_stat (fs, &iatt, stat);
-        }
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_stat, 3.4.2)
+int
+pub_glfs_h_stat(struct glfs *fs, struct glfs_object *object, struct stat *stat)
+{
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    inode_t *inode = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+
+    DECLARE_OLD_THIS;
+
+    /* validate in args */
+    if ((fs == NULL) || (object == NULL)) {
+        errno = EINVAL;
+        return -1;
+    }
+
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    inode = glfs_resolve_inode(fs, subvol, object);
+    if (!inode) {
+        errno = ESTALE;
+        goto out;
+    }
+
+    /* populate loc */
+    GLFS_LOC_FILL_INODE(inode, loc, out);
+
+    /* fop/op */
+    ret = syncop_stat(subvol, &loc, &iatt, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    /* populate out args */
+    if (!ret && stat) {
+        glfs_iatt_to_stat(fs, &iatt, stat);
+    }
 out:
-        loc_wipe (&loc);
+    loc_wipe(&loc);
 
-        if (inode)
-                inode_unref (inode);
+    if (inode)
+        inode_unref(inode);
 
-        priv_glfs_subvol_done (fs, subvol);
-
-        return ret;
-}
+    glfs_subvol_done(fs, subvol);
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_stat, 3.4.2);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_getattrs, 3.4.2)
 int
-pub_glfs_h_getattrs (struct glfs *fs, struct glfs_object *object,
-                     struct stat *stat)
+pub_glfs_h_getattrs(struct glfs *fs, struct glfs_object *object,
+                    struct stat *stat)
 {
-        int                      ret = 0;
-        xlator_t                *subvol = NULL;
-        inode_t                 *inode = NULL;
-        struct iatt              iatt = {0, };
-
-        /* validate in args */
-        if ((fs == NULL) || (object == NULL)) {
-                errno = EINVAL;
-                return -1;
-        }
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    inode_t *inode = NULL;
+    struct iatt iatt = {
+        0,
+    };
+
+    /* validate in args */
+    if ((fs == NULL) || (object == NULL)) {
+        errno = EINVAL;
+        return -1;
+    }
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    inode = glfs_resolve_inode(fs, subvol, object);
+    if (!inode) {
+        ret = 0;
+        errno = ESTALE;
+        goto out;
+    }
+
+    /* fop/op */
+    ret = glfs_resolve_base(fs, subvol, inode, &iatt);
+
+    /* populate out args */
+    if (!ret && stat) {
+        glfs_iatt_to_stat(fs, &iatt, stat);
+    }
 
-        __glfs_entry_fs (fs);
+out:
+    if (inode)
+        inode_unref(inode);
 
-        /* get the active volume */
-        subvol = priv_glfs_active_subvol (fs);
-        if (!subvol) {
-                ret = -1;
-                errno = EIO;
-                goto out;
-        }
+    glfs_subvol_done(fs, subvol);
 
-        /* get/refresh the in arg objects inode in correlation to the xlator */
-        inode = glfs_resolve_inode (fs, subvol, object);
-        if (!inode) {
-                errno = ESTALE;
-                goto out;
-        }
+    __GLFS_EXIT_FS;
 
-        /* fop/op */
-        ret = glfs_resolve_base (fs, subvol, inode, &iatt);
+invalid_fs:
+    return ret;
+}
 
-        /* populate out args */
-        if (!ret && stat) {
-                glfs_iatt_to_stat (fs, &iatt, stat);
-        }
+int
+glfs_h_getxattrs_common(struct glfs *fs, struct glfs_object *object,
+                        dict_t **xattr, const char *name,
+                        gf_boolean_t is_listxattr)
+{
+    int ret = 0;
+    xlator_t *subvol = NULL;
+    inode_t *inode = NULL;
+    loc_t loc = {
+        0,
+    };
+
+    /* validate in args */
+    if ((fs == NULL) || (object == NULL)) {
+        errno = EINVAL;
+        return -1;
+    }
+
+    if (!is_listxattr) {
+        if (!name || *name == '\0') {
+            errno = EINVAL;
+            return -1;
+        }
+
+        if (strlen(name) > GF_XATTR_NAME_MAX) {
+            errno = ENAMETOOLONG;
+            return -1;
+        }
+    }
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    inode = glfs_resolve_inode(fs, subvol, object);
+    if (!inode) {
+        errno = ESTALE;
+        goto out;
+    }
+
+    /* populate loc */
+    GLFS_LOC_FILL_INODE(inode, loc, out);
+
+    ret = syncop_getxattr(subvol, &loc, xattr, name, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
 
 out:
-        if (inode)
-                inode_unref (inode);
-
-        priv_glfs_subvol_done (fs, subvol);
+    loc_wipe(&loc);
 
-        return ret;
-}
+    if (inode)
+        inode_unref(inode);
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_getattrs, 3.4.2);
+    glfs_subvol_done(fs, subvol);
 
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_getxattrs, 3.5.1)
 int
-pub_glfs_h_getxattrs (struct glfs *fs, struct glfs_object *object,
-                      const char *name, void *value, size_t size)
+pub_glfs_h_getxattrs(struct glfs *fs, struct glfs_object *object,
+                     const char *name, void *value, size_t size)
 {
-        int                 ret = 0;
-        xlator_t        *subvol = NULL;
-        inode_t         *inode = NULL;
-        loc_t            loc = {0, };
-        dict_t                *xattr = NULL;
-
-        /* validate in args */
-        if ((fs == NULL) || (object == NULL)) {
-                errno = EINVAL;
-                return -1;
-        }
-
-        __glfs_entry_fs (fs);
-
-        /* get the active volume */
-        subvol = priv_glfs_active_subvol (fs);
-        if (!subvol) {
-                ret = -1;
-                errno = EIO;
-                goto out;
-        }
+    int ret = -1;
+    dict_t *xattr = NULL;
 
-        /* get/refresh the in arg objects inode in correlation to the xlator */
-        inode = glfs_resolve_inode (fs, subvol, object);
-        if (!inode) {
-                errno = ESTALE;
-                goto out;
-        }
-
-        /* populate loc */
-        GLFS_LOC_FILL_INODE (inode, loc, out);
+    /* validate in args */
+    if ((fs == NULL) || (object == NULL)) {
+        errno = EINVAL;
+        return -1;
+    }
 
-        ret = syncop_getxattr (subvol, &loc, &xattr, name);
-        DECODE_SYNCOP_ERR (ret);
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
 
-        if (ret)
-                goto out;
+    ret = glfs_h_getxattrs_common(fs, object, &xattr, name, (name == NULL));
+    if (ret)
+        goto out;
 
-        /* If @name is NULL, means get all the xattrs (i.e listxattr). */
-        if (name)
-                ret = glfs_getxattr_process (value, size, xattr, name);
-        else
-                ret = glfs_listxattr_process (value, size, xattr);
+    /* If @name is NULL, means get all the xattrs (i.e listxattr). */
+    if (name)
+        ret = glfs_getxattr_process(value, size, xattr, name);
+    else
+        ret = glfs_listxattr_process(value, size, xattr);
 
 out:
-        loc_wipe (&loc);
-
-        if (inode)
-                inode_unref (inode);
+    if (xattr)
+        dict_unref(xattr);
 
-        priv_glfs_subvol_done (fs, subvol);
+    __GLFS_EXIT_FS;
 
-        return ret;
+invalid_fs:
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_getxattrs, 3.5.1);
-
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_setattrs, 3.4.2)
 int
-pub_glfs_h_setattrs (struct glfs *fs, struct glfs_object *object,
-                     struct stat *stat, int valid)
+pub_glfs_h_setattrs(struct glfs *fs, struct glfs_object *object,
+                    struct stat *stat, int valid)
 {
-        int              ret = -1;
-        xlator_t        *subvol = NULL;
-        inode_t         *inode = NULL;
-        loc_t            loc = {0, };
-        struct iatt      iatt = {0, };
-        int              glvalid = 0;
-
-        /* validate in args */
-        if ((fs == NULL) || (object == NULL) || (stat == NULL)) {
-                errno = EINVAL;
-                return -1;
-        }
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    inode_t *inode = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    int glvalid = 0;
+
+    /* validate in args */
+    if ((fs == NULL) || (object == NULL) || (stat == NULL)) {
+        errno = EINVAL;
+        return -1;
+    }
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    inode = glfs_resolve_inode(fs, subvol, object);
+    if (!inode) {
+        errno = ESTALE;
+        goto out;
+    }
+
+    /* map valid masks from in args */
+    glfs_iatt_from_stat(stat, valid, &iatt, &glvalid);
+
+    /* populate loc */
+    GLFS_LOC_FILL_INODE(inode, loc, out);
+
+    /* fop/op */
+    ret = syncop_setattr(subvol, &loc, &iatt, glvalid, 0, 0, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+out:
+    loc_wipe(&loc);
 
-        __glfs_entry_fs (fs);
+    if (inode)
+        inode_unref(inode);
 
-        /* get the active volume */
-        subvol = priv_glfs_active_subvol (fs);
-        if (!subvol) {
-                ret = -1;
-                errno = EIO;
-                goto out;
-        }
+    glfs_subvol_done(fs, subvol);
 
-        /* get/refresh the in arg objects inode in correlation to the xlator */
-        inode = glfs_resolve_inode (fs, subvol, object);
-        if (!inode) {
-                errno = ESTALE;
-                goto out;
-        }
+    __GLFS_EXIT_FS;
 
-        /* map valid masks from in args */
-        glfs_iatt_from_stat (stat, valid, &iatt, &glvalid);
+invalid_fs:
+    return ret;
+}
 
-        /* populate loc */
-        GLFS_LOC_FILL_INODE (inode, loc, out);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_setxattrs, 3.5.0)
+int
+pub_glfs_h_setxattrs(struct glfs *fs, struct glfs_object *object,
+                     const char *name, const void *value, size_t size,
+                     int flags)
+{
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    inode_t *inode = NULL;
+    loc_t loc = {
+        0,
+    };
+    dict_t *xattr = NULL;
+    void *value_cp = NULL;
+
+    /* validate in args */
+    if ((fs == NULL) || (object == NULL) || (name == NULL) || (value == NULL)) {
+        errno = EINVAL;
+        return -1;
+    }
+
+    if (!name || *name == '\0') {
+        errno = EINVAL;
+        return -1;
+    }
+
+    if (strlen(name) > GF_XATTR_NAME_MAX) {
+        errno = ENAMETOOLONG;
+        return -1;
+    }
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    inode = glfs_resolve_inode(fs, subvol, object);
+    if (!inode) {
+        errno = ESTALE;
+        goto out;
+    }
+
+    value_cp = gf_memdup(value, size);
+    GF_CHECK_ALLOC_AND_LOG(subvol->name, value_cp, ret,
+                           "Failed to"
+                           " duplicate setxattr value",
+                           out);
+
+    xattr = dict_for_key_value(name, value_cp, size, _gf_false);
+    if (!xattr) {
+        GF_FREE(value_cp);
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    /* populate loc */
+    GLFS_LOC_FILL_INODE(inode, loc, out);
+
+    /* fop/op */
+    ret = syncop_setxattr(subvol, &loc, xattr, flags, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
 
-        /* fop/op */
-        ret = syncop_setattr (subvol, &loc, &iatt, glvalid, 0, 0);
-        DECODE_SYNCOP_ERR (ret);
 out:
-        loc_wipe (&loc);
+    loc_wipe(&loc);
 
-        if (inode)
-                inode_unref (inode);
+    if (inode)
+        inode_unref(inode);
 
-        priv_glfs_subvol_done (fs, subvol);
+    if (xattr)
+        dict_unref(xattr);
 
-        return ret;
-}
+    glfs_subvol_done(fs, subvol);
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_setattrs, 3.4.2);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_removexattrs, 3.5.1)
 int
-pub_glfs_h_setxattrs (struct glfs *fs, struct glfs_object *object,
-                      const char *name, const void *value, size_t size,
-                      int flags)
+pub_glfs_h_removexattrs(struct glfs *fs, struct glfs_object *object,
+                        const char *name)
 {
-        int              ret = -1;
-        xlator_t        *subvol = NULL;
-        inode_t         *inode = NULL;
-        loc_t            loc = {0, };
-        dict_t          *xattr = NULL;
-
-        /* validate in args */
-        if ((fs == NULL) || (object == NULL) ||
-                 (name == NULL) || (value == NULL)) {
-                errno = EINVAL;
-                return -1;
-        }
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    inode_t *inode = NULL;
+    loc_t loc = {
+        0,
+    };
+
+    /* validate in args */
+    if ((fs == NULL) || (object == NULL) || (name == NULL)) {
+        errno = EINVAL;
+        return -1;
+    }
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    inode = glfs_resolve_inode(fs, subvol, object);
+    if (!inode) {
+        errno = ESTALE;
+        goto out;
+    }
+
+    /* populate loc */
+    GLFS_LOC_FILL_INODE(inode, loc, out);
+
+    /* fop/op */
+    ret = syncop_removexattr(subvol, &loc, name, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
 
-        __glfs_entry_fs (fs);
+out:
+    loc_wipe(&loc);
 
-        /* get the active volume */
-        subvol = priv_glfs_active_subvol (fs);
-        if (!subvol) {
-                ret = -1;
-                errno = EIO;
-                goto out;
-        }
+    if (inode)
+        inode_unref(inode);
 
-        /* get/refresh the in arg objects inode in correlation to the xlator */
-        inode = glfs_resolve_inode (fs, subvol, object);
-        if (!inode) {
-                errno = ESTALE;
-                goto out;
-        }
+    glfs_subvol_done(fs, subvol);
 
-        xattr = dict_for_key_value (name, value, size);
-        if (!xattr) {
-                ret = -1;
-                errno = ENOMEM;
-                goto out;
-        }
+    __GLFS_EXIT_FS;
 
-        /* populate loc */
-        GLFS_LOC_FILL_INODE (inode, loc, out);
+invalid_fs:
+    return ret;
+}
 
-        /* fop/op */
-        ret = syncop_setxattr (subvol, &loc, xattr, flags);
-        DECODE_SYNCOP_ERR (ret);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_open, 3.4.2)
+struct glfs_fd *
+pub_glfs_h_open(struct glfs *fs, struct glfs_object *object, int flags)
+{
+    int ret = -1;
+    struct glfs_fd *glfd = NULL;
+    xlator_t *subvol = NULL;
+    inode_t *inode = NULL;
+    loc_t loc = {
+        0,
+    };
+    dict_t *fop_attr = NULL;
+
+    /* validate in args */
+    if ((fs == NULL) || (object == NULL)) {
+        errno = EINVAL;
+        return NULL;
+    }
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        errno = EIO;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    inode = glfs_resolve_inode(fs, subvol, object);
+    if (!inode) {
+        errno = ESTALE;
+        goto out;
+    }
+
+    /* check types to open */
+    if (IA_ISDIR(inode->ia_type)) {
+        ret = -1;
+        errno = EISDIR;
+        goto out;
+    }
+
+    if (!IA_ISREG(inode->ia_type)) {
+        ret = -1;
+        errno = EINVAL;
+        goto out;
+    }
+
+    glfd = glfs_fd_new(fs);
+    if (!glfd) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    glfd->fd = fd_create(inode, getpid());
+    if (!glfd->fd) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+    glfd->fd->flags = flags;
+
+    /* populate loc */
+    GLFS_LOC_FILL_INODE(inode, loc, out);
+
+    /* fop/op */
+    ret = get_fop_attr_thrd_key(&fop_attr);
+    if (ret)
+        gf_msg_debug("gfapi", 0, "Getting leaseid from thread failed");
+
+    ret = syncop_open(subvol, &loc, flags, glfd->fd, fop_attr, NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    glfd->fd->flags = flags;
 
 out:
-        loc_wipe (&loc);
-
-        if (inode)
-                inode_unref (inode);
+    loc_wipe(&loc);
 
-        if (xattr)
-                dict_unref (xattr);
+    if (inode)
+        inode_unref(inode);
+    if (fop_attr)
+        dict_unref(fop_attr);
 
-        priv_glfs_subvol_done (fs, subvol);
+    if (ret && glfd) {
+        GF_REF_PUT(glfd);
+        glfd = NULL;
+    } else if (glfd) {
+        glfd_set_state_bind(glfd);
+    }
 
-        return ret;
-}
+    glfs_subvol_done(fs, subvol);
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_setxattrs, 3.5.0);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return glfd;
+}
 
-int
-pub_glfs_h_removexattrs (struct glfs *fs, struct glfs_object *object,
-                         const char *name)
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_creat, 3.4.2)
+struct glfs_object *
+pub_glfs_h_creat(struct glfs *fs, struct glfs_object *parent, const char *path,
+                 int flags, mode_t mode, struct stat *stat)
 {
-        int              ret = -1;
-        xlator_t        *subvol = NULL;
-        inode_t         *inode = NULL;
-        loc_t            loc = {0, };
-
-        /* validate in args */
-        if ((fs == NULL) || (object == NULL) || (name == NULL)) {
-                errno = EINVAL;
-                return -1;
-        }
-
-        __glfs_entry_fs (fs);
-
-        /* get the active volume */
-        subvol = priv_glfs_active_subvol (fs);
-        if (!subvol) {
-                ret = -1;
-                errno = EIO;
-                goto out;
-        }
-
-        /* get/refresh the in arg objects inode in correlation to the xlator */
-        inode = glfs_resolve_inode (fs, subvol, object);
-        if (!inode) {
-                errno = ESTALE;
-                goto out;
+    int ret = -1;
+    fd_t *fd = NULL;
+    xlator_t *subvol = NULL;
+    inode_t *inode = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    uuid_t gfid;
+    dict_t *xattr_req = NULL;
+    struct glfs_object *object = NULL;
+
+    /* validate in args */
+    if ((fs == NULL) || (parent == NULL) || (path == NULL)) {
+        errno = EINVAL;
+        return NULL;
+    }
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    inode = glfs_resolve_inode(fs, subvol, parent);
+    if (!inode) {
+        ret = -1;
+        errno = ESTALE;
+        goto out;
+    }
+
+    xattr_req = dict_new();
+    if (!xattr_req) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    gf_uuid_generate(gfid);
+    ret = dict_set_gfuuid(xattr_req, "gfid-req", gfid, true);
+    if (ret) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    GLFS_LOC_FILL_PINODE(inode, loc, ret, errno, out, path);
+
+    fd = fd_create(loc.inode, getpid());
+    if (!fd) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+    fd->flags = flags;
+
+    /* fop/op */
+    ret = syncop_create(subvol, &loc, flags, mode, fd, &iatt, xattr_req, NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    /* populate out args */
+    if (ret == 0) {
+        ret = glfs_loc_link(&loc, &iatt);
+        if (ret != 0) {
+            goto out;
         }
 
-        /* populate loc */
-        GLFS_LOC_FILL_INODE (inode, loc, out);
+        if (stat)
+            glfs_iatt_to_stat(fs, &iatt, stat);
 
-        /* fop/op */
-        ret = syncop_removexattr (subvol, &loc, name, 0);
-        DECODE_SYNCOP_ERR (ret);
+        ret = glfs_create_object(&loc, &object);
+    }
 
 out:
-        loc_wipe (&loc);
-
-        if (inode)
-                inode_unref (inode);
+    if (ret && object != NULL) {
+        /* Release the held reference */
+        glfs_h_close(object);
+        object = NULL;
+    }
 
-        priv_glfs_subvol_done (fs, subvol);
+    loc_wipe(&loc);
 
-        return ret;
-}
+    if (inode)
+        inode_unref(inode);
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_removexattrs, 3.5.1);
+    if (xattr_req)
+        dict_unref(xattr_req);
 
+    if (fd)
+        fd_unref(fd);
 
-struct glfs_fd *
-pub_glfs_h_open (struct glfs *fs, struct glfs_object *object, int flags)
-{
-        int              ret = -1;
-        struct glfs_fd  *glfd = NULL;
-        xlator_t        *subvol = NULL;
-        inode_t         *inode = NULL;
-        loc_t            loc = {0, };
-
-        /* validate in args */
-        if ((fs == NULL) || (object == NULL)) {
-                errno = EINVAL;
-                return NULL;
-        }
+    glfs_subvol_done(fs, subvol);
 
-        __glfs_entry_fs (fs);
+    __GLFS_EXIT_FS;
 
-        /* get the active volume */
-        subvol = priv_glfs_active_subvol (fs);
-        if (!subvol) {
-                errno = EIO;
-                goto out;
-        }
+invalid_fs:
+    return object;
+}
 
-        /* get/refresh the in arg objects inode in correlation to the xlator */
-        inode = glfs_resolve_inode (fs, subvol, object);
-        if (!inode) {
-                errno = ESTALE;
-                goto out;
-        }
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_creat_open, 6.6)
+struct glfs_object *
+pub_glfs_h_creat_open(struct glfs *fs, struct glfs_object *parent,
+                      const char *path, int flags, mode_t mode,
+                      struct stat *stat, struct glfs_fd **out_fd)
+{
+    int ret = -1;
+    struct glfs_fd *glfd = NULL;
+    xlator_t *subvol = NULL;
+    inode_t *inode = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    uuid_t gfid;
+    dict_t *xattr_req = NULL;
+    struct glfs_object *object = NULL;
+    dict_t *fop_attr = NULL;
+
+    /* validate in args */
+    if ((fs == NULL) || (parent == NULL) || (path == NULL) ||
+        (out_fd == NULL)) {
+        errno = EINVAL;
+        return NULL;
+    }
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    inode = glfs_resolve_inode(fs, subvol, parent);
+    if (!inode) {
+        ret = -1;
+        goto out;
+    }
+
+    xattr_req = dict_new();
+    if (!xattr_req) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    gf_uuid_generate(gfid);
+    ret = dict_set_gfuuid(xattr_req, "gfid-req", gfid, true);
+    if (ret) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    GLFS_LOC_FILL_PINODE(inode, loc, ret, errno, out, path);
+
+    glfd = glfs_fd_new(fs);
+    if (!glfd) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    glfd->fd = fd_create(loc.inode, getpid());
+    if (!glfd->fd) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+    glfd->fd->flags = flags;
+
+    ret = get_fop_attr_thrd_key(&fop_attr);
+    if (ret)
+        gf_msg_debug("gfapi", 0, "Getting leaseid from thread failed");
+
+    /* fop/op */
+    ret = syncop_create(subvol, &loc, flags, mode, glfd->fd, &iatt, xattr_req,
+                        NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    /* populate out args */
+    if (ret == 0) {
+        glfd->fd->flags = flags;
 
-        /* check types to open */
-        if (IA_ISDIR (inode->ia_type)) {
-                ret = -1;
-                errno = EISDIR;
-                goto out;
+        ret = glfs_loc_link(&loc, &iatt);
+        if (ret != 0) {
+            goto out;
         }
 
-        if (!IA_ISREG (inode->ia_type)) {
-                ret = -1;
-                errno = EINVAL;
-                goto out;
-        }
+        if (stat)
+            glfs_iatt_to_stat(fs, &iatt, stat);
 
-        glfd = glfs_fd_new (fs);
-        if (!glfd) {
-                ret = -1;
-                errno = ENOMEM;
-                goto out;
-        }
+        ret = glfs_create_object(&loc, &object);
+    }
 
-        glfd->fd = fd_create (inode, getpid());
-        if (!glfd->fd) {
-                ret = -1;
-                errno = ENOMEM;
-                goto out;
-        }
-        glfd->fd->flags = flags;
+out:
+    if (ret && object != NULL) {
+        /* Release the held reference */
+        glfs_h_close(object);
+        object = NULL;
+    }
 
-        /* populate loc */
-        GLFS_LOC_FILL_INODE (inode, loc, out);
+    loc_wipe(&loc);
 
-        /* fop/op */
-        ret = syncop_open (subvol, &loc, flags, glfd->fd);
-        DECODE_SYNCOP_ERR (ret);
+    if (inode)
+        inode_unref(inode);
 
-        glfd->fd->flags = flags;
-        fd_bind (glfd->fd);
-        glfs_fd_bind (glfd);
+    if (fop_attr)
+        dict_unref(fop_attr);
 
-out:
-        loc_wipe (&loc);
+    if (xattr_req)
+        dict_unref(xattr_req);
 
-        if (inode)
-                inode_unref (inode);
+    if (ret && glfd) {
+        GF_REF_PUT(glfd);
+    } else if (glfd) {
+        glfd_set_state_bind(glfd);
+        *out_fd = glfd;
+    }
 
-        if (ret && glfd) {
-                glfs_fd_destroy (glfd);
-                glfd = NULL;
-        }
+    glfs_subvol_done(fs, subvol);
 
-        priv_glfs_subvol_done (fs, subvol);
+    __GLFS_EXIT_FS;
 
-        return glfd;
+invalid_fs:
+    return object;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_open, 3.4.2);
-
-
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_mkdir, 3.4.2)
 struct glfs_object *
-pub_glfs_h_creat (struct glfs *fs, struct glfs_object *parent, const char *path,
-                  int flags, mode_t mode, struct stat *stat)
+pub_glfs_h_mkdir(struct glfs *fs, struct glfs_object *parent, const char *path,
+                 mode_t mode, struct stat *stat)
 {
-        int                 ret = -1;
-        struct glfs_fd     *glfd = NULL;
-        xlator_t           *subvol = NULL;
-        inode_t            *inode = NULL;
-        loc_t               loc = {0, };
-        struct iatt         iatt = {0, };
-        uuid_t              gfid;
-        dict_t             *xattr_req = NULL;
-        struct glfs_object *object = NULL;
-
-        /* validate in args */
-        if ((fs == NULL) || (parent == NULL) || (path == NULL)) {
-                errno = EINVAL;
-                return NULL;
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    inode_t *inode = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    uuid_t gfid;
+    dict_t *xattr_req = NULL;
+    struct glfs_object *object = NULL;
+
+    /* validate in args */
+    if ((fs == NULL) || (parent == NULL) || (path == NULL)) {
+        errno = EINVAL;
+        return NULL;
+    }
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    inode = glfs_resolve_inode(fs, subvol, parent);
+    if (!inode) {
+        errno = ESTALE;
+        goto out;
+    }
+
+    xattr_req = dict_new();
+    if (!xattr_req) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    gf_uuid_generate(gfid);
+    ret = dict_set_gfuuid(xattr_req, "gfid-req", gfid, true);
+    if (ret) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    GLFS_LOC_FILL_PINODE(inode, loc, ret, errno, out, path);
+
+    /* fop/op */
+    ret = syncop_mkdir(subvol, &loc, mode, &iatt, xattr_req, NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    /* populate out args */
+    if (ret == 0) {
+        ret = glfs_loc_link(&loc, &iatt);
+        if (ret != 0) {
+            goto out;
         }
 
-        __glfs_entry_fs (fs);
-
-        /* get the active volume */
-        subvol = priv_glfs_active_subvol (fs);
-        if (!subvol) {
-                ret = -1;
-                errno = EIO;
-                goto out;
-        }
+        if (stat)
+            glfs_iatt_to_stat(fs, &iatt, stat);
 
-        /* get/refresh the in arg objects inode in correlation to the xlator */
-        inode = glfs_resolve_inode (fs, subvol, parent);
-        if (!inode) {
-                errno = ESTALE;
-                goto out;
-        }
+        ret = glfs_create_object(&loc, &object);
+    }
 
-        xattr_req = dict_new ();
-        if (!xattr_req) {
-                ret = -1;
-                errno = ENOMEM;
-                goto out;
-        }
+out:
+    if (ret && object != NULL) {
+        glfs_h_close(object);
+        object = NULL;
+    }
 
-        uuid_generate (gfid);
-        ret = dict_set_static_bin (xattr_req, "gfid-req", gfid, 16);
-        if (ret) {
-                ret = -1;
-                errno = ENOMEM;
-                goto out;
-        }
+    loc_wipe(&loc);
 
-        GLFS_LOC_FILL_PINODE (inode, loc, ret, errno, out, path);
+    if (inode)
+        inode_unref(inode);
 
-        glfd = glfs_fd_new (fs);
-        if (!glfd) {
-                 ret = -1;
-                 errno = ENOMEM;
-                 goto out;
-        }
+    if (xattr_req)
+        dict_unref(xattr_req);
 
-        glfd->fd = fd_create (loc.inode, getpid());
-        if (!glfd->fd) {
-                ret = -1;
-                errno = ENOMEM;
-                goto out;
-        }
-        glfd->fd->flags = flags;
+    glfs_subvol_done(fs, subvol);
 
-        /* fop/op */
-        ret = syncop_create (subvol, &loc, flags, mode, glfd->fd,
-                             xattr_req, &iatt);
-        DECODE_SYNCOP_ERR (ret);
-
-        /* populate out args */
-        if (ret == 0) {
-                /* TODO: If the inode existed in the cache (say file already
-                   exists), then the glfs_loc_link will not update the
-                   loc.inode, as a result we will have a 0000 GFID that we
-                   would copy out to the object, this needs to be fixed.
-                */
-                ret = glfs_loc_link (&loc, &iatt);
-                if (ret != 0) {
-                        goto out;
-                }
+    __GLFS_EXIT_FS;
 
-                if (stat)
-                        glfs_iatt_to_stat (fs, &iatt, stat);
+invalid_fs:
+    return object;
+}
 
-                ret = glfs_create_object (&loc, &object);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_mknod, 3.4.2)
+struct glfs_object *
+pub_glfs_h_mknod(struct glfs *fs, struct glfs_object *parent, const char *path,
+                 mode_t mode, dev_t dev, struct stat *stat)
+{
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    inode_t *inode = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    uuid_t gfid;
+    dict_t *xattr_req = NULL;
+    struct glfs_object *object = NULL;
+
+    /* validate in args */
+    if ((fs == NULL) || (parent == NULL) || (path == NULL)) {
+        errno = EINVAL;
+        return NULL;
+    }
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    inode = glfs_resolve_inode(fs, subvol, parent);
+    if (!inode) {
+        errno = ESTALE;
+        goto out;
+    }
+
+    xattr_req = dict_new();
+    if (!xattr_req) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    gf_uuid_generate(gfid);
+    ret = dict_set_gfuuid(xattr_req, "gfid-req", gfid, true);
+    if (ret) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    GLFS_LOC_FILL_PINODE(inode, loc, ret, errno, out, path);
+
+    /* fop/op */
+    ret = syncop_mknod(subvol, &loc, mode, dev, &iatt, xattr_req, NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    /* populate out args */
+    if (ret == 0) {
+        ret = glfs_loc_link(&loc, &iatt);
+        if (ret != 0) {
+            goto out;
         }
 
-        glfd->fd->flags = flags;
-        fd_bind (glfd->fd);
-        glfs_fd_bind (glfd);
+        if (stat)
+            glfs_iatt_to_stat(fs, &iatt, stat);
 
+        ret = glfs_create_object(&loc, &object);
+    }
 out:
-        if (ret && object != NULL) {
-                glfs_h_close (object);
-                object = NULL;
-        }
+    if (ret && object != NULL) {
+        glfs_h_close(object);
+        object = NULL;
+    }
 
-        loc_wipe(&loc);
+    loc_wipe(&loc);
 
-        if (inode)
-                inode_unref (inode);
+    if (inode)
+        inode_unref(inode);
 
-        if (xattr_req)
-                dict_unref (xattr_req);
+    if (xattr_req)
+        dict_unref(xattr_req);
 
-        if (ret && glfd) {
-                glfs_fd_destroy (glfd);
-                glfd = NULL;
-        }
+    glfs_subvol_done(fs, subvol);
 
-        priv_glfs_subvol_done (fs, subvol);
+    __GLFS_EXIT_FS;
 
-        return object;
+invalid_fs:
+    return object;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_creat, 3.4.2);
-
-
-struct glfs_object *
-pub_glfs_h_mkdir (struct glfs *fs, struct glfs_object *parent, const char *path,
-                  mode_t mode, struct stat *stat)
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_unlink, 3.4.2)
+int
+pub_glfs_h_unlink(struct glfs *fs, struct glfs_object *parent, const char *path)
 {
-        int                 ret = -1;
-        xlator_t           *subvol = NULL;
-        inode_t            *inode = NULL;
-        loc_t               loc = {0, };
-        struct iatt         iatt = {0, };
-        uuid_t              gfid;
-        dict_t             *xattr_req = NULL;
-        struct glfs_object *object = NULL;
-
-        /* validate in args */
-        if ((fs == NULL) || (parent == NULL) || (path == NULL)) {
-                errno = EINVAL;
-                return NULL;
-        }
-
-        __glfs_entry_fs (fs);
-
-        /* get the active volume */
-        subvol = priv_glfs_active_subvol (fs);
-        if (!subvol) {
-                ret = -1;
-                errno = EIO;
-                goto out;
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    inode_t *inode = NULL;
+    loc_t loc = {
+        0,
+    };
+
+    /* validate in args */
+    if ((fs == NULL) || (parent == NULL) || (path == NULL)) {
+        errno = EINVAL;
+        return -1;
+    }
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    inode = glfs_resolve_inode(fs, subvol, parent);
+    if (!inode) {
+        errno = ESTALE;
+        goto out;
+    }
+
+    ret = glfs_resolve_at(fs, subvol, inode, path, &loc, NULL, 0, 0);
+    if (ret != 0) {
+        goto out;
+    }
+
+    if (!IA_ISDIR(loc.inode->ia_type)) {
+        ret = syncop_unlink(subvol, &loc, NULL, NULL);
+        DECODE_SYNCOP_ERR(ret);
+        if (ret != 0) {
+            goto out;
         }
-
-        /* get/refresh the in arg objects inode in correlation to the xlator */
-        inode = glfs_resolve_inode (fs, subvol, parent);
-        if (!inode) {
-                errno = ESTALE;
-                goto out;
+    } else {
+        ret = syncop_rmdir(subvol, &loc, 0, NULL, NULL);
+        DECODE_SYNCOP_ERR(ret);
+        if (ret != 0) {
+            goto out;
         }
+    }
 
-        xattr_req = dict_new ();
-        if (!xattr_req) {
-                ret = -1;
-                errno = ENOMEM;
-                goto out;
-        }
+    if (ret == 0)
+        ret = glfs_loc_unlink(&loc);
 
-        uuid_generate (gfid);
-        ret = dict_set_static_bin (xattr_req, "gfid-req", gfid, 16);
-        if (ret) {
-                ret = -1;
-                errno = ENOMEM;
-                goto out;
-        }
+out:
+    loc_wipe(&loc);
 
-        GLFS_LOC_FILL_PINODE (inode, loc, ret, errno, out, path);
+    if (inode)
+        inode_unref(inode);
 
-        /* fop/op */
-        ret = syncop_mkdir (subvol, &loc, mode, xattr_req, &iatt);
-        DECODE_SYNCOP_ERR (ret);
+    glfs_subvol_done(fs, subvol);
 
-        /* populate out args */
-        if ( ret == 0 )  {
-                ret = glfs_loc_link (&loc, &iatt);
-                if (ret != 0) {
-                        goto out;
-                }
+    __GLFS_EXIT_FS;
 
-                if (stat)
-                        glfs_iatt_to_stat (fs, &iatt, stat);
+invalid_fs:
+    return ret;
+}
 
-                ret = glfs_create_object (&loc, &object);
-        }
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_opendir, 3.4.2)
+struct glfs_fd *
+pub_glfs_h_opendir(struct glfs *fs, struct glfs_object *object)
+{
+    int ret = -1;
+    struct glfs_fd *glfd = NULL;
+    xlator_t *subvol = NULL;
+    inode_t *inode = NULL;
+    loc_t loc = {
+        0,
+    };
+
+    /* validate in args */
+    if ((fs == NULL) || (object == NULL)) {
+        errno = EINVAL;
+        return NULL;
+    }
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    inode = glfs_resolve_inode(fs, subvol, object);
+    if (!inode) {
+        errno = ESTALE;
+        goto out;
+    }
+
+    if (!IA_ISDIR(inode->ia_type)) {
+        ret = -1;
+        errno = ENOTDIR;
+        goto out;
+    }
+
+    glfd = glfs_fd_new(fs);
+    if (!glfd)
+        goto out;
+
+    INIT_LIST_HEAD(&glfd->entries);
+
+    glfd->fd = fd_create(inode, getpid());
+    if (!glfd->fd) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    GLFS_LOC_FILL_INODE(inode, loc, out);
+
+    /* fop/op */
+    ret = syncop_opendir(subvol, &loc, glfd->fd, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
 
 out:
-        if (ret && object != NULL) {
-                glfs_h_close (object);
-                object = NULL;
-        }
+    loc_wipe(&loc);
 
-        loc_wipe(&loc);
+    if (inode)
+        inode_unref(inode);
 
-        if (inode)
-                inode_unref (inode);
+    if (ret && glfd) {
+        GF_REF_PUT(glfd);
+        glfd = NULL;
+    } else if (glfd) {
+        glfd_set_state_bind(glfd);
+    }
 
-        if (xattr_req)
-                dict_unref (xattr_req);
+    glfs_subvol_done(fs, subvol);
 
-        priv_glfs_subvol_done (fs, subvol);
+    __GLFS_EXIT_FS;
 
-        return object;
+invalid_fs:
+    return glfd;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_mkdir, 3.4.2);
-
-
-struct glfs_object *
-pub_glfs_h_mknod (struct glfs *fs, struct glfs_object *parent, const char *path,
-                  mode_t mode, dev_t dev, struct stat *stat)
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_access, 3.6.0)
+int
+pub_glfs_h_access(struct glfs *fs, struct glfs_object *object, int mask)
 {
-        int                 ret = -1;
-        xlator_t           *subvol = NULL;
-        inode_t            *inode = NULL;
-        loc_t               loc = {0, };
-        struct iatt         iatt = {0, };
-        uuid_t              gfid;
-        dict_t             *xattr_req = NULL;
-        struct glfs_object *object = NULL;
-
-        /* validate in args */
-        if ((fs == NULL) || (parent == NULL) || (path == NULL)) {
-                errno = EINVAL;
-                return NULL;
-        }
-
-        __glfs_entry_fs (fs);
-
-        /* get the active volume */
-        subvol = priv_glfs_active_subvol (fs);
-        if (!subvol) {
-                ret = -1;
-                errno = EIO;
-                goto out;
-        }
-
-        /* get/refresh the in arg objects inode in correlation to the xlator */
-        inode = glfs_resolve_inode (fs, subvol, parent);
-        if (!inode) {
-                errno = ESTALE;
-                goto out;
-        }
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    inode_t *inode = NULL;
+    loc_t loc = {
+        0,
+    };
+
+    DECLARE_OLD_THIS;
+
+    /* validate in args */
+    if ((fs == NULL) || (object == NULL)) {
+        errno = EINVAL;
+        return ret;
+    }
 
-        xattr_req = dict_new ();
-        if (!xattr_req) {
-                ret = -1;
-                errno = ENOMEM;
-                goto out;
-        }
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
 
-        uuid_generate (gfid);
-        ret = dict_set_static_bin (xattr_req, "gfid-req", gfid, 16);
-        if (ret) {
-                ret = -1;
-                errno = ENOMEM;
-                goto out;
-        }
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
 
-        GLFS_LOC_FILL_PINODE (inode, loc, ret, errno, out, path);
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    inode = glfs_resolve_inode(fs, subvol, object);
+    if (!inode) {
+        errno = ESTALE;
+        goto out;
+    }
 
-        /* fop/op */
-        ret = syncop_mknod (subvol, &loc, mode, dev, xattr_req, &iatt);
-        DECODE_SYNCOP_ERR (ret);
+    GLFS_LOC_FILL_INODE(inode, loc, out);
 
-        /* populate out args */
-        if (ret == 0) {
-                ret = glfs_loc_link (&loc, &iatt);
-                if (ret != 0) {
-                        goto out;
-                }
+    /* fop/op */
 
-                if (stat)
-                        glfs_iatt_to_stat (fs, &iatt, stat);
+    ret = syncop_access(subvol, &loc, mask, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
 
-                ret = glfs_create_object (&loc, &object);
-        }
 out:
-        if (ret && object != NULL) {
-                glfs_h_close (object);
-                object = NULL;
-        }
+    loc_wipe(&loc);
 
-        loc_wipe(&loc);
+    if (inode)
+        inode_unref(inode);
 
-        if (inode)
-                inode_unref (inode);
+    glfs_subvol_done(fs, subvol);
 
-        if (xattr_req)
-                dict_unref (xattr_req);
+    __GLFS_EXIT_FS;
 
-        priv_glfs_subvol_done (fs, subvol);
-
-        return object;
+invalid_fs:
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_mknod, 3.4.2);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_extract_handle, 3.4.2)
+ssize_t
+pub_glfs_h_extract_handle(struct glfs_object *object, unsigned char *handle,
+                          int len)
+{
+    ssize_t ret = -1;
 
+    /* validate in args */
+    if (object == NULL) {
+        errno = EINVAL;
+        goto out;
+    }
 
-int
-pub_glfs_h_unlink (struct glfs *fs, struct glfs_object *parent, const char *path)
-{
-        int                 ret = -1;
-        xlator_t           *subvol = NULL;
-        inode_t            *inode = NULL;
-        loc_t               loc = {0, };
-
-        /* validate in args */
-        if ((fs == NULL) || (parent == NULL) || (path == NULL)) {
-                errno = EINVAL;
-                return -1;
-        }
+    if (!handle || !len) {
+        ret = GFAPI_HANDLE_LENGTH;
+        goto out;
+    }
 
-        __glfs_entry_fs (fs);
+    if (len < GFAPI_HANDLE_LENGTH) {
+        errno = ERANGE;
+        goto out;
+    }
 
-        /* get the active volume */
-        subvol = priv_glfs_active_subvol (fs);
-        if ( !subvol ) {
-                ret = -1;
-                errno = EIO;
-                goto out;
-        }
+    memcpy(handle, object->gfid, GFAPI_HANDLE_LENGTH);
 
-        /* get/refresh the in arg objects inode in correlation to the xlator */
-        inode = glfs_resolve_inode (fs, subvol, parent);
-        if (!inode) {
-                errno = ESTALE;
-                goto out;
-        }
+    ret = GFAPI_HANDLE_LENGTH;
 
-        ret = priv_glfs_resolve_at (fs, subvol, inode, path, &loc, NULL, 0 , 0);
-        if (ret != 0) {
-                goto out;
-        }
+out:
+    return ret;
+}
 
-        if (!IA_ISDIR(loc.inode->ia_type)) {
-                ret = syncop_unlink (subvol, &loc);
-                DECODE_SYNCOP_ERR (ret);
-                if (ret != 0) {
-                        goto out;
-                }
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_create_from_handle, 3.4.2)
+struct glfs_object *
+pub_glfs_h_create_from_handle(struct glfs *fs, unsigned char *handle, int len,
+                              struct stat *stat)
+{
+    loc_t loc = {
+        0,
+    };
+    int ret = -1;
+    struct iatt iatt = {
+        0,
+    };
+    inode_t *newinode = NULL;
+    xlator_t *subvol = NULL;
+    struct glfs_object *object = NULL;
+    uint64_t ctx_value = LOOKUP_NOT_NEEDED;
+    gf_boolean_t lookup_needed = _gf_false;
+
+    /* validate in args */
+    if ((fs == NULL) || (handle == NULL) || (len != GFAPI_HANDLE_LENGTH)) {
+        errno = EINVAL;
+        return NULL;
+    }
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        errno = EIO;
+        goto out;
+    }
+
+    memcpy(loc.gfid, handle, GFAPI_HANDLE_LENGTH);
+
+    /* make sure the gfid received is valid */
+    GF_VALIDATE_OR_GOTO("glfs_h_create_from_handle",
+                        !(gf_uuid_is_null(loc.gfid)), out);
+
+    newinode = inode_find(subvol->itable, loc.gfid);
+    if (newinode) {
+        if (!stat) /* No need of lookup */
+            goto found;
+
+        lookup_needed = inode_needs_lookup(newinode, THIS);
+        if (lookup_needed) {
+            loc.inode = newinode;
         } else {
-                ret = syncop_rmdir (subvol, &loc, 0);
-                DECODE_SYNCOP_ERR (ret);
-                if (ret != 0) {
-                        goto out;
-                }
-        }
-
-        if (ret == 0)
-                ret = glfs_loc_unlink (&loc);
+            /* populate loc */
+            GLFS_LOC_FILL_INODE(newinode, loc, fill_out);
+
+            /* fop/op */
+            ret = syncop_stat(subvol, &loc, &iatt, NULL, NULL);
+            DECODE_SYNCOP_ERR(ret);
+
+            if (ret) {
+            fill_out:
+                /* Drop the reference hold in inode_find */
+                inode_unref(newinode);
+                goto out;
+            }
+
+            glfs_iatt_to_stat(fs, &iatt, stat);
+            goto found;
+        }
+    } else {
+        loc.inode = inode_new(subvol->itable);
+        if (!loc.inode) {
+            errno = ENOMEM;
+            goto out;
+        }
+    }
+
+    ret = syncop_lookup(subvol, &loc, &iatt, 0, 0, 0);
+    DECODE_SYNCOP_ERR(ret);
+    if (ret) {
+        gf_smsg(subvol->name, GF_LOG_WARNING, errno,
+                API_MSG_INODE_REFRESH_FAILED, "gfid=%s", uuid_utoa(loc.gfid),
+                "error=%s", strerror(errno), NULL);
+        goto out;
+    }
+
+    newinode = inode_link(loc.inode, 0, 0, &iatt);
+    if (newinode) {
+        if (newinode == loc.inode) {
+            inode_ctx_set(newinode, THIS, &ctx_value);
+        }
+        inode_lookup(newinode);
+    } else {
+        gf_smsg(subvol->name, GF_LOG_WARNING, errno, API_MSG_INODE_LINK_FAILED,
+                "gfid=%s", uuid_utoa(loc.gfid), NULL);
+        goto out;
+    }
+
+    /* populate stat */
+    if (stat)
+        glfs_iatt_to_stat(fs, &iatt, stat);
+
+found:
+    object = GF_CALLOC(1, sizeof(struct glfs_object), glfs_mt_glfs_object_t);
+    if (object == NULL) {
+        errno = ENOMEM;
+        ret = -1;
+        goto out;
+    }
+
+    /* populate the return object */
+    object->inode = newinode;
+    gf_uuid_copy(object->gfid, object->inode->gfid);
 
 out:
-        loc_wipe (&loc);
+    /* TODO: Check where the inode ref is being held? */
+    loc_wipe(&loc);
 
-        if (inode)
-                inode_unref (inode);
+    glfs_subvol_done(fs, subvol);
 
-        priv_glfs_subvol_done (fs, subvol);
+    __GLFS_EXIT_FS;
 
-        return ret;
+invalid_fs:
+    return object;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_unlink, 3.4.2);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_close, 3.4.2)
+int
+pub_glfs_h_close(struct glfs_object *object)
+{
+    /* since glfs_h_* objects hold a reference to inode
+     * it is safe to keep lookup count to '0' */
+    inode_forget(object->inode, 0);
+    inode_unref(object->inode);
+    GF_FREE(object);
 
+    return 0;
+}
 
-struct glfs_fd *
-pub_glfs_h_opendir (struct glfs *fs, struct glfs_object *object)
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_truncate, 3.4.2)
+int
+pub_glfs_h_truncate(struct glfs *fs, struct glfs_object *object, off_t offset)
 {
-        int              ret = -1;
-        struct glfs_fd  *glfd = NULL;
-        xlator_t        *subvol = NULL;
-        inode_t         *inode = NULL;
-        loc_t            loc = {0, };
-
-        /* validate in args */
-        if ((fs == NULL) || (object == NULL)) {
-                errno = EINVAL;
-                return NULL;
-        }
+    loc_t loc = {
+        0,
+    };
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    inode_t *inode = NULL;
+
+    DECLARE_OLD_THIS;
+
+    /* validate in args */
+    if (object == NULL) {
+        errno = EINVAL;
+        return -1;
+    }
+
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    inode = glfs_resolve_inode(fs, subvol, object);
+    if (!inode) {
+        errno = ESTALE;
+        goto out;
+    }
+
+    GLFS_LOC_FILL_INODE(inode, loc, out);
+
+    /* fop/op */
+    ret = syncop_truncate(subvol, &loc, (off_t)offset, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    /* populate out args */
+    if (ret == 0)
+        ret = glfs_loc_unlink(&loc);
 
-        __glfs_entry_fs (fs);
-
-        /* get the active volume */
-        subvol = priv_glfs_active_subvol (fs);
-        if (!subvol) {
-                ret = -1;
-                errno = EIO;
-                goto out;
-        }
+out:
+    loc_wipe(&loc);
 
-        /* get/refresh the in arg objects inode in correlation to the xlator */
-        inode = glfs_resolve_inode (fs, subvol, object);
-        if (!inode) {
-                errno = ESTALE;
-                goto out;
-        }
+    if (inode)
+        inode_unref(inode);
 
-        if (!IA_ISDIR (inode->ia_type)) {
-                ret = -1;
-                errno = ENOTDIR;
-                goto out;
-        }
+    glfs_subvol_done(fs, subvol);
 
-        glfd = glfs_fd_new (fs);
-        if (!glfd)
-                goto out;
+    __GLFS_EXIT_FS;
 
-        INIT_LIST_HEAD (&glfd->entries);
+invalid_fs:
+    return ret;
+}
 
-        glfd->fd = fd_create (inode, getpid());
-        if (!glfd->fd) {
-                ret = -1;
-                errno = ENOMEM;
-                goto out;
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_symlink, 3.4.2)
+struct glfs_object *
+pub_glfs_h_symlink(struct glfs *fs, struct glfs_object *parent,
+                   const char *name, const char *data, struct stat *stat)
+{
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    inode_t *inode = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    uuid_t gfid;
+    dict_t *xattr_req = NULL;
+    struct glfs_object *object = NULL;
+
+    DECLARE_OLD_THIS;
+
+    /* validate in args */
+    if ((parent == NULL) || (name == NULL) || (data == NULL)) {
+        errno = EINVAL;
+        return NULL;
+    }
+
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    inode = glfs_resolve_inode(fs, subvol, parent);
+    if (!inode) {
+        errno = ESTALE;
+        goto out;
+    }
+
+    xattr_req = dict_new();
+    if (!xattr_req) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    gf_uuid_generate(gfid);
+    ret = dict_set_gfuuid(xattr_req, "gfid-req", gfid, true);
+    if (ret) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    GLFS_LOC_FILL_PINODE(inode, loc, ret, errno, out, name);
+
+    /* fop/op */
+    ret = syncop_symlink(subvol, &loc, data, &iatt, xattr_req, NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    /* populate out args */
+    if (ret == 0) {
+        ret = glfs_loc_link(&loc, &iatt);
+        if (ret != 0) {
+            goto out;
         }
 
-        GLFS_LOC_FILL_INODE (inode, loc, out);
+        if (stat)
+            glfs_iatt_to_stat(fs, &iatt, stat);
 
-        /* fop/op */
-        ret = syncop_opendir (subvol, &loc, glfd->fd);
-        DECODE_SYNCOP_ERR (ret);
+        ret = glfs_create_object(&loc, &object);
+    }
 
 out:
-        loc_wipe (&loc);
+    if (ret && object != NULL) {
+        pub_glfs_h_close(object);
+        object = NULL;
+    }
 
-        if (inode)
-                inode_unref (inode);
+    loc_wipe(&loc);
 
-        if (ret && glfd) {
-                glfs_fd_destroy (glfd);
-                glfd = NULL;
-        } else {
-                fd_bind (glfd->fd);
-                glfs_fd_bind (glfd);
-        }
+    if (inode)
+        inode_unref(inode);
 
-        priv_glfs_subvol_done (fs, subvol);
+    if (xattr_req)
+        dict_unref(xattr_req);
 
-        return glfd;
-}
+    glfs_subvol_done(fs, subvol);
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_opendir, 3.4.2);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return object;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_readlink, 3.4.2)
 int
-pub_glfs_h_access (struct glfs *fs, struct glfs_object *object, int mask)
+pub_glfs_h_readlink(struct glfs *fs, struct glfs_object *object, char *buf,
+                    size_t bufsiz)
 {
-	int              ret = -1;
-	xlator_t        *subvol = NULL;
-	inode_t         *inode = NULL;
-	loc_t            loc = {0, };
+    loc_t loc = {
+        0,
+    };
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    inode_t *inode = NULL;
+    char *linkval = NULL;
+
+    DECLARE_OLD_THIS;
+
+    /* validate in args */
+    if ((object == NULL) || (buf == NULL)) {
+        errno = EINVAL;
+        return -1;
+    }
+
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    inode = glfs_resolve_inode(fs, subvol, object);
+    if (!inode) {
+        errno = ESTALE;
+        goto out;
+    }
+
+    GLFS_LOC_FILL_INODE(inode, loc, out);
+
+    /* fop/op */
+    ret = syncop_readlink(subvol, &loc, &linkval, bufsiz, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    /* populate out args */
+    if (ret > 0)
+        memcpy(buf, linkval, ret);
 
-	/* validate in args */
-	if ((fs == NULL) || (object == NULL)) {
-		errno = EINVAL;
-		goto out;
-	}
-
-	__glfs_entry_fs (fs);
-
-	/* get the active volume */
-	subvol = priv_glfs_active_subvol (fs);
-	if (!subvol) {
-		ret = -1;
-		errno = EIO;
-		goto out;
-	}
+out:
+    loc_wipe(&loc);
 
-	/* get/refresh the in arg objects inode in correlation to the xlator */
-	inode = glfs_resolve_inode (fs, subvol, object);
-	if (!inode) {
-		errno = ESTALE;
-		goto out;
-	}
+    if (inode)
+        inode_unref(inode);
 
+    if (linkval)
+        GF_FREE(linkval);
 
-	GLFS_LOC_FILL_INODE (inode, loc, out);
+    glfs_subvol_done(fs, subvol);
 
-	/* fop/op */
+    __GLFS_EXIT_FS;
 
-	ret = syncop_access (subvol, &loc, mask);
-        DECODE_SYNCOP_ERR (ret);
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_link, 3.4.2)
+int
+pub_glfs_h_link(struct glfs *fs, struct glfs_object *linksrc,
+                struct glfs_object *parent, const char *name)
+{
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    inode_t *inode = NULL;
+    inode_t *pinode = NULL;
+    loc_t oldloc = {
+        0,
+    };
+    loc_t newloc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+
+    DECLARE_OLD_THIS;
+
+    /* validate in args */
+    if ((linksrc == NULL) || (parent == NULL) || (name == NULL)) {
+        errno = EINVAL;
+        return -1;
+    }
+
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    inode = glfs_resolve_inode(fs, subvol, linksrc);
+    if (!inode) {
+        errno = ESTALE;
+        goto out;
+    }
+
+    if (inode->ia_type == IA_IFDIR) {
+        ret = -1;
+        errno = EISDIR;
+        goto out;
+    }
+
+    GLFS_LOC_FILL_INODE(inode, oldloc, out);
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    pinode = glfs_resolve_inode(fs, subvol, parent);
+    if (!pinode) {
+        errno = ESTALE;
+        goto out;
+    }
+
+    /* setup newloc based on parent */
+    newloc.parent = inode_ref(pinode);
+    newloc.name = name;
+    ret = glfs_loc_touchup(&newloc);
+    if (ret != 0) {
+        errno = EINVAL;
+        goto out;
+    }
+
+    /* Filling the inode of the hard link to be same as that of the
+     * original file
+     */
+    newloc.inode = inode_ref(inode);
+
+    /* fop/op */
+    ret = syncop_link(subvol, &oldloc, &newloc, &iatt, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    if (ret == 0)
+        ret = glfs_loc_link(&newloc, &iatt);
 out:
-	loc_wipe (&loc);
+    loc_wipe(&oldloc);
+    loc_wipe(&newloc);
 
-	if (inode)
-		inode_unref (inode);
+    if (inode)
+        inode_unref(inode);
 
+    if (pinode)
+        inode_unref(pinode);
 
-	priv_glfs_subvol_done (fs, subvol);
-
-	return ret;
-}
+    glfs_subvol_done(fs, subvol);
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_access, 3.6.0);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
-ssize_t
-pub_glfs_h_extract_handle (struct glfs_object *object, unsigned char *handle,
-                           int len)
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_rename, 3.4.2)
+int
+pub_glfs_h_rename(struct glfs *fs, struct glfs_object *olddir,
+                  const char *oldname, struct glfs_object *newdir,
+                  const char *newname)
 {
-        ssize_t ret = -1;
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    inode_t *oldpinode = NULL;
+    inode_t *newpinode = NULL;
+    loc_t oldloc = {
+        0,
+    };
+    loc_t newloc = {
+        0,
+    };
+    struct iatt oldiatt = {
+        0,
+    };
+    struct iatt newiatt = {
+        0,
+    };
+
+    DECLARE_OLD_THIS;
+
+    /* validate in args */
+    if ((olddir == NULL) || (oldname == NULL) || (newdir == NULL) ||
+        (newname == NULL)) {
+        errno = EINVAL;
+        return -1;
+    }
+
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    oldpinode = glfs_resolve_inode(fs, subvol, olddir);
+    if (!oldpinode) {
+        errno = ESTALE;
+        goto out;
+    }
+
+    ret = glfs_resolve_at(fs, subvol, oldpinode, oldname, &oldloc, &oldiatt, 0,
+                          0);
+    if (ret != 0) {
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    newpinode = glfs_resolve_inode(fs, subvol, newdir);
+    if (!newpinode) {
+        errno = ESTALE;
+        goto out;
+    }
+
+    ret = glfs_resolve_at(fs, subvol, newpinode, newname, &newloc, &newiatt, 0,
+                          0);
+
+    if (ret && errno != ENOENT && newloc.parent)
+        goto out;
+
+    if (newiatt.ia_type != IA_INVAL) {
+        if ((oldiatt.ia_type == IA_IFDIR) != (newiatt.ia_type == IA_IFDIR)) {
+            /* Either both old and new must be dirs,
+             * or both must be non-dirs. Else, fail.
+             */
+            ret = -1;
+            errno = EEXIST;
+            goto out;
+        }
+    }
+
+    /* TODO: check if new or old is a prefix of the other, and fail EINVAL */
+
+    ret = syncop_rename(subvol, &oldloc, &newloc, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    if (ret == 0) {
+        inode_rename(oldloc.parent->table, oldloc.parent, oldloc.name,
+                     newloc.parent, newloc.name, oldloc.inode, &oldiatt);
+
+        if (newloc.inode && !inode_has_dentry(newloc.inode))
+            inode_forget(newloc.inode, 0);
+    }
 
-        /* validate in args */
-        if (object == NULL) {
-                errno = EINVAL;
-                goto out;
-        }
+out:
+    loc_wipe(&oldloc);
+    loc_wipe(&newloc);
 
-        if (!handle || !len) {
-                ret = GFAPI_HANDLE_LENGTH;
-                goto out;
-        }
+    if (oldpinode)
+        inode_unref(oldpinode);
 
-        if (len < GFAPI_HANDLE_LENGTH)
-        {
-                errno = ERANGE;
-                goto out;
-        }
+    if (newpinode)
+        inode_unref(newpinode);
 
-        memcpy (handle, object->gfid, GFAPI_HANDLE_LENGTH);
+    glfs_subvol_done(fs, subvol);
 
-        ret = GFAPI_HANDLE_LENGTH;
+    __GLFS_EXIT_FS;
 
-out:
-        return ret;
+invalid_fs:
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_extract_handle, 3.4.2);
-
-
+/*
+ * Given a handle/gfid, find if the corresponding inode is present in
+ * the inode table. If yes create and return the corresponding glfs_object.
+ */
 struct glfs_object *
-pub_glfs_h_create_from_handle (struct glfs *fs, unsigned char *handle, int len,
-                               struct stat *stat)
+glfs_h_find_handle(struct glfs *fs, unsigned char *handle, int len)
 {
-        loc_t               loc = {0, };
-        int                 ret = -1;
-        struct iatt         iatt = {0, };
-        inode_t            *newinode = NULL;
-        xlator_t           *subvol = NULL;
-        struct glfs_object *object = NULL;
-
-        /* validate in args */
-        if ((fs == NULL) || (handle == NULL) || (len != GFAPI_HANDLE_LENGTH)) {
-                errno = EINVAL;
-                return NULL;
-        }
+    inode_t *newinode = NULL;
+    xlator_t *subvol = NULL;
+    struct glfs_object *object = NULL;
+    uuid_t gfid;
+
+    /* validate in args */
+    if ((fs == NULL) || (handle == NULL) || (len != GFAPI_HANDLE_LENGTH)) {
+        errno = EINVAL;
+        return NULL;
+    }
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        errno = EIO;
+        goto out;
+    }
+
+    memcpy(gfid, handle, GFAPI_HANDLE_LENGTH);
+
+    /* make sure the gfid received is valid */
+    GF_VALIDATE_OR_GOTO("glfs_h_find_handle", !(gf_uuid_is_null(gfid)), out);
+
+    newinode = inode_find(subvol->itable, gfid);
+    if (!newinode) {
+        goto out;
+    }
+
+    object = GF_CALLOC(1, sizeof(struct glfs_object), glfs_mt_glfs_object_t);
+    if (object == NULL) {
+        errno = ENOMEM;
+        goto out;
+    }
+
+    /* populate the return object. The ref taken here
+     * is un'refed when the application does glfs_h_close() */
+    object->inode = inode_ref(newinode);
+    gf_uuid_copy(object->gfid, object->inode->gfid);
 
-        __glfs_entry_fs (fs);
+out:
+    /* inode_find takes a reference. Unref it. */
+    if (newinode)
+        inode_unref(newinode);
 
-        /* get the active volume */
-        subvol = priv_glfs_active_subvol (fs);
-        if (!subvol) {
-                errno = EIO;
-                goto out;
-        }
+    glfs_subvol_done(fs, subvol);
 
-        memcpy (loc.gfid, handle, GFAPI_HANDLE_LENGTH);
+    __GLFS_EXIT_FS;
 
-        newinode = inode_find (subvol->itable, loc.gfid);
-        if (newinode)
-                loc.inode = newinode;
-        else {
-                loc.inode = inode_new (subvol->itable);
-                if (!loc.inode) {
-                        errno = ENOMEM;
-                        goto out;
-                }
-        }
+invalid_fs:
+    return object;
+}
 
-        ret = syncop_lookup (subvol, &loc, 0, &iatt, 0, 0);
-        DECODE_SYNCOP_ERR (ret);
-        if (ret) {
-                gf_log (subvol->name, GF_LOG_WARNING,
-                        "inode refresh of %s failed: %s",
-                        uuid_utoa (loc.gfid), strerror (errno));
-                goto out;
-        }
+static void
+glfs_free_upcall_inode(void *to_free)
+{
+    struct glfs_upcall_inode *arg = to_free;
 
-        newinode = inode_link (loc.inode, 0, 0, &iatt);
-        if (newinode)
-                inode_lookup (newinode);
-        else {
-                gf_log (subvol->name, GF_LOG_WARNING,
-                        "inode linking of %s failed: %s",
-                        uuid_utoa (loc.gfid), strerror (errno));
-                errno = EINVAL;
-                goto out;
-        }
+    if (!arg)
+        return;
 
-        /* populate stat */
-        if (stat)
-                glfs_iatt_to_stat (fs, &iatt, stat);
+    if (arg->object)
+        glfs_h_close(arg->object);
+    if (arg->p_object)
+        glfs_h_close(arg->p_object);
+    if (arg->oldp_object)
+        glfs_h_close(arg->oldp_object);
 
-        object = GF_CALLOC (1, sizeof(struct glfs_object),
-                            glfs_mt_glfs_object_t);
-        if (object == NULL) {
-                errno = ENOMEM;
-                ret = -1;
-                goto out;
-        }
+    GF_FREE(arg);
+}
 
-        /* populate the return object */
-        object->inode = newinode;
-        uuid_copy (object->gfid, object->inode->gfid);
+int
+glfs_h_poll_cache_invalidation(struct glfs *fs, struct glfs_upcall *up_arg,
+                               struct gf_upcall *upcall_data)
+{
+    int ret = -1;
+    struct glfs_object *p_object = NULL;
+    struct glfs_object *oldp_object = NULL;
+    struct glfs_object *object = NULL;
+    struct gf_upcall_cache_invalidation *ca_data = NULL;
+    struct glfs_upcall_inode *up_inode_arg = NULL;
+
+    ca_data = upcall_data->data;
+    GF_VALIDATE_OR_GOTO("glfs_h_poll_cache_invalidation", ca_data, out);
+
+    object = glfs_h_find_handle(fs, upcall_data->gfid, GFAPI_HANDLE_LENGTH);
+    if (!object) {
+        /* The reason handle creation will fail is because we
+         * couldn't find the inode in the gfapi inode table.
+         *
+         * But since application would have taken inode_ref, the
+         * only case when this can happen is when it has closed
+         * the handle and hence will no more be interested in
+         * the upcall for this particular gfid.
+         */
+        gf_smsg(THIS->name, GF_LOG_DEBUG, errno, API_MSG_CREATE_HANDLE_FAILED,
+                "gfid=%s", uuid_utoa(upcall_data->gfid), NULL);
+        errno = ESTALE;
+        goto out;
+    }
+
+    up_inode_arg = GF_CALLOC(1, sizeof(struct glfs_upcall_inode),
+                             glfs_mt_upcall_inode_t);
+    GF_VALIDATE_OR_GOTO("glfs_h_poll_cache_invalidation", up_inode_arg, out);
+
+    up_inode_arg->object = object;
+    up_inode_arg->flags = ca_data->flags;
+    up_inode_arg->expire_time_attr = ca_data->expire_time_attr;
+
+    /* XXX: Update stat as well in case of UP_*_TIMES.
+     * This will be addressed as part of INODE_UPDATE */
+    if (ca_data->flags & GFAPI_INODE_UPDATE_FLAGS) {
+        glfs_iatt_to_stat(fs, &ca_data->stat, &up_inode_arg->buf);
+    }
+
+    if (ca_data->flags & GFAPI_UP_PARENT_TIMES) {
+        p_object = glfs_h_find_handle(fs, ca_data->p_stat.ia_gfid,
+                                      GFAPI_HANDLE_LENGTH);
+        if (!p_object) {
+            gf_smsg(THIS->name, GF_LOG_DEBUG, errno,
+                    API_MSG_CREATE_HANDLE_FAILED, "gfid=%s",
+                    uuid_utoa(ca_data->p_stat.ia_gfid), NULL);
+            errno = ESTALE;
+            goto out;
+        }
+
+        glfs_iatt_to_stat(fs, &ca_data->p_stat, &up_inode_arg->p_buf);
+    }
+    up_inode_arg->p_object = p_object;
+
+    /* In case of RENAME, update old parent as well */
+    if (ca_data->flags & GFAPI_UP_RENAME) {
+        oldp_object = glfs_h_find_handle(fs, ca_data->oldp_stat.ia_gfid,
+                                         GFAPI_HANDLE_LENGTH);
+        if (!oldp_object) {
+            gf_smsg(THIS->name, GF_LOG_DEBUG, errno,
+                    API_MSG_CREATE_HANDLE_FAILED, "gfid=%s",
+                    uuid_utoa(ca_data->oldp_stat.ia_gfid), NULL);
+            errno = ESTALE;
+            /* By the time we receive upcall old parent_dir may
+             * have got removed. We still need to send upcall
+             * for the file/dir and current parent handles. */
+            up_inode_arg->oldp_object = NULL;
+            ret = 0;
+        }
+
+        glfs_iatt_to_stat(fs, &ca_data->oldp_stat, &up_inode_arg->oldp_buf);
+    }
+    up_inode_arg->oldp_object = oldp_object;
+
+    up_arg->reason = GLFS_UPCALL_INODE_INVALIDATE;
+    up_arg->event = up_inode_arg;
+    up_arg->free_event = glfs_free_upcall_inode;
+
+    ret = 0;
 
 out:
-        /* TODO: Check where the inode ref is being held? */
-        loc_wipe (&loc);
-
-        priv_glfs_subvol_done (fs, subvol);
-
-        return object;
+    if (ret) {
+        /* Close p_object and oldp_object as well if being referenced.*/
+        if (object)
+            glfs_h_close(object);
+
+        /* Set reason to prevent applications from using ->event */
+        up_arg->reason = GLFS_UPCALL_EVENT_NULL;
+        GF_FREE(up_inode_arg);
+    }
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_create_from_handle, 3.4.2);
+void
+glfs_release_upcall(void *ptr)
+{
+    struct glfs_upcall *to_free = ptr;
 
+    if (to_free->event)
+        to_free->free_event(to_free->event);
+}
 
+/*
+ * This API is used to poll for upcall events stored in the upcall list.
+ * Current users of this API is NFS-Ganesha. In case of any event received, it
+ * will be mapped appropriately into 'glfs_upcall' along with the handle object
+ * to be passed to NFS-Ganesha.
+ *
+ * On success, applications need to check if up_arg is not-NULL or errno is not
+ * ENOENT. glfs_upcall_get_reason() can be used to decide what kind of event
+ * has been received.
+ *
+ * Current supported upcall_events:
+ *      GLFS_UPCALL_INODE_INVALIDATE
+ *
+ * After processing the event, applications need to free 'up_arg' by calling
+ * glfs_free().
+ *
+ * Also similar to I/Os, the application should ideally stop polling before
+ * calling glfs_fini(..). Hence making an assumption that 'fs' & ctx structures
+ * cannot be freed while in this routine.
+ */
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_poll_upcall, 3.7.16)
 int
-pub_glfs_h_close (struct glfs_object *object)
+pub_glfs_h_poll_upcall(struct glfs *fs, struct glfs_upcall **up_arg)
 {
-        /* Release the held reference */
-        inode_unref (object->inode);
-        GF_FREE (object);
+    upcall_entry *u_list = NULL;
+    upcall_entry *tmp = NULL;
+    xlator_t *subvol = NULL;
+    glusterfs_ctx_t *ctx = NULL;
+    int ret = -1;
+    struct gf_upcall *upcall_data = NULL;
+
+    DECLARE_OLD_THIS;
+
+    if (!up_arg) {
+        errno = EINVAL;
+        goto err;
+    }
+
+    __GLFS_ENTRY_VALIDATE_FS(fs, err);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        errno = EIO;
+        goto restore;
+    }
+
+    /* Ideally applications should stop polling before calling
+     * 'glfs_fini'. Yet cross check if cleanup has started. */
+    pthread_mutex_lock(&fs->mutex);
+    {
+        ctx = fs->ctx;
+
+        if (ctx->cleanup_started) {
+            pthread_mutex_unlock(&fs->mutex);
+            goto out;
+        }
+
+        fs->pin_refcnt++;
+
+        /* once we call this function, the applications seems to be
+         * interested in events, enable caching them */
+        fs->cache_upcalls = _gf_true;
+    }
+    pthread_mutex_unlock(&fs->mutex);
+
+    pthread_mutex_lock(&fs->upcall_list_mutex);
+    {
+        list_for_each_entry_safe(u_list, tmp, &fs->upcall_list, upcall_list)
+        {
+            list_del_init(&u_list->upcall_list);
+            upcall_data = &u_list->upcall_data;
+            break;
+        }
+    }
+    /* No other thread can delete this entry. So unlock it */
+    pthread_mutex_unlock(&fs->upcall_list_mutex);
+
+    if (upcall_data) {
+        switch (upcall_data->event_type) {
+            case GF_UPCALL_CACHE_INVALIDATION:
+                *up_arg = GLFS_CALLOC(1, sizeof(struct gf_upcall),
+                                      glfs_release_upcall,
+                                      glfs_mt_upcall_entry_t);
+                if (!*up_arg) {
+                    errno = ENOMEM;
+                    break; /* goto free u_list */
+                }
 
-        return 0;
-}
+                /* XXX: Need to revisit this to support
+                 * GLFS_UPCALL_INODE_UPDATE if required. */
+                ret = glfs_h_poll_cache_invalidation(fs, *up_arg, upcall_data);
+                if (ret || (*up_arg)->reason == GLFS_UPCALL_EVENT_NULL) {
+                    /* It could so happen that the file which got
+                     * upcall notification may have got deleted by
+                     * the same client. Irrespective of the error,
+                     * return with an error or success+ENOENT. */
+                    if ((*up_arg)->reason == GLFS_UPCALL_EVENT_NULL)
+                        errno = ENOENT;
+
+                    GLFS_FREE(*up_arg);
+                    *up_arg = NULL;
+                }
+                break;
+            case GF_UPCALL_RECALL_LEASE:
+                gf_log("glfs_h_poll_upcall", GF_LOG_DEBUG,
+                       "UPCALL_RECALL_LEASE is not implemented yet");
+            /* fallthrough till we support leases */
+            case GF_UPCALL_EVENT_NULL:
+                /* no 'default:' label, to force handling all upcall events */
+                errno = ENOENT;
+                break;
+        }
+
+        GF_FREE(u_list->upcall_data.data);
+        GF_FREE(u_list);
+    } else {
+        /* fs->upcall_list was empty, no upcall events cached */
+        errno = ENOENT;
+    }
+
+    ret = 0;
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_close, 3.4.2);
+out:
+    pthread_mutex_lock(&fs->mutex);
+    {
+        fs->pin_refcnt--;
+    }
+    pthread_mutex_unlock(&fs->mutex);
+
+    glfs_subvol_done(fs, subvol);
+
+restore:
+    __GLFS_EXIT_FS;
+err:
+    return ret;
+}
 
+static gf_boolean_t log_upcall370 = _gf_true; /* log once */
 
+/* The old glfs_h_poll_upcall interface requires intimate knowledge of the
+ * structures that are returned to the calling application. This is not
+ * recommended, as the returned structures need to returned correctly (handles
+ * closed, memory free'd with the unavailable GF_FREE(), and possibly more.)
+ *
+ * To the best of our knowledge, only NFS-Ganesha uses the upcall events
+ * through gfapi. We keep this backwards compatibility function around so that
+ * applications using the existing implementation do not break.
+ *
+ * WARNING: this function will be removed in the future.
+ */
+GFAPI_SYMVER_PUBLIC(glfs_h_poll_upcall370, glfs_h_poll_upcall, 3.7.0)
 int
-pub_glfs_h_truncate (struct glfs *fs, struct glfs_object *object, off_t offset)
+pub_glfs_h_poll_upcall370(struct glfs *fs, struct glfs_callback_arg *up_arg)
 {
-        loc_t               loc = {0, };
-        int                 ret = -1;
-        xlator_t           *subvol = NULL;
-        inode_t            *inode = NULL;
-
-        /* validate in args */
-        if ((fs == NULL) || (object == NULL)) {
-                errno = EINVAL;
-                return -1;
-        }
-
-        __glfs_entry_fs (fs);
-
-        /* get the active volume */
-        subvol = priv_glfs_active_subvol (fs);
-        if (!subvol) {
+    struct glfs_upcall *upcall = NULL;
+    int ret = -1;
+
+    if (log_upcall370) {
+        log_upcall370 = _gf_false;
+        gf_log(THIS->name, GF_LOG_WARNING,
+               "this application is "
+               "compiled against an old version of libgfapi, it "
+               "should use glfs_free() to release the structure "
+               "returned by glfs_h_poll_upcall() - for more details, "
+               "see http://review.gluster.org/14701");
+    }
+
+    ret = pub_glfs_h_poll_upcall(fs, &upcall);
+    if (ret == 0) {
+        up_arg->fs = fs;
+        if ((errno == ENOENT) || !upcall || !upcall->event) {
+            up_arg->reason = GLFS_UPCALL_EVENT_NULL;
+            goto out;
+        }
+
+        up_arg->reason = upcall->reason;
+
+        if (upcall->reason == GLFS_UPCALL_INODE_INVALIDATE) {
+            struct glfs_callback_inode_arg *cb_inode = NULL;
+            struct glfs_upcall_inode *up_inode = NULL;
+
+            cb_inode = GF_CALLOC(1, sizeof(struct glfs_callback_inode_arg),
+                                 glfs_mt_upcall_inode_t);
+            if (!cb_inode) {
+                errno = ENOMEM;
                 ret = -1;
-                errno = EIO;
                 goto out;
-        }
+            }
 
-        /* get/refresh the in arg objects inode in correlation to the xlator */
-        inode = glfs_resolve_inode (fs, subvol, object);
-        if (!inode) {
-                errno = ESTALE;
-                goto out;
-        }
+            up_inode = upcall->event;
 
-        GLFS_LOC_FILL_INODE (inode, loc, out);
+            /* copy attributes one by one, the memory layout might
+             * be different between the old glfs_callback_inode_arg
+             * and new glfs_upcall_inode */
+            cb_inode->object = up_inode->object;
+            cb_inode->flags = up_inode->flags;
+            memcpy(&cb_inode->buf, &up_inode->buf, sizeof(struct stat));
+            cb_inode->expire_time_attr = up_inode->expire_time_attr;
+            cb_inode->p_object = up_inode->p_object;
+            memcpy(&cb_inode->p_buf, &up_inode->p_buf, sizeof(struct stat));
+            cb_inode->oldp_object = up_inode->oldp_object;
+            memcpy(&cb_inode->oldp_buf, &up_inode->oldp_buf,
+                   sizeof(struct stat));
 
-        /* fop/op */
-        ret = syncop_truncate (subvol, &loc, (off_t)offset);
-        DECODE_SYNCOP_ERR (ret);
-
-        /* populate out args */
-        if (ret == 0)
-                ret = glfs_loc_unlink (&loc);
+            up_arg->event_arg = cb_inode;
+        }
+    }
 
 out:
-        loc_wipe (&loc);
-
-        if (inode)
-                inode_unref (inode);
-
-        priv_glfs_subvol_done (fs, subvol);
+    if (upcall) {
+        /* we can not use glfs_free() here, objects need to stay */
+        GF_FREE(upcall->event);
+        GF_FREE(upcall);
+    }
 
-        return ret;
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_truncate, 3.4.2);
-
+#ifdef HAVE_ACL_LIBACL_H
+#include <glusterfs/glusterfs-acl.h>
+#include <acl/libacl.h>
 
-struct glfs_object *
-pub_glfs_h_symlink (struct glfs *fs, struct glfs_object *parent,
-                    const char *name, const char *data, struct stat *stat)
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_acl_set, 3.7.0)
+int
+pub_glfs_h_acl_set(struct glfs *fs, struct glfs_object *object,
+                   const acl_type_t type, const acl_t acl)
 {
-        int                 ret = -1;
-        xlator_t           *subvol = NULL;
-        inode_t            *inode = NULL;
-        loc_t               loc = {0, };
-        struct iatt         iatt = {0, };
-        uuid_t              gfid;
-        dict_t             *xattr_req = NULL;
-        struct glfs_object *object = NULL;
-
-        /* validate in args */
-        if ((fs == NULL) || (parent == NULL) || (name == NULL) ||
-                (data == NULL)) {
-                errno = EINVAL;
-                return NULL;
-        }
+    int ret = -1;
+    char *acl_s = NULL;
+    const char *acl_key = NULL;
+    struct glfs_object *new_object = NULL;
 
-        __glfs_entry_fs (fs);
+    DECLARE_OLD_THIS;
 
-        /* get the active volume */
-        subvol = priv_glfs_active_subvol (fs);
-        if (!subvol) {
-                ret = -1;
-                errno = EIO;
-                goto out;
-        }
+    if (!object || !acl) {
+        errno = EINVAL;
+        return ret;
+    }
 
-        /* get/refresh the in arg objects inode in correlation to the xlator */
-        inode = glfs_resolve_inode (fs, subvol, parent);
-        if (!inode) {
-                errno = ESTALE;
-                goto out;
-        }
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
 
-        xattr_req = dict_new ();
-        if (!xattr_req) {
-                ret = -1;
-                errno = ENOMEM;
-                goto out;
-        }
+    acl_key = gf_posix_acl_get_key(type);
+    if (!acl_key)
+        goto out;
 
-        uuid_generate (gfid);
-        ret = dict_set_static_bin (xattr_req, "gfid-req", gfid, 16);
-        if (ret) {
-                ret = -1;
-                errno = ENOMEM;
-                goto out;
-        }
+    acl_s = acl_to_any_text(acl, NULL, ',', TEXT_ABBREVIATE | TEXT_NUMERIC_IDS);
+    if (!acl_s)
+        goto out;
 
-        GLFS_LOC_FILL_PINODE (inode, loc, ret, errno, out, name);
-
-        /* fop/op */
-        ret = syncop_symlink (subvol, &loc, data, xattr_req, &iatt);
-        DECODE_SYNCOP_ERR (ret);
-
-        /* populate out args */
-        if (ret == 0) {
-                /* TODO: If the inode existed in the cache (say file already
-                 * exists), then the glfs_loc_link will not update the
-                 * loc.inode, as a result we will have a 0000 GFID that we
-                 * would copy out to the object, this needs to be fixed.
-                 */
-                ret = glfs_loc_link (&loc, &iatt);
-                if (ret != 0) {
-                        goto out;
-                }
+    if (IA_ISLNK(object->inode->ia_type)) {
+        new_object = glfs_h_resolve_symlink(fs, object);
+        if (new_object == NULL)
+            goto out;
+    } else
+        new_object = object;
 
-                if (stat)
-                        glfs_iatt_to_stat (fs, &iatt, stat);
+    ret = pub_glfs_h_setxattrs(fs, new_object, acl_key, acl_s,
+                               strlen(acl_s) + 1, 0);
 
-                ret = glfs_create_object (&loc, &object);
-        }
+    acl_free(acl_s);
 
 out:
-        if (ret && object != NULL) {
-                pub_glfs_h_close (object);
-                object = NULL;
-        }
-
-        loc_wipe(&loc);
-
-        if (inode)
-                inode_unref (inode);
-
-        if (xattr_req)
-                dict_unref (xattr_req);
+    if (IA_ISLNK(object->inode->ia_type) && new_object)
+        glfs_h_close(new_object);
 
-        priv_glfs_subvol_done (fs, subvol);
+    __GLFS_EXIT_FS;
 
-        return object;
+invalid_fs:
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_symlink, 3.4.2);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_acl_get, 3.7.0)
+acl_t
+pub_glfs_h_acl_get(struct glfs *fs, struct glfs_object *object,
+                   const acl_type_t type)
+{
+    int ret = 0;
+    acl_t acl = NULL;
+    char *acl_s = NULL;
+    dict_t *xattr = NULL;
+    const char *acl_key = NULL;
+    struct glfs_object *new_object = NULL;
 
+    DECLARE_OLD_THIS;
 
-int
-pub_glfs_h_readlink (struct glfs *fs, struct glfs_object *object, char *buf,
-                     size_t bufsiz)
-{
-        loc_t               loc = {0, };
-        int                 ret = -1;
-        xlator_t           *subvol = NULL;
-        inode_t            *inode = NULL;
-        char               *linkval = NULL;
-
-        /* validate in args */
-        if ((fs == NULL) || (object == NULL) || (buf == NULL)) {
-                errno = EINVAL;
-                return -1;
-        }
+    if (!object) {
+        errno = EINVAL;
+        return NULL;
+    }
 
-        __glfs_entry_fs (fs);
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
 
-        /* get the active volume */
-        subvol = priv_glfs_active_subvol (fs);
-        if (!subvol) {
-                ret = -1;
-                errno = EIO;
-                goto out;
-        }
+    acl_key = gf_posix_acl_get_key(type);
+    if (!acl_key)
+        goto out;
 
-        /* get/refresh the in arg objects inode in correlation to the xlator */
-        inode = glfs_resolve_inode (fs, subvol, object);
-        if (!inode) {
-                errno = ESTALE;
-                goto out;
-        }
+    if (IA_ISLNK(object->inode->ia_type)) {
+        new_object = glfs_h_resolve_symlink(fs, object);
+        if (new_object == NULL)
+            goto out;
+    } else
+        new_object = object;
 
-        GLFS_LOC_FILL_INODE (inode, loc, out);
+    ret = glfs_h_getxattrs_common(fs, new_object, &xattr, acl_key, _gf_false);
+    if (ret)
+        goto out;
 
-        /* fop/op */
-        ret = syncop_readlink (subvol, &loc, &linkval, bufsiz);
-        DECODE_SYNCOP_ERR (ret);
+    ret = dict_get_str(xattr, (char *)acl_key, &acl_s);
+    if (ret)
+        goto out;
 
-        /* populate out args */
-        if (ret > 0)
-                memcpy (buf, linkval, ret);
+    acl = acl_from_text(acl_s);
 
 out:
-        loc_wipe (&loc);
+    if (xattr)
+        dict_unref(xattr);
 
-        if (inode)
-                inode_unref (inode);
+    if (IA_ISLNK(object->inode->ia_type) && new_object)
+        glfs_h_close(new_object);
 
-        if (linkval)
-                GF_FREE (linkval);
+    __GLFS_EXIT_FS;
 
-        priv_glfs_subvol_done (fs, subvol);
-
-        return ret;
+invalid_fs:
+    return acl;
+}
+#else /* !HAVE_ACL_LIBACL_H */
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_acl_get, 3.7.0)
+acl_t
+pub_glfs_h_acl_get(struct glfs *fs, struct glfs_object *object,
+                   const acl_type_t type)
+{
+    errno = ENOTSUP;
+    return NULL;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_readlink, 3.4.2);
-
-
-extern int
-priv_glfs_loc_touchup (loc_t *);
-
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_acl_set, 3.7.0)
 int
-pub_glfs_h_link (struct glfs *fs, struct glfs_object *linksrc,
-             struct glfs_object *parent, const char *name)
+pub_glfs_h_acl_set(struct glfs *fs, struct glfs_object *object,
+                   const acl_type_t type, const acl_t acl)
 {
-        int                 ret = -1;
-        xlator_t           *subvol = NULL;
-        inode_t            *inode = NULL;
-        inode_t            *pinode = NULL;
-        loc_t               oldloc = {0, };
-        loc_t               newloc = {0, };
-
-        /* validate in args */
-        if ((fs == NULL) || (linksrc == NULL) || (parent == NULL) ||
-                (name == NULL)) {
-                errno = EINVAL;
-                return -1;
-        }
-
-        __glfs_entry_fs (fs);
-
-        /* get the active volume */
-        subvol = priv_glfs_active_subvol (fs);
-        if (!subvol) {
-                ret = -1;
-                errno = EIO;
-                goto out;
-        }
-
-        /* get/refresh the in arg objects inode in correlation to the xlator */
-        inode = glfs_resolve_inode (fs, subvol, linksrc);
-        if (!inode) {
-                errno = ESTALE;
-                goto out;
-        }
-
-        if (inode->ia_type == IA_IFDIR) {
-                ret = -1;
-                errno = EISDIR;
-                goto out;
-        }
+    errno = ENOTSUP;
+    return -1;
+}
+#endif
 
-        GLFS_LOC_FILL_INODE (inode, oldloc, out);
+/* The API to perform read using anonymous fd */
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_anonymous_read, 3.7.0)
+ssize_t
+pub_glfs_h_anonymous_read(struct glfs *fs, struct glfs_object *object,
+                          const void *buf, size_t count, off_t offset)
+{
+    struct iovec iov = {
+        0,
+    };
+    ssize_t ret = 0;
 
-        /* get/refresh the in arg objects inode in correlation to the xlator */
-        pinode = glfs_resolve_inode (fs, subvol, parent);
-        if (!pinode) {
-                errno = ESTALE;
-                goto out;
-        }
+    /* validate in args */
+    if ((fs == NULL) || (object == NULL)) {
+        errno = EINVAL;
+        return -1;
+    }
 
-        /* setup newloc based on parent */
-        newloc.parent = inode_ref (pinode);
-        newloc.name = name;
-        ret = priv_glfs_loc_touchup (&newloc);
-        if (ret != 0) {
-                errno = EINVAL;
-                goto out;
-        }
+    iov.iov_base = (void *)buf;
+    iov.iov_len = count;
 
-        /* Filling the inode of the hard link to be same as that of the
-         * original file
-         */
-        newloc.inode = inode_ref (inode);
+    ret = glfs_anonymous_preadv(fs, object, &iov, 1, offset, 0);
 
-        /* fop/op */
-        ret = syncop_link (subvol, &oldloc, &newloc);
-        DECODE_SYNCOP_ERR (ret);
+    return ret;
+}
 
-        if (ret == 0)
-                /* TODO: No iatt to pass as there has been no lookup */
-                ret = glfs_loc_link (&newloc, NULL);
-out:
-        loc_wipe (&oldloc);
-        loc_wipe (&newloc);
+/* The API to perform write using anonymous fd */
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_anonymous_write, 3.7.0)
+ssize_t
+pub_glfs_h_anonymous_write(struct glfs *fs, struct glfs_object *object,
+                           const void *buf, size_t count, off_t offset)
+{
+    struct iovec iov = {
+        0,
+    };
+    ssize_t ret = 0;
 
-        if (inode)
-                inode_unref (inode);
+    /* validate in args */
+    if ((fs == NULL) || (object == NULL)) {
+        errno = EINVAL;
+        return -1;
+    }
 
-        if (pinode)
-                inode_unref (pinode);
+    iov.iov_base = (void *)buf;
+    iov.iov_len = count;
 
-        priv_glfs_subvol_done (fs, subvol);
+    ret = glfs_anonymous_pwritev(fs, object, &iov, 1, offset, 0);
 
-        return ret;
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_link, 3.4.2);
-
-
-int
-pub_glfs_h_rename (struct glfs *fs, struct glfs_object *olddir,
-                   const char *oldname, struct glfs_object *newdir,
-                   const char *newname)
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_object_copy, 3.11.0)
+struct glfs_object *
+pub_glfs_object_copy(struct glfs_object *src)
 {
-        int                 ret = -1;
-        xlator_t           *subvol = NULL;
-        inode_t            *oldpinode = NULL;
-        inode_t            *newpinode = NULL;
-        loc_t               oldloc = {0, };
-        loc_t               newloc = {0, };
-        struct iatt         oldiatt = {0, };
-        struct iatt         newiatt = {0, };
-
-        /* validate in args */
-        if ((fs == NULL) || (olddir == NULL) || (oldname == NULL) ||
-                (newdir == NULL) || (newname == NULL)) {
-                errno = EINVAL;
-                return -1;
-        }
+    struct glfs_object *object = NULL;
 
-        __glfs_entry_fs (fs);
+    GF_VALIDATE_OR_GOTO("glfs_dup_object", src, out);
 
-        /* get the active volume */
-        subvol = priv_glfs_active_subvol (fs);
-        if ( !subvol ) {
-                ret = -1;
-                errno = EIO;
-                goto out;
-        }
+    object = GF_CALLOC(1, sizeof(struct glfs_object), glfs_mt_glfs_object_t);
+    if (object == NULL) {
+        errno = ENOMEM;
+        gf_smsg(THIS->name, GF_LOG_WARNING, errno, API_MSG_CREATE_HANDLE_FAILED,
+                "glfs_dup_object gfid=%s", uuid_utoa(src->inode->gfid), NULL);
+        return NULL;
+    }
 
-        /* get/refresh the in arg objects inode in correlation to the xlator */
-        oldpinode = glfs_resolve_inode (fs, subvol, olddir);
-        if (!oldpinode) {
-                errno = ESTALE;
-                goto out;
-        }
-
-        ret = priv_glfs_resolve_at (fs, subvol, oldpinode, oldname, &oldloc,
-                                    &oldiatt, 0 , 0);
-        if (ret != 0) {
-                goto out;
-        }
-
-        /* get/refresh the in arg objects inode in correlation to the xlator */
-        newpinode = glfs_resolve_inode (fs, subvol, newdir);
-        if (!newpinode) {
-                errno = ESTALE;
-                goto out;
-        }
+    object->inode = inode_ref(src->inode);
+    gf_uuid_copy(object->gfid, src->inode->gfid);
 
-        ret = priv_glfs_resolve_at (fs, subvol, newpinode, newname, &newloc,
-                                    &newiatt, 0, 0);
+out:
+    return object;
+}
 
-        if (ret && errno != ENOENT && newloc.parent)
-                goto out;
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_xreaddirplus_get_object, 3.11.0)
+struct glfs_object *
+pub_glfs_xreaddirplus_get_object(struct glfs_xreaddirp_stat *xstat)
+{
+    GF_VALIDATE_OR_GOTO("glfs_xreaddirplus_get_object", xstat, out);
 
-        if (newiatt.ia_type != IA_INVAL) {
-                if ((oldiatt.ia_type == IA_IFDIR) !=
-                        (newiatt.ia_type == IA_IFDIR)) {
-                        /* Either both old and new must be dirs,
-                         * or both must be non-dirs. Else, fail.
-                         */
-                        ret = -1;
-                        errno = EEXIST;
-                        goto out;
-                }
-        }
+    if (!(xstat->flags_handled & GFAPI_XREADDIRP_HANDLE))
+        gf_smsg(THIS->name, GF_LOG_ERROR, errno, API_MSG_HANDLE_NOT_SET,
+                "GFAPI_XREADDIRP_HANDLE xstat=%p", xstat, "handle=%x",
+                xstat->flags_handled, NULL);
 
-        /* TODO: check if new or old is a prefix of the other, and fail EINVAL */
+    return xstat->object;
 
-        ret = syncop_rename (subvol, &oldloc, &newloc);
-        DECODE_SYNCOP_ERR (ret);
+out:
+    return NULL;
+}
 
-        if (ret == 0)
-                inode_rename (oldloc.parent->table, oldloc.parent, oldloc.name,
-                              newloc.parent, newloc.name, oldloc.inode,
-                              &oldiatt);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_lease, 4.0.0)
+int
+pub_glfs_h_lease(struct glfs *fs, struct glfs_object *object,
+                 struct glfs_lease *lease)
+{
+    int ret = -1;
+    xlator_t *subvol = NULL;
+    inode_t *inode = NULL;
+    loc_t loc = {
+        0,
+    };
+    struct gf_lease gf_lease = {
+        0,
+    };
+
+    /* validate in args */
+    if ((fs == NULL) || (object == NULL)) {
+        errno = EINVAL;
+        return -1;
+    }
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* get the active volume */
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    /* get/refresh the in arg objects inode in correlation to the xlator */
+    inode = glfs_resolve_inode(fs, subvol, object);
+    if (!inode) {
+        errno = ESTALE;
+        goto out;
+    }
+
+    /* populate loc */
+    GLFS_LOC_FILL_INODE(inode, loc, out);
+
+    glfs_lease_to_gf_lease(lease, &gf_lease);
+
+    ret = syncop_lease(subvol, &loc, &gf_lease, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    gf_lease_to_glfs_lease(&gf_lease, lease);
 
 out:
-        loc_wipe (&oldloc);
-        loc_wipe (&newloc);
+    loc_wipe(&loc);
 
-        if (oldpinode)
-                inode_unref (oldpinode);
+    if (inode)
+        inode_unref(inode);
 
-        if (newpinode)
-                inode_unref (newpinode);
+    glfs_subvol_done(fs, subvol);
 
-        priv_glfs_subvol_done (fs, subvol);
+    __GLFS_EXIT_FS;
 
-        return ret;
+invalid_fs:
+    return ret;
 }
-
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_h_rename, 3.4.2);
-
diff --git a/api/src/glfs-handles.h b/api/src/glfs-handles.h
index ad963455e85..4d039b9c76b 100644
--- a/api/src/glfs-handles.h
+++ b/api/src/glfs-handles.h
@@ -1,5 +1,5 @@
 /*
-  Copyright (c) 2013 Red Hat, Inc. <http://www.redhat.com>
+  Copyright (c) 2013-2018 Red Hat, Inc. <http://www.redhat.com>
   This file is part of GlusterFS.
 
   This file is licensed to you under your choice of the GNU Lesser
@@ -46,27 +46,39 @@
  *
  */
 
-/* Values for valid falgs to be used when using XXXsetattr, to set multiple
- attribute values passed via the related stat structure.
- */
-#define GFAPI_SET_ATTR_MODE  0x1
-#define GFAPI_SET_ATTR_UID   0x2
-#define GFAPI_SET_ATTR_GID   0x4
-#define GFAPI_SET_ATTR_SIZE  0x8
-#define GFAPI_SET_ATTR_ATIME 0x10
-#define GFAPI_SET_ATTR_MTIME 0x20
-
 /* Handle length for object handles returned from glfs_h_extract_handle or
  * glfs_h_create_from_handle */
 #define GFAPI_HANDLE_LENGTH 16
 
+/* These flags should be in sync to the ones defined in upcall.h */
+#define GFAPI_UP_NLINK 0x00000001 /* update nlink */
+#define GFAPI_UP_MODE 0x00000002  /* update mode and ctime */
+#define GFAPI_UP_OWN 0x00000004   /* update mode,uid,gid and ctime */
+#define GFAPI_UP_SIZE 0x00000008  /* update fsize */
+#define GFAPI_UP_TIMES 0x00000010 /* update all times */
+#define GFAPI_UP_ATIME 0x00000020 /* update atime only */
+#define GFAPI_UP_PERM                                                          \
+    0x00000040 /* update fields needed for                                     \
+                  permission checking */
+#define GFAPI_UP_RENAME                                                        \
+    0x00000080 /* this is a rename op -                                        \
+                  delete the cache entry */
+#define GFAPI_UP_FORGET                                                        \
+    0x00000100                           /* inode_forget on server side -      \
+                                            invalidate the cache entry */
+#define GFAPI_UP_PARENT_TIMES 0x00000200 /* update parent dir times */
+
+#define GFAPI_INODE_UPDATE_FLAGS                                               \
+    (GFAPI_UP_NLINK | GFAPI_UP_MODE | GFAPI_UP_OWN | GFAPI_UP_SIZE |           \
+     GFAPI_UP_TIMES | GFAPI_UP_ATIME)
+
 /* Portability non glibc c++ build systems */
 #ifndef __THROW
-# if defined __cplusplus
-#  define __THROW       throw ()
-# else
-#  define __THROW
-# endif
+#if defined __cplusplus
+#define __THROW throw()
+#else
+#define __THROW
+#endif
 #endif
 
 __BEGIN_DECLS
@@ -82,88 +94,261 @@ __BEGIN_DECLS
 struct glfs_object;
 typedef struct glfs_object glfs_object_t;
 
-/* Handle based operations */
-/* Operations that generate handles */
-struct glfs_object *glfs_h_lookupat (struct glfs *fs,
-				     struct glfs_object *parent,
-				     const char *path,
-                                     struct stat *stat) __THROW;
+/* Functions for getting details about the glfs_upcall_inode
+ *
+ * None of the pointers returned by the below functions should be free()'d,
+ * glfs_free()'d or glfs_h_close()'d by the application.
+ *
+ * Releasing of the structures is done by passing the glfs_upcall pointer
+ * to glfs_free().
+ */
+struct glfs_upcall_inode;
+typedef struct glfs_upcall_inode glfs_upcall_inode_t;
+
+glfs_object_t *
+glfs_upcall_inode_get_object(glfs_upcall_inode_t *arg) __THROW
+    GFAPI_PUBLIC(glfs_upcall_inode_get_object, 3.7.16);
+
+uint64_t
+glfs_upcall_inode_get_flags(glfs_upcall_inode_t *arg) __THROW
+    GFAPI_PUBLIC(glfs_upcall_inode_get_flags, 3.7.16);
+
+struct stat *
+glfs_upcall_inode_get_stat(glfs_upcall_inode_t *arg) __THROW
+    GFAPI_PUBLIC(glfs_upcall_inode_get_stat, 3.7.16);
 
-struct glfs_object *glfs_h_creat (struct glfs *fs, struct glfs_object *parent,
-				  const char *path, int flags, mode_t mode,
-				  struct stat *sb) __THROW;
+uint64_t
+glfs_upcall_inode_get_expire(glfs_upcall_inode_t *arg) __THROW
+    GFAPI_PUBLIC(glfs_upcall_inode_get_expire, 3.7.16);
 
-struct glfs_object *glfs_h_mkdir (struct glfs *fs, struct glfs_object *parent,
-				  const char *path, mode_t flags,
-				  struct stat *sb) __THROW;
+glfs_object_t *
+glfs_upcall_inode_get_pobject(glfs_upcall_inode_t *arg) __THROW
+    GFAPI_PUBLIC(glfs_upcall_inode_get_pobject, 3.7.16);
 
-struct glfs_object *glfs_h_mknod (struct glfs *fs, struct glfs_object *parent,
-				  const char *path, mode_t mode, dev_t dev,
-				  struct stat *sb) __THROW;
+struct stat *
+glfs_upcall_inode_get_pstat(glfs_upcall_inode_t *arg) __THROW
+    GFAPI_PUBLIC(glfs_upcall_inode_get_pstat, 3.7.16);
 
-struct glfs_object *glfs_h_symlink (struct glfs *fs, struct glfs_object *parent,
-				    const char *name, const char *data,
-				    struct stat *stat) __THROW;
+glfs_object_t *
+glfs_upcall_inode_get_oldpobject(glfs_upcall_inode_t *arg) __THROW
+    GFAPI_PUBLIC(glfs_upcall_inode_get_oldpobject, 3.7.16);
+
+struct stat *
+glfs_upcall_inode_get_oldpstat(glfs_upcall_inode_t *arg) __THROW
+    GFAPI_PUBLIC(glfs_upcall_inode_get_oldpstat, 3.7.16);
+
+/* Handle based operations */
+/* Operations that generate handles */
+glfs_object_t *
+glfs_h_lookupat(glfs_t *fs, glfs_object_t *parent, const char *path,
+                struct stat *stat, int follow) __THROW
+    GFAPI_PUBLIC(glfs_h_lookupat, 3.7.4);
+
+glfs_object_t *
+glfs_h_creat(glfs_t *fs, glfs_object_t *parent, const char *path, int flags,
+             mode_t mode, struct stat *sb) __THROW
+    GFAPI_PUBLIC(glfs_h_creat, 3.4.2);
+
+glfs_object_t *
+glfs_h_mkdir(glfs_t *fs, glfs_object_t *parent, const char *path, mode_t flags,
+             struct stat *sb) __THROW GFAPI_PUBLIC(glfs_h_mkdir, 3.4.2);
+
+glfs_object_t *
+glfs_h_mknod(glfs_t *fs, glfs_object_t *parent, const char *path, mode_t mode,
+             dev_t dev, struct stat *sb) __THROW
+    GFAPI_PUBLIC(glfs_h_mknod, 3.4.2);
+
+glfs_object_t *
+glfs_h_symlink(glfs_t *fs, glfs_object_t *parent, const char *name,
+               const char *data, struct stat *stat) __THROW
+    GFAPI_PUBLIC(glfs_h_symlink, 3.4.2);
 
 /* Operations on the actual objects */
-int glfs_h_unlink (struct glfs *fs, struct glfs_object *parent,
-		   const char *path) __THROW;
+int
+glfs_h_unlink(glfs_t *fs, glfs_object_t *parent, const char *path) __THROW
+    GFAPI_PUBLIC(glfs_h_unlink, 3.4.2);
+
+int
+glfs_h_close(glfs_object_t *object) __THROW GFAPI_PUBLIC(glfs_h_close, 3.4.2);
 
-int glfs_h_close (struct glfs_object *object) __THROW;
+int
+glfs_caller_specific_init(void *uid_caller_key, void *gid_caller_key,
+                          void *future) __THROW
+    GFAPI_PUBLIC(glfs_caller_specific_init, 3.5.0);
 
-int glfs_caller_specific_init (void *uid_caller_key, void *gid_caller_key,
-			       void *future) __THROW;
+int
+glfs_h_truncate(glfs_t *fs, glfs_object_t *object, off_t offset) __THROW
+    GFAPI_PUBLIC(glfs_h_truncate, 3.4.2);
 
-int glfs_h_truncate (struct glfs *fs, struct glfs_object *object,
-                     off_t offset) __THROW;
+int
+glfs_h_stat(glfs_t *fs, glfs_object_t *object, struct stat *stat) __THROW
+    GFAPI_PUBLIC(glfs_h_stat, 3.4.2);
 
-int glfs_h_stat(struct glfs *fs, struct glfs_object *object,
-                struct stat *stat) __THROW;
+int
+glfs_h_statfs(glfs_t *fs, glfs_object_t *object, struct statvfs *stat) __THROW
+    GFAPI_PUBLIC(glfs_h_statfs, 3.7.0);
 
-int glfs_h_getattrs (struct glfs *fs, struct glfs_object *object,
-		     struct stat *stat) __THROW;
+int
+glfs_h_getattrs(glfs_t *fs, glfs_object_t *object, struct stat *stat) __THROW
+    GFAPI_PUBLIC(glfs_h_getattrs, 3.4.2);
 
-int glfs_h_getxattrs (struct glfs *fs, struct glfs_object *object,
-		      const char *name, void *value,
-		      size_t size) __THROW;
+int
+glfs_h_getxattrs(glfs_t *fs, glfs_object_t *object, const char *name,
+                 void *value, size_t size) __THROW
+    GFAPI_PUBLIC(glfs_h_getxattrs, 3.5.1);
 
-int glfs_h_setattrs (struct glfs *fs, struct glfs_object *object,
-		     struct stat *sb, int valid) __THROW;
+int
+glfs_h_setattrs(glfs_t *fs, glfs_object_t *object, struct stat *sb,
+                int valid) __THROW GFAPI_PUBLIC(glfs_h_setattrs, 3.4.2);
 
-int glfs_h_setxattrs (struct glfs *fs, struct glfs_object *object,
-		      const char *name, const void *value,
-		      size_t size, int flags) __THROW;
+int
+glfs_h_setxattrs(glfs_t *fs, glfs_object_t *object, const char *name,
+                 const void *value, size_t size, int flags) __THROW
+    GFAPI_PUBLIC(glfs_h_setxattrs, 3.5.0);
 
-int glfs_h_readlink (struct glfs *fs, struct glfs_object *object, char *buf,
-		     size_t bufsiz) __THROW;
+int
+glfs_h_readlink(glfs_t *fs, glfs_object_t *object, char *buf,
+                size_t bufsiz) __THROW GFAPI_PUBLIC(glfs_h_readlink, 3.4.2);
 
-int glfs_h_link (struct glfs *fs, struct glfs_object *linktgt,
-		 struct glfs_object *parent, const char *name) __THROW;
+int
+glfs_h_link(glfs_t *fs, glfs_object_t *linktgt, glfs_object_t *parent,
+            const char *name) __THROW GFAPI_PUBLIC(glfs_h_link, 3.4.2);
 
-int glfs_h_rename (struct glfs *fs, struct glfs_object *olddir,
-		   const char *oldname, struct glfs_object *newdir,
-		   const char *newname) __THROW;
+int
+glfs_h_rename(glfs_t *fs, glfs_object_t *olddir, const char *oldname,
+              glfs_object_t *newdir, const char *newname) __THROW
+    GFAPI_PUBLIC(glfs_h_rename, 3.4.2);
 
-int glfs_h_removexattrs (struct glfs *fs, struct glfs_object *object,
-			 const char *name) __THROW;
+int
+glfs_h_removexattrs(glfs_t *fs, glfs_object_t *object, const char *name) __THROW
+    GFAPI_PUBLIC(glfs_h_removexattrs, 3.5.1);
 
 /* Operations enabling opaque invariant handle to object transitions */
-ssize_t glfs_h_extract_handle (struct glfs_object *object,
-			       unsigned char *handle, int len) __THROW;
+ssize_t
+glfs_h_extract_handle(glfs_object_t *object, unsigned char *handle,
+                      int len) __THROW
+    GFAPI_PUBLIC(glfs_h_extract_handle, 3.4.2);
 
-struct glfs_object *glfs_h_create_from_handle (struct glfs *fs,
-					       unsigned char *handle, int len,
-					       struct stat *stat) __THROW;
+/* Given a handle, looks up the inode and creates glfs_object.
+ * In addition, if provided 'stat', copies the inode attributes
+ */
+glfs_object_t *
+glfs_h_create_from_handle(glfs_t *fs, unsigned char *handle, int len,
+                          struct stat *stat) __THROW
+    GFAPI_PUBLIC(glfs_h_create_from_handle, 3.4.2);
 
 /* Operations enabling object handles to fd transitions */
-struct glfs_fd *glfs_h_opendir (struct glfs *fs,
-                                struct glfs_object *object) __THROW;
+glfs_fd_t *
+glfs_h_opendir(glfs_t *fs, glfs_object_t *object) __THROW
+    GFAPI_PUBLIC(glfs_h_opendir, 3.4.2);
 
-struct glfs_fd *glfs_h_open (struct glfs *fs, struct glfs_object *object,
-			     int flags) __THROW;
+glfs_fd_t *
+glfs_h_open(glfs_t *fs, glfs_object_t *object, int flags) __THROW
+    GFAPI_PUBLIC(glfs_h_open, 3.4.2);
 
 int
-glfs_h_access (struct glfs *fs, struct glfs_object *object, int mask) __THROW;
+glfs_h_access(glfs_t *fs, glfs_object_t *object, int mask) __THROW
+    GFAPI_PUBLIC(glfs_h_access, 3.6.0);
+
+struct glfs_object *
+glfs_h_creat_open(struct glfs *fs, struct glfs_object *parent, const char *path,
+                  int flags, mode_t mode, struct stat *stat,
+                  struct glfs_fd **out_fd) __THROW
+    GFAPI_PUBLIC(glfs_h_creat_open, 6.6);
+/*
+  SYNOPSIS
+
+  glfs_h_poll_upcall: Poll for upcall events given a 'glfs' object.
+
+  DESCRIPTION
+
+  This API is used to poll for upcall events stored in the
+  upcall list. Current users of this API is NFS-Ganesha.
+  In case of any event received, it will be mapped appropriately
+  into 'glfs_upcall' along with the handle('glfs_object') to be
+  passed to NFS-Ganesha.
+
+  In case of success, applications need to check the value of
+  cbk->handle to be NON NULL before processing the upcall
+  events.
+
+  PARAMETERS
+
+  @fs: glfs object to poll the upcall events for
+  @cbk: Pointer that will contain an upcall event for use by the application.
+        Application is responsible for free'ing the structure with glfs_free().
+
+  RETURN VALUES
+
+  0   : Success.
+  -1  : Error condition, mostly due to out of memory.
+
+*/
+
+int
+glfs_h_poll_upcall(glfs_t *fs, glfs_upcall_t **cbk) __THROW
+    GFAPI_PUBLIC(glfs_h_poll_upcall, 3.7.16);
+
+int
+glfs_h_acl_set(glfs_t *fs, glfs_object_t *object, const acl_type_t type,
+               const acl_t acl) __THROW GFAPI_PUBLIC(glfs_h_acl_set, 3.7.0);
+
+acl_t
+glfs_h_acl_get(glfs_t *fs, glfs_object_t *object, const acl_type_t type) __THROW
+    GFAPI_PUBLIC(glfs_h_acl_get, 3.7.0);
+
+size_t
+glfs_h_anonymous_write(glfs_t *fs, glfs_object_t *object, const void *buf,
+                       size_t count, off_t offset) __THROW
+    GFAPI_PUBLIC(glfs_h_anonymous_write, 3.7.0);
+
+ssize_t
+glfs_h_anonymous_read(glfs_t *fs, glfs_object_t *object, const void *buf,
+                      size_t count, off_t offset) __THROW
+    GFAPI_PUBLIC(glfs_h_anonymous_read, 3.7.0);
+
+/*
+ * Caution: The object returned by this object gets freed as part
+ * of 'glfs_free(xstat)'. Make sure to have a copy using 'glfs_object_copy()'
+ * to use post that.
+ */
+glfs_object_t *
+glfs_xreaddirplus_get_object(struct glfs_xreaddirp_stat *xstat) __THROW
+    GFAPI_PUBLIC(glfs_xreaddirplus_get_object, 3.11.0);
+
+/* Applications should close the object returned by this routine
+ * explicitly using 'glfs_h_close()'
+ */
+glfs_object_t *
+glfs_object_copy(glfs_object_t *src) __THROW
+    GFAPI_PUBLIC(glfs_object_copy, 3.11.0);
+
+int
+glfs_h_lease(glfs_t *fs, glfs_object_t *object, glfs_lease_t *lease) __THROW
+    GFAPI_PUBLIC(glfs_h_lease, 4.0.0);
+
+glfs_object_t *
+glfs_h_find_handle(glfs_t *fs, unsigned char *handle, int len) __THROW
+    GFAPI_PUBLIC(glfs_h_lease, 4.0.0);
+
+/* Functions for getting details about the glfs_upcall_lease
+ *
+ * None of the pointers returned by the below functions should be free()'d,
+ * glfs_free()'d or glfs_h_close()'d by the application.
+ *
+ * Releasing of the structures is done by passing the glfs_upcall pointer
+ * to glfs_free().
+ */
+struct glfs_upcall_lease;
+typedef struct glfs_upcall_lease glfs_upcall_lease_t;
+
+glfs_object_t *
+glfs_upcall_lease_get_object(glfs_upcall_lease_t *arg) __THROW
+    GFAPI_PUBLIC(glfs_upcall_lease_get_object, 4.1.6);
+
+uint32_t
+glfs_upcall_lease_get_lease_type(glfs_upcall_lease_t *arg) __THROW
+    GFAPI_PUBLIC(glfs_upcall_lease_get_lease_type, 4.1.6);
 
 __END_DECLS
 
diff --git a/api/src/glfs-internal.h b/api/src/glfs-internal.h
index afcbb4553ba..7cc3b18a104 100644
--- a/api/src/glfs-internal.h
+++ b/api/src/glfs-internal.h
@@ -1,5 +1,5 @@
 /*
-  Copyright (c) 2012 Red Hat, Inc. <http://www.redhat.com>
+  Copyright (c) 2012-2018 Red Hat, Inc. <http://www.redhat.com>
   This file is part of GlusterFS.
 
   This file is licensed to you under your choice of the GNU Lesser
@@ -8,158 +8,484 @@
   cases as published by the Free Software Foundation.
 */
 
-
 #ifndef _GLFS_INTERNAL_H
 #define _GLFS_INTERNAL_H
 
-#include "xlator.h"
+#include <glusterfs/xlator.h>
+#include <glusterfs/glusterfs.h>
+#include <glusterfs/upcall-utils.h>
+#include "glfs-handles.h"
+#include <glusterfs/refcount.h>
+#include <glusterfs/syncop.h>
 
 #define GLFS_SYMLINK_MAX_FOLLOW 2048
 
 #define DEFAULT_REVAL_COUNT 1
 
 /*
- * syncop_xxx() calls are executed in two ways, one is inside a synctask where
- * the executing function will do 'swapcontext' and the other is without
- * synctask where the executing thread is made to wait using pthread_cond_wait.
- * Executing thread may change when syncop_xxx() is executed inside a synctask.
- * This leads to errno_location change i.e. errno may give errno of
- * non-executing thread. So errno is not touched inside a synctask execution.
- * All gfapi calls are executed using the second way of executing syncop_xxx()
- * where the executing thread waits using pthread_cond_wait so it is ok to set
- * errno in these cases. The following macro makes syncop_xxx() behave just
- * like a system call, where -1 is returned and errno is set when a failure
- * occurs.
+ * According to  pthread mutex and conditional variable ( cond,
+ * child_down_count, upcall mutex and mutex) initialization of struct glfs
+ * members, below GLFS_INIT_* flags are set in 'pthread_flags' member of struct
+ * glfs. The flags are set from glfs_init() and  glfs_new_from_ctx() functions
+ * as part of fs inititialization.
+ *
+ * These flag bits are validated in glfs_fini() to destroy all or partially
+ * initialized mutex and conditional variables of glfs object.
+ * If you introduce new pthread mutex or conditional variable in glfs object,
+ * please make sure you have a flag bit intorduced here for proper cleanup
+ * in glfs_fini().
+ *
  */
-#define DECODE_SYNCOP_ERR(ret) do {  \
-        if (ret < 0) {          \
-                errno = -ret;   \
-                ret = -1;       \
-        }                       \
-        } while (0)
-
-#define ESTALE_RETRY(ret,errno,reval,loc,label) do {	\
-	if (ret == -1 && errno == ESTALE) {	        \
-		if (reval < DEFAULT_REVAL_COUNT) {	\
-			reval++;			\
-			loc_wipe (loc);			\
-			goto label;			\
-		}					\
-	}						\
-	} while (0)
-
-#define GLFS_LOC_FILL_INODE(oinode, loc, label) do {   \
-	loc.inode = inode_ref (oinode);                \
-	uuid_copy (loc.gfid, oinode->gfid);            \
-	ret = priv_glfs_loc_touchup (&loc);            \
-	if (ret != 0) {                                \
-		errno = EINVAL;                        \
-		goto label;                            \
-	}                                              \
-	} while (0)
-
-#define GLFS_LOC_FILL_PINODE(pinode, loc, ret, errno, label, path) do {   \
-	loc.inode = inode_new (pinode->table);                            \
-	if (!loc.inode) {                                                 \
-		ret = -1;                                                 \
-		errno = ENOMEM;                                           \
-		goto label;                                               \
-	}                                                                 \
-	loc.parent = inode_ref (pinode);                                  \
-	loc.name = path;                                                  \
-	ret = priv_glfs_loc_touchup (&loc);                               \
-	if (ret != 0) {                                                   \
-		errno = EINVAL;                                           \
-		goto label;                                               \
-	}                                                                 \
-	} while (0)
 
-struct glfs;
+#define PTHREAD_MUTEX_INIT(mutex, attr, flags, mask, label)                    \
+    do {                                                                       \
+        int __ret = -1;                                                        \
+        __ret = pthread_mutex_init(mutex, attr);                               \
+        if (__ret == 0)                                                        \
+            flags |= mask;                                                     \
+        else                                                                   \
+            goto label;                                                        \
+    } while (0)
+
+#define PTHREAD_MUTEX_DESTROY(mutex, flags, mask)                              \
+    do {                                                                       \
+        if (flags & mask)                                                      \
+            (void)pthread_mutex_destroy(mutex);                                \
+    } while (0)
+
+#define PTHREAD_COND_INIT(cond, attr, flags, mask, label)                      \
+    do {                                                                       \
+        int __ret = -1;                                                        \
+        __ret = pthread_cond_init(cond, attr);                                 \
+        if (__ret == 0)                                                        \
+            flags |= mask;                                                     \
+        else                                                                   \
+            goto label;                                                        \
+    } while (0)
+
+#define PTHREAD_COND_DESTROY(cond, flags, mask)                                \
+    do {                                                                       \
+        if (flags & mask)                                                      \
+            (void)pthread_cond_destroy(cond);                                  \
+    } while (0)
+
+#define GLFS_INIT_MUTEX 0x00000001        /* pthread_mutex_flag */
+#define GLFS_INIT_COND 0x00000002         /* pthread_cond_flag */
+#define GLFS_INIT_COND_CHILD 0x00000004   /* pthread_cond_child_down_flag */
+#define GLFS_INIT_MUTEX_UPCALL 0x00000008 /* pthread_mutex_upcall_flag */
+
+#ifndef GF_DARWIN_HOST_OS
+#ifndef GFAPI_PUBLIC
+#define GFAPI_PUBLIC(sym, ver) /**/
+#endif
+#ifndef GFAPI_PRIVATE
+#define GFAPI_PRIVATE(sym, ver) /**/
+#endif
+#if __GNUC__ >= 10
+#define GFAPI_SYMVER_PUBLIC_DEFAULT(fn, ver)                                   \
+    __attribute__((__symver__(STR(fn) "@@GFAPI_" STR(ver))))
+
+#define GFAPI_SYMVER_PRIVATE_DEFAULT(fn, ver)                                  \
+    __attribute__((__symver__(STR(fn) "@@GFAPI_PRIVATE_" STR(ver))))
+
+#define GFAPI_SYMVER_PUBLIC(fn1, fn2, ver)                                     \
+    __attribute__((__symver__(STR(fn2) "@GFAPI_" STR(ver))))
+
+#define GFAPI_SYMVER_PRIVATE(fn1, fn2, ver)                                    \
+    __attribute__((__symver__(STR(fn2) "@GFAPI_PRIVATE_" STR(ver))))
+
+#else
+#define GFAPI_SYMVER_PUBLIC_DEFAULT(fn, ver)                                   \
+    asm(".symver pub_" STR(fn) ", " STR(fn) "@@GFAPI_" STR(ver));
+
+#define GFAPI_SYMVER_PRIVATE_DEFAULT(fn, ver)                                  \
+    asm(".symver priv_" STR(fn) ", " STR(fn) "@@GFAPI_PRIVATE_" STR(ver));
+
+#define GFAPI_SYMVER_PUBLIC(fn1, fn2, ver)                                     \
+    asm(".symver pub_" STR(fn1) ", " STR(fn2) "@GFAPI_" STR(ver));
+
+#define GFAPI_SYMVER_PRIVATE(fn1, fn2, ver)                                    \
+    asm(".symver priv_" STR(fn1) ", " STR(fn2) "@GFAPI_PRIVATE_" STR(ver));
+#endif
+#define STR(str) #str
+#else
+#ifndef GFAPI_PUBLIC
+#define GFAPI_PUBLIC(sym, ver) __asm("_" __STRING(sym) "$GFAPI_" __STRING(ver));
+#endif
+#ifndef GFAPI_PRIVATE
+#define GFAPI_PRIVATE(sym, ver)                                                \
+    __asm("_" __STRING(sym) "$GFAPI_PRIVATE_" __STRING(ver));
+#endif
+#define GFAPI_SYMVER_PUBLIC_DEFAULT(fn, dotver)  /**/
+#define GFAPI_SYMVER_PRIVATE_DEFAULT(fn, dotver) /**/
+#define GFAPI_SYMVER_PUBLIC(fn1, fn2, dotver)    /**/
+#define GFAPI_SYMVER_PRIVATE(fn1, fn2, dotver)   /**/
+#endif
+
+#define ESTALE_RETRY(ret, errno, reval, loc, label)                            \
+    do {                                                                       \
+        if (ret == -1 && errno == ESTALE) {                                    \
+            if (reval < DEFAULT_REVAL_COUNT) {                                 \
+                reval++;                                                       \
+                loc_wipe(loc);                                                 \
+                goto label;                                                    \
+            }                                                                  \
+        }                                                                      \
+    } while (0)
+
+#define GLFS_LOC_FILL_INODE(oinode, loc, label)                                \
+    do {                                                                       \
+        loc.inode = inode_ref(oinode);                                         \
+        gf_uuid_copy(loc.gfid, oinode->gfid);                                  \
+        ret = glfs_loc_touchup(&loc);                                          \
+        if (ret != 0) {                                                        \
+            errno = EINVAL;                                                    \
+            goto label;                                                        \
+        }                                                                      \
+    } while (0)
+
+#define GLFS_LOC_FILL_PINODE(pinode, loc, ret, errno, label, path)             \
+    do {                                                                       \
+        loc.inode = inode_new(pinode->table);                                  \
+        if (!loc.inode) {                                                      \
+            ret = -1;                                                          \
+            errno = ENOMEM;                                                    \
+            goto label;                                                        \
+        }                                                                      \
+        loc.parent = inode_ref(pinode);                                        \
+        loc.name = path;                                                       \
+        ret = glfs_loc_touchup(&loc);                                          \
+        if (ret != 0) {                                                        \
+            errno = EINVAL;                                                    \
+            goto label;                                                        \
+        }                                                                      \
+    } while (0)
 
-typedef int (*glfs_init_cbk) (struct glfs *fs, int ret);
+struct glfs;
 
-struct glfs {
-	char               *volname;
-        uuid_t              vol_uuid;
+struct _upcall_entry {
+    struct list_head upcall_list;
+    struct gf_upcall upcall_data;
+};
+typedef struct _upcall_entry upcall_entry;
 
-	glusterfs_ctx_t    *ctx;
+typedef int (*glfs_init_cbk)(struct glfs *fs, int ret);
 
-	pthread_t           poller;
+struct glfs {
+    char *volname;
+    uuid_t vol_uuid;
+
+    glusterfs_ctx_t *ctx;
+
+    pthread_t poller;
+
+    glfs_init_cbk init_cbk;
+    pthread_mutex_t mutex;
+    pthread_cond_t cond;
+    pthread_cond_t child_down_cond; /* for broadcasting CHILD_DOWN */
+    int init;
+    int ret;
+    int err;
+
+    xlator_t *active_subvol; /* active graph */
+    xlator_t *mip_subvol;    /* graph for which migration is in
+                              * progress */
+    xlator_t *next_subvol;   /* Any new graph is put to
+                              * next_subvol, the graph in
+                              * next_subvol can either be moved
+                              * to mip_subvol (if any IO picks it
+                              * up for migration), or be
+                              * destroyed (if there is a new
+                              * graph, and this was never picked
+                              * for migration) */
+    xlator_t *old_subvol;
+
+    char *oldvolfile;
+    ssize_t oldvollen;
+
+    inode_t *cwd;
+
+    uint32_t dev_id; /* Used to fill st_dev in struct stat */
+
+    struct list_head openfds;
+
+    gf_boolean_t migration_in_progress;
+
+    gf_boolean_t cache_upcalls; /* add upcalls to the upcall_list? */
+    struct list_head upcall_list;
+    pthread_mutex_t upcall_list_mutex; /* mutex for upcall entry list */
+
+    uint32_t pin_refcnt;
+    uint32_t pthread_flags; /* GLFS_INIT_* # defines set this flag */
+
+    uint32_t upcall_events; /* Mask of upcall events application
+                             * is interested in */
+    glfs_upcall_cbk up_cbk; /* upcall cbk function to be registered */
+    void *up_data;          /* Opaque data provided by application
+                             * during upcall registration */
+    struct list_head waitq; /* waiting synctasks */
+};
 
-	glfs_init_cbk       init_cbk;
-	pthread_mutex_t     mutex;
-	pthread_cond_t      cond;
-	int                 init;
-	int                 ret;
-	int                 err;
+/* This enum is used to maintain the state of glfd. In case of async fops
+ * fd might be closed before the actual fop is complete. Therefore we need
+ * to track whether the fd is closed or not, instead actually closing it.*/
+enum glfs_fd_state { GLFD_INIT, GLFD_OPEN, GLFD_CLOSE };
 
-	xlator_t           *active_subvol;
-	xlator_t           *next_subvol;
-	xlator_t           *old_subvol;
+struct glfs_fd {
+    struct list_head openfds;
+    struct list_head list;
+    GF_REF_DECL;
+    struct glfs *fs;
+    enum glfs_fd_state state;
+    off_t offset;
+    fd_t *fd; /* Currently guared by @fs->mutex. TODO: per-glfd lock */
+    struct list_head entries;
+    gf_dirent_t *next;
+    struct dirent *readdirbuf;
+    gf_lkowner_t lk_owner;
+    glfs_leaseid_t lease_id; /* Stores lease_id of client in glfd */
+    gf_lock_t lock;          /* lock taken before updating fd state */
+    glfs_recall_cbk cbk;
+    void *cookie;
+};
 
-	char               *oldvolfile;
-	ssize_t             oldvollen;
+/* glfs object handle introduced for the alternate gfapi implementation based
+   on glfs handles/gfid/inode
+*/
+struct glfs_object {
+    inode_t *inode;
+    uuid_t gfid;
+};
 
-	inode_t            *cwd;
+struct glfs_upcall {
+    struct glfs *fs;                /* glfs object */
+    enum glfs_upcall_reason reason; /* Upcall event type */
+    void *event;                    /* changes based in the event type */
+    void (*free_event)(void *);     /* free event after the usage */
+};
 
-	uint32_t            dev_id; /* Used to fill st_dev in struct stat */
+struct glfs_upcall_inode {
+    struct glfs_object *object;      /* Object which need to be acted upon */
+    int flags;                       /* Cache UPDATE/INVALIDATE flags */
+    struct stat buf;                 /* Latest stat of this entry */
+    unsigned int expire_time_attr;   /* the amount of time for which
+                                      * the application need to cache
+                                      * this entry */
+    struct glfs_object *p_object;    /* parent Object to be updated */
+    struct stat p_buf;               /* Latest stat of parent dir handle */
+    struct glfs_object *oldp_object; /* Old parent Object to be updated */
+    struct stat oldp_buf;            /* Latest stat of old parent dir handle */
+};
 
-	struct list_head    openfds;
+struct glfs_upcall_lease {
+    struct glfs_object *object; /* Object which need to be acted upon */
+    uint32_t lease_type;        /* Lease type to which client can downgrade to*/
+};
 
-	gf_boolean_t        migration_in_progress;
+struct glfs_upcall_lease_fd {
+    uint32_t lease_type; /* Lease type to which client can downgrade to*/
+    void *fd_cookie;     /* Object which need to be acted upon */
 };
 
-struct glfs_fd {
-	struct list_head   openfds;
-	struct glfs       *fs;
-	off_t              offset;
-	fd_t              *fd; /* Currently guared by @fs->mutex. TODO: per-glfd lock */
-	struct list_head   entries;
-	gf_dirent_t       *next;
-	struct dirent     *readdirbuf;
+struct glfs_xreaddirp_stat {
+    struct stat
+        st; /* Stat for that dirent - corresponds to GFAPI_XREADDIRP_STAT */
+    struct glfs_object *object; /* handled for GFAPI_XREADDIRP_HANDLE */
+    uint32_t flags_handled;     /* final set of flags successfulyy handled */
 };
 
-/* glfs object handle introduced for the alternate gfapi implementation based
-   on glfs handles/gfid/inode
-*/
-struct glfs_object {
-        inode_t         *inode;
-        uuid_t          gfid;
+#define DEFAULT_EVENT_POOL_SIZE 16384
+#define GF_MEMPOOL_COUNT_OF_DICT_T 4096
+#define GF_MEMPOOL_COUNT_OF_DATA_T (GF_MEMPOOL_COUNT_OF_DICT_T * 4)
+#define GF_MEMPOOL_COUNT_OF_DATA_PAIR_T (GF_MEMPOOL_COUNT_OF_DICT_T * 4)
+
+#define GF_MEMPOOL_COUNT_OF_LRU_BUF_T 256
+
+typedef void(glfs_mem_release_t)(void *ptr);
+
+struct glfs_mem_header {
+    uint32_t magic;
+    size_t nmemb;
+    size_t size;
+    glfs_mem_release_t *release;
 };
 
-#define DEFAULT_EVENT_POOL_SIZE           16384
-#define GF_MEMPOOL_COUNT_OF_DICT_T        4096
-#define GF_MEMPOOL_COUNT_OF_DATA_T        (GF_MEMPOOL_COUNT_OF_DICT_T * 4)
-#define GF_MEMPOOL_COUNT_OF_DATA_PAIR_T   (GF_MEMPOOL_COUNT_OF_DICT_T * 4)
+#define GLFS_MEM_HEADER_SIZE (sizeof(struct glfs_mem_header))
+#define GLFS_MEM_HEADER_MAGIC 0x20170830
 
-#define GF_MEMPOOL_COUNT_OF_LRU_BUF_T     256
+static inline void *
+__glfs_calloc(size_t nmemb, size_t size, glfs_mem_release_t release,
+              uint32_t type, const char *typestr)
+{
+    struct glfs_mem_header *header = NULL;
 
-int glfs_mgmt_init (struct glfs *fs);
-void priv_glfs_init_done (struct glfs *fs, int ret);
-int glfs_process_volfp (struct glfs *fs, FILE *fp);
-int glfs_resolve (struct glfs *fs, xlator_t *subvol, const char *path, loc_t *loc,
-		  struct iatt *iatt, int reval);
-int glfs_lresolve (struct glfs *fs, xlator_t *subvol, const char *path, loc_t *loc,
-		   struct iatt *iatt, int reval);
-fd_t *glfs_resolve_fd (struct glfs *fs, xlator_t *subvol, struct glfs_fd *glfd);
+    header = __gf_calloc(nmemb, (size + GLFS_MEM_HEADER_SIZE), type, typestr);
+    if (!header)
+        return NULL;
 
-fd_t *__glfs_migrate_fd (struct glfs *fs, xlator_t *subvol, struct glfs_fd *glfd);
+    header->magic = GLFS_MEM_HEADER_MAGIC;
+    header->nmemb = nmemb;
+    header->size = size;
+    header->release = release;
 
-int glfs_first_lookup (xlator_t *subvol);
+    return header + 1;
+}
 
-static inline void
-__glfs_entry_fs (struct glfs *fs)
+static inline void *
+__glfs_malloc(size_t size, glfs_mem_release_t release, uint32_t type,
+              const char *typestr)
 {
-	THIS = fs->ctx->master;
+    struct glfs_mem_header *header = NULL;
+
+    header = __gf_malloc((size + GLFS_MEM_HEADER_SIZE), type, typestr);
+    if (!header)
+        return NULL;
+
+    header->magic = GLFS_MEM_HEADER_MAGIC;
+    header->nmemb = 1;
+    header->size = size;
+    header->release = release;
+
+    return header + 1;
 }
 
+static inline void *
+__glfs_realloc(void *ptr, size_t size)
+{
+    struct glfs_mem_header *old_header = NULL;
+    struct glfs_mem_header *new_header = NULL;
+    struct glfs_mem_header tmp_header;
+    void *new_ptr = NULL;
+
+    GF_ASSERT(NULL != ptr);
+
+    old_header = (struct glfs_mem_header *)(ptr - GLFS_MEM_HEADER_SIZE);
+    GF_ASSERT(old_header->magic == GLFS_MEM_HEADER_MAGIC);
+    tmp_header = *old_header;
+
+    new_ptr = __gf_realloc(old_header, (size + GLFS_MEM_HEADER_SIZE));
+    if (!new_ptr)
+        return NULL;
+
+    new_header = (struct glfs_mem_header *)new_ptr;
+    *new_header = tmp_header;
+    new_header->size = size;
+
+    return new_header + 1;
+}
 
 static inline void
-__glfs_entry_fd (struct glfs_fd *fd)
+__glfs_free(void *free_ptr)
 {
-	THIS = fd->fd->inode->table->xl->ctx->master;
+    struct glfs_mem_header *header = NULL;
+    void *release_ptr = NULL;
+    int i = 0;
+
+    if (!free_ptr)
+        return;
+
+    header = (struct glfs_mem_header *)(free_ptr - GLFS_MEM_HEADER_SIZE);
+    GF_ASSERT(header->magic == GLFS_MEM_HEADER_MAGIC);
+
+    if (header->release) {
+        release_ptr = free_ptr;
+        for (i = 0; i < header->nmemb; i++) {
+            header->release(release_ptr);
+            release_ptr += header->size;
+        }
+    }
+
+    __gf_free(header);
 }
 
+#define GLFS_CALLOC(nmemb, size, release, type)                                \
+    __glfs_calloc(nmemb, size, release, type, #type)
+
+#define GLFS_MALLOC(size, release, type)                                       \
+    __glfs_malloc(size, release, type, #type)
+
+#define GLFS_REALLOC(ptr, size) __glfs_realloc(ptr, size)
+
+#define GLFS_FREE(free_ptr) __glfs_free(free_ptr)
+
+int
+glfs_mgmt_init(struct glfs *fs);
+void
+glfs_init_done(struct glfs *fs, int ret) GFAPI_PRIVATE(glfs_init_done, 3.4.0);
+int
+glfs_process_volfp(struct glfs *fs, FILE *fp);
+int
+glfs_resolve(struct glfs *fs, xlator_t *subvol, const char *path, loc_t *loc,
+             struct iatt *iatt, int reval) GFAPI_PRIVATE(glfs_resolve, 3.7.0);
+int
+glfs_lresolve(struct glfs *fs, xlator_t *subvol, const char *path, loc_t *loc,
+              struct iatt *iatt, int reval);
+fd_t *
+glfs_resolve_fd(struct glfs *fs, xlator_t *subvol, struct glfs_fd *glfd);
+
+fd_t *
+__glfs_migrate_fd(struct glfs *fs, xlator_t *subvol, struct glfs_fd *glfd);
+
+int
+glfs_first_lookup(xlator_t *subvol);
+
+void
+glfs_process_upcall_event(struct glfs *fs, void *data)
+    GFAPI_PRIVATE(glfs_process_upcall_event, 3.7.0);
+
+#define __GLFS_ENTRY_VALIDATE_FS(fs, label)                                    \
+    do {                                                                       \
+        if (!fs) {                                                             \
+            errno = EINVAL;                                                    \
+            goto label;                                                        \
+        }                                                                      \
+        old_THIS = THIS;                                                       \
+        THIS = fs->ctx->master;                                                \
+    } while (0)
+
+#define __GLFS_EXIT_FS                                                         \
+    do {                                                                       \
+        THIS = old_THIS;                                                       \
+    } while (0)
+
+#define __GLFS_ENTRY_VALIDATE_FD(glfd, label)                                  \
+    do {                                                                       \
+        if (!glfd || !glfd->fd || !glfd->fd->inode ||                          \
+            glfd->state != GLFD_OPEN) {                                        \
+            errno = EBADF;                                                     \
+            goto label;                                                        \
+        }                                                                      \
+        old_THIS = THIS;                                                       \
+        THIS = glfd->fd->inode->table->xl->ctx->master;                        \
+    } while (0)
+
+#define __GLFS_LOCK_WAIT(fs)                                                   \
+    do {                                                                       \
+        struct synctask *task = NULL;                                          \
+                                                                               \
+        task = synctask_get();                                                 \
+                                                                               \
+        if (task) {                                                            \
+            list_add_tail(&task->waitq, &fs->waitq);                           \
+            pthread_mutex_unlock(&fs->mutex);                                  \
+            synctask_yield(task, NULL);                                        \
+            pthread_mutex_lock(&fs->mutex);                                    \
+        } else {                                                               \
+            /* non-synctask */                                                 \
+            pthread_cond_wait(&fs->cond, &fs->mutex);                          \
+        }                                                                      \
+    } while (0)
+
+#define __GLFS_SYNCTASK_WAKE(fs)                                               \
+    do {                                                                       \
+        struct synctask *waittask = NULL;                                      \
+                                                                               \
+        while (!list_empty(&fs->waitq)) {                                      \
+            waittask = list_entry(fs->waitq.next, struct synctask, waitq);     \
+            list_del_init(&waittask->waitq);                                   \
+            synctask_wake(waittask);                                           \
+        }                                                                      \
+    } while (0)
 
 /*
   By default all lock attempts from user context must
@@ -168,63 +494,93 @@ __glfs_entry_fd (struct glfs_fd *fd)
   we can give up the mutex during syncop calls so
   that bottom up calls (particularly CHILD_UP notify)
   can do a mutex_lock() on @glfs without deadlocking
-  the filesystem
+  the filesystem.
+
+  All the fops should wait for graph migration to finish
+  before starting the fops. Therefore these functions should
+  call glfs_lock with wait_for_migration as true. But waiting
+  for migration to finish in call-back path can result thread
+  dead-locks. The reason for this is we only have finite
+  number of epoll threads. so if we wait on epoll threads
+  there will not be any thread left to handle outstanding
+  rpc replies.
 */
 static inline int
-glfs_lock (struct glfs *fs)
+glfs_lock(struct glfs *fs, gf_boolean_t wait_for_migration)
 {
-	pthread_mutex_lock (&fs->mutex);
+    pthread_mutex_lock(&fs->mutex);
 
-	while (!fs->init)
-		pthread_cond_wait (&fs->cond, &fs->mutex);
+    while (!fs->init)
+        __GLFS_LOCK_WAIT(fs);
 
-	while (fs->migration_in_progress)
-		pthread_cond_wait (&fs->cond, &fs->mutex);
+    while (wait_for_migration && fs->migration_in_progress)
+        __GLFS_LOCK_WAIT(fs);
 
-	return 0;
+    return 0;
 }
 
-
 static inline void
-glfs_unlock (struct glfs *fs)
+glfs_unlock(struct glfs *fs)
 {
-	pthread_mutex_unlock (&fs->mutex);
+    pthread_mutex_unlock(&fs->mutex);
 }
 
-
-void glfs_fd_destroy (struct glfs_fd *glfd);
-
-struct glfs_fd *glfs_fd_new (struct glfs *fs);
-void glfs_fd_bind (struct glfs_fd *glfd);
-
-xlator_t *priv_glfs_active_subvol (struct glfs *fs);
-xlator_t *__glfs_active_subvol (struct glfs *fs);
-void priv_glfs_subvol_done (struct glfs *fs, xlator_t *subvol);
-
-inode_t *glfs_refresh_inode (xlator_t *subvol, inode_t *inode);
-
-inode_t *glfs_cwd_get (struct glfs *fs);
-int glfs_cwd_set (struct glfs *fs, inode_t *inode);
-inode_t *glfs_resolve_inode (struct glfs *fs, xlator_t *subvol,
-			     struct glfs_object *object);
-int glfs_create_object (loc_t *loc, struct glfs_object **retobject);
-int __glfs_cwd_set (struct glfs *fs, inode_t *inode);
-
-int glfs_resolve_base (struct glfs *fs, xlator_t *subvol, inode_t *inode,
-		       struct iatt *iatt);
-int priv_glfs_resolve_at (struct glfs *fs, xlator_t *subvol, inode_t *at,
-                          const char *origpath, loc_t *loc, struct iatt *iatt,
-                          int follow, int reval);
-int priv_glfs_loc_touchup (loc_t *loc);
-void glfs_iatt_to_stat (struct glfs *fs, struct iatt *iatt, struct stat *stat);
-int glfs_loc_link (loc_t *loc, struct iatt *iatt);
-int glfs_loc_unlink (loc_t *loc);
-dict_t *dict_for_key_value (const char *name, const char *value, size_t size);
-int glfs_getxattr_process (void *value, size_t size, dict_t *xattr,
-			   const char *name);
+struct glfs_fd *
+glfs_fd_new(struct glfs *fs);
+void
+glfs_fd_bind(struct glfs_fd *glfd);
+void
+glfd_set_state_bind(struct glfs_fd *glfd);
+
+xlator_t *
+glfs_active_subvol(struct glfs *fs) GFAPI_PRIVATE(glfs_active_subvol, 3.4.0);
+xlator_t *
+__glfs_active_subvol(struct glfs *fs);
+void
+glfs_subvol_done(struct glfs *fs, xlator_t *subvol)
+    GFAPI_PRIVATE(glfs_subvol_done, 3.4.0);
+
+inode_t *
+glfs_refresh_inode(xlator_t *subvol, inode_t *inode);
+
+inode_t *
+glfs_cwd_get(struct glfs *fs);
+int
+glfs_cwd_set(struct glfs *fs, inode_t *inode);
+inode_t *
+glfs_resolve_inode(struct glfs *fs, xlator_t *subvol,
+                   struct glfs_object *object);
+int
+glfs_create_object(loc_t *loc, struct glfs_object **retobject);
+int
+__glfs_cwd_set(struct glfs *fs, inode_t *inode);
+
+int
+glfs_resolve_base(struct glfs *fs, xlator_t *subvol, inode_t *inode,
+                  struct iatt *iatt);
+
+int
+glfs_resolve_at(struct glfs *fs, xlator_t *subvol, inode_t *at,
+                const char *origpath, loc_t *loc, struct iatt *iatt, int follow,
+                int reval) GFAPI_PRIVATE(glfs_resolve_at, 3.4.0);
+int
+glfs_loc_touchup(loc_t *loc) GFAPI_PRIVATE(glfs_loc_touchup, 3.4.0);
+void
+glfs_iatt_to_stat(struct glfs *fs, struct iatt *iatt, struct stat *stat);
+void
+glfs_iatt_from_stat(struct stat *stat, int valid, struct iatt *iatt,
+                    int *gvalid);
+int
+glfs_loc_link(loc_t *loc, struct iatt *iatt);
+int
+glfs_loc_unlink(loc_t *loc);
+int
+glfs_getxattr_process(void *value, size_t size, dict_t *xattr,
+                      const char *name);
 
 /* Sends RPC call to glusterd to fetch required volume info */
-int glfs_get_volume_info (struct glfs *fs);
+int
+glfs_get_volume_info(struct glfs *fs);
 
 /*
   SYNOPSIS
@@ -250,7 +606,8 @@ int glfs_get_volume_info (struct glfs *fs);
        NULL   : Otherwise.
 */
 
-struct glfs *priv_glfs_new_from_ctx (glusterfs_ctx_t *ctx);
+struct glfs *
+glfs_new_from_ctx(glusterfs_ctx_t *ctx) GFAPI_PRIVATE(glfs_new_from_ctx, 3.7.0);
 
 /*
   SYNOPSIS
@@ -261,9 +618,9 @@ struct glfs *priv_glfs_new_from_ctx (glusterfs_ctx_t *ctx);
   DESCRIPTION
 
        The glfs_t object allocated by glfs_new_from_ctx() must be released
-       by the caller using this routine. The usage is restricted to NFS
-       MOUNT over UDP i.e.
-       __mnt3udp_get_export_subdir_inode ()
+       by the caller using this routine. The usage can be found
+       at glfs_fini() or NFS, MOUNT over UDP i.e.
+                        __mnt3udp_get_export_subdir_inode ()
                                 => glfs_resolve_at().
 
   PARAMETERS
@@ -275,13 +632,125 @@ struct glfs *priv_glfs_new_from_ctx (glusterfs_ctx_t *ctx);
        void
 */
 
-void priv_glfs_free_from_ctx (struct glfs *fs);
+void
+glfs_free_from_ctx(struct glfs *fs) GFAPI_PRIVATE(glfs_free_from_ctx, 3.7.0);
+
+int
+glfs_recall_lease_fd(struct glfs *fs, struct gf_upcall *up_data);
+
+int
+glfs_get_upcall_cache_invalidation(struct gf_upcall *to_up_data,
+                                   struct gf_upcall *from_up_data);
+int
+glfs_h_poll_cache_invalidation(struct glfs *fs, struct glfs_upcall *up_arg,
+                               struct gf_upcall *upcall_data);
+
+ssize_t
+glfs_anonymous_preadv(struct glfs *fs, struct glfs_object *object,
+                      const struct iovec *iovec, int iovcnt, off_t offset,
+                      int flags);
+ssize_t
+glfs_anonymous_pwritev(struct glfs *fs, struct glfs_object *object,
+                       const struct iovec *iovec, int iovcnt, off_t offset,
+                       int flags);
+
+struct glfs_object *
+glfs_h_resolve_symlink(struct glfs *fs, struct glfs_object *object);
+
+/* Deprecated structures that were passed to client applications, replaced by
+ * accessor functions. Do not use these in new applications, and update older
+ * usage.
+ *
+ * See http://review.gluster.org/14701 for more details.
+ *
+ * WARNING: These structures will be removed in the future.
+ */
+struct glfs_callback_arg {
+    struct glfs *fs;
+    enum glfs_upcall_reason reason;
+    void *event_arg;
+};
 
-#define GFAPI_SYMVER_PUBLIC_DEFAULT(fn, dotver) \
-        asm(".symver pub_"STR(fn)", "STR(fn)"@@GFAPI_"STR(dotver))
+struct glfs_callback_inode_arg {
+    struct glfs_object *object;      /* Object which need to be acted upon */
+    int flags;                       /* Cache UPDATE/INVALIDATE flags */
+    struct stat buf;                 /* Latest stat of this entry */
+    unsigned int expire_time_attr;   /* the amount of time for which
+                                      * the application need to cache
+                                      * this entry
+                                      */
+    struct glfs_object *p_object;    /* parent Object to be updated */
+    struct stat p_buf;               /* Latest stat of parent dir handle */
+    struct glfs_object *oldp_object; /* Old parent Object
+                                      * to be updated */
+    struct stat oldp_buf;            /* Latest stat of old parent
+                                      * dir handle */
+};
+struct dirent *
+glfs_readdirbuf_get(struct glfs_fd *glfd);
 
-#define GFAPI_SYMVER_PRIVATE_DEFAULT(fn, dotver) \
-        asm(".symver priv_"STR(fn)", "STR(fn)"@@GFAPI_PRIVATE_"STR(dotver))
-#define STR(str) #str
+gf_dirent_t *
+glfd_entry_next(struct glfs_fd *glfd, int plus);
+
+void
+gf_dirent_to_dirent(gf_dirent_t *gf_dirent, struct dirent *dirent);
+
+void
+gf_lease_to_glfs_lease(struct gf_lease *gf_lease, struct glfs_lease *lease);
+
+void
+glfs_lease_to_gf_lease(struct glfs_lease *lease, struct gf_lease *gf_lease);
 
+void
+glfs_release_upcall(void *ptr);
+
+int
+get_fop_attr_glfd(dict_t **fop_attr, struct glfs_fd *glfd);
+
+int
+set_fop_attr_glfd(struct glfs_fd *glfd);
+
+int
+get_fop_attr_thrd_key(dict_t **fop_attr);
+
+void
+unset_fop_attr(dict_t **fop_attr);
+
+/*
+  SYNOPSIS
+  glfs_statx: Fetch extended file attributes for the given path.
+
+  DESCRIPTION
+  This function fetches extended file attributes for the given path.
+
+  PARAMETERS
+  @fs: The 'virtual mount' object referencing a volume, under which file exists.
+  @path: Path of the file within the virtual mount.
+  @mask: Requested extended file attributes mask, (See mask defines above)
+
+  RETURN VALUES
+  -1 : Failure. @errno will be set with the type of failure.
+  0  : Filled in statxbuf with appropriate masks for valid items in the
+       structure.
+
+  ERRNO VALUES
+    EINVAL: fs is invalid
+    EINVAL: mask has unsupported bits set
+    Other errors as returned by stat(2)
+ */
+
+int
+glfs_statx(struct glfs *fs, const char *path, unsigned int mask,
+           struct glfs_stat *statxbuf) GFAPI_PRIVATE(glfs_statx, 6.0);
+
+void
+glfs_iatt_from_statx(struct iatt *, const struct glfs_stat *)
+    GFAPI_PRIVATE(glfs_iatt_from_statx, 6.0);
+
+/*
+ * This API is a per thread setting, similar to glfs_setfs{u/g}id, because of
+ * the call to syncopctx_setfspid.
+ */
+int
+glfs_setfspid(struct glfs *, pid_t) GFAPI_PRIVATE(glfs_setfspid, 6.1);
 #endif /* !_GLFS_INTERNAL_H */
diff --git a/api/src/glfs-master.c b/api/src/glfs-master.c
index 89017bab84d..100dcc16cc0 100644
--- a/api/src/glfs-master.c
+++ b/api/src/glfs-master.c
@@ -1,5 +1,5 @@
 /*
-  Copyright (c) 2012 Red Hat, Inc. <http://www.redhat.com>
+  Copyright (c) 2012-2016 Red Hat, Inc. <http://www.redhat.com>
   This file is part of GlusterFS.
 
   This file is licensed to you under your choice of the GNU Lesser
@@ -8,153 +8,176 @@
   cases as published by the Free Software Foundation.
 */
 
-#include <unistd.h>
-#include <string.h>
-#include <stdlib.h>
 #include <stdio.h>
-#include <inttypes.h>
-#include <limits.h>
 
-#ifndef _CONFIG_H
-#define _CONFIG_H
-#include "config.h"
-#endif
-
-#include "xlator.h"
-#include "glusterfs.h"
+#include <glusterfs/glusterfs.h>
 
 #include "glfs-internal.h"
 #include "glfs-mem-types.h"
-
-
-extern void
-glfs_subvol_done (struct glfs *, xlator_t *);
+#include "gfapi-messages.h"
 
 int
-graph_setup (struct glfs *fs, glusterfs_graph_t *graph)
+graph_setup(struct glfs *fs, glusterfs_graph_t *graph)
 {
-	xlator_t      *new_subvol = NULL;
-	xlator_t      *old_subvol = NULL;
-	inode_table_t *itable = NULL;
-	int            ret = -1;
-
-	new_subvol = graph->top;
-
-	/* This is called in a bottom-up context, it should specifically
-	   NOT be glfs_lock()
-	*/
-	pthread_mutex_lock (&fs->mutex);
-	{
-		if (new_subvol->switched ||
-		    new_subvol == fs->active_subvol ||
-		    new_subvol == fs->next_subvol) {
-			/* Spurious CHILD_UP event on old graph */
-			ret = 0;
-			goto unlock;
-		}
-
-		if (!new_subvol->itable) {
-			itable = inode_table_new (131072, new_subvol);
-			if (!itable) {
-				errno = ENOMEM;
-				ret = -1;
-				goto unlock;
-			}
-
-			new_subvol->itable = itable;
-		}
-
-		old_subvol = fs->next_subvol;
-		fs->next_subvol = new_subvol;
-		fs->next_subvol->winds++; /* first ref */
-		ret = 0;
-	}
+    xlator_t *new_subvol = NULL;
+    xlator_t *old_subvol = NULL;
+    inode_table_t *itable = NULL;
+    int ret = -1;
+
+    new_subvol = graph->top;
+
+    /* This is called in a bottom-up context, it should specifically
+       NOT be glfs_lock()
+    */
+    pthread_mutex_lock(&fs->mutex);
+    {
+        if (new_subvol->switched || new_subvol == fs->active_subvol ||
+            new_subvol == fs->next_subvol || new_subvol == fs->mip_subvol) {
+            /* Spurious CHILD_UP event on old graph */
+            ret = 0;
+            goto unlock;
+        }
+
+        if (!new_subvol->itable) {
+            itable = inode_table_new(131072, new_subvol);
+            if (!itable) {
+                errno = ENOMEM;
+                ret = -1;
+                goto unlock;
+            }
+
+            new_subvol->itable = itable;
+        }
+
+        old_subvol = fs->next_subvol;
+        fs->next_subvol = new_subvol;
+        fs->next_subvol->winds++; /* first ref */
+        ret = 0;
+    }
 unlock:
-	pthread_mutex_unlock (&fs->mutex);
+    pthread_mutex_unlock(&fs->mutex);
 
-	if (old_subvol)
-		/* wasn't picked up so far, skip */
-		glfs_subvol_done (fs, old_subvol);
+    if (old_subvol)
+        /* wasn't picked up so far, skip */
+        glfs_subvol_done(fs, old_subvol);
 
-	return ret;
+    return ret;
 }
 
-
-extern void
-glfs_init_done (struct glfs *fs, int ret);
-
 int
-notify (xlator_t *this, int event, void *data, ...)
+notify(xlator_t *this, int event, void *data, ...)
 {
-	glusterfs_graph_t   *graph = NULL;
-	struct glfs	    *fs = NULL;
-
-	graph = data;
-	fs = this->private;
-
-	switch (event) {
-	case GF_EVENT_GRAPH_NEW:
-		gf_log (this->name, GF_LOG_INFO, "New graph %s (%d) coming up",
-			uuid_utoa ((unsigned char *)graph->graph_uuid),
-			graph->id);
-		break;
-	case GF_EVENT_CHILD_UP:
-		graph_setup (fs, graph);
-		glfs_init_done (fs, 0);
-		break;
-	case GF_EVENT_CHILD_DOWN:
-		graph_setup (fs, graph);
-		glfs_init_done (fs, 1);
-		break;
-	case GF_EVENT_CHILD_CONNECTING:
-		break;
-	default:
-		gf_log (this->name, GF_LOG_DEBUG,
-			"got notify event %d", event);
-		break;
-	}
-
-	return 0;
+    glusterfs_graph_t *graph = NULL;
+    struct glfs *fs = NULL;
+
+    graph = data;
+    fs = this->private;
+
+    switch (event) {
+        case GF_EVENT_GRAPH_NEW:
+            gf_smsg(this->name, GF_LOG_INFO, 0, API_MSG_NEW_GRAPH,
+                    "graph-uuid=%s",
+                    uuid_utoa((unsigned char *)graph->graph_uuid), "id=%d",
+                    graph->id, NULL);
+            break;
+        case GF_EVENT_CHILD_UP:
+            pthread_mutex_lock(&fs->mutex);
+            {
+                graph->used = 1;
+            }
+            pthread_mutex_unlock(&fs->mutex);
+            graph_setup(fs, graph);
+            glfs_init_done(fs, 0);
+            break;
+        case GF_EVENT_CHILD_DOWN:
+            pthread_mutex_lock(&fs->mutex);
+            {
+                graph->used = 0;
+                pthread_cond_broadcast(&fs->child_down_cond);
+            }
+            pthread_mutex_unlock(&fs->mutex);
+            glfs_init_done(fs, 1);
+            break;
+        case GF_EVENT_CHILD_CONNECTING:
+            break;
+        case GF_EVENT_UPCALL:
+            glfs_process_upcall_event(fs, data);
+            break;
+        default:
+            gf_msg_debug(this->name, 0, "got notify event %d", event);
+            break;
+    }
+
+    return 0;
 }
 
-
 int
-mem_acct_init (xlator_t *this)
+mem_acct_init(xlator_t *this)
 {
-	int ret = -1;
+    int ret = -1;
 
-	if (!this)
-		return ret;
+    if (!this)
+        return ret;
 
-	ret = xlator_mem_acct_init (this, glfs_mt_end + 1);
-	if (ret) {
-		gf_log (this->name, GF_LOG_ERROR, "Failed to initialise "
-                        "memory accounting");
-		return ret;
-	}
+    ret = xlator_mem_acct_init(this, glfs_mt_end + 1);
+    if (ret) {
+        gf_smsg(this->name, GF_LOG_ERROR, ENOMEM, API_MSG_MEM_ACCT_INIT_FAILED,
+                NULL);
+        return ret;
+    }
 
-	return 0;
+    return 0;
 }
 
-
 int
-init (xlator_t *this)
+init(xlator_t *this)
 {
-	return 0;
+    return 0;
 }
 
-
 void
-fini (xlator_t *this)
+fini(xlator_t *this)
+{
+}
+
+/* place-holder fops */
+int
+glfs_forget(xlator_t *this, inode_t *inode)
 {
+    return 0;
+}
 
+int
+glfs_release(xlator_t *this, fd_t *fd)
+{
+    return 0;
 }
 
+int
+glfs_releasedir(xlator_t *this, fd_t *fd)
+{
+    return 0;
+}
 
 struct xlator_dumpops dumpops;
 
-
 struct xlator_fops fops;
 
-
-struct xlator_cbks cbks;
+struct xlator_cbks cbks = {
+    .forget = glfs_forget,
+    .release = glfs_release,
+    .releasedir = glfs_releasedir,
+};
+
+xlator_api_t xlator_api = {
+    .init = init,
+    .fini = fini,
+    .notify = notify,
+    .mem_acct_init = mem_acct_init,
+    .op_version = {1},
+    .dumpops = &dumpops,
+    .fops = &fops,
+    .cbks = &cbks,
+    .identifier = "glfs-api",
+    .category = GF_MAINTAINED,
+};
diff --git a/api/src/glfs-mem-types.h b/api/src/glfs-mem-types.h
index 76f4fc774cc..bfa325a3ad9 100644
--- a/api/src/glfs-mem-types.h
+++ b/api/src/glfs-mem-types.h
@@ -1,5 +1,5 @@
 /*
-  Copyright (c) 2012 Red Hat, Inc. <http://www.redhat.com>
+  Copyright (c) 2012-2017 Red Hat, Inc. <http://www.redhat.com>
   This file is part of GlusterFS.
 
   This file is licensed to you under your choice of the GNU Lesser
@@ -11,22 +11,25 @@
 #ifndef _GLFS_MEM_TYPES_H
 #define _GLFS_MEM_TYPES_H
 
-#include "mem-types.h"
+#include <glusterfs/mem-types.h>
 
 #define GF_MEM_TYPE_START (gf_common_mt_end + 1)
 
 enum glfs_mem_types_ {
-        glfs_mt_glfs_t = GF_MEM_TYPE_START,
-        glfs_mt_call_pool_t,
-        glfs_mt_xlator_t,
-	glfs_mt_glfs_fd_t,
-	glfs_mt_glfs_io_t,
-	glfs_mt_volfile_t,
-	glfs_mt_xlator_cmdline_option_t,
-        glfs_mt_server_cmdline_t,
-	glfs_mt_glfs_object_t,
-	glfs_mt_readdirbuf_t,
-	glfs_mt_end
-
+    glfs_mt_call_pool_t = GF_MEM_TYPE_START,
+    glfs_mt_xlator_t,
+    glfs_mt_glfs_fd_t,
+    glfs_mt_glfs_io_t,
+    glfs_mt_volfile_t,
+    glfs_mt_xlator_cmdline_option_t,
+    glfs_mt_server_cmdline_t,
+    glfs_mt_glfs_object_t,
+    glfs_mt_readdirbuf_t,
+    glfs_mt_upcall_entry_t,
+    glfs_mt_acl_t,
+    glfs_mt_upcall_inode_t,
+    glfs_mt_realpath_t,
+    glfs_mt_xreaddirp_stat_t,
+    glfs_mt_end
 };
 #endif
diff --git a/api/src/glfs-mgmt.c b/api/src/glfs-mgmt.c
index ea017f31af6..7c82b8cd162 100644
--- a/api/src/glfs-mgmt.c
+++ b/api/src/glfs-mgmt.c
@@ -1,5 +1,5 @@
 /*
-  Copyright (c) 2012 Red Hat, Inc. <http://www.redhat.com>
+  Copyright (c) 2012-2018 Red Hat, Inc. <http://www.redhat.com>
   This file is part of GlusterFS.
 
   This file is licensed to you under your choice of the GNU Lesser
@@ -8,7 +8,6 @@
   cases as published by the Free Software Foundation.
 */
 
-
 #include <stdio.h>
 #include <sys/types.h>
 #include <sys/wait.h>
@@ -16,839 +15,1035 @@
 #include <signal.h>
 #include <pthread.h>
 
-#ifndef _CONFIG_H
-#define _CONFIG_H
-#include "config.h"
-#endif /* _CONFIG_H */
-
-#include "glusterfs.h"
+#include <glusterfs/glusterfs.h>
 #include "glfs.h"
-#include "stack.h"
-#include "dict.h"
-#include "event.h"
-#include "defaults.h"
+#include <glusterfs/dict.h>
 
 #include "rpc-clnt.h"
 #include "protocol-common.h"
-#include "glusterfs3.h"
-#include "portmap-xdr.h"
-#include "xdr-common.h"
 #include "xdr-generic.h"
+#include "rpc-common-xdr.h"
 
-#include "syncop.h"
-#include "xlator.h"
+#include <glusterfs/syncop.h>
 
 #include "glfs-internal.h"
-#include "glfs-mem-types.h"
+#include "gfapi-messages.h"
+#include <glusterfs/syscall.h>
 
-int glfs_volfile_fetch (struct glfs *fs);
-int32_t glfs_get_volume_info_rpc (call_frame_t *frame, xlator_t *this,
-                                  struct glfs *fs);
+int
+glfs_volfile_fetch(struct glfs *fs);
+int32_t
+glfs_get_volume_info_rpc(call_frame_t *frame, xlator_t *this, struct glfs *fs);
 
 int
-glfs_process_volfp (struct glfs *fs, FILE *fp)
+glfs_process_volfp(struct glfs *fs, FILE *fp)
 {
-	glusterfs_graph_t  *graph = NULL;
-	int		    ret = -1;
-	xlator_t	   *trav = NULL;
-	glusterfs_ctx_t	   *ctx = NULL;
-
-	ctx = fs->ctx;
-	graph = glusterfs_graph_construct (fp);
-	if (!graph) {
-		gf_log ("glfs", GF_LOG_ERROR, "failed to construct the graph");
-		goto out;
-	}
-
-	for (trav = graph->first; trav; trav = trav->next) {
-		if (strcmp (trav->type, "mount/fuse") == 0) {
-			gf_log ("glfs", GF_LOG_ERROR,
-				"fuse xlator cannot be specified "
-				"in volume file");
-			goto out;
-		}
-	}
-
-	ret = glusterfs_graph_prepare (graph, ctx);
-	if (ret) {
-		glusterfs_graph_destroy (graph);
-		goto out;
-	}
-
-	ret = glusterfs_graph_activate (graph, ctx);
-
-	if (ret) {
-		glusterfs_graph_destroy (graph);
-		goto out;
-	}
-
-	ret = 0;
+    glusterfs_graph_t *graph = NULL;
+    int ret = -1;
+    xlator_t *trav = NULL;
+    glusterfs_ctx_t *ctx = NULL;
+
+    ctx = fs->ctx;
+    graph = glusterfs_graph_construct(fp);
+    if (!graph) {
+        gf_smsg("glfs", GF_LOG_ERROR, errno, API_MSG_GRAPH_CONSTRUCT_FAILED,
+                NULL);
+        goto out;
+    }
+
+    for (trav = graph->first; trav; trav = trav->next) {
+        if (strcmp(trav->type, "mount/api") == 0) {
+            gf_smsg("glfs", GF_LOG_ERROR, EINVAL, API_MSG_API_XLATOR_ERROR,
+                    NULL);
+            goto out;
+        }
+    }
+
+    ret = glusterfs_graph_prepare(graph, ctx, fs->volname);
+    if (ret) {
+        glusterfs_graph_destroy(graph);
+        goto out;
+    }
+
+    ret = glusterfs_graph_activate(graph, ctx);
+
+    if (ret) {
+        glusterfs_graph_destroy(graph);
+        goto out;
+    }
+
+    gf_log_dump_graph(fp, graph);
+
+    ret = 0;
 out:
-	if (fp)
-		fclose (fp);
+    if (fp)
+        fclose(fp);
 
-	if (!ctx->active) {
-		ret = -1;
-	}
+    if (!ctx->active) {
+        ret = -1;
+    }
 
-	return ret;
+    return ret;
 }
 
-
 int
-mgmt_cbk_spec (struct rpc_clnt *rpc, void *mydata, void *data)
+mgmt_cbk_spec(struct rpc_clnt *rpc, void *mydata, void *data)
 {
-	struct glfs *fs = NULL;
-	xlator_t    *this = NULL;
+    struct glfs *fs = NULL;
+    xlator_t *this = NULL;
 
-	this = mydata;
-	fs = this->private;
+    this = mydata;
+    fs = this->private;
 
-	glfs_volfile_fetch (fs);
+    glfs_volfile_fetch(fs);
 
-	return 0;
+    return 0;
 }
 
-
 int
-mgmt_cbk_event (struct rpc_clnt *rpc, void *mydata, void *data)
+mgmt_cbk_event(struct rpc_clnt *rpc, void *mydata, void *data)
 {
-	return 0;
+    return 0;
 }
 
+static int
+mgmt_cbk_statedump(struct rpc_clnt *rpc, void *mydata, void *data)
+{
+    struct glfs *fs = NULL;
+    xlator_t *this = NULL;
+    gf_statedump target_pid = {
+        0,
+    };
+    struct iovec *iov = NULL;
+    int ret = -1;
+
+    this = mydata;
+    if (!this) {
+        gf_smsg("glfs", GF_LOG_ERROR, EINVAL, API_MSG_NULL, "mydata", NULL);
+        errno = EINVAL;
+        goto out;
+    }
+
+    fs = this->private;
+    if (!fs) {
+        gf_smsg("glfs", GF_LOG_ERROR, EINVAL, API_MSG_NULL, "glfs", NULL);
+        errno = EINVAL;
+        goto out;
+    }
+
+    iov = (struct iovec *)data;
+    if (!iov) {
+        gf_smsg("glfs", GF_LOG_ERROR, EINVAL, API_MSG_NULL, "iovec data", NULL);
+        errno = EINVAL;
+        goto out;
+    }
+
+    ret = xdr_to_generic(*iov, &target_pid, (xdrproc_t)xdr_gf_statedump);
+    if (ret < 0) {
+        gf_smsg("glfs", GF_LOG_ERROR, EINVAL, API_MSG_DECODE_XDR_FAILED, NULL);
+        goto out;
+    }
+
+    gf_msg_trace("glfs", 0, "statedump requested for pid: %d", target_pid.pid);
+
+    if ((uint64_t)getpid() == target_pid.pid) {
+        gf_msg_debug("glfs", 0, "Taking statedump for pid: %d", target_pid.pid);
+
+        ret = glfs_sysrq(fs, GLFS_SYSRQ_STATEDUMP);
+        if (ret < 0) {
+            gf_smsg("glfs", GF_LOG_INFO, 0, API_MSG_STATEDUMP_FAILED, NULL);
+        }
+    }
+out:
+    return ret;
+}
 
-rpcclnt_cb_actor_t mgmt_cbk_actors[GF_CBK_MAXVALUE] = {
-	[GF_CBK_FETCHSPEC] = {"FETCHSPEC", GF_CBK_FETCHSPEC, mgmt_cbk_spec },
-	[GF_CBK_EVENT_NOTIFY] = {"EVENTNOTIFY", GF_CBK_EVENT_NOTIFY,
-				 mgmt_cbk_event},
+static rpcclnt_cb_actor_t mgmt_cbk_actors[GF_CBK_MAXVALUE] = {
+    [GF_CBK_FETCHSPEC] = {"FETCHSPEC", mgmt_cbk_spec, GF_CBK_FETCHSPEC},
+    [GF_CBK_EVENT_NOTIFY] = {"EVENTNOTIFY", mgmt_cbk_event,
+                             GF_CBK_EVENT_NOTIFY},
+    [GF_CBK_STATEDUMP] = {"STATEDUMP", mgmt_cbk_statedump, GF_CBK_STATEDUMP},
 };
 
-
-struct rpcclnt_cb_program mgmt_cbk_prog = {
-	.progname  = "GlusterFS Callback",
-	.prognum   = GLUSTER_CBK_PROGRAM,
-	.progver   = GLUSTER_CBK_VERSION,
-	.actors	   = mgmt_cbk_actors,
-	.numactors = GF_CBK_MAXVALUE,
+static struct rpcclnt_cb_program mgmt_cbk_prog = {
+    .progname = "GlusterFS Callback",
+    .prognum = GLUSTER_CBK_PROGRAM,
+    .progver = GLUSTER_CBK_VERSION,
+    .actors = mgmt_cbk_actors,
+    .numactors = GF_CBK_MAXVALUE,
 };
 
-char *clnt_handshake_procs[GF_HNDSK_MAXVALUE] = {
-	[GF_HNDSK_NULL]		= "NULL",
-	[GF_HNDSK_SETVOLUME]	= "SETVOLUME",
-	[GF_HNDSK_GETSPEC]	= "GETSPEC",
-	[GF_HNDSK_PING]		= "PING",
-	[GF_HNDSK_EVENT_NOTIFY] = "EVENTNOTIFY",
-        [GF_HNDSK_GET_VOLUME_INFO] = "GETVOLUMEINFO",
+static char *clnt_handshake_procs[GF_HNDSK_MAXVALUE] = {
+    [GF_HNDSK_NULL] = "NULL",
+    [GF_HNDSK_SETVOLUME] = "SETVOLUME",
+    [GF_HNDSK_GETSPEC] = "GETSPEC",
+    [GF_HNDSK_PING] = "PING",
+    [GF_HNDSK_EVENT_NOTIFY] = "EVENTNOTIFY",
+    [GF_HNDSK_GET_VOLUME_INFO] = "GETVOLUMEINFO",
 };
 
-rpc_clnt_prog_t clnt_handshake_prog = {
-	.progname  = "GlusterFS Handshake",
-	.prognum   = GLUSTER_HNDSK_PROGRAM,
-	.progver   = GLUSTER_HNDSK_VERSION,
-	.procnames = clnt_handshake_procs,
+static rpc_clnt_prog_t clnt_handshake_prog = {
+    .progname = "GlusterFS Handshake",
+    .prognum = GLUSTER_HNDSK_PROGRAM,
+    .progver = GLUSTER_HNDSK_VERSION,
+    .procnames = clnt_handshake_procs,
 };
 
-
 int
-mgmt_submit_request (void *req, call_frame_t *frame,
-		     glusterfs_ctx_t *ctx,
-		     rpc_clnt_prog_t *prog, int procnum,
-		     fop_cbk_fn_t cbkfn, xdrproc_t xdrproc)
+mgmt_submit_request(void *req, call_frame_t *frame, glusterfs_ctx_t *ctx,
+                    rpc_clnt_prog_t *prog, int procnum, fop_cbk_fn_t cbkfn,
+                    xdrproc_t xdrproc)
 {
-	int			ret	    = -1;
-	int			count	   = 0;
-	struct iovec		iov	    = {0, };
-	struct iobuf		*iobuf = NULL;
-	struct iobref		*iobref = NULL;
-	ssize_t			xdr_size = 0;
-
-	iobref = iobref_new ();
-	if (!iobref) {
-		goto out;
-	}
-
-	if (req) {
-		xdr_size = xdr_sizeof (xdrproc, req);
-
-		iobuf = iobuf_get2 (ctx->iobuf_pool, xdr_size);
-		if (!iobuf) {
-			goto out;
-		};
-
-		iobref_add (iobref, iobuf);
-
-		iov.iov_base = iobuf->ptr;
-		iov.iov_len  = iobuf_pagesize (iobuf);
-
-		/* Create the xdr payload */
-		ret = xdr_serialize_generic (iov, req, xdrproc);
-		if (ret == -1) {
-			gf_log (THIS->name, GF_LOG_WARNING,
-				"failed to create XDR payload");
-			goto out;
-		}
-		iov.iov_len = ret;
-		count = 1;
-	}
-
-	/* Send the msg */
-	ret = rpc_clnt_submit (ctx->mgmt, prog, procnum, cbkfn,
-			       &iov, count,
-			       NULL, 0, iobref, frame, NULL, 0, NULL, 0, NULL);
+    int ret = -1;
+    int count = 0;
+    struct iovec iov = {
+        0,
+    };
+    struct iobuf *iobuf = NULL;
+    struct iobref *iobref = NULL;
+    ssize_t xdr_size = 0;
+
+    iobref = iobref_new();
+    if (!iobref) {
+        goto out;
+    }
+
+    if (req) {
+        xdr_size = xdr_sizeof(xdrproc, req);
+
+        iobuf = iobuf_get2(ctx->iobuf_pool, xdr_size);
+        if (!iobuf) {
+            goto out;
+        };
+
+        iobref_add(iobref, iobuf);
+
+        iov.iov_base = iobuf->ptr;
+        iov.iov_len = iobuf_pagesize(iobuf);
+
+        /* Create the xdr payload */
+        ret = xdr_serialize_generic(iov, req, xdrproc);
+        if (ret == -1) {
+            gf_smsg(THIS->name, GF_LOG_WARNING, 0, API_MSG_XDR_PAYLOAD_FAILED,
+                    NULL);
+            goto out;
+        }
+        iov.iov_len = ret;
+        count = 1;
+    }
+
+    /* Send the msg */
+    ret = rpc_clnt_submit(ctx->mgmt, prog, procnum, cbkfn, &iov, count, NULL, 0,
+                          iobref, frame, NULL, 0, NULL, 0, NULL);
 
 out:
-	if (iobref)
-		iobref_unref (iobref);
+    if (iobref)
+        iobref_unref(iobref);
 
-	if (iobuf)
-		iobuf_unref (iobuf);
-	return ret;
+    if (iobuf)
+        iobuf_unref(iobuf);
+    return ret;
 }
 
 /*
  * Callback routine for 'GF_HNDSK_GET_VOLUME_INFO' rpc request
  */
 int
-mgmt_get_volinfo_cbk (struct rpc_req *req, struct iovec *iov,
-                      int count, void *myframe)
+mgmt_get_volinfo_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                     void *myframe)
 {
-        int                        ret                  = 0;
-        char                       *volume_id_str       = NULL;
-        dict_t                     *dict                = NULL;
-        char                       key[1024]            = {0};
-        gf_get_volume_info_rsp     rsp                  = {0,};
-        call_frame_t               *frame               = NULL;
-        glusterfs_ctx_t            *ctx                 = NULL;
-        struct glfs                *fs                  = NULL;
-        struct syncargs            *args;
-
-        frame = myframe;
-        ctx = frame->this->ctx;
-        args = frame->local;
-
-        if (!ctx) {
-                gf_log (frame->this->name, GF_LOG_ERROR, "NULL context");
-                errno = EINVAL;
-                ret = -1;
-                goto out;
-        }
-
-        fs = ((xlator_t *)ctx->master)->private;
-
-        if (-1 == req->rpc_status) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "GET_VOLUME_INFO RPC call is not successfull");
-                errno = EINVAL;
-                ret = -1;
-                goto out;
-        }
-
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_get_volume_info_rsp);
-
-        if (ret < 0) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response for GET_VOLUME_INFO");
-                goto out;
-        }
-
-        gf_log (frame->this->name, GF_LOG_DEBUG,
-                "Received resp to GET_VOLUME_INFO RPC: %d", rsp.op_ret);
-
-        if (rsp.op_ret == -1) {
-                errno = rsp.op_errno;
-                ret = -1;
-                goto out;
-        }
-
-        if (!rsp.dict.dict_len) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "Response received for GET_VOLUME_INFO RPC call is not valid");
-                ret = -1;
-                errno = EINVAL;
-                goto out;
-        }
-
-        dict = dict_new ();
-
-        if (!dict) {
-                ret = -1;
-                errno = ENOMEM;
-                goto out;
-        }
-
-        ret = dict_unserialize (rsp.dict.dict_val,
-                                rsp.dict.dict_len,
-                                &dict);
-
-        if (ret) {
-                errno = ENOMEM;
-                goto out;
-        }
-
-        snprintf (key, sizeof (key), "volume_id");
-        ret = dict_get_str (dict, key, &volume_id_str);
-        if (ret) {
-                errno = EINVAL;
-                goto out;
-        }
-
-        ret = 0;
+    int ret = 0;
+    char *volume_id_str = NULL;
+    dict_t *dict = NULL;
+    gf_get_volume_info_rsp rsp = {
+        0,
+    };
+    call_frame_t *frame = NULL;
+    glusterfs_ctx_t *ctx = NULL;
+    struct glfs *fs = NULL;
+    struct syncargs *args;
+
+    frame = myframe;
+    ctx = frame->this->ctx;
+    args = frame->local;
+
+    if (!ctx) {
+        gf_smsg(frame->this->name, GF_LOG_ERROR, EINVAL, API_MSG_NULL,
+                "context", NULL);
+        errno = EINVAL;
+        ret = -1;
+        goto out;
+    }
+
+    fs = ((xlator_t *)ctx->master)->private;
+
+    if (-1 == req->rpc_status) {
+        gf_smsg(frame->this->name, GF_LOG_ERROR, EINVAL,
+                API_MSG_CALL_NOT_SUCCESSFUL, NULL);
+        errno = EINVAL;
+        ret = -1;
+        goto out;
+    }
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_get_volume_info_rsp);
+
+    if (ret < 0) {
+        gf_smsg(frame->this->name, GF_LOG_ERROR, 0,
+                API_MSG_XDR_RESPONSE_DECODE_FAILED, NULL);
+        goto out;
+    }
+
+    gf_msg_debug(frame->this->name, 0,
+                 "Received resp to GET_VOLUME_INFO "
+                 "RPC: %d",
+                 rsp.op_ret);
+
+    if (rsp.op_ret == -1) {
+        errno = rsp.op_errno;
+        ret = -1;
+        goto out;
+    }
+
+    if (!rsp.dict.dict_len) {
+        gf_smsg(frame->this->name, GF_LOG_ERROR, EINVAL, API_MSG_CALL_NOT_VALID,
+                NULL);
+        ret = -1;
+        errno = EINVAL;
+        goto out;
+    }
+
+    dict = dict_new();
+
+    if (!dict) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &dict);
+
+    if (ret) {
+        errno = ENOMEM;
+        goto out;
+    }
+
+    ret = dict_get_str_sizen(dict, "volume_id", &volume_id_str);
+    if (ret) {
+        errno = EINVAL;
+        goto out;
+    }
+
+    ret = 0;
 out:
-        if (volume_id_str) {
-                gf_log (frame->this->name, GF_LOG_DEBUG,
-                        "Volume Id: %s", volume_id_str);
-                pthread_mutex_lock (&fs->mutex);
-                uuid_parse (volume_id_str, fs->vol_uuid);
-                pthread_mutex_unlock (&fs->mutex);
-        }
+    if (volume_id_str) {
+        gf_msg_debug(frame->this->name, 0, "Volume Id: %s", volume_id_str);
+        pthread_mutex_lock(&fs->mutex);
+        gf_uuid_parse(volume_id_str, fs->vol_uuid);
+        pthread_mutex_unlock(&fs->mutex);
+    }
 
-        if (ret) {
-               gf_log (frame->this->name, GF_LOG_ERROR,
-                       "In GET_VOLUME_INFO cbk, received error: %s",
-                       strerror(errno));
-        }
+    if (ret) {
+        gf_smsg(frame->this->name, GF_LOG_ERROR, errno,
+                API_MSG_GET_VOLINFO_CBK_FAILED, "error=%s", strerror(errno),
+                NULL);
+    }
 
-        if (dict)
-                dict_destroy (dict);
+    if (dict)
+        dict_unref(dict);
 
-        if (rsp.dict.dict_val)
-                free (rsp.dict.dict_val);
+    if (rsp.dict.dict_val)
+        free(rsp.dict.dict_val);
 
-        if (rsp.op_errstr && *rsp.op_errstr)
-                free (rsp.op_errstr);
+    if (rsp.op_errstr)
+        free(rsp.op_errstr);
 
-        gf_log (frame->this->name, GF_LOG_DEBUG, "Returning: %d", ret);
+    gf_msg_debug(frame->this->name, 0, "Returning: %d", ret);
 
-        __wake (args);
+    __wake(args);
 
-        return ret;
+    return ret;
 }
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_get_volumeid, 3.5.0)
 int
-pub_glfs_get_volumeid (struct glfs *fs, char *volid, size_t size)
+pub_glfs_get_volumeid(struct glfs *fs, char *volid, size_t size)
 {
-        /* TODO: Define a global macro to store UUID size */
-        size_t uuid_size = 16;
-
-        pthread_mutex_lock (&fs->mutex);
-        {
-                /* check if the volume uuid is initialized */
-                if (!uuid_is_null (fs->vol_uuid)) {
-                        pthread_mutex_unlock (&fs->mutex);
-                        goto done;
-                }
+    /* TODO: Define a global macro to store UUID size */
+    size_t uuid_size = 16;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    pthread_mutex_lock(&fs->mutex);
+    {
+        /* check if the volume uuid is initialized */
+        if (!gf_uuid_is_null(fs->vol_uuid)) {
+            pthread_mutex_unlock(&fs->mutex);
+            goto done;
         }
-        pthread_mutex_unlock (&fs->mutex);
+    }
+    pthread_mutex_unlock(&fs->mutex);
 
-        /* Need to fetch volume_uuid */
-        glfs_get_volume_info (fs);
+    /* Need to fetch volume_uuid */
+    glfs_get_volume_info(fs);
 
-        if (uuid_is_null (fs->vol_uuid)) {
-                gf_log (THIS->name, GF_LOG_ERROR, "Unable to fetch volume UUID");
-                return -1;
-        }
+    if (gf_uuid_is_null(fs->vol_uuid)) {
+        gf_smsg(THIS->name, GF_LOG_ERROR, EINVAL, API_MSG_FETCH_VOLUUID_FAILED,
+                NULL);
+        goto out;
+    }
 
 done:
-        if (!volid || !size) {
-                gf_log (THIS->name, GF_LOG_DEBUG, "volumeid/size is null");
-                return uuid_size;
-        }
+    if (!volid || !size) {
+        gf_msg_debug(THIS->name, 0, "volumeid/size is null");
+        __GLFS_EXIT_FS;
+        return uuid_size;
+    }
 
-        if (size < uuid_size) {
-                gf_log (THIS->name, GF_LOG_ERROR, "Insufficient size passed");
-                errno = ERANGE;
-                return -1;
-        }
+    if (size < uuid_size) {
+        gf_smsg(THIS->name, GF_LOG_ERROR, ERANGE, API_MSG_INSUFF_SIZE, NULL);
+        errno = ERANGE;
+        goto out;
+    }
 
-        memcpy (volid, fs->vol_uuid, uuid_size);
+    memcpy(volid, fs->vol_uuid, uuid_size);
 
-        return uuid_size;
-}
+    __GLFS_EXIT_FS;
+
+    return uuid_size;
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_get_volumeid, 3.5.0);
+out:
+    __GLFS_EXIT_FS;
+
+invalid_fs:
+    return -1;
+}
 
 int
-glfs_get_volume_info (struct glfs *fs)
+glfs_get_volume_info(struct glfs *fs)
 {
-        call_frame_t     *frame = NULL;
-        glusterfs_ctx_t  *ctx   = NULL;
-        struct syncargs  args   = {0, };
-        int              ret    = 0;
+    call_frame_t *frame = NULL;
+    glusterfs_ctx_t *ctx = NULL;
+    struct syncargs args = {
+        0,
+    };
+    int ret = 0;
 
-        ctx = fs->ctx;
-        frame = create_frame (THIS, ctx->pool);
-        frame->local = &args;
+    ctx = fs->ctx;
+    frame = create_frame(THIS, ctx->pool);
+    if (!frame) {
+        gf_smsg("glfs", GF_LOG_ERROR, ENOMEM, API_MSG_FRAME_CREAT_FAILED, NULL);
+        ret = -1;
+        goto out;
+    }
 
-        __yawn ((&args));
+    frame->local = &args;
 
-        ret = glfs_get_volume_info_rpc (frame, THIS, fs);
-        if (ret)
-                goto out;
+    __yawn((&args));
+
+    ret = glfs_get_volume_info_rpc(frame, THIS, fs);
+    if (ret)
+        goto out;
 
-        __yield ((&args));
+    __yield((&args));
 
-        frame->local = NULL;
-        STACK_DESTROY (frame->root);
+    frame->local = NULL;
+    STACK_DESTROY(frame->root);
 
 out:
-        return ret;
+    return ret;
 }
 
 int32_t
-glfs_get_volume_info_rpc (call_frame_t *frame, xlator_t *this,
-                          struct glfs *fs)
+glfs_get_volume_info_rpc(call_frame_t *frame, xlator_t *this, struct glfs *fs)
 {
-        gf_get_volume_info_req  req       = {{0,}};
-        int                     ret       = 0;
-        glusterfs_ctx_t         *ctx      = NULL;
-        dict_t                  *dict     = NULL;
-        int32_t                 flags     = 0;
-
-        if (!frame || !this || !fs) {
-                ret = -1;
-                goto out;
-        }
-
-        ctx = fs->ctx;
-
-        dict = dict_new ();
-        if (!dict) {
-                ret = -1;
-                goto out;
-        }
-
-        if (fs->volname) {
-                ret = dict_set_str (dict, "volname", fs->volname);
-                if (ret)
-                        goto out;
-        }
-
-        // Set the flags for the fields which we are interested in
-        flags = (int32_t)GF_GET_VOLUME_UUID; //ctx->flags;
-        ret = dict_set_int32 (dict, "flags", flags);
-        if (ret) {
-                gf_log (frame->this->name, GF_LOG_ERROR, "failed to set flags");
-                goto out;
-        }
-
-        ret = dict_allocate_and_serialize (dict, &req.dict.dict_val,
-                                           &req.dict.dict_len);
-
-
-        ret = mgmt_submit_request (&req, frame, ctx, &clnt_handshake_prog,
-                                   GF_HNDSK_GET_VOLUME_INFO,
-                                   mgmt_get_volinfo_cbk,
-                                   (xdrproc_t)xdr_gf_get_volume_info_req);
+    gf_get_volume_info_req req = {{
+        0,
+    }};
+    int ret = 0;
+    glusterfs_ctx_t *ctx = NULL;
+    dict_t *dict = NULL;
+    int32_t flags = 0;
+
+    if (!frame || !this || !fs) {
+        ret = -1;
+        goto out;
+    }
+
+    ctx = fs->ctx;
+
+    dict = dict_new();
+    if (!dict) {
+        ret = -1;
+        goto out;
+    }
+
+    if (fs->volname) {
+        ret = dict_set_str(dict, "volname", fs->volname);
+        if (ret)
+            goto out;
+    }
+
+    // Set the flags for the fields which we are interested in
+    flags = (int32_t)GF_GET_VOLUME_UUID;  // ctx->flags;
+    ret = dict_set_int32(dict, "flags", flags);
+    if (ret) {
+        gf_smsg(frame->this->name, GF_LOG_ERROR, EINVAL,
+                API_MSG_DICT_SET_FAILED, "flags", NULL);
+        goto out;
+    }
+
+    ret = dict_allocate_and_serialize(dict, &req.dict.dict_val,
+                                      &req.dict.dict_len);
+
+    ret = mgmt_submit_request(&req, frame, ctx, &clnt_handshake_prog,
+                              GF_HNDSK_GET_VOLUME_INFO, mgmt_get_volinfo_cbk,
+                              (xdrproc_t)xdr_gf_get_volume_info_req);
 out:
-        if (dict) {
-                dict_unref (dict);
-        }
+    if (dict) {
+        dict_unref(dict);
+    }
 
-        GF_FREE (req.dict.dict_val);
+    GF_FREE(req.dict.dict_val);
 
-        return ret;
+    return ret;
 }
 
 static int
-glusterfs_oldvolfile_update (struct glfs *fs, char *volfile, ssize_t size)
+glusterfs_oldvolfile_update(struct glfs *fs, char *volfile, ssize_t size)
 {
-	int ret = -1;
-
-        pthread_mutex_lock (&fs->mutex);
-
-	fs->oldvollen = size;
-	if (!fs->oldvolfile) {
-		fs->oldvolfile = GF_CALLOC (1, size+1, glfs_mt_volfile_t);
-	} else {
-		fs->oldvolfile = GF_REALLOC (fs->oldvolfile, size+1);
-	}
-
-	if (!fs->oldvolfile) {
-		fs->oldvollen = 0;
-	} else {
-		memcpy (fs->oldvolfile, volfile, size);
-		fs->oldvollen = size;
-		ret = 0;
-	}
+    int ret = -1;
+
+    pthread_mutex_lock(&fs->mutex);
+
+    fs->oldvollen = size;
+    if (!fs->oldvolfile) {
+        fs->oldvolfile = CALLOC(1, size + 1);
+    } else {
+        fs->oldvolfile = REALLOC(fs->oldvolfile, size + 1);
+    }
+
+    if (!fs->oldvolfile) {
+        fs->oldvollen = 0;
+    } else {
+        memcpy(fs->oldvolfile, volfile, size);
+        fs->oldvollen = size;
+        ret = 0;
+    }
 
-        pthread_mutex_unlock (&fs->mutex);
+    pthread_mutex_unlock(&fs->mutex);
 
-	return ret;
+    return ret;
 }
 
-
 int
-mgmt_getspec_cbk (struct rpc_req *req, struct iovec *iov, int count,
-		  void *myframe)
+glfs_mgmt_getspec_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                      void *myframe)
 {
-	gf_getspec_rsp		 rsp   = {0,};
-	call_frame_t		*frame = NULL;
-	glusterfs_ctx_t		*ctx = NULL;
-	int			 ret   = 0;
-	ssize_t			 size = 0;
-	FILE			*tmpfp = NULL;
-	int			 need_retry = 0;
-	struct glfs		*fs = NULL;
-
-	frame = myframe;
-	ctx = frame->this->ctx;
-
-	if (!ctx) {
-		gf_log (frame->this->name, GF_LOG_ERROR, "NULL context");
-		errno = EINVAL;
-		ret = -1;
-		goto out;
-	}
-
-	fs = ((xlator_t *)ctx->master)->private;
-
-	if (-1 == req->rpc_status) {
-		ret = -1;
-		need_retry = 1;
-		goto out;
-	}
-
-	ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_getspec_rsp);
-	if (ret < 0) {
-		gf_log (frame->this->name, GF_LOG_ERROR, "XDR decoding error");
-		ret   = -1;
-		goto out;
-	}
-
-	if (-1 == rsp.op_ret) {
-		gf_log (frame->this->name, GF_LOG_ERROR,
-			"failed to get the 'volume file' from server");
-		ret = -1;
-		errno = rsp.op_errno;
-		goto out;
-	}
-
-	ret = 0;
-	size = rsp.op_ret;
-
-	if ((size == fs->oldvollen) &&
-	    (memcmp (fs->oldvolfile, rsp.spec, size) == 0)) {
-		gf_log (frame->this->name, GF_LOG_INFO,
-			"No change in volfile, continuing");
-		goto out;
-	}
-
-	tmpfp = tmpfile ();
-	if (!tmpfp) {
-		ret = -1;
-		goto out;
-	}
-
-	fwrite (rsp.spec, size, 1, tmpfp);
-	fflush (tmpfp);
-	if (ferror (tmpfp)) {
-		ret = -1;
-		goto out;
-	}
-
-	/*  Check if only options have changed. No need to reload the
-	*  volfile if topology hasn't changed.
-	*  glusterfs_volfile_reconfigure returns 3 possible return states
-	*  return 0	     =======> reconfiguration of options has succeeded
-	*  return 1	     =======> the graph has to be reconstructed and all the xlators should be inited
-	*  return -1(or -ve) =======> Some Internal Error occurred during the operation
-	*/
-
-	ret = glusterfs_volfile_reconfigure (fs->oldvollen, tmpfp, fs->ctx,
-					     fs->oldvolfile);
-	if (ret == 0) {
-		gf_log ("glusterfsd-mgmt", GF_LOG_DEBUG,
-			"No need to re-load volfile, reconfigure done");
-		ret = glusterfs_oldvolfile_update (fs, rsp.spec, size);
-		goto out;
-	}
-
-	if (ret < 0) {
-		gf_log ("glusterfsd-mgmt", GF_LOG_DEBUG,
-			"Reconfigure failed !!");
-		goto out;
-	}
-
-	ret = glfs_process_volfp (fs, tmpfp);
-	/* tmpfp closed */
-	tmpfp = NULL;
-	if (ret)
-		goto out;
-
-	ret = glusterfs_oldvolfile_update (fs, rsp.spec, size);
+    gf_getspec_rsp rsp = {
+        0,
+    };
+    call_frame_t *frame = NULL;
+    glusterfs_ctx_t *ctx = NULL;
+    int ret = 0;
+    ssize_t size = 0;
+    FILE *tmpfp = NULL;
+    int need_retry = 0;
+    struct glfs *fs = NULL;
+    dict_t *dict = NULL;
+    char *servers_list = NULL;
+    int tmp_fd = -1;
+    char template[] = "/tmp/gfapi.volfile.XXXXXX";
+
+    frame = myframe;
+    ctx = frame->this->ctx;
+
+    if (!ctx) {
+        gf_smsg(frame->this->name, GF_LOG_ERROR, EINVAL, API_MSG_NULL,
+                "context", NULL);
+        errno = EINVAL;
+        ret = -1;
+        goto out;
+    }
+
+    fs = ((xlator_t *)ctx->master)->private;
+
+    if (-1 == req->rpc_status) {
+        ret = -1;
+        need_retry = 1;
+        goto out;
+    }
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_getspec_rsp);
+    if (ret < 0) {
+        gf_smsg(frame->this->name, GF_LOG_ERROR, 0, API_MSG_XDR_DECODE_FAILED,
+                NULL);
+        ret = -1;
+        goto out;
+    }
+
+    if (-1 == rsp.op_ret) {
+        gf_smsg(frame->this->name, GF_LOG_ERROR, rsp.op_errno,
+                API_MSG_GET_VOLFILE_FAILED, "from server", NULL);
+        ret = -1;
+        errno = rsp.op_errno;
+        goto out;
+    }
+
+    if (!rsp.xdata.xdata_len) {
+        goto volfile;
+    }
+
+    dict = dict_new();
+    if (!dict) {
+        ret = -1;
+        errno = ENOMEM;
+        goto out;
+    }
+
+    ret = dict_unserialize(rsp.xdata.xdata_val, rsp.xdata.xdata_len, &dict);
+    if (ret) {
+        gf_log(frame->this->name, GF_LOG_ERROR,
+               "failed to unserialize xdata to dictionary");
+        goto out;
+    }
+    dict->extra_stdfree = rsp.xdata.xdata_val;
+
+    /* glusterd2 only */
+    ret = dict_get_str(dict, "servers-list", &servers_list);
+    if (ret) {
+        goto volfile;
+    }
+
+    gf_log(frame->this->name, GF_LOG_INFO,
+           "Received list of available volfile servers: %s", servers_list);
+
+    ret = gf_process_getspec_servers_list(&ctx->cmd_args, servers_list);
+    if (ret) {
+        gf_log(frame->this->name, GF_LOG_ERROR,
+               "Failed (%s) to process servers list: %s", strerror(errno),
+               servers_list);
+    }
+
+volfile:
+    ret = 0;
+    size = rsp.op_ret;
+
+    pthread_mutex_lock(&fs->mutex);
+    if ((size == fs->oldvollen) &&
+        (memcmp(fs->oldvolfile, rsp.spec, size) == 0)) {
+        pthread_mutex_unlock(&fs->mutex);
+        gf_smsg(frame->this->name, GF_LOG_INFO, 0, API_MSG_VOLFILE_INFO, NULL);
+        goto out;
+    }
+    pthread_mutex_unlock(&fs->mutex);
+
+    /* coverity[secure_temp] mkstemp uses 0600 as the mode and is safe */
+    tmp_fd = mkstemp(template);
+    if (-1 == tmp_fd) {
+        ret = -1;
+        goto out;
+    }
+
+    /* Calling unlink so that when the file is closed or program
+     * terminates the temporary file is deleted.
+     */
+    ret = sys_unlink(template);
+    if (ret < 0) {
+        gf_smsg(frame->this->name, GF_LOG_INFO, 0, API_MSG_UNABLE_TO_DEL,
+                "template=%s", template, NULL);
+        ret = 0;
+    }
+
+    tmpfp = fdopen(tmp_fd, "w+b");
+    if (!tmpfp) {
+        ret = -1;
+        goto out;
+    }
+
+    fwrite(rsp.spec, size, 1, tmpfp);
+    fflush(tmpfp);
+    if (ferror(tmpfp)) {
+        ret = -1;
+        goto out;
+    }
+
+    /*  Check if only options have changed. No need to reload the
+     *  volfile if topology hasn't changed.
+     *  glusterfs_volfile_reconfigure returns 3 possible return states
+     *  return 0	     =======> reconfiguration of options has succeeded
+     *  return 1	     =======> the graph has to be reconstructed and all
+     * the xlators should be inited return -1(or -ve) =======> Some Internal
+     * Error occurred during the operation
+     */
+
+    pthread_mutex_lock(&fs->mutex);
+    ret = gf_volfile_reconfigure(fs->oldvollen, tmpfp, fs->ctx, fs->oldvolfile);
+    pthread_mutex_unlock(&fs->mutex);
+
+    if (ret == 0) {
+        gf_msg_debug("glusterfsd-mgmt", 0,
+                     "No need to re-load "
+                     "volfile, reconfigure done");
+        ret = glusterfs_oldvolfile_update(fs, rsp.spec, size);
+        goto out;
+    }
+
+    if (ret < 0) {
+        gf_msg_debug("glusterfsd-mgmt", 0, "Reconfigure failed !!");
+        goto out;
+    }
+
+    ret = glfs_process_volfp(fs, tmpfp);
+    /* tmpfp closed */
+    tmpfp = NULL;
+    tmp_fd = -1;
+    if (ret)
+        goto out;
+
+    ret = glusterfs_oldvolfile_update(fs, rsp.spec, size);
 out:
-	STACK_DESTROY (frame->root);
-
-	if (rsp.spec)
-		free (rsp.spec);
-
-	// Stop if server is running at an unsupported op-version
-	if (ENOTSUP == ret) {
-		gf_log ("mgmt", GF_LOG_ERROR, "Server is operating at an "
-			"op-version which is not supported");
-		errno = ENOTSUP;
-		priv_glfs_init_done (fs, -1);
-	}
-
-	if (ret && ctx && !ctx->active) {
-		/* Do it only for the first time */
-		/* Failed to get the volume file, something wrong,
-		   restart the process */
-		gf_log ("glfs-mgmt", GF_LOG_ERROR,
-			"failed to fetch volume file (key:%s)",
-			ctx->cmd_args.volfile_id);
-		if (!need_retry) {
-			if (!errno)
-				errno = EINVAL;
-			priv_glfs_init_done (fs, -1);
-		}
-	}
-
-	if (tmpfp)
-		fclose (tmpfp);
-
-	return 0;
-}
-
-
-int
-glfs_volfile_fetch (struct glfs *fs)
-{
-	cmd_args_t	 *cmd_args = NULL;
-	gf_getspec_req	  req = {0, };
-	int		  ret = 0;
-	call_frame_t	 *frame = NULL;
-	glusterfs_ctx_t	 *ctx = NULL;
-        dict_t           *dict = NULL;
-
-	ctx = fs->ctx;
-	cmd_args = &ctx->cmd_args;
-
-	frame = create_frame (THIS, ctx->pool);
-
-	req.key = cmd_args->volfile_id;
-	req.flags = 0;
-
-        dict = dict_new ();
-        if (!dict) {
-                ret = -1;
-                goto out;
-        }
-
-        // Set the supported min and max op-versions, so glusterd can make a
-        // decision
-        ret = dict_set_int32 (dict, "min-op-version", GD_OP_VERSION_MIN);
-        if (ret) {
-                gf_log (THIS->name, GF_LOG_ERROR, "Failed to set min-op-version"
-                        " in request dict");
-                goto out;
+    STACK_DESTROY(frame->root);
+
+    if (rsp.spec)
+        free(rsp.spec);
+
+    if (dict)
+        dict_unref(dict);
+
+    // Stop if server is running at an unsupported op-version
+    if (ENOTSUP == ret) {
+        gf_smsg("mgmt", GF_LOG_ERROR, ENOTSUP, API_MSG_WRONG_OPVERSION, NULL);
+        errno = ENOTSUP;
+        glfs_init_done(fs, -1);
+    }
+
+    if (ret && ctx && !ctx->active) {
+        /* Do it only for the first time */
+        /* Failed to get the volume file, something wrong,
+           restart the process */
+        gf_smsg("glfs-mgmt", GF_LOG_ERROR, EINVAL, API_MSG_GET_VOLFILE_FAILED,
+                "key=%s", ctx->cmd_args.volfile_id, NULL);
+        if (!need_retry) {
+            if (!errno)
+                errno = EINVAL;
+            glfs_init_done(fs, -1);
         }
+    }
 
-        ret = dict_set_int32 (dict, "max-op-version", GD_OP_VERSION_MAX);
-        if (ret) {
-                gf_log (THIS->name, GF_LOG_ERROR, "Failed to set max-op-version"
-                        " in request dict");
-                goto out;
-        }
+    if (tmpfp)
+        fclose(tmpfp);
+    else if (tmp_fd != -1)
+        sys_close(tmp_fd);
 
-        ret = dict_allocate_and_serialize (dict, &req.xdata.xdata_val,
-                                           &req.xdata.xdata_len);
-        if (ret < 0) {
-                gf_log (THIS->name, GF_LOG_ERROR,
-                        "Failed to serialize dictionary");
-                goto out;
-        }
+    return 0;
+}
 
-	ret = mgmt_submit_request (&req, frame, ctx, &clnt_handshake_prog,
-				   GF_HNDSK_GETSPEC, mgmt_getspec_cbk,
-				   (xdrproc_t)xdr_gf_getspec_req);
+int
+glfs_volfile_fetch(struct glfs *fs)
+{
+    cmd_args_t *cmd_args = NULL;
+    gf_getspec_req req = {
+        0,
+    };
+    int ret = -1;
+    call_frame_t *frame = NULL;
+    glusterfs_ctx_t *ctx = NULL;
+    dict_t *dict = NULL;
+
+    ctx = fs->ctx;
+    cmd_args = &ctx->cmd_args;
+
+    req.key = cmd_args->volfile_id;
+    req.flags = 0;
+
+    dict = dict_new();
+    if (!dict) {
+        goto out;
+    }
+
+    // Set the supported min and max op-versions, so glusterd can make a
+    // decision
+    ret = dict_set_int32(dict, "min-op-version", GD_OP_VERSION_MIN);
+    if (ret) {
+        gf_smsg(THIS->name, GF_LOG_ERROR, EINVAL, API_MSG_DICT_SET_FAILED,
+                "min-op-version", NULL);
+        goto out;
+    }
+
+    ret = dict_set_int32(dict, "max-op-version", GD_OP_VERSION_MAX);
+    if (ret) {
+        gf_smsg(THIS->name, GF_LOG_ERROR, EINVAL, API_MSG_DICT_SET_FAILED,
+                "max-op-version", NULL);
+        goto out;
+    }
+
+    /* Ask for a list of volfile (glusterd2 only) servers */
+    if (GF_CLIENT_PROCESS == ctx->process_mode) {
+        req.flags = req.flags | GF_GETSPEC_FLAG_SERVERS_LIST;
+    }
+
+    ret = dict_allocate_and_serialize(dict, &req.xdata.xdata_val,
+                                      &req.xdata.xdata_len);
+    if (ret < 0) {
+        gf_smsg(THIS->name, GF_LOG_ERROR, 0, API_MSG_DICT_SERIALIZE_FAILED,
+                NULL);
+        goto out;
+    }
+
+    frame = create_frame(THIS, ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
+
+    ret = mgmt_submit_request(&req, frame, ctx, &clnt_handshake_prog,
+                              GF_HNDSK_GETSPEC, glfs_mgmt_getspec_cbk,
+                              (xdrproc_t)xdr_gf_getspec_req);
 out:
-        return ret;
-}
+    if (req.xdata.xdata_val)
+        GF_FREE(req.xdata.xdata_val);
+    if (dict)
+        dict_unref(dict);
 
+    return ret;
+}
 
 static int
-mgmt_rpc_notify (struct rpc_clnt *rpc, void *mydata, rpc_clnt_event_t event,
-		 void *data)
+mgmt_rpc_notify(struct rpc_clnt *rpc, void *mydata, rpc_clnt_event_t event,
+                void *data)
 {
-	xlator_t	*this = NULL;
-	glusterfs_ctx_t *ctx = NULL;
-        server_cmdline_t *server = NULL;
-        rpc_transport_t  *rpc_trans = NULL;
-	struct glfs	 *fs = NULL;
-	int		 ret = 0;
-
-	this = mydata;
-        rpc_trans = rpc->conn.trans;
-
-	ctx = this->ctx;
-	if (!ctx)
-		goto out;
-
-	fs = ((xlator_t *)ctx->master)->private;
-
-	switch (event) {
-	case RPC_CLNT_DISCONNECT:
-		if (!ctx->active) {
-                        gf_log ("glfs-mgmt", GF_LOG_ERROR,
-                                "failed to connect with remote-host: %s (%s)",
-                                ctx->cmd_args.volfile_server,
-                                strerror (errno));
-                        server = ctx->cmd_args.curr_server;
-                        if (server->list.next == &ctx->cmd_args.volfile_servers) {
-                                errno = ENOTCONN;
-                                gf_log("glfs-mgmt", GF_LOG_INFO,
-                                       "Exhausted all volfile servers");
-                                priv_glfs_init_done (fs, -1);
-                                break;
-                        }
-                        server = list_entry (server->list.next, typeof(*server),
-                                             list);
-                        ctx->cmd_args.curr_server = server;
-                        ctx->cmd_args.volfile_server_port = server->port;
-                        ctx->cmd_args.volfile_server = server->volfile_server;
-                        ctx->cmd_args.volfile_server_transport = server->transport;
-
-                        ret = dict_set_int32 (rpc_trans->options,
-                                              "remote-port",
-                                              server->port);
-                        if (ret != 0) {
-                                gf_log ("glfs-mgmt", GF_LOG_ERROR,
-                                        "failed to set remote-port: %d",
-                                        server->port);
-                                errno = ENOTCONN;
-                                priv_glfs_init_done (fs, -1);
-                                break;
-                        }
-
-                        ret = dict_set_str (rpc_trans->options,
-                                            "remote-host",
-                                            server->volfile_server);
-                        if (ret != 0) {
-                                gf_log ("glfs-mgmt", GF_LOG_ERROR,
-                                        "failed to set remote-host: %s",
-                                        server->volfile_server);
-                                errno = ENOTCONN;
-                                priv_glfs_init_done (fs, -1);
-                                break;
-                        }
-
-                        ret = dict_set_str (rpc_trans->options,
-                                            "transport-type",
-                                            server->transport);
-                        if (ret != 0) {
-                                gf_log ("glfs-mgmt", GF_LOG_ERROR,
-                                        "failed to set transport-type: %s",
-                                        server->transport);
-                                errno = ENOTCONN;
-                                priv_glfs_init_done (fs, -1);
-                                break;
-                        }
-                        gf_log ("glfs-mgmt", GF_LOG_INFO,
-                                "connecting to next volfile server %s"
-                                " at port %d with transport: %s",
-                                server->volfile_server, server->port,
-                                server->transport);
+    xlator_t *this = NULL;
+    glusterfs_ctx_t *ctx = NULL;
+    server_cmdline_t *server = NULL;
+    rpc_transport_t *rpc_trans = NULL;
+    struct glfs *fs = NULL;
+    int ret = 0;
+    struct dnscache6 *dnscache = NULL;
+
+    this = mydata;
+    rpc_trans = rpc->conn.trans;
+
+    ctx = this->ctx;
+    if (!ctx)
+        goto out;
+
+    fs = ((xlator_t *)ctx->master)->private;
+
+    switch (event) {
+        case RPC_CLNT_DISCONNECT:
+            if (!ctx->active) {
+                if (rpc_trans->connect_failed)
+                    gf_smsg("glfs-mgmt", GF_LOG_ERROR, 0,
+                            API_MSG_REMOTE_HOST_CONN_FAILED, "server=%s",
+                            ctx->cmd_args.volfile_server, NULL);
+                else
+                    gf_smsg("glfs-mgmt", GF_LOG_INFO, 0,
+                            API_MSG_REMOTE_HOST_CONN_FAILED, "server=%s",
+                            ctx->cmd_args.volfile_server, NULL);
+
+                if (!rpc->disabled) {
+                    /*
+                     * Check if dnscache is exhausted for current server
+                     * and continue until cache is exhausted
+                     */
+                    dnscache = rpc_trans->dnscache;
+                    if (dnscache && dnscache->next) {
+                        break;
+                    }
+                }
+                server = ctx->cmd_args.curr_server;
+                if (server->list.next == &ctx->cmd_args.volfile_servers) {
+                    errno = ENOTCONN;
+                    gf_smsg("glfs-mgmt", GF_LOG_INFO, ENOTCONN,
+                            API_MSG_VOLFILE_SERVER_EXHAUST, NULL);
+                    glfs_init_done(fs, -1);
+                    break;
                 }
-                break;
-	case RPC_CLNT_CONNECT:
-		rpc_clnt_set_connected (&((struct rpc_clnt*)ctx->mgmt)->conn);
-
-		ret = glfs_volfile_fetch (fs);
-		if (ret && (ctx->active == NULL)) {
-			/* Do it only for the first time */
-			/* Exit the process.. there are some wrong options */
-			gf_log ("glfs-mgmt", GF_LOG_ERROR,
-				"failed to fetch volume file (key:%s)",
-                                ctx->cmd_args.volfile_id);
-                        errno = EINVAL;
-                        priv_glfs_init_done (fs, -1);
+                server = list_entry(server->list.next, typeof(*server), list);
+                ctx->cmd_args.curr_server = server;
+                ctx->cmd_args.volfile_server_port = server->port;
+                ctx->cmd_args.volfile_server = server->volfile_server;
+                ctx->cmd_args.volfile_server_transport = server->transport;
+
+                ret = dict_set_str(rpc_trans->options, "transport-type",
+                                   server->transport);
+                if (ret != 0) {
+                    gf_smsg("glfs-mgmt", GF_LOG_ERROR, ENOTCONN,
+                            API_MSG_DICT_SET_FAILED, "transport-type=%s",
+                            server->transport, NULL);
+                    errno = ENOTCONN;
+                    glfs_init_done(fs, -1);
+                    break;
                 }
 
-                break;
-	default:
-		break;
-	}
+                if (strcmp(server->transport, "unix") == 0) {
+                    ret = dict_set_str(rpc_trans->options,
+                                       "transport.socket.connect-path",
+                                       server->volfile_server);
+                    if (ret != 0) {
+                        gf_smsg("glfs-mgmt", GF_LOG_ERROR, ENOTCONN,
+                                API_MSG_DICT_SET_FAILED,
+                                "socket.connect-path=%s",
+                                server->volfile_server, NULL);
+                        errno = ENOTCONN;
+                        glfs_init_done(fs, -1);
+                        break;
+                    }
+                    /* delete the remote-host and remote-port keys
+                     * in case they were set while looping through
+                     * list of volfile servers previously
+                     */
+                    dict_del(rpc_trans->options, "remote-host");
+                    dict_del(rpc_trans->options, "remote-port");
+                } else {
+                    ret = dict_set_int32(rpc_trans->options, "remote-port",
+                                         server->port);
+                    if (ret != 0) {
+                        gf_smsg("glfs-mgmt", GF_LOG_ERROR, ENOTCONN,
+                                API_MSG_DICT_SET_FAILED, "remote-port=%d",
+                                server->port, NULL);
+                        errno = ENOTCONN;
+                        glfs_init_done(fs, -1);
+                        break;
+                    }
+
+                    ret = dict_set_str(rpc_trans->options, "remote-host",
+                                       server->volfile_server);
+                    if (ret != 0) {
+                        gf_smsg("glfs-mgmt", GF_LOG_ERROR, ENOTCONN,
+                                API_MSG_DICT_SET_FAILED, "remote-host=%s",
+                                server->volfile_server, NULL);
+                        errno = ENOTCONN;
+                        glfs_init_done(fs, -1);
+                        break;
+                    }
+                    /* delete the "transport.socket.connect-path"
+                     * key in case if it was set while looping
+                     * through list of volfile servers previously
+                     */
+                    dict_del(rpc_trans->options,
+                             "transport.socket.connect-path");
+                }
+
+                gf_smsg("glfs-mgmt", GF_LOG_INFO, 0, API_MSG_VOLFILE_CONNECTING,
+                        "server=%s", server->volfile_server, "port=%d",
+                        server->port, "transport=%s", server->transport, NULL);
+            }
+            break;
+        case RPC_CLNT_CONNECT:
+            ret = glfs_volfile_fetch(fs);
+            if (ret && (ctx->active == NULL)) {
+                /* Do it only for the first time */
+                /* Exit the process.. there are some wrong options */
+                gf_smsg("glfs-mgmt", GF_LOG_ERROR, EINVAL,
+                        API_MSG_GET_VOLFILE_FAILED, "key=%s",
+                        ctx->cmd_args.volfile_id, NULL);
+                errno = EINVAL;
+                glfs_init_done(fs, -1);
+            }
+
+            break;
+        default:
+            break;
+    }
 out:
-	return 0;
+    return 0;
 }
 
-
 int
-glusterfs_mgmt_notify (int32_t op, void *data, ...)
+glusterfs_mgmt_notify(int32_t op, void *data, ...)
 {
-	int ret = 0;
+    int ret = 0;
 
-	switch (op)
-	{
-		case GF_EN_DEFRAG_STATUS:
-			break;
+    switch (op) {
+        case GF_EN_DEFRAG_STATUS:
+            break;
 
-		default:
-			break;
-	}
+        default:
+            break;
+    }
 
-	return ret;
+    return ret;
 }
 
-
 int
-glfs_mgmt_init (struct glfs *fs)
+glfs_mgmt_init(struct glfs *fs)
 {
-	cmd_args_t		*cmd_args = NULL;
-	struct rpc_clnt		*rpc = NULL;
-	dict_t			*options = NULL;
-	int			ret = -1;
-	int			port = GF_DEFAULT_BASE_PORT;
-	char			*host = NULL;
-	glusterfs_ctx_t		*ctx = NULL;
-
-	ctx = fs->ctx;
-	cmd_args = &ctx->cmd_args;
-
-	if (ctx->mgmt)
-		return 0;
-
-	if (cmd_args->volfile_server_port)
-		port = cmd_args->volfile_server_port;
-
-	host = "localhost";
-	if (cmd_args->volfile_server)
-		host = cmd_args->volfile_server;
-
-	ret = rpc_transport_inet_options_build (&options, host, port);
-	if (ret)
-		goto out;
-
-	rpc = rpc_clnt_new (options, THIS->ctx, THIS->name, 8);
-	if (!rpc) {
-		ret = -1;
-		gf_log (THIS->name, GF_LOG_WARNING,
-			"failed to create rpc clnt");
-		goto out;
-	}
-
-	ret = rpc_clnt_register_notify (rpc, mgmt_rpc_notify, THIS);
-	if (ret) {
-		gf_log (THIS->name, GF_LOG_WARNING,
-			"failed to register notify function");
-		goto out;
-	}
-
-	ret = rpcclnt_cbk_program_register (rpc, &mgmt_cbk_prog, THIS);
-	if (ret) {
-		gf_log (THIS->name, GF_LOG_WARNING,
-			"failed to register callback function");
-		goto out;
-	}
-
-	ctx->notify = glusterfs_mgmt_notify;
-
-	/* This value should be set before doing the 'rpc_clnt_start()' as
-	   the notify function uses this variable */
-	ctx->mgmt = rpc;
-
-	ret = rpc_clnt_start (rpc);
+    cmd_args_t *cmd_args = NULL;
+    struct rpc_clnt *rpc = NULL;
+    dict_t *options = NULL;
+    int ret = -1;
+    int port = GF_DEFAULT_BASE_PORT;
+    char *host = NULL;
+    glusterfs_ctx_t *ctx = NULL;
+
+    ctx = fs->ctx;
+    cmd_args = &ctx->cmd_args;
+
+    if (ctx->mgmt)
+        return 0;
+
+    options = dict_new();
+    if (!options)
+        goto out;
+
+    if (cmd_args->volfile_server_port)
+        port = cmd_args->volfile_server_port;
+
+    if (cmd_args->volfile_server) {
+        host = cmd_args->volfile_server;
+    } else if (cmd_args->volfile_server_transport &&
+               !strcmp(cmd_args->volfile_server_transport, "unix")) {
+        host = DEFAULT_GLUSTERD_SOCKFILE;
+    } else {
+        host = "localhost";
+    }
+
+    if (cmd_args->volfile_server_transport &&
+        !strcmp(cmd_args->volfile_server_transport, "unix")) {
+        ret = rpc_transport_unix_options_build(options, host, 0);
+    } else {
+        xlator_cmdline_option_t *opt = find_xlator_option_in_cmd_args_t(
+            "address-family", cmd_args);
+        ret = rpc_transport_inet_options_build(options, host, port,
+                                               (opt ? opt->value : NULL));
+    }
+
+    if (ret)
+        goto out;
+
+    rpc = rpc_clnt_new(options, THIS, THIS->name, 8);
+    if (!rpc) {
+        ret = -1;
+        gf_smsg(THIS->name, GF_LOG_WARNING, 0, API_MSG_CREATE_RPC_CLIENT_FAILED,
+                NULL);
+        goto out;
+    }
+
+    ret = rpc_clnt_register_notify(rpc, mgmt_rpc_notify, THIS);
+    if (ret) {
+        gf_smsg(THIS->name, GF_LOG_WARNING, 0, API_MSG_REG_NOTIFY_FUNC_FAILED,
+                NULL);
+        goto out;
+    }
+
+    ret = rpcclnt_cbk_program_register(rpc, &mgmt_cbk_prog, THIS);
+    if (ret) {
+        gf_smsg(THIS->name, GF_LOG_WARNING, 0, API_MSG_REG_CBK_FUNC_FAILED,
+                NULL);
+        goto out;
+    }
+
+    ctx->notify = glusterfs_mgmt_notify;
+
+    /* This value should be set before doing the 'rpc_clnt_start()' as
+       the notify function uses this variable */
+    ctx->mgmt = rpc;
+
+    ret = rpc_clnt_start(rpc);
 out:
-	return ret;
+    if (options)
+        dict_unref(options);
+    return ret;
 }
diff --git a/api/src/glfs-resolve.c b/api/src/glfs-resolve.c
index e3f14f9a364..8a393ecb464 100644
--- a/api/src/glfs-resolve.c
+++ b/api/src/glfs-resolve.c
@@ -1,5 +1,5 @@
 /*
-  Copyright (c) 2012 Red Hat, Inc. <http://www.redhat.com>
+  Copyright (c) 2012-2018 Red Hat, Inc. <http://www.redhat.com>
   This file is part of GlusterFS.
 
   This file is licensed to you under your choice of the GNU Lesser
@@ -8,7 +8,6 @@
   cases as published by the Free Software Foundation.
 */
 
-
 #include <unistd.h>
 #include <string.h>
 #include <stdlib.h>
@@ -16,1005 +15,1185 @@
 #include <inttypes.h>
 #include <limits.h>
 
-#ifndef _CONFIG_H
-#define _CONFIG_H
-#include "config.h"
-#endif
-
-#include "glusterfs.h"
-#include "logging.h"
-#include "stack.h"
-#include "event.h"
+#include <glusterfs/glusterfs.h>
+#include <glusterfs/logging.h>
+#include <glusterfs/stack.h>
+#include <glusterfs/gf-event.h>
 #include "glfs-mem-types.h"
-#include "common-utils.h"
-#include "syncop.h"
-#include "call-stub.h"
-
+#include <glusterfs/common-utils.h>
+#include <glusterfs/syncop.h>
+#include <glusterfs/call-stub.h>
+#include "gfapi-messages.h"
+#include <glusterfs/inode.h>
 #include "glfs-internal.h"
 
-#define graphid_str(subvol) (uuid_utoa((unsigned char *)subvol->graph->graph_uuid))
-
-
+#define graphid_str(subvol)                                                    \
+    (uuid_utoa((unsigned char *)subvol->graph->graph_uuid))
 int
-glfs_first_lookup_safe (xlator_t *subvol)
+glfs_first_lookup_safe(xlator_t *subvol)
 {
-	loc_t  loc = {0, };
-	int    ret = -1;
+    loc_t loc = {
+        0,
+    };
+    int ret = -1;
 
-	loc.inode = subvol->itable->root;
-	memset (loc.gfid, 0, 16);
-	loc.gfid[15] = 1;
-	loc.path = "/";
-	loc.name = "";
+    loc.inode = subvol->itable->root;
+    memset(loc.gfid, 0, 16);
+    loc.gfid[15] = 1;
+    loc.path = "/";
+    loc.name = "";
 
-	ret = syncop_lookup (subvol, &loc, 0, 0, 0, 0);
-        DECODE_SYNCOP_ERR (ret);
+    ret = syncop_lookup(subvol, &loc, 0, 0, 0, 0);
+    DECODE_SYNCOP_ERR(ret);
 
-	gf_log (subvol->name, GF_LOG_DEBUG, "first lookup complete %d", ret);
+    gf_msg_debug(subvol->name, 0, "first lookup complete %d", ret);
 
-	return ret;
+    return ret;
 }
 
-
 int
-__glfs_first_lookup (struct glfs *fs, xlator_t *subvol)
-{
-	int ret = -1;
-
-	fs->migration_in_progress = 1;
-	pthread_mutex_unlock (&fs->mutex);
-	{
-		ret = glfs_first_lookup_safe (subvol);
-	}
-	pthread_mutex_lock (&fs->mutex);
-	fs->migration_in_progress = 0;
-	pthread_cond_broadcast (&fs->cond);
-
-	return ret;
-}
-
-
-inode_t *
-glfs_refresh_inode_safe (xlator_t *subvol, inode_t *oldinode)
+__glfs_first_lookup(struct glfs *fs, xlator_t *subvol)
 {
-	loc_t        loc = {0, };
-	int          ret = -1;
-	struct iatt  iatt = {0, };
-	inode_t     *newinode = NULL;
+    int ret = -1;
 
+    fs->migration_in_progress = 1;
+    pthread_mutex_unlock(&fs->mutex);
+    {
+        ret = glfs_first_lookup_safe(subvol);
+    }
+    pthread_mutex_lock(&fs->mutex);
+    fs->migration_in_progress = 0;
+    pthread_cond_broadcast(&fs->cond);
 
-	if (!oldinode)
-		return NULL;
+    /* wake up other waiting tasks */
+    __GLFS_SYNCTASK_WAKE(fs);
 
-	if (oldinode->table->xl == subvol)
-		return inode_ref (oldinode);
-
-	newinode = inode_find (subvol->itable, oldinode->gfid);
-	if (newinode)
-		return newinode;
-
-	uuid_copy (loc.gfid, oldinode->gfid);
-	loc.inode = inode_new (subvol->itable);
-	if (!loc.inode)
-		return NULL;
+    return ret;
+}
 
-	ret = syncop_lookup (subvol, &loc, 0, &iatt, 0, 0);
-        DECODE_SYNCOP_ERR (ret);
+/**
+ * We have to check if need_lookup flag is set in both old and the new inodes.
+ * If its set in oldinode, then directly go ahead and do an explicit lookup.
+ * But if its not set in the oldinode, then check if the newinode is linked
+ * via readdirp. If so an explicit lookup is needed on the new inode, so that
+ * below xlators can set their respective contexts.
+ */
+inode_t *
+glfs_refresh_inode_safe(xlator_t *subvol, inode_t *oldinode,
+                        gf_boolean_t need_lookup)
+{
+    loc_t loc = {
+        0,
+    };
+    int ret = -1;
+    struct iatt iatt = {
+        0,
+    };
+    inode_t *newinode = NULL;
+    gf_boolean_t lookup_needed = _gf_false;
+    uint64_t ctx_value = LOOKUP_NOT_NEEDED;
+
+    if (!oldinode)
+        return NULL;
+
+    if (!need_lookup && oldinode->table->xl == subvol)
+        return inode_ref(oldinode);
+
+    newinode = inode_find(subvol->itable, oldinode->gfid);
+    if (!need_lookup && newinode) {
+        lookup_needed = inode_needs_lookup(newinode, THIS);
+        if (!lookup_needed)
+            return newinode;
+    }
+
+    gf_uuid_copy(loc.gfid, oldinode->gfid);
+    if (!newinode)
+        loc.inode = inode_new(subvol->itable);
+    else
+        loc.inode = newinode;
+
+    if (!loc.inode)
+        return NULL;
+
+    ret = syncop_lookup(subvol, &loc, &iatt, 0, 0, 0);
+    DECODE_SYNCOP_ERR(ret);
+
+    if (ret) {
+        gf_smsg(subvol->name, GF_LOG_WARNING, errno,
+                API_MSG_INODE_REFRESH_FAILED, "gfid=%s",
+                uuid_utoa(oldinode->gfid), "err=%s", strerror(errno), NULL);
+        loc_wipe(&loc);
+        return NULL;
+    }
+
+    newinode = inode_link(loc.inode, 0, 0, &iatt);
+    if (newinode) {
+        if (newinode == loc.inode)
+            inode_ctx_set(newinode, THIS, &ctx_value);
+        inode_lookup(newinode);
+    } else {
+        gf_smsg(subvol->name, GF_LOG_WARNING, errno, API_MSG_INODE_LINK_FAILED,
+                "gfid=%s", uuid_utoa((unsigned char *)&iatt.ia_gfid), NULL);
+    }
+
+    loc_wipe(&loc);
+
+    return newinode;
+}
 
-	if (ret) {
-		gf_log (subvol->name, GF_LOG_WARNING,
-			"inode refresh of %s failed: %s",
-			uuid_utoa (oldinode->gfid), strerror (errno));
-		loc_wipe (&loc);
-		return NULL;
-	}
+inode_t *
+__glfs_refresh_inode(struct glfs *fs, xlator_t *subvol, inode_t *inode,
+                     gf_boolean_t need_lookup)
+{
+    inode_t *newinode = NULL;
 
-	newinode = inode_link (loc.inode, 0, 0, &iatt);
-	if (newinode)
-		inode_lookup (newinode);
+    fs->migration_in_progress = 1;
+    pthread_mutex_unlock(&fs->mutex);
+    {
+        newinode = glfs_refresh_inode_safe(subvol, inode, need_lookup);
+    }
+    pthread_mutex_lock(&fs->mutex);
+    fs->migration_in_progress = 0;
+    pthread_cond_broadcast(&fs->cond);
 
-	loc_wipe (&loc);
+    /* wake up other waiting tasks */
+    __GLFS_SYNCTASK_WAKE(fs);
 
-	return newinode;
+    return newinode;
 }
 
-
-inode_t *
-__glfs_refresh_inode (struct glfs *fs, xlator_t *subvol, inode_t *inode)
+GFAPI_SYMVER_PRIVATE_DEFAULT(glfs_loc_touchup, 3.4.0)
+int
+priv_glfs_loc_touchup(loc_t *loc)
 {
-	inode_t *newinode = NULL;
-
-	fs->migration_in_progress = 1;
-	pthread_mutex_unlock (&fs->mutex);
-	{
-		newinode = glfs_refresh_inode_safe (subvol, inode);
-	}
-	pthread_mutex_lock (&fs->mutex);
-	fs->migration_in_progress = 0;
-	pthread_cond_broadcast (&fs->cond);
-
-	return newinode;
+    int ret = 0;
+
+    ret = loc_touchup(loc, loc->name);
+    if (ret < 0) {
+        errno = -ret;
+        ret = -1;
+    }
+
+    return ret;
 }
 
 int
-priv_glfs_loc_touchup (loc_t *loc)
+glfs_resolve_symlink(struct glfs *fs, xlator_t *subvol, inode_t *inode,
+                     char **lpath)
 {
-	char *path = NULL;
-	int   ret = -1;
-	char *bn = NULL;
-
-	if (loc->parent)
-		ret = inode_path (loc->parent, loc->name, &path);
-	else
-		ret = inode_path (loc->inode, 0, &path);
-
-	loc->path = path;
-
-	if (ret < 0 || !path) {
-		ret = -1;
-		errno = ENOMEM;
-		goto out;
-	}
-
-	bn = strrchr (path, '/');
-	if (bn)
-		bn++;
-	loc->name = bn;
-	ret = 0;
+    loc_t loc = {
+        0,
+    };
+    char *path = NULL;
+    char *rpath = NULL;
+    int ret = -1;
+
+    loc.inode = inode_ref(inode);
+    gf_uuid_copy(loc.gfid, inode->gfid);
+    ret = inode_path(inode, NULL, &rpath);
+    if (ret < 0)
+        goto out;
+    loc.path = rpath;
+
+    ret = syncop_readlink(subvol, &loc, &path, 4096, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+
+    if (ret < 0)
+        goto out;
+
+    if (lpath)
+        *lpath = path;
 out:
-	return ret;
+    loc_wipe(&loc);
+    return ret;
 }
 
-GFAPI_SYMVER_PRIVATE_DEFAULT(glfs_loc_touchup, 3.4.0);
-
 int
-glfs_resolve_symlink (struct glfs *fs, xlator_t *subvol, inode_t *inode,
-		      char **lpath)
+glfs_resolve_base(struct glfs *fs, xlator_t *subvol, inode_t *inode,
+                  struct iatt *iatt)
 {
-	loc_t  loc = {0, };
-	char  *path = NULL;
-	char  *rpath = NULL;
-	int    ret = -1;
-
-	loc.inode = inode_ref (inode);
-	uuid_copy (loc.gfid, inode->gfid);
-	ret = inode_path (inode, NULL, &rpath);
-	if (ret < 0)
-		goto out;
-	loc.path = rpath;
-
-	ret = syncop_readlink (subvol, &loc, &path, 4096);
-        DECODE_SYNCOP_ERR (ret);
-
-	if (ret < 0)
-		goto out;
-
-	if (lpath)
-		*lpath = path;
+    loc_t loc = {
+        0,
+    };
+    int ret = -1;
+    char *path = NULL;
+
+    loc.inode = inode_ref(inode);
+    gf_uuid_copy(loc.gfid, inode->gfid);
+
+    ret = inode_path(loc.inode, NULL, &path);
+    loc.path = path;
+    if (ret < 0)
+        goto out;
+
+    ret = syncop_lookup(subvol, &loc, iatt, NULL, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
 out:
-	loc_wipe (&loc);
-	return ret;
+    loc_wipe(&loc);
+
+    return ret;
 }
+/*
+ * This function can be used to call named lookup on root.
+ * If you use glfs_resolve_base, that will be a nameless lookup.
+ */
+static int
+glfs_resolve_root(struct glfs *fs, xlator_t *subvol, inode_t *inode,
+                  struct iatt *iatt)
+{
+    loc_t loc = {
+        0,
+    };
+    int ret = -1;
+    char *path = NULL;
+
+    loc.inode = inode_ref(inode);
+
+    ret = inode_path(loc.inode, ".", &path);
+    loc.path = path;
+    loc.name = ".";
+    /* Having a value in loc.name will help to bypass md-cache check for
+     * nameless lookup.
+     * TODO: Re-visit on nameless lookup and md-cache.
+     * Github issue : https://github.com/gluster/glusterfs/issues/232
+     */
+    loc.parent = inode_ref(inode);
+    if (ret < 0)
+        goto out;
+
+    ret = syncop_lookup(subvol, &loc, iatt, NULL, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+out:
+    loc_wipe(&loc);
 
+    return ret;
+}
 
-int
-glfs_resolve_base (struct glfs *fs, xlator_t *subvol, inode_t *inode,
-		   struct iatt *iatt)
+inode_t *
+glfs_resolve_component(struct glfs *fs, xlator_t *subvol, inode_t *parent,
+                       const char *component, struct iatt *iatt,
+                       int force_lookup)
 {
-	loc_t       loc = {0, };
-	int         ret = -1;
-	char       *path = NULL;
+    loc_t loc = {
+        0,
+    };
+    inode_t *inode = NULL;
+    inode_t *temp_parent = NULL;
+    int reval = 0;
+    int ret = -1;
+    int glret = -1;
+    struct iatt ciatt = {
+        0,
+    };
+    uuid_t gfid;
+    dict_t *xattr_req = NULL;
+    uint64_t ctx_value = LOOKUP_NOT_NEEDED;
+
+    loc.parent = inode_ref(parent);
+    gf_uuid_copy(loc.pargfid, parent->gfid);
+
+    if (__is_root_gfid(parent->gfid) &&
+        ((strcmp(component, ".") == 0) || (strcmp(component, "..") == 0) ||
+         (strcmp(component, "") == 0))) {
+        if (!force_lookup) {
+            inode = inode_ref(parent);
+        } else {
+            ret = glfs_resolve_root(fs, subvol, parent, &ciatt);
+            if (!ret)
+                inode = inode_ref(parent);
+        }
+        goto found;
+    }
+    /* *
+     * if the component name is either "." or "..", it will try to
+     * resolve that if inode has a proper parent (named lookup).
+     *
+     * Below condition works like this
+     *
+     * Example 1 :
+     * Path /out_dir/dir/in_dir/.
+     *     In put values :
+     *         parent = in_dir
+     *         component : "."
+     *
+     *      Out put values:
+     *         parent : dir
+     *         component : "in_dir"
+     *
+     * Example 2 :
+     * Path /out_dir/dir/in_dir/..
+     *     In put values :
+     *         parent = in_dir
+     *         component : ".."
+     *
+     *     Out put values:
+     *         parent : output_dir
+     *         component : "dir"
+     *
+     * In case of nameless lookup, both "." and ".." retained
+     */
+
+    if (strcmp(component, ".") == 0) {
+        loc.inode = inode_ref(parent);
+        temp_parent = inode_parent(loc.inode, 0, 0);
+        if (temp_parent) {
+            inode_unref(loc.parent);
+            loc.parent = temp_parent;
+            gf_uuid_copy(loc.pargfid, temp_parent->gfid);
+            inode_find_directory_name(loc.inode, &loc.name);
+        }
 
-	loc.inode = inode_ref (inode);
-	uuid_copy (loc.gfid, inode->gfid);
+    } else if (strcmp(component, "..") == 0) {
+        loc.inode = inode_parent(parent, 0, 0);
+        if (loc.inode) {
+            temp_parent = inode_parent(loc.inode, 0, 0);
+            if (temp_parent) {
+                inode_unref(loc.parent);
+                loc.parent = temp_parent;
+                gf_uuid_copy(loc.pargfid, temp_parent->gfid);
+                inode_find_directory_name(loc.inode, &loc.name);
+            } else if (__is_root_gfid(loc.inode->gfid)) {
+                inode_unref(loc.parent);
+                loc.parent = inode_ref(loc.inode);
+                gf_uuid_copy(loc.pargfid, loc.inode->gfid);
+                loc.name = ".";
+            } else {
+                inode_unref(loc.inode);
+                loc.inode = NULL;
+            }
+        }
+    } else
+        loc.inode = inode_grep(parent->table, parent, component);
 
-	ret = inode_path (loc.inode, NULL, &path);
-	loc.path = path;
-	if (ret < 0)
-		goto out;
+    if (!loc.name)
+        loc.name = component;
 
-	ret = syncop_lookup (subvol, &loc, NULL, iatt, NULL, NULL);
-        DECODE_SYNCOP_ERR (ret);
-out:
-	loc_wipe (&loc);
+    if (loc.inode) {
+        gf_uuid_copy(loc.gfid, loc.inode->gfid);
+        reval = 1;
 
-	return ret;
-}
+        if (!(force_lookup || inode_needs_lookup(loc.inode, THIS))) {
+            inode = inode_ref(loc.inode);
+            goto found;
+        }
+    } else {
+        gf_uuid_generate(gfid);
+        loc.inode = inode_new(parent->table);
+        if (!loc.inode) {
+            errno = ENOMEM;
+            goto out;
+        }
 
+        xattr_req = dict_new();
+        if (!xattr_req) {
+            errno = ENOMEM;
+            goto out;
+        }
 
-inode_t *
-glfs_resolve_component (struct glfs *fs, xlator_t *subvol, inode_t *parent,
-			const char *component, struct iatt *iatt,
-			int force_lookup)
-{
-	loc_t        loc = {0, };
-	inode_t     *inode = NULL;
-	int          reval = 0;
-	int          ret = -1;
-	int          glret = -1;
-	struct iatt  ciatt = {0, };
-	uuid_t       gfid;
-	dict_t      *xattr_req = NULL;
-
-	loc.name = component;
-
-	loc.parent = inode_ref (parent);
-	uuid_copy (loc.pargfid, parent->gfid);
-
-        /* /.. and /. should point back to /
-           we lookup using inode and gfid of root
-           Fill loc.name so that we make use md-cache.
-           md-cache is not valid for nameless lookups.
-        */
-        if (__is_root_gfid (parent->gfid)) {
-                if ((strcmp (component, ".") == 0) ||
-                    (strcmp (component, "..") == 0)) {
-                        loc.inode = inode_ref (parent);
-                        loc.name = ".";
-                }
-        } else {
-                if (strcmp (component, ".") == 0)
-                        loc.inode = inode_ref (parent);
-                else if (strcmp (component, "..") == 0)
-                        loc.inode = inode_parent (parent, 0, 0);
-                else
-                        loc.inode = inode_grep (parent->table, parent,
-                                                component);
+        ret = dict_set_gfuuid(xattr_req, "gfid-req", gfid, true);
+        if (ret) {
+            errno = ENOMEM;
+            goto out;
+        }
+    }
+
+    glret = priv_glfs_loc_touchup(&loc);
+    if (glret < 0) {
+        ret = -1;
+        goto out;
+    }
+
+    ret = syncop_lookup(subvol, &loc, &ciatt, NULL, xattr_req, NULL);
+    if (ret && reval) {
+        /*
+         * A stale mapping might exist for a dentry/inode that has been
+         * removed from another client.
+         */
+        if (-ret == ENOENT) {
+            inode_unlink(loc.inode, loc.parent, loc.name);
+            if (!inode_has_dentry(loc.inode))
+                inode_forget(loc.inode, 0);
         }
 
+        inode_unref(loc.inode);
+        gf_uuid_clear(loc.gfid);
+        loc.inode = inode_new(parent->table);
+        if (!loc.inode) {
+            errno = ENOMEM;
+            goto out;
+        }
 
-	if (loc.inode) {
-		uuid_copy (loc.gfid, loc.inode->gfid);
-		reval = 1;
-
-		if (!force_lookup) {
-			inode = inode_ref (loc.inode);
-			ciatt.ia_type = inode->ia_type;
-			goto found;
-		}
-	} else {
-		uuid_generate (gfid);
-		loc.inode = inode_new (parent->table);
-                if (!loc.inode) {
-                        errno = ENOMEM;
-                        goto out;
-                }
+        xattr_req = dict_new();
+        if (!xattr_req) {
+            errno = ENOMEM;
+            goto out;
+        }
 
-                xattr_req = dict_new ();
-                if (!xattr_req) {
-                        errno = ENOMEM;
-                        goto out;
-                }
+        gf_uuid_generate(gfid);
 
-                ret = dict_set_static_bin (xattr_req, "gfid-req", gfid, 16);
-                if (ret) {
-                        errno = ENOMEM;
-                        goto out;
-                }
+        ret = dict_set_gfuuid(xattr_req, "gfid-req", gfid, true);
+        if (ret) {
+            errno = ENOMEM;
+            goto out;
+        }
 
-	}
-
-	glret = priv_glfs_loc_touchup (&loc);
-	if (glret < 0) {
-		ret = -1;
-		goto out;
-	}
-
-        ret = syncop_lookup (subvol, &loc, xattr_req, &ciatt, NULL, NULL);
-        if (ret && reval) {
-                /*
-                 * A stale mapping might exist for a dentry/inode that has been
-                 * removed from another client.
-                 */
-                if (-ret == ENOENT)
-                        inode_unlink(loc.inode, loc.parent,
-                                     loc.name);
-		inode_unref (loc.inode);
-		loc.inode = inode_new (parent->table);
-		if (!loc.inode) {
-			errno = ENOMEM;
-			goto out;
-		}
-
-		xattr_req = dict_new ();
-		if (!xattr_req) {
-			errno = ENOMEM;
-			goto out;
-		}
-
-		uuid_generate (gfid);
-
-		ret = dict_set_static_bin (xattr_req, "gfid-req", gfid, 16);
-		if (ret) {
-			errno = ENOMEM;
-			goto out;
-		}
-
-		ret = syncop_lookup (subvol, &loc, xattr_req, &ciatt,
-				     NULL, NULL);
-	}
-        DECODE_SYNCOP_ERR (ret);
-	if (ret)
-		goto out;
-
-	inode = inode_link (loc.inode, loc.parent, component, &ciatt);
+        ret = syncop_lookup(subvol, &loc, &ciatt, NULL, xattr_req, NULL);
+    }
+    DECODE_SYNCOP_ERR(ret);
+    if (ret)
+        goto out;
+
+    inode = inode_link(loc.inode, loc.parent, component, &ciatt);
+
+    if (!inode) {
+        gf_smsg(subvol->name, GF_LOG_WARNING, errno, API_MSG_INODE_LINK_FAILED,
+                "gfid=%s", uuid_utoa((unsigned char *)&ciatt.ia_gfid), NULL);
+        goto out;
+    } else if (inode == loc.inode)
+        inode_ctx_set(inode, THIS, &ctx_value);
 found:
-	if (inode)
-		inode_lookup (inode);
-	if (iatt)
-		*iatt = ciatt;
+    if (inode) {
+        ciatt.ia_type = inode->ia_type;
+        inode_lookup(inode);
+    }
+    if (iatt)
+        *iatt = ciatt;
 out:
-	if (xattr_req)
-		dict_unref (xattr_req);
+    if (xattr_req)
+        dict_unref(xattr_req);
+    loc_wipe(&loc);
 
-	loc_wipe (&loc);
-
-	return inode;
+    return inode;
 }
 
-
+GFAPI_SYMVER_PRIVATE_DEFAULT(glfs_resolve_at, 3.4.0)
 int
-priv_glfs_resolve_at (struct glfs *fs, xlator_t *subvol, inode_t *at,
-		 const char *origpath, loc_t *loc, struct iatt *iatt,
-		 int follow, int reval)
+priv_glfs_resolve_at(struct glfs *fs, xlator_t *subvol, inode_t *at,
+                     const char *origpath, loc_t *loc, struct iatt *iatt,
+                     int follow, int reval)
 {
-	inode_t    *inode = NULL;
-	inode_t    *parent = NULL;
-	char       *saveptr = NULL;
-	char       *path = NULL;
-	char       *component = NULL;
-	char       *next_component = NULL;
-	int         ret = -1;
-	struct iatt ciatt = {0, };
-
-	path = gf_strdup (origpath);
-	if (!path) {
-		errno = ENOMEM;
-		return -1;
-	}
-
-	parent = NULL;
-	if (at && path[0] != '/') {
-		/* A relative resolution of a path which starts with '/'
-		   is equal to an absolute path resolution.
-		*/
-		inode = inode_ref (at);
-	} else {
-		inode = inode_ref (subvol->itable->root);
-
-		if (strcmp (path, "/") == 0)
-			glfs_resolve_base (fs, subvol, inode, &ciatt);
-	}
-
-	for (component = strtok_r (path, "/", &saveptr);
-	     component; component = next_component) {
-
-		next_component = strtok_r (NULL, "/", &saveptr);
-
-		if (parent)
-			inode_unref (parent);
-
-		parent = inode;
-
-		inode = glfs_resolve_component (fs, subvol, parent,
-						component, &ciatt,
-						/* force hard lookup on the last
-						   component, as the caller
-						   wants proper iatt filled
-						*/
-						(reval || (!next_component &&
-						iatt)));
-		if (!inode)
-			break;
-
-		if (IA_ISLNK (ciatt.ia_type) && (next_component || follow)) {
-			/* If the component is not the last piece,
-			   then following it is necessary even if
-			   not requested by the caller
-			*/
-			char *lpath = NULL;
-			loc_t sym_loc = {0,};
-
-			if (follow > GLFS_SYMLINK_MAX_FOLLOW) {
-				errno = ELOOP;
-				ret = -1;
-				if (inode) {
-					inode_unref (inode);
-					inode = NULL;
-				}
-				break;
-			}
-
-			ret = glfs_resolve_symlink (fs, subvol, inode, &lpath);
-			inode_unref (inode);
-			inode = NULL;
-			if (ret < 0)
-				break;
-
-			ret = priv_glfs_resolve_at (fs, subvol, parent, lpath,
-					       &sym_loc,
-					       /* followed iatt becomes the
-						  component iatt
-					       */
-					       &ciatt,
-					       /* always recurisvely follow while
-						  following symlink
-					       */
-					       follow + 1, reval);
-			if (ret == 0)
-				inode = inode_ref (sym_loc.inode);
-			loc_wipe (&sym_loc);
-			GF_FREE (lpath);
-		}
-
-		if (!next_component)
-			break;
-
-		if (!IA_ISDIR (ciatt.ia_type)) {
-			/* next_component exists and this component is
-			   not a directory
-			*/
-			inode_unref (inode);
-			inode = NULL;
-			ret = -1;
-			errno = ENOTDIR;
-			break;
-		}
-	}
-
-	if (parent && next_component)
-		/* resolution failed mid-way */
-		goto out;
-
-	/* At this point, all components up to the last parent directory
-	   have been resolved successfully (@parent). Resolution of basename
-	   might have failed (@inode) if at all.
-	*/
-
-	loc->parent = parent;
-	if (parent) {
-		uuid_copy (loc->pargfid, parent->gfid);
-		loc->name = component;
-	}
-
-	loc->inode = inode;
-	if (inode) {
-		uuid_copy (loc->gfid, inode->gfid);
-		if (iatt)
-			*iatt = ciatt;
-		ret = 0;
-	}
-
-        if (priv_glfs_loc_touchup (loc) < 0) {
+    inode_t *inode = NULL;
+    inode_t *parent = NULL;
+    char *saveptr = NULL;
+    char *path = NULL;
+    char *component = NULL;
+    char *next_component = NULL;
+    int ret = -1;
+    struct iatt ciatt = {
+        0,
+    };
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    if (origpath[0] == '\0') {
+        errno = EINVAL;
+        goto invalid_fs;
+    }
+
+    parent = NULL;
+    if (at && origpath[0] != '/') {
+        /* A relative resolution of a path which starts with '/'
+           is equal to an absolute path resolution.
+        */
+        inode = inode_ref(at);
+    } else {
+        inode = inode_ref(subvol->itable->root);
+
+        if (strcmp(origpath, "/") == 0)
+            glfs_resolve_root(fs, subvol, inode, &ciatt);
+    }
+
+    path = gf_strdup(origpath);
+    if (!path)
+        goto invalid_fs;
+
+    for (component = strtok_r(path, "/", &saveptr); component;
+         component = next_component) {
+        next_component = strtok_r(NULL, "/", &saveptr);
+
+        if (parent)
+            inode_unref(parent);
+        parent = inode;
+        inode = glfs_resolve_component(fs, subvol, parent, component, &ciatt,
+                                       /* force hard lookup on the last
+                                          component, as the caller
+                                          wants proper iatt filled
+                                       */
+                                       (reval || (!next_component && iatt)));
+        if (!inode) {
+            ret = -1;
+            break;
+        }
+
+        if (IA_ISLNK(ciatt.ia_type) && (next_component || follow)) {
+            /* If the component is not the last piece,
+               then following it is necessary even if
+               not requested by the caller
+            */
+            char *lpath = NULL;
+            loc_t sym_loc = {
+                0,
+            };
+
+            if (follow > GLFS_SYMLINK_MAX_FOLLOW) {
+                errno = ELOOP;
                 ret = -1;
+                if (inode) {
+                    inode_unref(inode);
+                    inode = NULL;
+                }
+                break;
+            }
+
+            ret = glfs_resolve_symlink(fs, subvol, inode, &lpath);
+            inode_unref(inode);
+            inode = NULL;
+            if (ret < 0)
+                break;
+
+            ret = priv_glfs_resolve_at(fs, subvol, parent, lpath, &sym_loc,
+                                       /* followed iatt becomes the
+                                          component iatt
+                                       */
+                                       &ciatt,
+                                       /* always recurisvely follow while
+                                          following symlink
+                                       */
+                                       follow + 1, reval);
+            if (ret == 0)
+                inode = inode_ref(sym_loc.inode);
+            loc_wipe(&sym_loc);
+            GF_FREE(lpath);
         }
-out:
-	GF_FREE (path);
 
-	/* do NOT loc_wipe here as only last component might be missing */
+        if (!next_component)
+            break;
+
+        if (!IA_ISDIR(ciatt.ia_type)) {
+            /* next_component exists and this component is
+               not a directory
+            */
+            inode_unref(inode);
+            inode = NULL;
+            ret = -1;
+            errno = ENOTDIR;
+            break;
+        }
+    }
+
+    if (parent && next_component)
+        /* resolution failed mid-way */
+        goto out;
+
+    /* At this point, all components up to the last parent directory
+       have been resolved successfully (@parent). Resolution of basename
+       might have failed (@inode) if at all.
+    */
+
+    loc->parent = parent;
+    if (parent) {
+        gf_uuid_copy(loc->pargfid, parent->gfid);
+        loc->name = component;
+    }
+
+    loc->inode = inode;
+    if (inode) {
+        gf_uuid_copy(loc->gfid, inode->gfid);
+        if (iatt)
+            *iatt = ciatt;
+        ret = 0;
+    }
+
+    if (priv_glfs_loc_touchup(loc) < 0) {
+        ret = -1;
+    }
+out:
+    GF_FREE(path);
+    __GLFS_EXIT_FS;
 
-	return ret;
+    /* do NOT loc_wipe here as only last component might be missing */
+invalid_fs:
+    return ret;
 }
 
-GFAPI_SYMVER_PRIVATE_DEFAULT(glfs_resolve_at, 3.4.0);
-
-
 int
-glfs_resolve_path (struct glfs *fs, xlator_t *subvol, const char *origpath,
-		   loc_t *loc, struct iatt *iatt, int follow, int reval)
+glfs_resolve_path(struct glfs *fs, xlator_t *subvol, const char *origpath,
+                  loc_t *loc, struct iatt *iatt, int follow, int reval)
 {
-	int ret = -1;
-	inode_t *cwd = NULL;
+    int ret = -1;
+    inode_t *cwd = NULL;
 
-	if (origpath[0] == '/')
-		return priv_glfs_resolve_at (fs, subvol, NULL, origpath, loc,
-                                             iatt, follow, reval);
+    if (origpath[0] == '/')
+        return priv_glfs_resolve_at(fs, subvol, NULL, origpath, loc, iatt,
+                                    follow, reval);
 
-	cwd = glfs_cwd_get (fs);
+    cwd = glfs_cwd_get(fs);
+    if (NULL == cwd) {
+        gf_smsg(subvol->name, GF_LOG_WARNING, EIO, API_MSG_GET_CWD_FAILED,
+                NULL);
+        errno = EIO;
+        goto out;
+    }
 
-	ret = priv_glfs_resolve_at (fs, subvol, cwd, origpath, loc, iatt,
-                                    follow, reval);
-	if (cwd)
-		inode_unref (cwd);
+    ret = priv_glfs_resolve_at(fs, subvol, cwd, origpath, loc, iatt, follow,
+                               reval);
+    if (cwd)
+        inode_unref(cwd);
 
-	return ret;
+out:
+    return ret;
 }
 
-
+GFAPI_SYMVER_PRIVATE_DEFAULT(glfs_resolve, 3.7.0)
 int
-glfs_resolve (struct glfs *fs, xlator_t *subvol, const char *origpath,
-	      loc_t *loc, struct iatt *iatt, int reval)
+priv_glfs_resolve(struct glfs *fs, xlator_t *subvol, const char *origpath,
+                  loc_t *loc, struct iatt *iatt, int reval)
 {
-	int ret = -1;
+    int ret = -1;
 
-	ret = glfs_resolve_path (fs, subvol, origpath, loc, iatt, 1, reval);
+    ret = glfs_resolve_path(fs, subvol, origpath, loc, iatt, 1, reval);
 
-	return ret;
+    return ret;
 }
 
-
 int
-glfs_lresolve (struct glfs *fs, xlator_t *subvol, const char *origpath,
-	       loc_t *loc, struct iatt *iatt, int reval)
+glfs_lresolve(struct glfs *fs, xlator_t *subvol, const char *origpath,
+              loc_t *loc, struct iatt *iatt, int reval)
 {
-	int ret = -1;
+    int ret = -1;
 
-	ret = glfs_resolve_path (fs, subvol, origpath, loc, iatt, 0, reval);
+    ret = glfs_resolve_path(fs, subvol, origpath, loc, iatt, 0, reval);
 
-	return ret;
+    return ret;
 }
 
-
 int
-glfs_migrate_fd_locks_safe (struct glfs *fs, xlator_t *oldsubvol, fd_t *oldfd,
-			    xlator_t *newsubvol, fd_t *newfd)
+glfs_migrate_fd_locks_safe(struct glfs *fs, xlator_t *oldsubvol, fd_t *oldfd,
+                           xlator_t *newsubvol, fd_t *newfd)
 {
-	dict_t *lockinfo = NULL;
-	int ret = 0;
-	char uuid1[64];
-
-	if (!oldfd->lk_ctx || fd_lk_ctx_empty (oldfd->lk_ctx))
-		return 0;
-
-	newfd->lk_ctx = fd_lk_ctx_ref (oldfd->lk_ctx);
-
-	ret = syncop_fgetxattr (oldsubvol, oldfd, &lockinfo,
-				GF_XATTR_LOCKINFO_KEY);
-        DECODE_SYNCOP_ERR (ret);
-	if (ret < 0) {
-		gf_log (fs->volname, GF_LOG_WARNING,
-			"fgetxattr (%s) failed (%s) on graph %s (%d)",
-			uuid_utoa_r (oldfd->inode->gfid, uuid1),
-			strerror (errno),
-			graphid_str (oldsubvol), oldsubvol->graph->id);
-		goto out;
-	}
-
-	if (!dict_get (lockinfo, GF_XATTR_LOCKINFO_KEY)) {
-		gf_log (fs->volname, GF_LOG_WARNING,
-			"missing lokinfo key (%s) on graph %s (%d)",
-			uuid_utoa_r (oldfd->inode->gfid, uuid1),
-			graphid_str (oldsubvol), oldsubvol->graph->id);
-		goto out;
-	}
-
-	ret = syncop_fsetxattr (newsubvol, newfd, lockinfo, 0);
-        DECODE_SYNCOP_ERR (ret);
-	if (ret < 0) {
-		gf_log (fs->volname, GF_LOG_WARNING,
-			"fsetxattr (%s) failed (%s) on graph %s (%d)",
-			uuid_utoa_r (newfd->inode->gfid, uuid1),
-			strerror (errno),
-			graphid_str (newsubvol), newsubvol->graph->id);
-		goto out;
-	}
+    dict_t *lockinfo = NULL;
+    int ret = 0;
+    char uuid1[64];
+
+    if (!oldfd->lk_ctx || fd_lk_ctx_empty(oldfd->lk_ctx))
+        return 0;
+
+    newfd->lk_ctx = fd_lk_ctx_ref(oldfd->lk_ctx);
+
+    ret = syncop_fgetxattr(oldsubvol, oldfd, &lockinfo, GF_XATTR_LOCKINFO_KEY,
+                           NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+    if (ret < 0) {
+        gf_smsg(fs->volname, GF_LOG_WARNING, errno, API_MSG_FGETXATTR_FAILED,
+                "gfid=%s", uuid_utoa_r(oldfd->inode->gfid, uuid1), "err=%s",
+                strerror(errno), "subvol=%s", graphid_str(oldsubvol), "id=%d",
+                oldsubvol->graph->id, NULL);
+        goto out;
+    }
+
+    if (!dict_get(lockinfo, GF_XATTR_LOCKINFO_KEY)) {
+        gf_smsg(fs->volname, GF_LOG_WARNING, 0, API_MSG_LOCKINFO_KEY_MISSING,
+                "gfid=%s", uuid_utoa_r(oldfd->inode->gfid, uuid1), "subvol=%s",
+                graphid_str(oldsubvol), "id=%d", oldsubvol->graph->id, NULL);
+        goto out;
+    }
+
+    ret = syncop_fsetxattr(newsubvol, newfd, lockinfo, 0, NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+    if (ret < 0) {
+        gf_smsg(fs->volname, GF_LOG_WARNING, 0, API_MSG_FSETXATTR_FAILED,
+                "gfid=%s", uuid_utoa_r(newfd->inode->gfid, uuid1), "err=%s",
+                strerror(errno), "subvol=%s", graphid_str(newsubvol), "id=%d",
+                newsubvol->graph->id, NULL);
+        goto out;
+    }
 out:
-	if (lockinfo)
-		dict_unref (lockinfo);
-	return ret;
+    if (lockinfo)
+        dict_unref(lockinfo);
+    return ret;
 }
 
-
 fd_t *
-glfs_migrate_fd_safe (struct glfs *fs, xlator_t *newsubvol, fd_t *oldfd)
+glfs_migrate_fd_safe(struct glfs *fs, xlator_t *newsubvol, fd_t *oldfd)
 {
-	fd_t *newfd = NULL;
-	inode_t *oldinode = NULL;
-	inode_t *newinode = NULL;
-	xlator_t *oldsubvol = NULL;
-	int ret = -1;
-	loc_t loc = {0, };
-	char uuid1[64];
-
-
-	oldinode = oldfd->inode;
-	oldsubvol = oldinode->table->xl;
-
-	if (oldsubvol == newsubvol)
-		return fd_ref (oldfd);
-
-	if (!oldsubvol->switched) {
-		ret = syncop_fsync (oldsubvol, oldfd, 0);
-                DECODE_SYNCOP_ERR (ret);
-		if (ret) {
-			gf_log (fs->volname, GF_LOG_WARNING,
-				"fsync() failed (%s) on %s graph %s (%d)",
-				strerror (errno),
-				uuid_utoa_r (oldfd->inode->gfid, uuid1),
-				graphid_str (oldsubvol), oldsubvol->graph->id);
-		}
-	}
-
-	newinode = glfs_refresh_inode_safe (newsubvol, oldinode);
-	if (!newinode) {
-		gf_log (fs->volname, GF_LOG_WARNING,
-			"inode (%s) refresh failed (%s) on graph %s (%d)",
-			uuid_utoa_r (oldinode->gfid, uuid1),
-			strerror (errno),
-			graphid_str (newsubvol), newsubvol->graph->id);
-		goto out;
-	}
-
-	newfd = fd_create (newinode, getpid());
-	if (!newfd) {
-		gf_log (fs->volname, GF_LOG_WARNING,
-			"fd_create (%s) failed (%s) on graph %s (%d)",
-			uuid_utoa_r (newinode->gfid, uuid1),
-			strerror (errno),
-			graphid_str (newsubvol), newsubvol->graph->id);
-		goto out;
-	}
-
-	loc.inode = inode_ref (newinode);
-
-        ret = inode_path (oldfd->inode, NULL, (char **)&loc.path);
-        if (ret < 0) {
-                gf_log (fs->volname, GF_LOG_INFO, "inode_path failed");
-                goto out;
+    fd_t *newfd = NULL;
+    inode_t *oldinode = NULL;
+    inode_t *newinode = NULL;
+    xlator_t *oldsubvol = NULL;
+    int ret = -1;
+    loc_t loc = {
+        0,
+    };
+    char uuid1[64];
+    dict_t *xdata = NULL;
+
+    oldinode = oldfd->inode;
+    oldsubvol = oldinode->table->xl;
+
+    if (oldsubvol == newsubvol)
+        return fd_ref(oldfd);
+
+    if (!oldsubvol->switched) {
+        xdata = dict_new();
+        if (!xdata || dict_set_int8(xdata, "last-fsync", 1)) {
+            gf_smsg(fs->volname, GF_LOG_WARNING, ENOMEM, API_MSG_FSYNC_FAILED,
+                    "err=%s", "last-fsync set failed", "gfid=%s",
+                    uuid_utoa_r(oldfd->inode->gfid, uuid1), "subvol=%s",
+                    graphid_str(oldsubvol), "id=%d", oldsubvol->graph->id,
+                    NULL);
         }
 
-        uuid_copy (loc.gfid, oldinode->gfid);
-
-
-	if (IA_ISDIR (oldinode->ia_type))
-		ret = syncop_opendir (newsubvol, &loc, newfd);
-	else
-		ret = syncop_open (newsubvol, &loc,
-				   oldfd->flags & ~(O_TRUNC|O_EXCL|O_CREAT),
-				   newfd);
-        DECODE_SYNCOP_ERR (ret);
-	loc_wipe (&loc);
-
-	if (ret) {
-		gf_log (fs->volname, GF_LOG_WARNING,
-			"syncop_open%s (%s) failed (%s) on graph %s (%d)",
-			IA_ISDIR (oldinode->ia_type) ? "dir" : "",
-			uuid_utoa_r (newinode->gfid, uuid1),
-			strerror (errno),
-			graphid_str (newsubvol), newsubvol->graph->id);
-		goto out;
-	}
-
-	ret = glfs_migrate_fd_locks_safe (fs, oldsubvol, oldfd, newsubvol,
-					  newfd);
-
-	if (ret) {
-		gf_log (fs->volname, GF_LOG_WARNING,
-			"lock migration (%s) failed (%s) on graph %s (%d)",
-			uuid_utoa_r (newinode->gfid, uuid1),
-			strerror (errno),
-			graphid_str (newsubvol), newsubvol->graph->id);
-		goto out;
-	}
-
-        newfd->flags = oldfd->flags;
-	fd_bind (newfd);
+        ret = syncop_fsync(oldsubvol, oldfd, 0, NULL, NULL, xdata, NULL);
+        DECODE_SYNCOP_ERR(ret);
+        if (ret) {
+            gf_smsg(fs->volname, GF_LOG_WARNING, errno, API_MSG_FSYNC_FAILED,
+                    "err=%s", strerror(errno), "gfid=%s",
+                    uuid_utoa_r(oldfd->inode->gfid, uuid1), "subvol=%s",
+                    graphid_str(oldsubvol), "id=%d", oldsubvol->graph->id,
+                    NULL);
+        }
+    }
+
+    newinode = glfs_refresh_inode_safe(newsubvol, oldinode, _gf_false);
+    if (!newinode) {
+        gf_smsg(fs->volname, GF_LOG_WARNING, errno,
+                API_MSG_INODE_REFRESH_FAILED, "gfid=%s",
+                uuid_utoa_r(oldinode->gfid, uuid1), "err=%s", strerror(errno),
+                "subvol=%s", graphid_str(newsubvol), "id=%d",
+                newsubvol->graph->id, NULL);
+        goto out;
+    }
+
+    newfd = fd_create(newinode, getpid());
+    if (!newfd) {
+        gf_smsg(fs->volname, GF_LOG_WARNING, errno,
+                API_MSG_FDCREATE_FAILED_ON_GRAPH, "gfid=%s",
+                uuid_utoa_r(newinode->gfid, uuid1), "err=%s", strerror(errno),
+                "subvol=%s", graphid_str(newsubvol), "id=%d",
+                newsubvol->graph->id, NULL);
+        goto out;
+    }
+
+    loc.inode = inode_ref(newinode);
+
+    ret = inode_path(oldfd->inode, NULL, (char **)&loc.path);
+    if (ret < 0) {
+        gf_smsg(fs->volname, GF_LOG_INFO, 0, API_MSG_INODE_PATH_FAILED, NULL);
+        goto out;
+    }
+
+    gf_uuid_copy(loc.gfid, oldinode->gfid);
+
+    if (IA_ISDIR(oldinode->ia_type))
+        ret = syncop_opendir(newsubvol, &loc, newfd, NULL, NULL);
+    else
+        ret = syncop_open(newsubvol, &loc,
+                          oldfd->flags & ~(O_TRUNC | O_EXCL | O_CREAT), newfd,
+                          NULL, NULL);
+    DECODE_SYNCOP_ERR(ret);
+    loc_wipe(&loc);
+
+    if (ret) {
+        gf_smsg(fs->volname, GF_LOG_WARNING, errno, API_MSG_SYNCOP_OPEN_FAILED,
+                "type=%s", IA_ISDIR(oldinode->ia_type) ? "dir" : "", "gfid=%s",
+                uuid_utoa_r(newinode->gfid, uuid1), "err=%s", strerror(errno),
+                "subvol=%s", graphid_str(newsubvol), "id=%d",
+                newsubvol->graph->id, NULL);
+        goto out;
+    }
+
+    ret = glfs_migrate_fd_locks_safe(fs, oldsubvol, oldfd, newsubvol, newfd);
+
+    if (ret) {
+        gf_smsg(fs->volname, GF_LOG_WARNING, errno, API_MSG_LOCK_MIGRATE_FAILED,
+                "gfid=%s", uuid_utoa_r(newinode->gfid, uuid1), "err=%s",
+                strerror(errno), "subvol=%s", graphid_str(newsubvol), "id=%d",
+                newsubvol->graph->id, NULL);
+        goto out;
+    }
+
+    newfd->flags = oldfd->flags;
+    fd_bind(newfd);
 out:
-	if (newinode)
-		inode_unref (newinode);
+    if (newinode)
+        inode_unref(newinode);
 
-	if (ret) {
-		fd_unref (newfd);
-		newfd = NULL;
-	}
+    if (ret) {
+        fd_unref(newfd);
+        newfd = NULL;
+    }
 
-	return newfd;
-}
+    if (xdata)
+        dict_unref(xdata);
 
+    return newfd;
+}
 
 fd_t *
-__glfs_migrate_fd (struct glfs *fs, xlator_t *newsubvol, struct glfs_fd *glfd)
+__glfs_migrate_fd(struct glfs *fs, xlator_t *newsubvol, struct glfs_fd *glfd)
 {
-	fd_t *oldfd = NULL;
-	fd_t *newfd = NULL;
+    fd_t *oldfd = NULL;
+    fd_t *newfd = NULL;
 
-	oldfd = glfd->fd;
+    oldfd = glfd->fd;
 
-	fs->migration_in_progress = 1;
-	pthread_mutex_unlock (&fs->mutex);
-	{
-		newfd = glfs_migrate_fd_safe (fs, newsubvol, oldfd);
-	}
-	pthread_mutex_lock (&fs->mutex);
-	fs->migration_in_progress = 0;
-	pthread_cond_broadcast (&fs->cond);
+    fs->migration_in_progress = 1;
+    pthread_mutex_unlock(&fs->mutex);
+    {
+        newfd = glfs_migrate_fd_safe(fs, newsubvol, oldfd);
+    }
+    pthread_mutex_lock(&fs->mutex);
+    fs->migration_in_progress = 0;
+    pthread_cond_broadcast(&fs->cond);
 
-	return newfd;
-}
+    /* wake up other waiting tasks */
+    __GLFS_SYNCTASK_WAKE(fs);
 
+    return newfd;
+}
 
 fd_t *
-__glfs_resolve_fd (struct glfs *fs, xlator_t *subvol, struct glfs_fd *glfd)
+__glfs_resolve_fd(struct glfs *fs, xlator_t *subvol, struct glfs_fd *glfd)
 {
-	fd_t *fd = NULL;
+    fd_t *fd = NULL;
 
-	if (glfd->fd->inode->table->xl == subvol)
-		return fd_ref (glfd->fd);
+    if (glfd->fd->inode->table->xl == subvol)
+        return fd_ref(glfd->fd);
 
-	fd = __glfs_migrate_fd (fs, subvol, glfd);
-	if (!fd)
-		return NULL;
+    fd = __glfs_migrate_fd(fs, subvol, glfd);
+    if (!fd)
+        return NULL;
 
-	if (subvol == fs->active_subvol) {
-		fd_unref (glfd->fd);
-		glfd->fd = fd_ref (fd);
-	}
+    if (subvol == fs->active_subvol) {
+        fd_unref(glfd->fd);
+        glfd->fd = fd_ref(fd);
+    }
 
-	return fd;
+    return fd;
 }
 
-
 fd_t *
-glfs_resolve_fd (struct glfs *fs, xlator_t *subvol, struct glfs_fd *glfd)
+glfs_resolve_fd(struct glfs *fs, xlator_t *subvol, struct glfs_fd *glfd)
 {
-	fd_t *fd = NULL;
+    fd_t *fd = NULL;
 
-	glfs_lock (fs);
-	{
-		fd = __glfs_resolve_fd (fs, subvol, glfd);
-	}
-	glfs_unlock (fs);
+    glfs_lock(fs, _gf_true);
+    {
+        fd = __glfs_resolve_fd(fs, subvol, glfd);
+    }
+    glfs_unlock(fs);
 
-	return fd;
+    return fd;
 }
 
-
 void
-__glfs_migrate_openfds (struct glfs *fs, xlator_t *subvol)
+__glfs_migrate_openfds(struct glfs *fs, xlator_t *subvol)
 {
-	struct glfs_fd *glfd = NULL;
-	fd_t *fd = NULL;
-
-	list_for_each_entry (glfd, &fs->openfds, openfds) {
-		if (uuid_is_null (glfd->fd->inode->gfid)) {
-			gf_log (fs->volname, GF_LOG_INFO,
-				"skipping openfd %p/%p in graph %s (%d)",
-				glfd, glfd->fd,	graphid_str(subvol),
-				subvol->graph->id);
-			/* create in progress, defer */
-			continue;
-		}
-
-		fd = __glfs_migrate_fd (fs, subvol, glfd);
-		if (fd) {
-			fd_unref (glfd->fd);
-			glfd->fd = fd;
-		}
-	}
-}
-
+    struct glfs_fd *glfd = NULL;
+    fd_t *fd = NULL;
+
+    list_for_each_entry(glfd, &fs->openfds, openfds)
+    {
+        if (gf_uuid_is_null(glfd->fd->inode->gfid)) {
+            gf_smsg(fs->volname, GF_LOG_INFO, 0, API_MSG_OPENFD_SKIPPED,
+                    "glfd=%p", glfd, "glfd->fd=%p", glfd->fd, "subvol=%s",
+                    graphid_str(subvol), "id=%d", subvol->graph->id, NULL);
+            /* create in progress, defer */
+            continue;
+        }
 
-xlator_t *
-__glfs_active_subvol (struct glfs *fs)
-{
-	xlator_t      *new_subvol = NULL;
-	int            ret = -1;
-	inode_t       *new_cwd = NULL;
-
-	if (!fs->next_subvol)
-		return fs->active_subvol;
-
-	new_subvol = fs->next_subvol;
-
-	ret = __glfs_first_lookup (fs, new_subvol);
-	if (ret) {
-		gf_log (fs->volname, GF_LOG_INFO,
-			"first lookup on graph %s (%d) failed (%s)",
-			graphid_str (new_subvol), new_subvol->graph->id,
-			strerror (errno));
-		return NULL;
-	}
-
-	if (fs->cwd) {
-		new_cwd = __glfs_refresh_inode (fs, new_subvol, fs->cwd);
-
-		if (!new_cwd) {
-			char buf1[64];
-			gf_log (fs->volname, GF_LOG_INFO,
-				"cwd refresh of %s graph %s (%d) failed (%s)",
-				uuid_utoa_r (fs->cwd->gfid, buf1),
-				graphid_str (new_subvol),
-				new_subvol->graph->id, strerror (errno));
-			return NULL;
-		}
-	}
-
-	__glfs_migrate_openfds (fs, new_subvol);
-
-	/* switching @active_subvol and @cwd
-	   should be atomic
-	*/
-	fs->old_subvol = fs->active_subvol;
-	fs->active_subvol = fs->next_subvol;
-	fs->next_subvol = NULL;
-
-	if (new_cwd) {
-		__glfs_cwd_set (fs, new_cwd);
-		inode_unref (new_cwd);
-	}
-
-	gf_log (fs->volname, GF_LOG_INFO, "switched to graph %s (%d)",
-		graphid_str (new_subvol), new_subvol->graph->id);
-
-	return new_subvol;
+        fd = __glfs_migrate_fd(fs, subvol, glfd);
+        if (fd) {
+            fd_unref(glfd->fd);
+            glfd->fd = fd;
+        }
+    }
 }
 
+/* Note that though it appears that this function executes under fs->mutex,
+ * it is not fully executed under fs->mutex. i.e. there are functions like
+ * __glfs_first_lookup, __glfs_refresh_inode, __glfs_migrate_openfds which
+ * unlocks fs->mutex before sending any network fop, and reacquire fs->mutex
+ * once the fop is complete. Hence the variable read from fs at the start of the
+ * function need not have the same value by the end of the function.
+ */
 xlator_t *
-priv_glfs_active_subvol (struct glfs *fs)
+__glfs_active_subvol(struct glfs *fs)
 {
-	xlator_t      *subvol = NULL;
-	xlator_t      *old_subvol = NULL;
+    xlator_t *new_subvol = NULL;
+    int ret = -1;
+    inode_t *new_cwd = NULL;
+
+    if (!fs->next_subvol)
+        return fs->active_subvol;
+
+    new_subvol = fs->mip_subvol = fs->next_subvol;
+    fs->next_subvol = NULL;
+
+    ret = __glfs_first_lookup(fs, new_subvol);
+    if (ret) {
+        gf_smsg(fs->volname, GF_LOG_INFO, errno,
+                API_MSG_FIRST_LOOKUP_GRAPH_FAILED, "subvol=%s",
+                graphid_str(new_subvol), "id=%d", new_subvol->graph->id,
+                "err=%s", strerror(errno), NULL);
+        return NULL;
+    }
+
+    if (fs->cwd) {
+        new_cwd = __glfs_refresh_inode(fs, new_subvol, fs->cwd, _gf_false);
+
+        if (!new_cwd) {
+            char buf1[64];
+            gf_smsg(fs->volname, GF_LOG_INFO, errno,
+                    API_MSG_CWD_GRAPH_REF_FAILED, "buf=%s",
+                    uuid_utoa_r(fs->cwd->gfid, buf1), "subvol=%s",
+                    graphid_str(new_subvol), "id=%d", new_subvol->graph->id,
+                    "err=%s", strerror(errno), NULL);
+            return NULL;
+        }
+    }
 
-	glfs_lock (fs);
-	{
-		subvol = __glfs_active_subvol (fs);
+    __glfs_migrate_openfds(fs, new_subvol);
+    /* TODO: Migrate the fds and inodes which have leases to the new graph
+     * (issue #350)*/
 
-		if (subvol)
-			subvol->winds++;
+    /* switching @active_subvol and @cwd
+       should be atomic
+    */
+    fs->old_subvol = fs->active_subvol;
+    fs->active_subvol = fs->mip_subvol;
+    fs->mip_subvol = NULL;
 
-		if (fs->old_subvol) {
-			old_subvol = fs->old_subvol;
-			fs->old_subvol = NULL;
-			old_subvol->switched = 1;
-		}
-	}
-	glfs_unlock (fs);
+    if (new_cwd) {
+        __glfs_cwd_set(fs, new_cwd);
+        inode_unref(new_cwd);
+    }
 
-	if (old_subvol)
-		priv_glfs_subvol_done (fs, old_subvol);
+    gf_smsg(fs->volname, GF_LOG_INFO, 0, API_MSG_SWITCHED_GRAPH, "subvol=%s",
+            graphid_str(new_subvol), "id=%d", new_subvol->graph->id, NULL);
 
-	return subvol;
+    return new_subvol;
 }
 
-GFAPI_SYMVER_PRIVATE_DEFAULT(glfs_active_subvol, 3.4.0);
-
+GFAPI_SYMVER_PRIVATE_DEFAULT(glfs_subvol_done, 3.4.0)
 void
-priv_glfs_subvol_done (struct glfs *fs, xlator_t *subvol)
+priv_glfs_subvol_done(struct glfs *fs, xlator_t *subvol)
 {
-	int ref = 0;
-	xlator_t *active_subvol = NULL;
-
-	if (!subvol)
-		return;
-
-	glfs_lock (fs);
-	{
-		ref = (--subvol->winds);
-		active_subvol = fs->active_subvol;
-	}
-	glfs_unlock (fs);
-
-	if (ref == 0) {
-		assert (subvol != active_subvol);
-		xlator_notify (subvol, GF_EVENT_PARENT_DOWN, subvol, NULL);
-	}
+    int ref = 0;
+    xlator_t *active_subvol = NULL;
+
+    if (!subvol)
+        return;
+
+    /* For decrementing subvol->wind ref count we need not check/wait for
+     * migration-in-progress flag.
+     * Also glfs_subvol_done is called in call-back path therefore waiting
+     * for migration-in-progress flag can lead to dead-lock.
+     */
+    glfs_lock(fs, _gf_false);
+    {
+        ref = (--subvol->winds);
+        active_subvol = fs->active_subvol;
+    }
+    glfs_unlock(fs);
+
+    if (ref == 0) {
+        assert(subvol != active_subvol);
+        xlator_notify(subvol, GF_EVENT_PARENT_DOWN, subvol, NULL);
+    }
 }
 
-GFAPI_SYMVER_PRIVATE_DEFAULT(glfs_subvol_done, 3.4.0);
+GFAPI_SYMVER_PRIVATE_DEFAULT(glfs_active_subvol, 3.4.0)
+xlator_t *
+priv_glfs_active_subvol(struct glfs *fs)
+{
+    xlator_t *subvol = NULL;
+    xlator_t *old_subvol = NULL;
+
+    glfs_lock(fs, _gf_true);
+    {
+        subvol = __glfs_active_subvol(fs);
+
+        if (subvol)
+            subvol->winds++;
+
+        if (fs->old_subvol) {
+            old_subvol = fs->old_subvol;
+            fs->old_subvol = NULL;
+            old_subvol->switched = 1;
+        }
+    }
+    glfs_unlock(fs);
+
+    if (old_subvol)
+        priv_glfs_subvol_done(fs, old_subvol);
+
+    return subvol;
+}
 
 int
-__glfs_cwd_set (struct glfs *fs, inode_t *inode)
+__glfs_cwd_set(struct glfs *fs, inode_t *inode)
 {
-	if (inode->table->xl != fs->active_subvol) {
-		inode = __glfs_refresh_inode (fs, fs->active_subvol, inode);
-		if (!inode)
-			return -1;
-	} else {
-		inode_ref (inode);
-	}
+    if (inode->table->xl != fs->active_subvol) {
+        inode = __glfs_refresh_inode(fs, fs->active_subvol, inode, _gf_false);
+        if (!inode)
+            return -1;
+    } else {
+        inode_ref(inode);
+    }
 
-	if (fs->cwd)
-		inode_unref (fs->cwd);
+    if (fs->cwd)
+        inode_unref(fs->cwd);
 
-	fs->cwd = inode;
+    fs->cwd = inode;
 
-	return 0;
+    return 0;
 }
 
-
 int
-glfs_cwd_set (struct glfs *fs, inode_t *inode)
+glfs_cwd_set(struct glfs *fs, inode_t *inode)
 {
-	int ret = 0;
+    int ret = 0;
 
-	glfs_lock (fs);
-	{
-		ret = __glfs_cwd_set (fs, inode);
-	}
-	glfs_unlock (fs);
+    glfs_lock(fs, _gf_true);
+    {
+        ret = __glfs_cwd_set(fs, inode);
+    }
+    glfs_unlock(fs);
 
-	return ret;
+    return ret;
 }
 
-
 inode_t *
-__glfs_cwd_get (struct glfs *fs)
+__glfs_cwd_get(struct glfs *fs)
 {
-	inode_t *cwd = NULL;
+    inode_t *cwd = NULL;
 
-	if (!fs->cwd)
-		return NULL;
+    if (!fs->cwd)
+        return NULL;
 
-	if (fs->cwd->table->xl == fs->active_subvol) {
-		cwd = inode_ref (fs->cwd);
-		return cwd;
-	}
+    if (fs->cwd->table->xl == fs->active_subvol) {
+        cwd = inode_ref(fs->cwd);
+        return cwd;
+    }
 
-	cwd = __glfs_refresh_inode (fs, fs->active_subvol, fs->cwd);
+    cwd = __glfs_refresh_inode(fs, fs->active_subvol, fs->cwd, _gf_false);
 
-	return cwd;
+    return cwd;
 }
 
 inode_t *
-glfs_cwd_get (struct glfs *fs)
+glfs_cwd_get(struct glfs *fs)
 {
-	inode_t *cwd = NULL;
+    inode_t *cwd = NULL;
 
-	glfs_lock (fs);
-	{
-		cwd = __glfs_cwd_get (fs);
-	}
-	glfs_unlock (fs);
+    glfs_lock(fs, _gf_true);
+    {
+        cwd = __glfs_cwd_get(fs);
+    }
+    glfs_unlock(fs);
 
-	return cwd;
+    return cwd;
 }
 
 inode_t *
-__glfs_resolve_inode (struct glfs *fs, xlator_t *subvol,
-		    struct glfs_object *object)
+__glfs_resolve_inode(struct glfs *fs, xlator_t *subvol,
+                     struct glfs_object *object)
 {
-	inode_t *inode = NULL;
+    inode_t *inode = NULL;
+    gf_boolean_t lookup_needed = _gf_false;
 
-	if (object->inode->table->xl == subvol)
-		return inode_ref (object->inode);
+    lookup_needed = inode_needs_lookup(object->inode, THIS);
 
-	inode = __glfs_refresh_inode (fs, fs->active_subvol,
-					object->inode);
-	if (!inode)
-		return NULL;
+    if (!lookup_needed && object->inode->table->xl == subvol)
+        return inode_ref(object->inode);
 
-	if (subvol == fs->active_subvol) {
-		inode_unref (object->inode);
-		object->inode = inode_ref (inode);
-	}
+    inode = __glfs_refresh_inode(fs, fs->active_subvol, object->inode,
+                                 lookup_needed);
+    if (!inode)
+        return NULL;
 
-	return inode;
+    if (subvol == fs->active_subvol) {
+        inode_unref(object->inode);
+        object->inode = inode_ref(inode);
+    }
+
+    return inode;
 }
 
 inode_t *
-glfs_resolve_inode (struct glfs *fs, xlator_t *subvol,
-		    struct glfs_object *object)
+glfs_resolve_inode(struct glfs *fs, xlator_t *subvol,
+                   struct glfs_object *object)
 {
-	inode_t *inode = NULL;
+    inode_t *inode = NULL;
 
-	glfs_lock (fs);
-	{
-		inode = __glfs_resolve_inode(fs, subvol, object);
-	}
-	glfs_unlock (fs);
+    glfs_lock(fs, _gf_true);
+    {
+        inode = __glfs_resolve_inode(fs, subvol, object);
+    }
+    glfs_unlock(fs);
 
-	return inode;
+    return inode;
 }
 
 int
-glfs_create_object (loc_t *loc, struct glfs_object **retobject)
+glfs_create_object(loc_t *loc, struct glfs_object **retobject)
 {
-	struct glfs_object *object = NULL;
+    struct glfs_object *object = NULL;
+
+    object = GF_CALLOC(1, sizeof(struct glfs_object), glfs_mt_glfs_object_t);
+    if (object == NULL) {
+        errno = ENOMEM;
+        return -1;
+    }
+
+    object->inode = loc->inode;
+    gf_uuid_copy(object->gfid, object->inode->gfid);
 
-	object = GF_CALLOC (1, sizeof(struct glfs_object),
-			    glfs_mt_glfs_object_t);
-	if (object == NULL) {
-		errno = ENOMEM;
-		return -1;
-	}
+    /* we hold the reference */
+    loc->inode = NULL;
 
-	object->inode = loc->inode;
-	uuid_copy (object->gfid, object->inode->gfid);
+    *retobject = object;
 
-	/* we hold the reference */
-	loc->inode = NULL;
+    return 0;
+}
 
-	*retobject = object;
+struct glfs_object *
+glfs_h_resolve_symlink(struct glfs *fs, struct glfs_object *object)
+{
+    xlator_t *subvol = NULL;
+    loc_t sym_loc = {
+        0,
+    };
+    struct iatt iatt = {
+        0,
+    };
+    char *lpath = NULL;
+    int ret = 0;
+    struct glfs_object *target_object = NULL;
+
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
+
+    ret = glfs_resolve_symlink(fs, subvol, object->inode, &lpath);
+    if (ret < 0)
+        goto out;
+
+    ret = glfs_resolve_at(fs, subvol, NULL, lpath, &sym_loc, &iatt,
+                          /* always recurisvely follow while
+                            following symlink
+                          */
+                          1, 0);
+    if (ret == 0)
+        ret = glfs_create_object(&sym_loc, &target_object);
 
-	return 0;
+out:
+    loc_wipe(&sym_loc);
+    GF_FREE(lpath);
+    return target_object;
 }
diff --git a/api/src/glfs.c b/api/src/glfs.c
index e74329caf9d..b4bf1423f6d 100644
--- a/api/src/glfs.c
+++ b/api/src/glfs.c
@@ -1,5 +1,5 @@
 /*
-  Copyright (c) 2012 Red Hat, Inc. <http://www.redhat.com>
+  Copyright (c) 2012-2018 Red Hat, Inc. <http://www.redhat.com>
   This file is part of GlusterFS.
 
   This file is licensed to you under your choice of the GNU Lesser
@@ -8,10 +8,8 @@
   cases as published by the Free Software Foundation.
 */
 
-
 /*
   TODO:
-  - merge locks in glfs_posix_lock for lock self-healing
   - set proper pid/lk_owner to call frames (currently buried in syncop)
   - fix logging.c/h to store logfp and loglevel in glusterfs_ctx_t and
     reach it via THIS.
@@ -32,858 +30,1777 @@
 #include <sys/types.h>
 #include <unistd.h>
 #include <limits.h>
-
-#ifndef _CONFIG_H
-#define _CONFIG_H
-#include "config.h"
+#ifdef GF_LINUX_HOST_OS
+#include <sys/prctl.h>
 #endif
 
-#include "glusterfs.h"
-#include "logging.h"
-#include "stack.h"
-#include "event.h"
+#include <glusterfs/glusterfs.h>
+#include <glusterfs/logging.h>
+#include <glusterfs/stack.h>
+#include <glusterfs/gf-event.h>
 #include "glfs-mem-types.h"
-#include "common-utils.h"
-#include "syncop.h"
-#include "call-stub.h"
+#include <glusterfs/common-utils.h>
+#include <glusterfs/syncop.h>
+#include <glusterfs/call-stub.h>
+#include <glusterfs/hashfn.h>
+#include "rpc-clnt.h"
+#include <glusterfs/statedump.h>
+#include <glusterfs/syscall.h>
 
+#include "gfapi-messages.h"
 #include "glfs.h"
 #include "glfs-internal.h"
-#include "hashfn.h"
-#include "rpc-clnt.h"
-
 
 static gf_boolean_t
-vol_assigned (cmd_args_t *args)
+vol_assigned(cmd_args_t *args)
 {
-	return args->volfile || args->volfile_server;
+    return args->volfile || args->volfile_server;
 }
 
-
 static int
-glusterfs_ctx_defaults_init (glusterfs_ctx_t *ctx)
+glusterfs_ctx_defaults_init(glusterfs_ctx_t *ctx)
 {
-	call_pool_t   *pool = NULL;
-	int	       ret = -1;
+    call_pool_t *pool = NULL;
+    int ret = -1;
 
-	if (!ctx) {
-		goto err;
+    if (!ctx) {
+        goto err;
     }
-    ret = xlator_mem_acct_init (THIS, glfs_mt_end + 1);
+
+    ret = xlator_mem_acct_init(THIS, glfs_mt_end + 1);
     if (ret != 0) {
-	    gf_log(THIS->name, GF_LOG_ERROR,
-              "Memory accounting init failed");
-	    return ret;
-    }
-
-	ctx->process_uuid = generate_glusterfs_ctx_id ();
-	if (!ctx->process_uuid) {
-		goto err;
-	}
-
-	ctx->page_size	= 128 * GF_UNIT_KB;
-
-	ctx->iobuf_pool = iobuf_pool_new ();
-	if (!ctx->iobuf_pool) {
-		goto err;
-	}
-
-	ctx->event_pool = event_pool_new (DEFAULT_EVENT_POOL_SIZE);
-	if (!ctx->event_pool) {
-		goto err;
-	}
-
-	ctx->env = syncenv_new (0, 0, 0);
-	if (!ctx->env) {
-		goto err;
-	}
-
-	pool = GF_CALLOC (1, sizeof (call_pool_t),
-			  glfs_mt_call_pool_t);
-	if (!pool) {
-		goto err;
-	}
-
-	/* frame_mem_pool size 112 * 4k */
-	pool->frame_mem_pool = mem_pool_new (call_frame_t, 4096);
-	if (!pool->frame_mem_pool) {
-		goto err;
-	}
-	/* stack_mem_pool size 256 * 1024 */
-	pool->stack_mem_pool = mem_pool_new (call_stack_t, 1024);
-	if (!pool->stack_mem_pool) {
-		goto err;
-	}
-
-	ctx->stub_mem_pool = mem_pool_new (call_stub_t, 1024);
-	if (!ctx->stub_mem_pool) {
-		goto err;
-	}
-
-	ctx->dict_pool = mem_pool_new (dict_t, GF_MEMPOOL_COUNT_OF_DICT_T);
-	if (!ctx->dict_pool)
-		goto err;
-
-	ctx->dict_pair_pool = mem_pool_new (data_pair_t,
-					    GF_MEMPOOL_COUNT_OF_DATA_PAIR_T);
-	if (!ctx->dict_pair_pool)
-		goto err;
-
-	ctx->dict_data_pool = mem_pool_new (data_t, GF_MEMPOOL_COUNT_OF_DATA_T);
-	if (!ctx->dict_data_pool)
-		goto err;
-
-        ctx->logbuf_pool = mem_pool_new (log_buf_t,
-                                         GF_MEMPOOL_COUNT_OF_LRU_BUF_T);
-        if (!ctx->logbuf_pool)
-                goto err;
-
-	INIT_LIST_HEAD (&pool->all_frames);
-	INIT_LIST_HEAD (&ctx->cmd_args.xlator_options);
-        INIT_LIST_HEAD (&ctx->cmd_args.volfile_servers);
-
-	LOCK_INIT (&pool->lock);
-	ctx->pool = pool;
-
-	pthread_mutex_init (&(ctx->lock), NULL);
-
-	ret = 0;
-err:
-	if (ret && pool) {
-		if (pool->frame_mem_pool)
-			mem_pool_destroy (pool->frame_mem_pool);
-		if (pool->stack_mem_pool)
-			mem_pool_destroy (pool->stack_mem_pool);
-		GF_FREE (pool);
-	}
-
-	if (ret && ctx) {
-		if (ctx->stub_mem_pool)
-			mem_pool_destroy (ctx->stub_mem_pool);
-		if (ctx->dict_pool)
-			mem_pool_destroy (ctx->dict_pool);
-		if (ctx->dict_data_pool)
-			mem_pool_destroy (ctx->dict_data_pool);
-		if (ctx->dict_pair_pool)
-			mem_pool_destroy (ctx->dict_pair_pool);
-                if (ctx->logbuf_pool)
-                        mem_pool_destroy (ctx->logbuf_pool);
-	}
-
-	return ret;
-}
+        gf_smsg(THIS->name, GF_LOG_ERROR, ENOMEM, API_MSG_MEM_ACCT_INIT_FAILED,
+                NULL);
+        return ret;
+    }
 
+    /* reset ret to -1 so that we don't need to explicitly
+     * set it in all error paths before "goto err"
+     */
 
-static int
-create_master (struct glfs *fs)
-{
-	int		 ret = 0;
-	xlator_t	*master = NULL;
+    ret = -1;
+
+    ctx->process_uuid = generate_glusterfs_ctx_id();
+    if (!ctx->process_uuid) {
+        goto err;
+    }
+
+    ctx->page_size = 128 * GF_UNIT_KB;
+
+    ctx->iobuf_pool = iobuf_pool_new();
+    if (!ctx->iobuf_pool) {
+        goto err;
+    }
+
+    ctx->event_pool = gf_event_pool_new(DEFAULT_EVENT_POOL_SIZE,
+                                        STARTING_EVENT_THREADS);
+    if (!ctx->event_pool) {
+        goto err;
+    }
+
+    ctx->env = syncenv_new(0, 0, 0);
+    if (!ctx->env) {
+        goto err;
+    }
+
+    pool = GF_CALLOC(1, sizeof(call_pool_t), glfs_mt_call_pool_t);
+    if (!pool) {
+        goto err;
+    }
 
-	master = GF_CALLOC (1, sizeof (*master),
-			    glfs_mt_xlator_t);
-	if (!master)
-		goto err;
+    /* frame_mem_pool size 112 * 4k */
+    pool->frame_mem_pool = mem_pool_new(call_frame_t, 4096);
+    if (!pool->frame_mem_pool) {
+        goto err;
+    }
+    /* stack_mem_pool size 256 * 1024 */
+    pool->stack_mem_pool = mem_pool_new(call_stack_t, 1024);
+    if (!pool->stack_mem_pool) {
+        goto err;
+    }
 
-	master->name = gf_strdup ("gfapi");
-	if (!master->name)
-		goto err;
+    ctx->stub_mem_pool = mem_pool_new(call_stub_t, 1024);
+    if (!ctx->stub_mem_pool) {
+        goto err;
+    }
 
-	if (xlator_set_type (master, "mount/api") == -1) {
-		gf_log ("glfs", GF_LOG_ERROR,
-			"master xlator for %s initialization failed",
-			fs->volname);
-		goto err;
-	}
+    ctx->dict_pool = mem_pool_new(dict_t, GF_MEMPOOL_COUNT_OF_DICT_T);
+    if (!ctx->dict_pool)
+        goto err;
 
-	master->ctx	 = fs->ctx;
-	master->private	 = fs;
-	master->options	 = get_new_dict ();
-	if (!master->options)
-		goto err;
+    ctx->dict_pair_pool = mem_pool_new(data_pair_t,
+                                       GF_MEMPOOL_COUNT_OF_DATA_PAIR_T);
+    if (!ctx->dict_pair_pool)
+        goto err;
 
+    ctx->dict_data_pool = mem_pool_new(data_t, GF_MEMPOOL_COUNT_OF_DATA_T);
+    if (!ctx->dict_data_pool)
+        goto err;
 
-	ret = xlator_init (master);
-	if (ret) {
-		gf_log ("glfs", GF_LOG_ERROR,
-			"failed to initialize gfapi translator");
-		goto err;
-	}
+    ctx->logbuf_pool = mem_pool_new(log_buf_t, GF_MEMPOOL_COUNT_OF_LRU_BUF_T);
+    if (!ctx->logbuf_pool)
+        goto err;
 
-	fs->ctx->master = master;
-	THIS = master;
+    INIT_LIST_HEAD(&pool->all_frames);
+    INIT_LIST_HEAD(&ctx->cmd_args.xlator_options);
+    INIT_LIST_HEAD(&ctx->cmd_args.volfile_servers);
 
-	return 0;
+    LOCK_INIT(&pool->lock);
+    ctx->pool = pool;
 
+    ret = 0;
 err:
-	if (master) {
-		xlator_destroy (master);
-	}
+    if (ret && pool) {
+        if (pool->frame_mem_pool)
+            mem_pool_destroy(pool->frame_mem_pool);
+        if (pool->stack_mem_pool)
+            mem_pool_destroy(pool->stack_mem_pool);
+        GF_FREE(pool);
+    }
+
+    if (ret && ctx) {
+        if (ctx->stub_mem_pool)
+            mem_pool_destroy(ctx->stub_mem_pool);
+        if (ctx->dict_pool)
+            mem_pool_destroy(ctx->dict_pool);
+        if (ctx->dict_data_pool)
+            mem_pool_destroy(ctx->dict_data_pool);
+        if (ctx->dict_pair_pool)
+            mem_pool_destroy(ctx->dict_pair_pool);
+        if (ctx->logbuf_pool)
+            mem_pool_destroy(ctx->logbuf_pool);
+    }
 
-	return -1;
+    return ret;
 }
 
+static int
+create_master(struct glfs *fs)
+{
+    int ret = 0;
+    xlator_t *master = NULL;
+
+    master = GF_CALLOC(1, sizeof(*master), glfs_mt_xlator_t);
+    if (!master)
+        goto err;
+
+    master->name = gf_strdup("gfapi");
+    if (!master->name)
+        goto err;
+
+    if (xlator_set_type(master, "mount/api") == -1) {
+        gf_smsg("glfs", GF_LOG_ERROR, 0, API_MSG_MASTER_XLATOR_INIT_FAILED,
+                "name=%s", fs->volname, NULL);
+        goto err;
+    }
+
+    master->ctx = fs->ctx;
+    master->private = fs;
+    master->options = dict_new();
+    if (!master->options)
+        goto err;
+
+    ret = xlator_init(master);
+    if (ret) {
+        gf_smsg("glfs", GF_LOG_ERROR, 0, API_MSG_GFAPI_XLATOR_INIT_FAILED,
+                NULL);
+        goto err;
+    }
+
+    fs->ctx->master = master;
+    THIS = master;
+
+    return 0;
+
+err:
+    if (master) {
+        xlator_destroy(master);
+    }
+
+    return -1;
+}
 
 static FILE *
-get_volfp (struct glfs *fs)
+get_volfp(struct glfs *fs)
 {
-	cmd_args_t  *cmd_args = NULL;
-	FILE	    *specfp = NULL;
+    cmd_args_t *cmd_args = NULL;
+    FILE *specfp = NULL;
 
-	cmd_args = &fs->ctx->cmd_args;
+    cmd_args = &fs->ctx->cmd_args;
 
-	if ((specfp = fopen (cmd_args->volfile, "r")) == NULL) {
-		gf_log ("glfs", GF_LOG_ERROR,
-			"volume file %s: %s",
-			cmd_args->volfile,
-			strerror (errno));
-		return NULL;
-	}
+    if ((specfp = fopen(cmd_args->volfile, "r")) == NULL) {
+        gf_smsg("glfs", GF_LOG_ERROR, errno, API_MSG_VOLFILE_OPEN_FAILED,
+                "file=%s", cmd_args->volfile, "err=%s", strerror(errno), NULL);
+        return NULL;
+    }
 
-	gf_log ("glfs", GF_LOG_DEBUG,
-		"loading volume file %s", cmd_args->volfile);
+    gf_msg_debug("glfs", 0, "loading volume file %s", cmd_args->volfile);
 
-	return specfp;
+    return specfp;
 }
 
-
 int
-glfs_volumes_init (struct glfs *fs)
+glfs_volumes_init(struct glfs *fs)
 {
-	FILE		   *fp = NULL;
-	cmd_args_t	   *cmd_args = NULL;
-	int		    ret = 0;
+    FILE *fp = NULL;
+    cmd_args_t *cmd_args = NULL;
+    int ret = 0;
+
+    cmd_args = &fs->ctx->cmd_args;
 
-	cmd_args = &fs->ctx->cmd_args;
+    if (!vol_assigned(cmd_args))
+        return -1;
 
-	if (!vol_assigned (cmd_args))
-		return -1;
+    if (sys_access(SECURE_ACCESS_FILE, F_OK) == 0) {
+        fs->ctx->secure_mgmt = 1;
+        fs->ctx->ssl_cert_depth = glusterfs_read_secure_access_file();
+    }
 
-	if (cmd_args->volfile_server) {
-		ret = glfs_mgmt_init (fs);
-		goto out;
-	}
+    if (cmd_args->volfile_server) {
+        ret = glfs_mgmt_init(fs);
+        goto out;
+    }
 
-	fp = get_volfp (fs);
+    fp = get_volfp(fs);
 
-	if (!fp) {
-		gf_log ("glfs", GF_LOG_ERROR,
-			"Cannot reach volume specification file");
-		ret = -1;
-		goto out;
-	}
+    if (!fp) {
+        gf_smsg("glfs", GF_LOG_ERROR, ENOENT, API_MSG_VOL_SPEC_FILE_ERROR,
+                NULL);
+        ret = -1;
+        goto out;
+    }
 
-	ret = glfs_process_volfp (fs, fp);
-	if (ret)
-		goto out;
+    ret = glfs_process_volfp(fs, fp);
+    if (ret)
+        goto out;
 
 out:
-	return ret;
+    return ret;
 }
 
-
 ///////////////////////////////////////////////////////////////////////////////
 
-
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_set_xlator_option, 3.4.0)
 int
-pub_glfs_set_xlator_option (struct glfs *fs, const char *xlator,
-                            const char *key, const char *value)
+pub_glfs_set_xlator_option(struct glfs *fs, const char *xlator, const char *key,
+                           const char *value)
 {
-	xlator_cmdline_option_t *option = NULL;
+    xlator_cmdline_option_t *option = NULL;
 
-	option = GF_CALLOC (1, sizeof (*option),
-			    glfs_mt_xlator_cmdline_option_t);
-	if (!option)
-		goto enomem;
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
 
-	INIT_LIST_HEAD (&option->cmd_args);
+    option = GF_CALLOC(1, sizeof(*option), glfs_mt_xlator_cmdline_option_t);
+    if (!option)
+        goto enomem;
 
-	option->volume = gf_strdup (xlator);
-	if (!option->volume)
-		goto enomem;
-	option->key = gf_strdup (key);
-	if (!option->key)
-		goto enomem;
-	option->value = gf_strdup (value);
-	if (!option->value)
-		goto enomem;
+    INIT_LIST_HEAD(&option->cmd_args);
 
-	list_add (&option->cmd_args, &fs->ctx->cmd_args.xlator_options);
+    option->volume = gf_strdup(xlator);
+    if (!option->volume)
+        goto enomem;
+    option->key = gf_strdup(key);
+    if (!option->key)
+        goto enomem;
+    option->value = gf_strdup(value);
+    if (!option->value)
+        goto enomem;
 
-	return 0;
-enomem:
-	errno = ENOMEM;
+    list_add(&option->cmd_args, &fs->ctx->cmd_args.xlator_options);
 
-	if (!option)
-		return -1;
+    __GLFS_EXIT_FS;
 
-	GF_FREE (option->volume);
-	GF_FREE (option->key);
-	GF_FREE (option->value);
-	GF_FREE (option);
+    return 0;
+enomem:
+    errno = ENOMEM;
 
-	return -1;
-}
+    if (!option) {
+        __GLFS_EXIT_FS;
+        return -1;
+    }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_set_xlator_option, 3.4.0);
+    GF_FREE(option->volume);
+    GF_FREE(option->key);
+    GF_FREE(option->value);
+    GF_FREE(option);
 
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return -1;
+}
+
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_unset_volfile_server, 3.5.1)
 int
-pub_glfs_unset_volfile_server (struct glfs *fs, const char *transport,
-                               const char *host, const int port)
+pub_glfs_unset_volfile_server(struct glfs *fs, const char *transport,
+                              const char *host, const int port)
 {
-        cmd_args_t       *cmd_args = NULL;
-        server_cmdline_t *server = NULL;
-        int               ret = -1;
+    cmd_args_t *cmd_args = NULL;
+    server_cmdline_t *server = NULL;
+    server_cmdline_t *tmp = NULL;
+    char *transport_val = NULL;
+    int port_val = 0;
+    int ret = -1;
+
+    if (!fs || !host) {
+        errno = EINVAL;
+        return ret;
+    }
 
-        if (!transport || !host || !port) {
-                errno = EINVAL;
-                return ret;
-        }
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
 
-        cmd_args = &fs->ctx->cmd_args;
-        list_for_each_entry(server, &cmd_args->curr_server->list, list) {
-                if ((!strcmp(server->volfile_server, host) &&
-                     !strcmp(server->transport, transport) &&
-                     (server->port == port))) {
-                        list_del (&server->list);
-                        ret = 0;
-                        goto out;
-                }
+    cmd_args = &fs->ctx->cmd_args;
+
+    if (transport) {
+        transport_val = gf_strdup(transport);
+    } else {
+        transport_val = gf_strdup(GF_DEFAULT_VOLFILE_TRANSPORT);
+    }
+
+    if (!transport_val) {
+        errno = ENOMEM;
+        goto out;
+    }
+
+    if (port) {
+        port_val = port;
+    } else {
+        port_val = GF_DEFAULT_BASE_PORT;
+    }
+
+    list_for_each_entry_safe(server, tmp, &cmd_args->curr_server->list, list)
+    {
+        if (!server->volfile_server || !server->transport)
+            continue;
+        if ((!strcmp(server->volfile_server, host) &&
+             !strcmp(server->transport, transport_val) &&
+             (server->port == port_val))) {
+            list_del(&server->list);
+            ret = 0;
+            goto out;
         }
+    }
 
 out:
-        return ret;
-}
-
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_unset_volfile_server, 3.5.1);
+    GF_FREE(transport_val);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_set_volfile_server, 3.4.0)
 int
-pub_glfs_set_volfile_server (struct glfs *fs, const char *transport,
-                             const char *host, int port)
+pub_glfs_set_volfile_server(struct glfs *fs, const char *transport,
+                            const char *host, int port)
 {
-        cmd_args_t            *cmd_args = NULL;
-        server_cmdline_t      *server = NULL;
-        server_cmdline_t      *tmp = NULL;
-        int                    ret = -1;
+    cmd_args_t *cmd_args = NULL;
+    int ret = -1;
+    char *server_host = NULL;
+    char *server_transport = NULL;
 
-        if (!transport || !host) {
-                errno = EINVAL;
-                return ret;
-        }
+    if (!fs || !host) {
+        errno = EINVAL;
+        return ret;
+    }
 
-        cmd_args = &fs->ctx->cmd_args;
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
 
-        cmd_args->max_connect_attempts = 1;
+    cmd_args = &fs->ctx->cmd_args;
+    cmd_args->max_connect_attempts = 1;
 
-        server = GF_CALLOC (1, sizeof (server_cmdline_t),
-                            glfs_mt_server_cmdline_t);
+    server_host = gf_strdup(host);
+    if (!server_host) {
+        errno = ENOMEM;
+        goto out;
+    }
 
-        if (!server) {
-                errno = ENOMEM;
-                goto out;
+    if (transport) {
+        /* volfile fetch support over tcp|unix only */
+        if (!strcmp(transport, "tcp") || !strcmp(transport, "unix")) {
+            server_transport = gf_strdup(transport);
+        } else if (!strcmp(transport, "rdma")) {
+            server_transport = gf_strdup(GF_DEFAULT_VOLFILE_TRANSPORT);
+            gf_smsg("glfs", GF_LOG_WARNING, EINVAL, API_MSG_TRANS_RDMA_DEP,
+                    NULL);
+        } else {
+            gf_smsg("glfs", GF_LOG_TRACE, EINVAL, API_MSG_TRANS_NOT_SUPPORTED,
+                    "transport=%s", transport, NULL);
+            goto out;
         }
+    } else {
+        server_transport = gf_strdup(GF_DEFAULT_VOLFILE_TRANSPORT);
+    }
 
-        INIT_LIST_HEAD (&server->list);
+    if (!server_transport) {
+        errno = ENOMEM;
+        goto out;
+    }
 
-        server->volfile_server = gf_strdup (host);
-        if (!server->volfile_server) {
-                errno = ENOMEM;
-                goto out;
-        }
+    if (!port) {
+        port = GF_DEFAULT_BASE_PORT;
+    }
 
-        server->transport = gf_strdup (transport);
-        if (!server->transport) {
-                errno = ENOMEM;
-                goto out;
-        }
+    if (!strcmp(server_transport, "unix")) {
+        port = 0;
+    }
 
-        server->port = port;
+    ret = gf_set_volfile_server_common(cmd_args, server_host, server_transport,
+                                       port);
+    if (ret) {
+        gf_log("glfs", GF_LOG_ERROR, "failed to set volfile server: %s",
+               strerror(errno));
+    }
 
-        if (!cmd_args->volfile_server) {
-                cmd_args->volfile_server = server->volfile_server;
-                cmd_args->volfile_server_transport = server->transport;
-                cmd_args->volfile_server_port = server->port;
-                cmd_args->curr_server = server;
-        }
+out:
+    if (server_host) {
+        GF_FREE(server_host);
+    }
 
-        list_for_each_entry(tmp, &cmd_args->volfile_servers, list) {
-                if ((!strcmp(tmp->volfile_server, host) &&
-                     !strcmp(tmp->transport, transport) &&
-                     (tmp->port == port))) {
-                        errno = EEXIST;
-                        ret = -1;
-                        goto out;
-                }
-        }
+    if (server_transport) {
+        GF_FREE(server_transport);
+    }
 
-        list_add_tail (&server->list, &cmd_args->volfile_servers);
+    __GLFS_EXIT_FS;
 
-        ret = 0;
+invalid_fs:
+    return ret;
+}
+
+/* *
+ * Used to free the arguments allocated by glfs_set_volfile_server()
+ */
+static void
+glfs_free_volfile_servers(cmd_args_t *cmd_args)
+{
+    server_cmdline_t *server = NULL;
+    server_cmdline_t *tmp = NULL;
+
+    GF_VALIDATE_OR_GOTO(THIS->name, cmd_args, out);
+
+    list_for_each_entry_safe(server, tmp, &cmd_args->volfile_servers, list)
+    {
+        list_del_init(&server->list);
+        GF_FREE(server->volfile_server);
+        GF_FREE(server->transport);
+        GF_FREE(server);
+    }
+    cmd_args->curr_server = NULL;
 out:
-        if (ret == -1) {
-                if (server) {
-                        GF_FREE (server->volfile_server);
-                        GF_FREE (server->transport);
-                        GF_FREE (server);
-                }
-        }
+    return;
+}
 
-        return ret;
+static void
+glfs_free_xlator_options(cmd_args_t *cmd_args)
+{
+    xlator_cmdline_option_t *xo = NULL;
+    xlator_cmdline_option_t *tmp_xo = NULL;
+
+    if (!&(cmd_args->xlator_options))
+        return;
+
+    list_for_each_entry_safe(xo, tmp_xo, &cmd_args->xlator_options, cmd_args)
+    {
+        list_del_init(&xo->cmd_args);
+        GF_FREE(xo->volume);
+        GF_FREE(xo->key);
+        GF_FREE(xo->value);
+        GF_FREE(xo);
+    }
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_set_volfile_server, 3.4.0);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_setfsuid, 3.4.2)
+int
+pub_glfs_setfsuid(uid_t fsuid)
+{
+    /* TODO:
+     * - Set the THIS and restore it appropriately
+     */
+    return syncopctx_setfsuid(&fsuid);
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_setfsgid, 3.4.2)
+int
+pub_glfs_setfsgid(gid_t fsgid)
+{
+    /* TODO:
+     * - Set the THIS and restore it appropriately
+     */
+    return syncopctx_setfsgid(&fsgid);
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_setfsgroups, 3.4.2)
 int
-pub_glfs_setfsuid (uid_t fsuid)
+pub_glfs_setfsgroups(size_t size, const gid_t *list)
 {
-	return syncopctx_setfsuid (&fsuid);
+    /* TODO:
+     * - Set the THIS and restore it appropriately
+     */
+    return syncopctx_setfsgroups(size, list);
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_setfsuid, 3.4.2);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_setfsleaseid, 4.0.0)
+int
+pub_glfs_setfsleaseid(glfs_leaseid_t leaseid)
+{
+    int ret = -1;
+    char *gleaseid = NULL;
+
+    gleaseid = gf_leaseid_get();
+    if (gleaseid) {
+        if (leaseid)
+            memcpy(gleaseid, leaseid, LEASE_ID_SIZE);
+        else /* reset leaseid */
+            memset(gleaseid, 0, LEASE_ID_SIZE);
+        ret = 0;
+    }
 
+    if (ret)
+        gf_log("glfs", GF_LOG_ERROR, "failed to set leaseid: %s",
+               strerror(errno));
+    return ret;
+}
 
 int
-pub_glfs_setfsgid (gid_t fsgid)
+get_fop_attr_glfd(dict_t **fop_attr, struct glfs_fd *glfd)
 {
-	return syncopctx_setfsgid (&fsgid);
+    char *leaseid = NULL;
+    int ret = 0;
+    gf_boolean_t dict_create = _gf_false;
+
+    leaseid = GF_MALLOC(LEASE_ID_SIZE, gf_common_mt_char);
+    GF_CHECK_ALLOC_AND_LOG("gfapi", leaseid, ret, "lease id alloc failed", out);
+    memcpy(leaseid, glfd->lease_id, LEASE_ID_SIZE);
+    if (*fop_attr == NULL) {
+        *fop_attr = dict_new();
+        dict_create = _gf_true;
+    }
+    GF_CHECK_ALLOC_AND_LOG("gfapi", *fop_attr, ret, "dict_new failed", out);
+    ret = dict_set_bin(*fop_attr, "lease-id", leaseid, LEASE_ID_SIZE);
+out:
+    if (ret) {
+        GF_FREE(leaseid);
+        if (dict_create) {
+            if (*fop_attr)
+                dict_unref(*fop_attr);
+            *fop_attr = NULL;
+        }
+    }
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_setfsgid, 3.4.2);
+int
+set_fop_attr_glfd(struct glfs_fd *glfd)
+{
+    char *lease_id = NULL;
+    int ret = -1;
 
+    lease_id = gf_existing_leaseid();
+    if (lease_id) {
+        memcpy(glfd->lease_id, lease_id, LEASE_ID_SIZE);
+        ret = 0;
+    }
+    return ret;
+}
 
 int
-pub_glfs_setfsgroups (size_t size, const gid_t *list)
+get_fop_attr_thrd_key(dict_t **fop_attr)
 {
-	return syncopctx_setfsgroups(size, list);
-}
+    char *existing_leaseid = NULL, *leaseid = NULL;
+    int ret = 0;
+    gf_boolean_t dict_create = _gf_false;
+
+    existing_leaseid = gf_existing_leaseid();
+    if (existing_leaseid) {
+        leaseid = GF_MALLOC(LEASE_ID_SIZE, gf_common_mt_char);
+        GF_CHECK_ALLOC_AND_LOG("gfapi", leaseid, ret, "lease id alloc failed",
+                               out);
+        memcpy(leaseid, existing_leaseid, LEASE_ID_SIZE);
+        if (*fop_attr == NULL) {
+            *fop_attr = dict_new();
+            dict_create = _gf_true;
+        }
+        GF_CHECK_ALLOC_AND_LOG("gfapi", *fop_attr, ret, "dict_new failed", out);
+        ret = dict_set_bin(*fop_attr, "lease-id", leaseid, LEASE_ID_SIZE);
+    }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_setfsgroups, 3.4.2);
+out:
+    if (ret) {
+        GF_FREE(leaseid);
+        if (dict_create) {
+            if (*fop_attr)
+                dict_unref(*fop_attr);
+            *fop_attr = NULL;
+        }
+    }
+    return ret;
+}
 
+void
+unset_fop_attr(dict_t **fop_attr)
+{
+    char *lease_id = NULL;
+    lease_id = gf_existing_leaseid();
+    if (lease_id)
+        memset(lease_id, 0, LEASE_ID_SIZE);
+    if (*fop_attr) {
+        dict_unref(*fop_attr);
+        *fop_attr = NULL;
+    }
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_from_glfd, 3.4.0)
 struct glfs *
-pub_glfs_from_glfd (struct glfs_fd *glfd)
+pub_glfs_from_glfd(struct glfs_fd *glfd)
 {
-	return glfd->fs;
+    if (glfd == NULL) {
+        errno = EBADF;
+        return NULL;
+    }
+
+    return glfd->fs;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_from_glfd, 3.4.0);
+static void
+glfs_fd_destroy(struct glfs_fd *glfd)
+{
+    if (!glfd)
+        return;
+
+    glfs_lock(glfd->fs, _gf_true);
+    {
+        list_del_init(&glfd->openfds);
+    }
+    glfs_unlock(glfd->fs);
+
+    if (glfd->fd) {
+        fd_unref(glfd->fd);
+        glfd->fd = NULL;
+    }
+
+    GF_FREE(glfd->readdirbuf);
 
+    GF_FREE(glfd);
+}
 
 struct glfs_fd *
-glfs_fd_new (struct glfs *fs)
+glfs_fd_new(struct glfs *fs)
 {
-	struct glfs_fd  *glfd = NULL;
+    struct glfs_fd *glfd = NULL;
 
-	glfd = GF_CALLOC (1, sizeof (*glfd), glfs_mt_glfs_fd_t);
-	if (!glfd)
-		return NULL;
+    glfd = GF_CALLOC(1, sizeof(*glfd), glfs_mt_glfs_fd_t);
+    if (!glfd)
+        return NULL;
 
-	glfd->fs = fs;
+    glfd->fs = fs;
 
-	INIT_LIST_HEAD (&glfd->openfds);
+    INIT_LIST_HEAD(&glfd->openfds);
 
-	return glfd;
-}
+    GF_REF_INIT(glfd, glfs_fd_destroy);
 
+    return glfd;
+}
 
 void
-glfs_fd_bind (struct glfs_fd *glfd)
+glfs_fd_bind(struct glfs_fd *glfd)
 {
-	struct glfs *fs = NULL;
+    struct glfs *fs = NULL;
 
-	fs = glfd->fs;
+    fs = glfd->fs;
 
-	glfs_lock (fs);
-	{
-		list_add_tail (&glfd->openfds, &fs->openfds);
-	}
-	glfs_unlock (fs);
+    glfs_lock(fs, _gf_true);
+    {
+        list_add_tail(&glfd->openfds, &fs->openfds);
+    }
+    glfs_unlock(fs);
 }
 
-void
-glfs_fd_destroy (struct glfs_fd *glfd)
+static void *
+glfs_poller(void *data)
 {
-	if (!glfd)
-		return;
+    struct glfs *fs = NULL;
+
+    fs = data;
+
+    gf_event_dispatch(fs->ctx->event_pool);
+
+    return NULL;
+}
+
+static struct glfs *
+glfs_new_fs(const char *volname)
+{
+    struct glfs *fs = NULL;
+
+    fs = CALLOC(1, sizeof(*fs));
+    if (!fs)
+        return NULL;
+
+    INIT_LIST_HEAD(&fs->openfds);
+    INIT_LIST_HEAD(&fs->upcall_list);
+    INIT_LIST_HEAD(&fs->waitq);
+
+    PTHREAD_MUTEX_INIT(&fs->mutex, NULL, fs->pthread_flags, GLFS_INIT_MUTEX,
+                       err);
+
+    PTHREAD_COND_INIT(&fs->cond, NULL, fs->pthread_flags, GLFS_INIT_COND, err);
 
-	glfs_lock (glfd->fs);
-	{
-		list_del_init (&glfd->openfds);
-	}
-	glfs_unlock (glfd->fs);
+    PTHREAD_COND_INIT(&fs->child_down_cond, NULL, fs->pthread_flags,
+                      GLFS_INIT_COND_CHILD, err);
 
-	if (glfd->fd)
-		fd_unref (glfd->fd);
+    PTHREAD_MUTEX_INIT(&fs->upcall_list_mutex, NULL, fs->pthread_flags,
+                       GLFS_INIT_MUTEX_UPCALL, err);
 
-	GF_FREE (glfd->readdirbuf);
+    fs->volname = strdup(volname);
+    if (!fs->volname)
+        goto err;
 
-	GF_FREE (glfd);
+    fs->pin_refcnt = 0;
+    fs->upcall_events = 0;
+    fs->up_cbk = NULL;
+    fs->up_data = NULL;
+
+    return fs;
+
+err:
+    glfs_free_from_ctx(fs);
+    return NULL;
 }
 
+extern xlator_t global_xlator;
+extern glusterfs_ctx_t *global_ctx;
+extern pthread_mutex_t global_ctx_mutex;
 
-static void *
-glfs_poller (void *data)
+static int
+glfs_init_global_ctx()
 {
-	struct glfs  *fs = NULL;
+    int ret = 0;
+    glusterfs_ctx_t *ctx = NULL;
+
+    pthread_mutex_lock(&global_ctx_mutex);
+    {
+        if (global_xlator.ctx)
+            goto unlock;
+
+        ctx = glusterfs_ctx_new();
+        if (!ctx) {
+            ret = -1;
+            goto unlock;
+        }
 
-	fs = data;
+        gf_log_globals_init(ctx, GF_LOG_NONE);
 
-	event_dispatch (fs->ctx->event_pool);
+        global_ctx = ctx;
+        global_xlator.ctx = global_ctx;
 
-	return NULL;
+        ret = glusterfs_ctx_defaults_init(ctx);
+        if (ret) {
+            global_ctx = NULL;
+            global_xlator.ctx = NULL;
+            goto unlock;
+        }
+    }
+unlock:
+    pthread_mutex_unlock(&global_ctx_mutex);
+
+    if (ret)
+        FREE(ctx);
+
+    return ret;
 }
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_new, 3.4.0)
+struct glfs *
+pub_glfs_new(const char *volname)
+{
+    if (!volname) {
+        errno = EINVAL;
+        return NULL;
+    }
+
+    struct glfs *fs = NULL;
+    int i = 0;
+    int ret = -1;
+    glusterfs_ctx_t *ctx = NULL;
+    xlator_t *old_THIS = NULL;
+    char pname[16] = "";
+    char msg[32] = "";
+
+    if (volname[0] == '/' || volname[0] == '-') {
+        if (strncmp(volname, "/snaps/", 7) == 0) {
+            goto label;
+        }
+        errno = EINVAL;
+        return NULL;
+    }
+
+    for (i = 0; i < strlen(volname); i++) {
+        if (!isalnum(volname[i]) && (volname[i] != '_') &&
+            (volname[i] != '-')) {
+            errno = EINVAL;
+            return NULL;
+        }
+    }
+
+label:
+    /*
+     * Do this as soon as possible in case something else depends on
+     * pool allocations.
+     */
+    mem_pools_init();
+
+    fs = glfs_new_fs(volname);
+    if (!fs)
+        goto out;
+
+    ctx = glusterfs_ctx_new();
+    if (!ctx)
+        goto out;
+
+    /* first globals init, for gf_mem_acct_enable_set () */
 
+    ret = glusterfs_globals_init(ctx);
+    if (ret)
+        goto out;
+
+    old_THIS = THIS;
+    ret = glfs_init_global_ctx();
+    if (ret)
+        goto out;
+
+    /* then ctx_defaults_init, for xlator_mem_acct_init(THIS) */
+
+    ret = glusterfs_ctx_defaults_init(ctx);
+    if (ret)
+        goto out;
+
+    fs->ctx = ctx;
+    fs->ctx->process_mode = GF_CLIENT_PROCESS;
+
+    ret = glfs_set_logging(fs, "/dev/null", 0);
+    if (ret)
+        goto out;
+
+    fs->ctx->cmd_args.volfile_id = gf_strdup(volname);
+    if (!(fs->ctx->cmd_args.volfile_id)) {
+        ret = -1;
+        goto out;
+    }
+
+    ret = -1;
+#ifdef GF_LINUX_HOST_OS
+    ret = prctl(PR_GET_NAME, (unsigned long)pname, 0, 0, 0);
+#endif
+    if (ret)
+        fs->ctx->cmd_args.process_name = gf_strdup("gfapi");
+    else {
+        snprintf(msg, sizeof(msg), "gfapi.%s", pname);
+        fs->ctx->cmd_args.process_name = gf_strdup(msg);
+    }
+    ret = 0;
+
+out:
+    if (ret) {
+        if (fs) {
+            glfs_fini(fs);
+            fs = NULL;
+        } else {
+            /* glfs_fini() calls mem_pools_fini() too */
+            mem_pools_fini();
+        }
+    }
+
+    if (old_THIS)
+        THIS = old_THIS;
+
+    return fs;
+}
+
+GFAPI_SYMVER_PRIVATE_DEFAULT(glfs_new_from_ctx, 3.7.0)
 struct glfs *
-pub_glfs_new (const char *volname)
+priv_glfs_new_from_ctx(glusterfs_ctx_t *ctx)
 {
-	struct glfs     *fs = NULL;
-	int              ret = -1;
-	glusterfs_ctx_t *ctx = NULL;
+    struct glfs *fs = NULL;
 
-	ctx = glusterfs_ctx_new ();
-	if (!ctx) {
-		return NULL;
-	}
+    if (!ctx)
+        goto out;
 
-	/* first globals init, for gf_mem_acct_enable_set () */
-	ret = glusterfs_globals_init (ctx);
-	if (ret)
-		return NULL;
+    fs = glfs_new_fs("");
+    if (!fs)
+        goto out;
 
-	THIS->ctx = ctx;
+    fs->ctx = ctx;
 
-	/* then ctx_defaults_init, for xlator_mem_acct_init(THIS) */
-	ret = glusterfs_ctx_defaults_init (ctx);
-	if (ret)
-		return NULL;
+out:
+    return fs;
+}
+
+GFAPI_SYMVER_PRIVATE_DEFAULT(glfs_free_from_ctx, 3.7.0)
+void
+priv_glfs_free_from_ctx(struct glfs *fs)
+{
+    upcall_entry *u_list = NULL;
+    upcall_entry *tmp = NULL;
+
+    if (!fs)
+        return;
+
+    /* cleanup upcall structures */
+    list_for_each_entry_safe(u_list, tmp, &fs->upcall_list, upcall_list)
+    {
+        list_del_init(&u_list->upcall_list);
+        GF_FREE(u_list->upcall_data.data);
+        GF_FREE(u_list);
+    }
 
-	fs = GF_CALLOC (1, sizeof (*fs), glfs_mt_glfs_t);
-	if (!fs)
-		return NULL;
-	fs->ctx = ctx;
+    PTHREAD_MUTEX_DESTROY(&fs->mutex, fs->pthread_flags, GLFS_INIT_MUTEX);
 
-	glfs_set_logging (fs, "/dev/null", 0);
+    PTHREAD_COND_DESTROY(&fs->cond, fs->pthread_flags, GLFS_INIT_COND);
 
-	fs->ctx->cmd_args.volfile_id = gf_strdup (volname);
+    PTHREAD_COND_DESTROY(&fs->child_down_cond, fs->pthread_flags,
+                         GLFS_INIT_COND_CHILD);
 
-	fs->volname = gf_strdup (volname);
+    PTHREAD_MUTEX_DESTROY(&fs->upcall_list_mutex, fs->pthread_flags,
+                          GLFS_INIT_MUTEX_UPCALL);
 
-	pthread_mutex_init (&fs->mutex, NULL);
-	pthread_cond_init (&fs->cond, NULL);
+    if (fs->oldvolfile)
+        FREE(fs->oldvolfile);
 
-	INIT_LIST_HEAD (&fs->openfds);
+    FREE(fs->volname);
 
-	return fs;
+    FREE(fs);
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_new, 3.4.0);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_set_volfile, 3.4.0)
+int
+pub_glfs_set_volfile(struct glfs *fs, const char *volfile)
+{
+    cmd_args_t *cmd_args = NULL;
+
+    cmd_args = &fs->ctx->cmd_args;
 
+    if (vol_assigned(cmd_args))
+        return -1;
 
-struct glfs *
-priv_glfs_new_from_ctx (glusterfs_ctx_t *ctx)
+    cmd_args->volfile = gf_strdup(volfile);
+    if (!cmd_args->volfile)
+        return -1;
+    return 0;
+}
+
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_set_logging, 3.4.0)
+int
+pub_glfs_set_logging(struct glfs *fs, const char *logfile, int loglevel)
 {
-        struct glfs    *fs = NULL;
+    int ret = -1;
+    char *tmplog = NULL;
 
-        if (!ctx)
-                return NULL;
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
 
-        fs = GF_CALLOC (1, sizeof (*fs), glfs_mt_glfs_t);
-        if (!fs)
-                return NULL;
-        fs->ctx = ctx;
+    if (!logfile) {
+        ret = gf_set_log_file_path(&fs->ctx->cmd_args, fs->ctx);
+        if (ret)
+            goto out;
+        tmplog = fs->ctx->cmd_args.log_file;
+    } else {
+        tmplog = (char *)logfile;
+    }
 
-        (void) pthread_cond_init (&fs->cond, NULL);
+    /* finish log set parameters before init */
+    if (loglevel >= 0)
+        gf_log_set_loglevel(fs->ctx, loglevel);
 
-        (void) pthread_mutex_init (&fs->mutex, NULL);
+    ret = gf_log_init(fs->ctx, tmplog, NULL);
+    if (ret)
+        goto out;
 
-        INIT_LIST_HEAD (&fs->openfds);
+    ret = gf_log_inject_timer_event(fs->ctx);
+    if (ret)
+        goto out;
 
-        return fs;
+out:
+    __GLFS_EXIT_FS;
+
+invalid_fs:
+    return ret;
 }
 
-GFAPI_SYMVER_PRIVATE_DEFAULT(glfs_new_from_ctx, 3.7.0);
+int
+glfs_init_wait(struct glfs *fs)
+{
+    int ret = -1;
+
+    /* Always a top-down call, use glfs_lock() */
+    glfs_lock(fs, _gf_true);
+    {
+        while (!fs->init)
+            pthread_cond_wait(&fs->cond, &fs->mutex);
+        ret = fs->ret;
+        errno = fs->err;
+    }
+    glfs_unlock(fs);
 
+    return ret;
+}
 
+GFAPI_SYMVER_PRIVATE_DEFAULT(glfs_init_done, 3.4.0)
 void
-priv_glfs_free_from_ctx (struct glfs *fs)
+priv_glfs_init_done(struct glfs *fs, int ret)
 {
-        if (!fs)
-                return;
+    glfs_init_cbk init_cbk;
 
-        (void) pthread_cond_destroy (&fs->cond);
+    if (!fs) {
+        gf_smsg("glfs", GF_LOG_ERROR, EINVAL, API_MSG_GLFS_FSOBJ_NULL, NULL);
+        goto out;
+    }
 
-        (void) pthread_mutex_destroy (&fs->mutex);
+    init_cbk = fs->init_cbk;
 
-        GF_FREE (fs);
-}
+    /* Always a bottom-up call, use mutex_lock() */
+    pthread_mutex_lock(&fs->mutex);
+    {
+        fs->init = 1;
+        fs->ret = ret;
+        fs->err = errno;
 
-GFAPI_SYMVER_PRIVATE_DEFAULT(glfs_free_from_ctx, 3.7.0);
+        if (!init_cbk)
+            pthread_cond_broadcast(&fs->cond);
+    }
+    pthread_mutex_unlock(&fs->mutex);
 
+    if (init_cbk)
+        init_cbk(fs, ret);
+out:
+    return;
+}
 
 int
-pub_glfs_set_volfile (struct glfs *fs, const char *volfile)
+glfs_init_common(struct glfs *fs)
 {
-	cmd_args_t  *cmd_args = NULL;
+    int ret = -1;
 
-	cmd_args = &fs->ctx->cmd_args;
+    ret = create_master(fs);
+    if (ret)
+        return ret;
 
-	if (vol_assigned (cmd_args))
-		return -1;
+    ret = gf_thread_create(&fs->poller, NULL, glfs_poller, fs, "glfspoll");
+    if (ret)
+        return ret;
 
-	cmd_args->volfile = gf_strdup (volfile);
+    ret = glfs_volumes_init(fs);
+    if (ret)
+        return ret;
 
-	return 0;
+    fs->dev_id = gf_dm_hashfn(fs->volname, strlen(fs->volname));
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_set_volfile, 3.4.0);
+int
+glfs_init_async(struct glfs *fs, glfs_init_cbk cbk)
+{
+    int ret = -1;
+
+    if (!fs || !fs->ctx) {
+        gf_smsg("glfs", GF_LOG_ERROR, EINVAL, API_MSG_FS_NOT_INIT, NULL);
+        errno = EINVAL;
+        return ret;
+    }
+
+    fs->init_cbk = cbk;
+
+    ret = glfs_init_common(fs);
 
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_init, 3.4.0)
 int
-pub_glfs_set_logging (struct glfs *fs, const char *logfile, int loglevel)
+pub_glfs_init(struct glfs *fs)
 {
-        int  ret = 0;
-        char *tmplog = NULL;
+    int ret = -1;
 
-        if (!logfile) {
-                ret = gf_set_log_file_path (&fs->ctx->cmd_args);
-                if (ret)
-                        goto out;
-                tmplog = fs->ctx->cmd_args.log_file;
-        } else {
-                tmplog = (char *)logfile;
-        }
+    DECLARE_OLD_THIS;
 
-        /* finish log set parameters before init */
-        if (loglevel >= 0)
-                gf_log_set_loglevel (loglevel);
+    if (!fs || !fs->ctx) {
+        gf_smsg("glfs", GF_LOG_ERROR, EINVAL, API_MSG_FS_NOT_INIT, NULL);
+        errno = EINVAL;
+        return ret;
+    }
 
-        ret = gf_log_init (fs->ctx, tmplog, NULL);
-        if (ret)
-                goto out;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
 
-        ret = gf_log_inject_timer_event (fs->ctx);
-        if (ret)
-                goto out;
+    ret = glfs_init_common(fs);
+    if (ret)
+        goto out;
 
+    ret = glfs_init_wait(fs);
 out:
-        return ret;
+    __GLFS_EXIT_FS;
+
+    /* Set the initial current working directory to "/" */
+    if (ret >= 0) {
+        ret = glfs_chdir(fs, "/");
+    }
+
+invalid_fs:
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_set_logging, 3.4.0);
+static int
+glusterfs_ctx_destroy(glusterfs_ctx_t *ctx)
+{
+    call_pool_t *pool = NULL;
+    int ret = 0;
+    glusterfs_graph_t *trav_graph = NULL;
+    glusterfs_graph_t *tmp = NULL;
+
+    if (ctx == NULL)
+        return 0;
+
+    if (ctx->cmd_args.curr_server)
+        glfs_free_volfile_servers(&ctx->cmd_args);
+
+    glfs_free_xlator_options(&ctx->cmd_args);
+
+    /* For all the graphs, crawl through the xlator_t structs and free
+     * all its members except for the mem_acct member,
+     * as GF_FREE will be referencing it.
+     */
+    list_for_each_entry_safe(trav_graph, tmp, &ctx->graphs, list)
+    {
+        xlator_tree_free_members(trav_graph->first);
+    }
+
+    /* Free the memory pool */
+    if (ctx->stub_mem_pool)
+        mem_pool_destroy(ctx->stub_mem_pool);
+    if (ctx->dict_pool)
+        mem_pool_destroy(ctx->dict_pool);
+    if (ctx->dict_data_pool)
+        mem_pool_destroy(ctx->dict_data_pool);
+    if (ctx->dict_pair_pool)
+        mem_pool_destroy(ctx->dict_pair_pool);
+    if (ctx->logbuf_pool)
+        mem_pool_destroy(ctx->logbuf_pool);
+
+    pool = ctx->pool;
+    if (pool) {
+        if (pool->frame_mem_pool)
+            mem_pool_destroy(pool->frame_mem_pool);
+        if (pool->stack_mem_pool)
+            mem_pool_destroy(pool->stack_mem_pool);
+        LOCK_DESTROY(&pool->lock);
+        GF_FREE(pool);
+    }
+
+    /* Free the event pool */
+    ret = gf_event_pool_destroy(ctx->event_pool);
+
+    /* Free the iobuf pool */
+    iobuf_pool_destroy(ctx->iobuf_pool);
+
+    GF_FREE(ctx->process_uuid);
+    GF_FREE(ctx->cmd_args.volfile_id);
+    GF_FREE(ctx->cmd_args.process_name);
+
+    LOCK_DESTROY(&ctx->lock);
+    pthread_mutex_destroy(&ctx->notify_lock);
+    pthread_cond_destroy(&ctx->notify_cond);
 
+    /* Free all the graph structs and its containing xlator_t structs
+     * from this point there should be no reference to GF_FREE/GF_CALLOC
+     * as it will try to access mem_acct and the below function would
+     * have freed the same.
+     */
+    list_for_each_entry_safe(trav_graph, tmp, &ctx->graphs, list)
+    {
+        glusterfs_graph_destroy_residual(trav_graph);
+    }
+
+    GF_FREE(ctx->statedump_path);
+    FREE(ctx);
+
+    return ret;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fini, 3.4.0)
 int
-glfs_init_wait (struct glfs *fs)
+pub_glfs_fini(struct glfs *fs)
 {
-	int   ret = -1;
+    int ret = -1;
+    int countdown = 100;
+    xlator_t *subvol = NULL;
+    glusterfs_ctx_t *ctx = NULL;
+    glusterfs_graph_t *graph = NULL;
+    call_pool_t *call_pool = NULL;
+    int fs_init = 0;
+    int err = -1;
+    struct synctask *waittask = NULL;
+
+    DECLARE_OLD_THIS;
+
+    if (!fs) {
+        errno = EINVAL;
+        goto invalid_fs;
+    }
 
-	/* Always a top-down call, use glfs_lock() */
-	glfs_lock (fs);
-	{
-		while (!fs->init)
-			pthread_cond_wait (&fs->cond,
-					   &fs->mutex);
-		ret = fs->ret;
-		errno = fs->err;
-	}
-	glfs_unlock (fs);
+    ctx = fs->ctx;
+    if (!ctx) {
+        goto free_fs;
+    }
 
-	return ret;
-}
+    THIS = fs->ctx->master;
 
+    if (ctx->mgmt) {
+        rpc_clnt_disable(ctx->mgmt);
+    }
 
-void
-priv_glfs_init_done (struct glfs *fs, int ret)
-{
-	glfs_init_cbk init_cbk;
+    call_pool = fs->ctx->pool;
 
-	if (!fs) {
-		gf_log ("glfs", GF_LOG_ERROR,
-			"fs is NULL");
-		goto out;
-	}
+    /* Wake up any suspended synctasks */
+    while (!list_empty(&fs->waitq)) {
+        waittask = list_entry(fs->waitq.next, struct synctask, waitq);
+        list_del_init(&waittask->waitq);
+        synctask_wake(waittask);
+    }
 
-	init_cbk = fs->init_cbk;
+    while (countdown--) {
+        /* give some time for background frames to finish */
+        pthread_mutex_lock(&fs->mutex);
+        {
+            /* Do we need to increase countdown? */
+            if ((!call_pool->cnt) && (!fs->pin_refcnt)) {
+                gf_msg_trace("glfs", 0,
+                             "call_pool_cnt - %" PRId64
+                             ","
+                             "pin_refcnt - %d",
+                             call_pool->cnt, fs->pin_refcnt);
+
+                ctx->cleanup_started = 1;
+                pthread_mutex_unlock(&fs->mutex);
+                break;
+            }
+        }
+        pthread_mutex_unlock(&fs->mutex);
+        gf_nanosleep(100000 * GF_US_IN_NS);
+    }
 
-	/* Always a bottom-up call, use mutex_lock() */
-	pthread_mutex_lock (&fs->mutex);
-	{
-		fs->init = 1;
-		fs->ret = ret;
-		fs->err = errno;
+    /* leaked frames may exist, we ignore */
 
-		if (!init_cbk)
-			pthread_cond_broadcast (&fs->cond);
-	}
-	pthread_mutex_unlock (&fs->mutex);
+    /*We deem glfs_fini as successful if there are no pending frames in the call
+     *pool*/
+    ret = (call_pool->cnt == 0) ? 0 : -1;
 
-	if (init_cbk)
-		init_cbk (fs, ret);
-out:
-	return;
+    pthread_mutex_lock(&fs->mutex);
+    {
+        fs_init = fs->init;
+    }
+    pthread_mutex_unlock(&fs->mutex);
+
+    if (fs_init != 0) {
+        subvol = glfs_active_subvol(fs);
+        if (subvol) {
+            /* PARENT_DOWN within glfs_subvol_done() is issued
+               only on graph switch (new graph should activiate
+               and decrement the extra @winds count taken in
+               glfs_graph_setup()
+
+               Since we are explicitly destroying,
+               PARENT_DOWN is necessary
+            */
+            xlator_notify(subvol, GF_EVENT_PARENT_DOWN, subvol, 0);
+            /* Here we wait for GF_EVENT_CHILD_DOWN before exiting,
+               in case of asynchrnous cleanup
+            */
+            graph = subvol->graph;
+            err = pthread_mutex_lock(&fs->mutex);
+            if (err != 0) {
+                gf_smsg("glfs", GF_LOG_ERROR, err, API_MSG_FSMUTEX_LOCK_FAILED,
+                        "error=%s", strerror(err), NULL);
+                goto fail;
+            }
+            /* check and wait for CHILD_DOWN for active subvol*/
+            {
+                while (graph->used) {
+                    err = pthread_cond_wait(&fs->child_down_cond, &fs->mutex);
+                    if (err != 0)
+                        gf_smsg("glfs", GF_LOG_INFO, err,
+                                API_MSG_COND_WAIT_FAILED, "name=%s",
+                                subvol->name, "err=%s", strerror(err), NULL);
+                }
+            }
+
+            err = pthread_mutex_unlock(&fs->mutex);
+            if (err != 0) {
+                gf_smsg("glfs", GF_LOG_ERROR, err,
+                        API_MSG_FSMUTEX_UNLOCK_FAILED, "error=%s",
+                        strerror(err), NULL);
+                goto fail;
+            }
+        }
+        glfs_subvol_done(fs, subvol);
+    }
+
+    ctx->cleanup_started = 1;
+
+    if (fs_init != 0) {
+        /* Destroy all the inode tables of all the graphs.
+         * NOTE:
+         * - inode objects should be destroyed before calling fini()
+         *   of each xlator, as fini() and forget() of the xlators
+         *   can share few common locks or data structures, calling
+         *   fini first might destroy those required by forget
+         *   ( eg: in quick-read)
+         * - The call to inode_table_destroy_all is not required when
+         *   the cleanup during graph switch is implemented to perform
+         *   inode table destroy.
+         */
+        inode_table_destroy_all(ctx);
+
+        /* Call fini() of all the xlators in the active graph
+         * NOTE:
+         * - xlator fini() should be called before destroying any of
+         *   the threads. (eg: fini() in protocol-client uses timer
+         *   thread) */
+        glusterfs_graph_deactivate(ctx->active);
+
+        /* Join the syncenv_processor threads and cleanup
+         * syncenv resources*/
+        syncenv_destroy(ctx->env);
+
+        /* Join the poller thread */
+        if (gf_event_dispatch_destroy(ctx->event_pool) < 0)
+            ret = -1;
+    }
+
+    /* Avoid dispatching events to mgmt after freed,
+     * unreference mgmt after the event_dispatch_destroy */
+    if (ctx->mgmt) {
+        rpc_clnt_unref(ctx->mgmt);
+        ctx->mgmt = NULL;
+    }
+
+    /* log infra has to be brought down before destroying
+     * timer registry, as logging uses timer infra
+     */
+    if (gf_log_fini(ctx) != 0)
+        ret = -1;
+
+    /* Join the timer thread */
+    if (fs_init != 0) {
+        gf_timer_registry_destroy(ctx);
+    }
+
+    /* Destroy the context and the global pools */
+    if (glusterfs_ctx_destroy(ctx) != 0)
+        ret = -1;
+
+free_fs:
+    glfs_free_from_ctx(fs);
+
+    /*
+     * Do this as late as possible in case anything else has (or
+     * grows) a dependency on mem-pool allocations.
+     */
+    mem_pools_fini();
+
+fail:
+    if (!ret)
+        ret = err;
+
+    __GLFS_EXIT_FS;
+
+invalid_fs:
+    return ret;
 }
 
-GFAPI_SYMVER_PRIVATE_DEFAULT(glfs_init_done, 3.4.0);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_get_volfile, 3.6.0)
+ssize_t
+pub_glfs_get_volfile(struct glfs *fs, void *buf, size_t len)
+{
+    ssize_t res = -1;
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    glfs_lock(fs, _gf_true);
+    if (len >= fs->oldvollen) {
+        gf_msg_trace("glfs", 0, "copying %zu to %p", len, buf);
+        memcpy(buf, fs->oldvolfile, len);
+        res = len;
+    } else {
+        res = len - fs->oldvollen;
+        gf_msg_trace("glfs", 0, "buffer is %zd too short", -res);
+    }
+    glfs_unlock(fs);
 
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return res;
+}
+
+GFAPI_SYMVER_PRIVATE_DEFAULT(glfs_ipc, 3.12.0)
 int
-glfs_init_common (struct glfs *fs)
+priv_glfs_ipc(struct glfs *fs, int opcode, void *xd_in, void **xd_out)
 {
-	int  ret = -1;
+    xlator_t *subvol = NULL;
+    int ret = -1;
 
-	ret = create_master (fs);
-	if (ret)
-		return ret;
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
 
-	ret = gf_thread_create (&fs->poller, NULL, glfs_poller, fs);
-	if (ret)
-		return ret;
+    subvol = glfs_active_subvol(fs);
+    if (!subvol) {
+        ret = -1;
+        errno = EIO;
+        goto out;
+    }
 
-	ret = glfs_volumes_init (fs);
-	if (ret)
-		return ret;
+    ret = syncop_ipc(subvol, opcode, (dict_t *)xd_in, (dict_t **)xd_out);
+    DECODE_SYNCOP_ERR(ret);
 
-	fs->dev_id = gf_dm_hashfn (fs->volname, strlen (fs->volname));
-	return ret;
-}
+out:
+    glfs_subvol_done(fs, subvol);
+    __GLFS_EXIT_FS;
 
+invalid_fs:
+    return ret;
+}
 
+GFAPI_SYMVER_PRIVATE_DEFAULT(glfs_setfspid, 6.1)
 int
-glfs_init_async (struct glfs *fs, glfs_init_cbk cbk)
+priv_glfs_setfspid(struct glfs *fs, pid_t pid)
 {
-	int  ret = -1;
+    cmd_args_t *cmd_args = NULL;
+    int ret = 0;
 
-	if (!fs || !fs->ctx) {
-		gf_log ("glfs", GF_LOG_ERROR,
-			"fs is not properly initialized.");
-		errno = EINVAL;
-		return ret;
-	}
+    cmd_args = &fs->ctx->cmd_args;
+    cmd_args->client_pid = pid;
+    cmd_args->client_pid_set = 1;
+    ret = syncopctx_setfspid(&pid);
 
-	fs->init_cbk = cbk;
+    return ret;
+}
 
-	ret = glfs_init_common (fs);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_free, 3.7.16)
+void
+pub_glfs_free(void *ptr)
+{
+    GLFS_FREE(ptr);
+}
 
-	return ret;
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_upcall_get_fs, 3.7.16)
+struct glfs *
+pub_glfs_upcall_get_fs(struct glfs_upcall *arg)
+{
+    return arg->fs;
 }
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_upcall_get_reason, 3.7.16)
+enum glfs_upcall_reason
+pub_glfs_upcall_get_reason(struct glfs_upcall *arg)
+{
+    return arg->reason;
+}
 
-int
-pub_glfs_init (struct glfs *fs)
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_upcall_get_event, 3.7.16)
+void *
+pub_glfs_upcall_get_event(struct glfs_upcall *arg)
 {
-	int  ret = -1;
+    return arg->event;
+}
 
-	if (!fs || !fs->ctx) {
-		gf_log ("glfs", GF_LOG_ERROR,
-			"fs is not properly initialized.");
-		errno = EINVAL;
-		return ret;
-	}
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_upcall_inode_get_object, 3.7.16)
+struct glfs_object *
+pub_glfs_upcall_inode_get_object(struct glfs_upcall_inode *arg)
+{
+    return arg->object;
+}
 
-	ret = glfs_init_common (fs);
-	if (ret)
-		return ret;
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_upcall_inode_get_flags, 3.7.16)
+uint64_t
+pub_glfs_upcall_inode_get_flags(struct glfs_upcall_inode *arg)
+{
+    return arg->flags;
+}
 
-	ret = glfs_init_wait (fs);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_upcall_inode_get_stat, 3.7.16)
+struct stat *
+pub_glfs_upcall_inode_get_stat(struct glfs_upcall_inode *arg)
+{
+    return &arg->buf;
+}
 
-	return ret;
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_upcall_inode_get_expire, 3.7.16)
+uint64_t
+pub_glfs_upcall_inode_get_expire(struct glfs_upcall_inode *arg)
+{
+    return arg->expire_time_attr;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_init, 3.4.0);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_upcall_inode_get_pobject, 3.7.16)
+struct glfs_object *
+pub_glfs_upcall_inode_get_pobject(struct glfs_upcall_inode *arg)
+{
+    return arg->p_object;
+}
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_upcall_inode_get_pstat, 3.7.16)
+struct stat *
+pub_glfs_upcall_inode_get_pstat(struct glfs_upcall_inode *arg)
+{
+    return &arg->p_buf;
+}
 
-extern xlator_t *
-priv_glfs_active_subvol (struct glfs *);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_upcall_inode_get_oldpobject, 3.7.16)
+struct glfs_object *
+pub_glfs_upcall_inode_get_oldpobject(struct glfs_upcall_inode *arg)
+{
+    return arg->oldp_object;
+}
+
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_upcall_inode_get_oldpstat, 3.7.16)
+struct stat *
+pub_glfs_upcall_inode_get_oldpstat(struct glfs_upcall_inode *arg)
+{
+    return &arg->oldp_buf;
+}
+
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_upcall_lease_get_object, 4.1.6)
+struct glfs_object *
+pub_glfs_upcall_lease_get_object(struct glfs_upcall_lease *arg)
+{
+    return arg->object;
+}
+
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_upcall_lease_get_lease_type, 4.1.6)
+uint32_t
+pub_glfs_upcall_lease_get_lease_type(struct glfs_upcall_lease *arg)
+{
+    return arg->lease_type;
+}
+
+/* definitions of the GLFS_SYSRQ_* chars are in glfs.h */
+static struct glfs_sysrq_help {
+    char sysrq;
+    char *msg;
+} glfs_sysrq_help[] = {{GLFS_SYSRQ_HELP, "(H)elp"},
+                       {GLFS_SYSRQ_STATEDUMP, "(S)tatedump"},
+                       {0, NULL}};
 
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_sysrq, 3.10.0)
 int
-pub_glfs_fini (struct glfs *fs)
+pub_glfs_sysrq(struct glfs *fs, char sysrq)
 {
-        int             ret = -1;
-        int             countdown = 100;
-        xlator_t        *subvol = NULL;
-        glusterfs_ctx_t *ctx = NULL;
-        call_pool_t     *call_pool = NULL;
-        int             fs_init = 0;
+    glusterfs_ctx_t *ctx = NULL;
+    int ret = 0;
+    int msg_len;
+    char msg[1024] = {
+        0,
+    }; /* should not exceed 1024 chars */
+
+    if (!fs || !fs->ctx) {
+        ret = -1;
+        errno = EINVAL;
+        goto out;
+    }
 
-        ctx = fs->ctx;
+    ctx = fs->ctx;
 
-        if (ctx->mgmt) {
-                rpc_clnt_disable (ctx->mgmt);
-                ctx->mgmt = NULL;
-        }
+    switch (sysrq) {
+        case GLFS_SYSRQ_HELP: {
+            struct glfs_sysrq_help *usage = NULL;
 
-        __glfs_entry_fs (fs);
+            for (usage = glfs_sysrq_help; usage->sysrq; usage++) {
+                msg_len = strlen(msg);
+                snprintf(msg + msg_len, /* append to msg */
+                         sizeof(msg) - msg_len - 2,
+                         /* - 2 for the " " + terminating \0 */
+                         " %s", usage->msg);
+            }
 
-        call_pool = fs->ctx->pool;
+            /* not really an 'error', but make sure it gets logged */
+            gf_log("glfs", GF_LOG_ERROR, "available events: %s", msg);
 
-        while (countdown--) {
-                /* give some time for background frames to finish */
-                if (!call_pool->cnt)
-                        break;
-                usleep (100000);
+            break;
         }
-        /* leaked frames may exist, we ignore */
+        case GLFS_SYSRQ_STATEDUMP:
+            gf_proc_dump_info(SIGUSR1, ctx);
+            break;
+        default:
+            gf_smsg("glfs", GF_LOG_ERROR, ENOTSUP, API_MSG_INVALID_SYSRQ,
+                    "sysrq=%c", sysrq, NULL);
+            errno = ENOTSUP;
+            ret = -1;
+    }
+out:
+    return ret;
+}
 
-        /*We deem glfs_fini as successful if there are no pending frames in the call
-         *pool*/
-        ret = (call_pool->cnt == 0)? 0: -1;
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_upcall_register, 3.13.0)
+int
+pub_glfs_upcall_register(struct glfs *fs, uint32_t event_list,
+                         glfs_upcall_cbk cbk, void *data)
+{
+    int ret = 0;
+
+    /* list of supported upcall events */
+    uint32_t up_events = (GLFS_EVENT_INODE_INVALIDATE |
+                          GLFS_EVENT_RECALL_LEASE);
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    GF_VALIDATE_OR_GOTO(THIS->name, cbk, out);
+
+    /* Event list should be either GLFS_EVENT_ANY
+     * or list of supported individual events (up_events)
+     */
+    if ((event_list != GLFS_EVENT_ANY) && (event_list & ~up_events)) {
+        errno = EINVAL;
+        ret = -1;
+        gf_smsg(THIS->name, GF_LOG_ERROR, errno, API_MSG_INVALID_ARG,
+                "event_list=(0x%08x)", event_list, NULL);
+        goto out;
+    }
 
-        pthread_mutex_lock (&fs->mutex);
-        {
-                fs_init = fs->init;
+    /* in case other thread does unregister */
+    pthread_mutex_lock(&fs->mutex);
+    {
+        if (event_list & GLFS_EVENT_INODE_INVALIDATE) {
+            /* @todo: Check if features.cache-invalidation is
+             * enabled.
+             */
+            fs->upcall_events |= GF_UPCALL_CACHE_INVALIDATION;
+            ret |= GLFS_EVENT_INODE_INVALIDATE;
         }
-        pthread_mutex_unlock (&fs->mutex);
-
-        if (fs_init != 0) {
-                subvol = priv_glfs_active_subvol (fs);
-                if (subvol) {
-                        /* PARENT_DOWN within priv_glfs_subvol_done() is issued only
-                           on graph switch (new graph should activiate and
-                           decrement the extra @winds count taken in glfs_graph_setup()
-
-                           Since we are explicitly destroying, PARENT_DOWN is necessary
-                        */
-                        xlator_notify (subvol, GF_EVENT_PARENT_DOWN, subvol, 0);
-                        /* TBD: wait for CHILD_DOWN before exiting, in case of
-                           asynchronous cleanup like graceful socket
-                           disconnection in the future.
-                        */
-                }
-                priv_glfs_subvol_done (fs, subvol);
+        if (event_list & GLFS_EVENT_RECALL_LEASE) {
+            /* @todo: Check if features.leases is enabled */
+            fs->upcall_events |= GF_UPCALL_RECALL_LEASE;
+            ret |= GLFS_EVENT_RECALL_LEASE;
         }
+        /* Override cbk function if existing */
+        fs->up_cbk = cbk;
+        fs->up_data = data;
+        fs->cache_upcalls = _gf_true;
+    }
+    pthread_mutex_unlock(&fs->mutex);
 
-        if (gf_log_fini(ctx) != 0)
-                ret = -1;
+out:
+    __GLFS_EXIT_FS;
 
-        return ret;
+invalid_fs:
+    return ret;
 }
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_fini, 3.4.0);
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_upcall_unregister, 3.13.0)
+int
+pub_glfs_upcall_unregister(struct glfs *fs, uint32_t event_list)
+{
+    int ret = 0;
+    /* list of supported upcall events */
+    uint32_t up_events = (GLFS_EVENT_INODE_INVALIDATE |
+                          GLFS_EVENT_RECALL_LEASE);
+
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    /* Event list should be either GLFS_EVENT_ANY
+     * or list of supported individual events (up_events)
+     */
+    if ((event_list != GLFS_EVENT_ANY) && (event_list & ~up_events)) {
+        errno = EINVAL;
+        ret = -1;
+        gf_smsg(THIS->name, GF_LOG_ERROR, errno, API_MSG_INVALID_ARG,
+                "event_list=(0x%08x)", event_list, NULL);
+        goto out;
+    }
 
+    pthread_mutex_lock(&fs->mutex);
+    {
+        /* We already checked if event_list contains list of supported
+         * upcall events. No other specific checks needed as of now for
+         * unregister */
+        fs->upcall_events &= ~(event_list);
+        ret |= ((event_list == GLFS_EVENT_ANY) ? up_events : event_list);
+
+        /* If there are no upcall events registered, reset cbk */
+        if (fs->upcall_events == 0) {
+            fs->up_cbk = NULL;
+            fs->up_data = NULL;
+            fs->cache_upcalls = _gf_false;
+        }
+    }
+    pthread_mutex_unlock(&fs->mutex);
 
-ssize_t
-pub_glfs_get_volfile (struct glfs *fs, void *buf, size_t len)
+out:
+    __GLFS_EXIT_FS;
+
+invalid_fs:
+    return ret;
+}
+
+GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_set_statedump_path, 7.0)
+int
+pub_glfs_set_statedump_path(struct glfs *fs, const char *path)
 {
-        ssize_t         res;
+    struct stat st;
+    int ret;
+    DECLARE_OLD_THIS;
+    __GLFS_ENTRY_VALIDATE_FS(fs, invalid_fs);
+
+    if (!path) {
+        gf_log("glfs", GF_LOG_ERROR, "path is NULL");
+        errno = EINVAL;
+        goto err;
+    }
+
+    /* If path is not present OR, if it is directory AND has enough permission
+     * to create files, then proceed */
+    ret = sys_stat(path, &st);
+    if (ret && errno != ENOENT) {
+        gf_log("glfs", GF_LOG_ERROR, "%s: not a valid path (%s)", path,
+               strerror(errno));
+        errno = EINVAL;
+        goto err;
+    }
 
-        glfs_lock(fs);
-        if (len >= fs->oldvollen) {
-                gf_log ("glfs", GF_LOG_TRACE, "copying %lu to %p", len, buf);
-                memcpy(buf,fs->oldvolfile,len);
-                res = len;
+    if (!ret) {
+        /* file is present, now check other things */
+        if (!S_ISDIR(st.st_mode)) {
+            gf_log("glfs", GF_LOG_ERROR, "%s: path is not directory", path);
+            errno = EINVAL;
+            goto err;
         }
-        else {
-                res = len - fs->oldvollen;
-                gf_log ("glfs", GF_LOG_TRACE, "buffer is %ld too short", -res);
+        if (sys_access(path, W_OK | X_OK) < 0) {
+            gf_log("glfs", GF_LOG_ERROR,
+                   "%s: path doesn't have write permission", path);
+            errno = EPERM;
+            goto err;
         }
-        glfs_unlock(fs);
+    }
 
-        return res;
-}
+    /* If set, it needs to be freed, so we don't have leak */
+    GF_FREE(fs->ctx->statedump_path);
 
-GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_get_volfile, 3.6.0);
+    fs->ctx->statedump_path = gf_strdup(path);
+    if (!fs->ctx->statedump_path) {
+        gf_log("glfs", GF_LOG_ERROR,
+               "%s: failed to set statedump path, no memory", path);
+        errno = ENOMEM;
+        goto err;
+    }
+
+    __GLFS_EXIT_FS;
 
+    return 0;
+err:
+    __GLFS_EXIT_FS;
+
+invalid_fs:
+    return -1;
+}
diff --git a/api/src/glfs.h b/api/src/glfs.h
index 6d896f0ea13..279d11d58ee 100644
--- a/api/src/glfs.h
+++ b/api/src/glfs.h
@@ -1,5 +1,5 @@
 /*
-  Copyright (c) 2012 Red Hat, Inc. <http://www.redhat.com>
+  Copyright (c) 2012-2018 Red Hat, Inc. <http://www.redhat.com>
   This file is part of GlusterFS.
 
   This file is licensed to you under your choice of the GNU Lesser
@@ -8,7 +8,6 @@
   cases as published by the Free Software Foundation.
 */
 
-
 #ifndef _GLFS_H
 #define _GLFS_H
 
@@ -21,6 +20,17 @@
   both the library and the application.
 */
 
+/* Values for valid flags to be used when using XXXsetattr, to set multiple
+ attribute values passed via the related stat structure.
+ */
+
+#define GFAPI_SET_ATTR_MODE 0x1
+#define GFAPI_SET_ATTR_UID 0x2
+#define GFAPI_SET_ATTR_GID 0x4
+#define GFAPI_SET_ATTR_SIZE 0x8
+#define GFAPI_SET_ATTR_ATIME 0x10
+#define GFAPI_SET_ATTR_MTIME 0x20
+
 #ifndef _FILE_OFFSET_BITS
 #define _FILE_OFFSET_BITS 64
 #endif
@@ -41,14 +51,64 @@
 #include <sys/cdefs.h>
 #include <dirent.h>
 #include <sys/statvfs.h>
+#include <stdint.h>
+#include <sys/time.h>
+
+/*
+ * For off64_t to be defined, we need both
+ * __USE_LARGEFILE64 to be true and __off64_t_defnined to be
+ * false. But, making __USE_LARGEFILE64 true causes other issues
+ * such as redinition of stat and fstat to stat64 and fstat64
+ * respectively which again causes compilation issues.
+ * Without off64_t being defined, this will not compile as
+ * copy_file_range uses off64_t. Hence define it here. First
+ * check whether __off64_t_defined is true or not. <unistd.h>
+ * sets that flag when it defines off64_t. If __off64_t_defined
+ * is false and __USE_FILE_OFFSET64 is true, then go on to define
+ * off64_t using __off64_t.
+ */
+#ifndef GF_BSD_HOST_OS
+#if defined(__USE_FILE_OFFSET64) && !defined(__off64_t_defined)
+typedef __off64_t off64_t;
+#endif /* defined(__USE_FILE_OFFSET64) && !defined(__off64_t_defined) */
+#else
+#include <stdio.h>
+#ifndef _OFF64_T_DECLARED
+/*
+ * Including <stdio.h> (done above) should actually define
+ * _OFF64_T_DECLARED with off64_t data type being available
+ * for consumption. But, off64_t data type is not recognizable
+ * for FreeBSD versions less than 11. Hence, int64_t is typedefed
+ * to off64_t.
+ */
+#define _OFF64_T_DECLARED
+typedef int64_t off64_t;
+#endif /* _OFF64_T_DECLARED */
+#endif /* GF_BSD_HOST_OS */
+
+#if defined(HAVE_SYS_ACL_H) || (defined(USE_POSIX_ACLS) && USE_POSIX_ACLS)
+#include <sys/acl.h>
+#else
+typedef void *acl_t;
+typedef int acl_type_t;
+#endif
 
 /* Portability non glibc c++ build systems */
 #ifndef __THROW
-# if defined __cplusplus
-#  define __THROW       throw ()
-# else
-#  define __THROW
-# endif
+#if defined __cplusplus
+#define __THROW throw()
+#else
+#define __THROW
+#endif
+#endif
+
+#ifndef GF_DARWIN_HOST_OS
+#define GFAPI_PUBLIC(sym, ver)  /**/
+#define GFAPI_PRIVATE(sym, ver) /**/
+#else
+#define GFAPI_PUBLIC(sym, ver) __asm("_" __STRING(sym) "$GFAPI_" __STRING(ver))
+#define GFAPI_PRIVATE(sym, ver)                                                \
+    __asm("_" __STRING(sym) "$GFAPI_PRIVATE_" __STRING(ver))
 #endif
 
 __BEGIN_DECLS
@@ -57,7 +117,6 @@ __BEGIN_DECLS
 struct glfs;
 typedef struct glfs glfs_t;
 
-
 /*
   SYNOPSIS
 
@@ -78,8 +137,8 @@ typedef struct glfs glfs_t;
 
   @volname: Name of the volume. This identifies the server-side volume and
             the fetched volfile (equivalent of --volfile-id command line
-	    parameter to glusterfsd). When used with glfs_set_volfile() the
-	    @volname has no effect (except for appearing in log messages).
+            parameter to glusterfsd). When used with glfs_set_volfile() the
+            @volname has no effect (except for appearing in log messages).
 
   RETURN VALUES
 
@@ -88,8 +147,8 @@ typedef struct glfs glfs_t;
 
 */
 
-glfs_t *glfs_new (const char *volname) __THROW;
-
+glfs_t *
+glfs_new(const char *volname) __THROW GFAPI_PUBLIC(glfs_new, 3.4.0);
 
 /*
   SYNOPSIS
@@ -118,8 +177,9 @@ glfs_t *glfs_new (const char *volname) __THROW;
 
 */
 
-int glfs_set_volfile (glfs_t *fs, const char *volfile);
-
+int
+glfs_set_volfile(glfs_t *fs, const char *volfile) __THROW
+    GFAPI_PUBLIC(glfs_set_volfile, 3.4.0);
 
 /*
   SYNOPSIS
@@ -144,15 +204,16 @@ int glfs_set_volfile (glfs_t *fs, const char *volfile);
        specification file.
 
   @transport: String specifying the transport used to connect to the
-              management daemon. Specifying NULL will result in the usage
-              of the default (tcp) transport type. Permitted values
-              are those what you specify as transport-type in a volume
-              specification file (e.g "tcp", "rdma" etc.)
+              management daemon. Specifying NULL will result in the
+              usage of the default (tcp) transport type. Permitted
+              values are "tcp" or "unix".
 
   @host:      String specifying the address where to find the management daemon.
+              Socket path, while using Unix domain socket as transport type.
               This would either be
-              - FQDN (e.g: "storage01.company.com") or
-              - ASCII (e.g: "192.168.22.1")
+              - FQDN (e.g : "storage01.company.com") or
+              - ASCII (e.g : "192.168.22.1") or
+              - Socket path (e.g : "/var/run/glusterd.socket")
 
   NOTE: This API is special, multiple calls to this function with different
         volfile servers, port or transport-type would create a list of volfile
@@ -169,10 +230,15 @@ int glfs_set_volfile (glfs_t *fs, const char *volfile);
 
 */
 
-int glfs_set_volfile_server (glfs_t *fs, const char *transport,
-                             const char *host, int port) __THROW;
-int glfs_unset_volfile_server (glfs_t *fs, const char *transport,
-                               const char *host, int port) __THROW;
+int
+glfs_set_volfile_server(glfs_t *fs, const char *transport, const char *host,
+                        int port) __THROW
+    GFAPI_PUBLIC(glfs_set_volfile_server, 3.4.0);
+
+int
+glfs_unset_volfile_server(glfs_t *fs, const char *transport, const char *host,
+                          int port) __THROW
+    GFAPI_PUBLIC(glfs_unset_volfile_server, 3.5.1);
 /*
   SYNOPSIS
 
@@ -202,8 +268,9 @@ int glfs_unset_volfile_server (glfs_t *fs, const char *transport,
 
 */
 
-int glfs_set_logging (glfs_t *fs, const char *logfile, int loglevel) __THROW;
-
+int
+glfs_set_logging(glfs_t *fs, const char *logfile, int loglevel) __THROW
+    GFAPI_PUBLIC(glfs_set_logging, 3.4.0);
 
 /*
   SYNOPSIS
@@ -229,8 +296,8 @@ int glfs_set_logging (glfs_t *fs, const char *logfile, int loglevel) __THROW;
 
 */
 
-int glfs_init (glfs_t *fs) __THROW;
-
+int
+glfs_init(glfs_t *fs) __THROW GFAPI_PUBLIC(glfs_init, 3.4.0);
 
 /*
   SYNOPSIS
@@ -262,7 +329,8 @@ int glfs_init (glfs_t *fs) __THROW;
    0 : Success.
 */
 
-int glfs_fini (glfs_t *fs) __THROW;
+int
+glfs_fini(glfs_t *fs) __THROW GFAPI_PUBLIC(glfs_fini, 3.4.0);
 
 /*
   SYNOPSIS
@@ -292,8 +360,9 @@ int glfs_fini (glfs_t *fs) __THROW;
       <0: volfile length exceeds @len by N bytes (@buf unchanged)
 */
 
-ssize_t glfs_get_volfile (glfs_t *fs, void *buf, size_t len) __THROW;
-
+ssize_t
+glfs_get_volfile(glfs_t *fs, void *buf, size_t len) __THROW
+    GFAPI_PUBLIC(glfs_get_volfile, 3.6.0);
 
 /*
   SYNOPSIS
@@ -306,9 +375,9 @@ ssize_t glfs_get_volfile (glfs_t *fs, void *buf, size_t len) __THROW;
        the management server (glusterd) to fetch volume uuid and stores it
        in the glusterfs_context linked to the glfs object fs which can be used
        in the subsequent calls. Later it parses that UUID to convert it from
-       cannonical string format into an opaque byte array and copy it into
-       the volid array. Incase if either of the input parameters, volid or size,
-       is NULL, number of bytes required to copy the volume UUID is returned.
+       canonical string format into an opaque byte array and copy it into
+       the volid array. In case if either of the input parameters, volid or
+  size, is NULL, number of bytes required to copy the volume UUID is returned.
 
   PARAMETERS
 
@@ -323,8 +392,9 @@ ssize_t glfs_get_volfile (glfs_t *fs, void *buf, size_t len) __THROW;
         Others : length of the volume UUID stored.
 */
 
-int glfs_get_volumeid (struct glfs *fs, char *volid, size_t size);
-
+int
+glfs_get_volumeid(glfs_t *fs, char *volid, size_t size) __THROW
+    GFAPI_PUBLIC(glfs_get_volumeid, 3.5.0);
 
 /*
  * FILE OPERATION
@@ -346,6 +416,119 @@ struct glfs_fd;
 typedef struct glfs_fd glfs_fd_t;
 
 /*
+ * Mask for request/result items in the struct glfs_stat.
+ *
+ * Query request/result mask for glfs_stat() (family of functions) and
+ * struct glfs_stat::glfs_st_mask.
+ *
+ * These bits should be set in the mask argument of glfs_stat() (family of
+ * functions) to request particular items when calling glfs_stat().
+ *
+ * NOTE: Lower order 32 bits are used to reflect statx(2) bits. For Gluster
+ * specific attrs/extensions, use higher order 32 bits.
+ *
+ */
+#define GLFS_STAT_TYPE 0x0000000000000001U   /* Want/got stx_mode & S_IFMT */
+#define GLFS_STAT_MODE 0x0000000000000002U   /* Want/got stx_mode & ~S_IFMT */
+#define GLFS_STAT_NLINK 0x0000000000000004U  /* Want/got stx_nlink */
+#define GLFS_STAT_UID 0x0000000000000008U    /* Want/got stx_uid */
+#define GLFS_STAT_GID 0x0000000000000010U    /* Want/got stx_gid */
+#define GLFS_STAT_ATIME 0x0000000000000020U  /* Want/got stx_atime */
+#define GLFS_STAT_MTIME 0x0000000000000040U  /* Want/got stx_mtime */
+#define GLFS_STAT_CTIME 0x0000000000000080U  /* Want/got stx_ctime */
+#define GLFS_STAT_INO 0x0000000000000100U    /* Want/got stx_ino */
+#define GLFS_STAT_SIZE 0x0000000000000200U   /* Want/got stx_size */
+#define GLFS_STAT_BLOCKS 0x0000000000000400U /* Want/got stx_blocks */
+#define GLFS_STAT_BASIC_STATS                                                  \
+    0x00000000000007ffU /* Items in the normal stat struct */
+#define GLFS_STAT_BTIME 0x0000000000000800U /* Want/got stx_btime */
+#define GLFS_STAT_ALL 0x0000000000000fffU   /* All currently supported flags */
+#define GLFS_STAT_RESERVED                                                     \
+    0x8000000000000000U /* Reserved to denote future expansion */
+
+/* Macros for checking validity of struct glfs_stat members.*/
+#define GLFS_STAT_TYPE_VALID(stmask) (stmask & GLFS_STAT_TYPE)
+#define GLFS_STAT_MODE_VALID(stmask) (stmask & GLFS_STAT_MODE)
+#define GLFS_STAT_NLINK_VALID(stmask) (stmask & GLFS_STAT_NLINK)
+#define GLFS_STAT_UID_VALID(stmask) (stmask & GLFS_STAT_UID)
+#define GLFS_STAT_GID_VALID(stmask) (stmask & GLFS_STAT_GID)
+#define GLFS_STAT_ATIME_VALID(stmask) (stmask & GLFS_STAT_ATIME)
+#define GLFS_STAT_MTIME_VALID(stmask) (stmask & GLFS_STAT_MTIME)
+#define GLFS_STAT_CTIME_VALID(stmask) (stmask & GLFS_STAT_CTIME)
+#define GLFS_STAT_INO_VALID(stmask) (stmask & GLFS_STAT_INO)
+#define GLFS_STAT_SIZE_VALID(stmask) (stmask & GLFS_STAT_SIZE)
+#define GLFS_STAT_BLOCKS_VALID(stmask) (stmask & GLFS_STAT_BLOCKS)
+#define GLFS_STAT_BTIME_VALID(stmask) (stmask & GLFS_STAT_BTIME)
+#define GLFS_STAT_GFID_VALID(stmask) (stmask & GLFS_STAT_GFID)
+
+/*
+ * Attributes to be found in glfs_st_attributes and masked in
+ * glfs_st_attributes_mask.
+ *
+ * These give information about the features or the state of a file that might
+ * be of use to programs.
+ *
+ * NOTE: Lower order 32 bits are used to reflect statx(2) attribute bits. For
+ * Gluster specific attrs, use higher order 32 bits.
+ *
+ * NOTE: We do not support any file attributes or state as yet!
+ */
+#define GLFS_STAT_ATTR_RESERVED                                                \
+    0x8000000000000000U /* Reserved to denote future expansion */
+
+/* Extended file attribute structure.
+ *
+ * The caller passes a mask of what they're specifically interested in as a
+ * parameter to glfs_stat().  What glfs_stat() actually got will be indicated
+ * in glfs_st_mask upon return.
+ *
+ * For each bit in the mask argument:
+ *
+ * - if the datum is not supported:
+ *
+ *   - the bit will be cleared, and
+ *
+ *   - the datum value is undefined
+ *
+ * - otherwise, if explicitly requested:
+ *
+ *   - the field will be filled in and the bit will be set;
+ *
+ * - otherwise, if not requested, but available in, it will be filled in
+ * anyway, and the bit will be set upon return;
+ *
+ * - otherwise the field and the bit will be cleared before returning.
+ *
+ */
+
+struct glfs_stat {
+    uint64_t glfs_st_mask;       /* What results were written [uncond] */
+    uint64_t glfs_st_attributes; /* Flags conveying information about the file
+                                    [uncond] */
+    uint64_t glfs_st_attributes_mask; /* Mask to show what's supported in
+                                         st_attributes [ucond] */
+    struct timespec glfs_st_atime;    /* Last access time */
+    struct timespec glfs_st_btime;    /* File creation time */
+    struct timespec glfs_st_ctime;    /* Last attribute change time */
+    struct timespec glfs_st_mtime;    /* Last data modification time */
+    ino_t glfs_st_ino;                /* Inode number */
+    off_t glfs_st_size;               /* File size */
+    blkcnt_t glfs_st_blocks;          /* Number of 512-byte blocks allocated */
+    uint32_t glfs_st_rdev_major; /* Device ID of special file [if bdev/cdev] */
+    uint32_t glfs_st_rdev_minor;
+    uint32_t glfs_st_dev_major; /* ID of device containing file [uncond] */
+    uint32_t glfs_st_dev_minor;
+    blksize_t glfs_st_blksize; /* Preferred general I/O size [uncond] */
+    nlink_t glfs_st_nlink;     /* Number of hard links */
+    uid_t glfs_st_uid;         /* User ID of owner */
+    gid_t glfs_st_gid;         /* Group ID of owner */
+    mode_t glfs_st_mode;       /* File mode */
+};
+
+#define GLFS_LEASE_ID_SIZE 16 /* 128bits */
+typedef char glfs_leaseid_t[GLFS_LEASE_ID_SIZE];
+
+/*
  * PER THREAD IDENTITY MODIFIERS
  *
  * The following operations enable to set a per thread identity context
@@ -362,15 +545,27 @@ typedef struct glfs_fd glfs_fd_t;
  *    caller
  *  - The groups once set, need to be unset by setting the size to 0 (in which
  *    case the list argument is a do not care)
+ *  - In case of leases feature enables, setfsleaseid is used to set and reset
+ *    leaseid before and after every I/O operation.
  *  - Once a process for a thread of operation choses to set the IDs, all glfs
  *    calls made from that thread would default to the IDs set for the thread.
  *    As a result use these APIs with care and ensure that the set IDs are
  *    reverted to global process defaults as required.
  *
  */
-int glfs_setfsuid (uid_t fsuid) __THROW;
-int glfs_setfsgid (gid_t fsgid) __THROW;
-int glfs_setfsgroups (size_t size, const gid_t *list) __THROW;
+int
+glfs_setfsuid(uid_t fsuid) __THROW GFAPI_PUBLIC(glfs_setfsuid, 3.4.2);
+
+int
+glfs_setfsgid(gid_t fsgid) __THROW GFAPI_PUBLIC(glfs_setfsgid, 3.4.2);
+
+int
+glfs_setfsgroups(size_t size, const gid_t *list) __THROW
+    GFAPI_PUBLIC(glfs_setfsgroups, 3.4.2);
+
+int
+glfs_setfsleaseid(glfs_leaseid_t leaseid) __THROW
+    GFAPI_PUBLIC(glfs_setfsleaseid, 4.0.0);
 
 /*
   SYNOPSIS
@@ -397,8 +592,9 @@ int glfs_setfsgroups (size_t size, const gid_t *list) __THROW;
 
  */
 
-glfs_fd_t *glfs_open (glfs_t *fs, const char *path, int flags) __THROW;
-
+glfs_fd_t *
+glfs_open(glfs_t *fs, const char *path, int flags) __THROW
+    GFAPI_PUBLIC(glfs_open, 3.4.0);
 
 /*
   SYNOPSIS
@@ -426,15 +622,20 @@ glfs_fd_t *glfs_open (glfs_t *fs, const char *path, int flags) __THROW;
 
  */
 
-glfs_fd_t *glfs_creat (glfs_t *fs, const char *path, int flags,
-		       mode_t mode) __THROW;
+glfs_fd_t *
+glfs_creat(glfs_t *fs, const char *path, int flags, mode_t mode) __THROW
+    GFAPI_PUBLIC(glfs_creat, 3.4.0);
 
-int glfs_close (glfs_fd_t *fd) __THROW;
+int
+glfs_close(glfs_fd_t *fd) __THROW GFAPI_PUBLIC(glfs_close, 3.4.0);
 
-glfs_t *glfs_from_glfd (glfs_fd_t *fd) __THROW;
+glfs_t *
+glfs_from_glfd(glfs_fd_t *fd) __THROW GFAPI_PUBLIC(glfs_from_glfd, 3.4.0);
 
-int glfs_set_xlator_option (glfs_t *fs, const char *xlator, const char *key,
-			    const char *value) __THROW;
+int
+glfs_set_xlator_option(glfs_t *fs, const char *xlator, const char *key,
+                       const char *value) __THROW
+    GFAPI_PUBLIC(glfs_set_xlator_option, 3.4.0);
 
 /*
 
@@ -457,208 +658,828 @@ int glfs_set_xlator_option (glfs_t *fs, const char *xlator, const char *key,
   time of issuing the async IO call. This can be used by the
   caller to differentiate different instances of the async requests
   in a common callback function.
+
+  @prestat and @poststat are allocated on the stack, that are auto destroyed
+  post the callback function returns.
 */
 
-typedef void (*glfs_io_cbk) (glfs_fd_t *fd, ssize_t ret, void *data);
+typedef void (*glfs_io_cbk)(glfs_fd_t *fd, ssize_t ret,
+                            struct glfs_stat *prestat,
+                            struct glfs_stat *poststat, void *data);
 
 // glfs_{read,write}[_async]
 
-ssize_t glfs_read (glfs_fd_t *fd, void *buf,
-                   size_t count, int flags) __THROW;
-ssize_t glfs_write (glfs_fd_t *fd, const void *buf,
-                    size_t count, int flags) __THROW;
-int glfs_read_async (glfs_fd_t *fd, void *buf, size_t count, int flags,
-		     glfs_io_cbk fn, void *data) __THROW;
-int glfs_write_async (glfs_fd_t *fd, const void *buf, size_t count, int flags,
-		      glfs_io_cbk fn, void *data) __THROW;
+ssize_t
+glfs_read(glfs_fd_t *fd, void *buf, size_t count, int flags) __THROW
+    GFAPI_PUBLIC(glfs_read, 3.4.0);
+
+ssize_t
+glfs_write(glfs_fd_t *fd, const void *buf, size_t count, int flags) __THROW
+    GFAPI_PUBLIC(glfs_write, 3.4.0);
+
+int
+glfs_read_async(glfs_fd_t *fd, void *buf, size_t count, int flags,
+                glfs_io_cbk fn, void *data) __THROW
+    GFAPI_PUBLIC(glfs_read_async, 6.0);
+
+int
+glfs_write_async(glfs_fd_t *fd, const void *buf, size_t count, int flags,
+                 glfs_io_cbk fn, void *data) __THROW
+    GFAPI_PUBLIC(glfs_write_async, 6.0);
 
 // glfs_{read,write}v[_async]
 
-ssize_t glfs_readv (glfs_fd_t *fd, const struct iovec *iov, int iovcnt,
-		    int flags) __THROW;
-ssize_t glfs_writev (glfs_fd_t *fd, const struct iovec *iov, int iovcnt,
-		     int flags) __THROW;
-int glfs_readv_async (glfs_fd_t *fd, const struct iovec *iov, int count,
-		      int flags, glfs_io_cbk fn, void *data) __THROW;
-int glfs_writev_async (glfs_fd_t *fd, const struct iovec *iov, int count,
-		       int flags, glfs_io_cbk fn, void *data) __THROW;
+ssize_t
+glfs_readv(glfs_fd_t *fd, const struct iovec *iov, int iovcnt,
+           int flags) __THROW GFAPI_PUBLIC(glfs_readv, 3.4.0);
+
+ssize_t
+glfs_writev(glfs_fd_t *fd, const struct iovec *iov, int iovcnt,
+            int flags) __THROW GFAPI_PUBLIC(glfs_writev, 3.4.0);
+
+int
+glfs_readv_async(glfs_fd_t *fd, const struct iovec *iov, int count, int flags,
+                 glfs_io_cbk fn, void *data) __THROW
+    GFAPI_PUBLIC(glfs_readv_async, 6.0);
+
+int
+glfs_writev_async(glfs_fd_t *fd, const struct iovec *iov, int count, int flags,
+                  glfs_io_cbk fn, void *data) __THROW
+    GFAPI_PUBLIC(glfs_writev_async, 6.0);
 
 // glfs_p{read,write}[_async]
 
-ssize_t glfs_pread (glfs_fd_t *fd, void *buf, size_t count, off_t offset,
-		    int flags) __THROW;
-ssize_t glfs_pwrite (glfs_fd_t *fd, const void *buf, size_t count,
-		     off_t offset, int flags) __THROW;
-int glfs_pread_async (glfs_fd_t *fd, void *buf, size_t count, off_t offset,
-		      int flags, glfs_io_cbk fn, void *data) __THROW;
-int glfs_pwrite_async (glfs_fd_t *fd, const void *buf, int count, off_t offset,
-		       int flags, glfs_io_cbk fn, void *data) __THROW;
+ssize_t
+glfs_pread(glfs_fd_t *fd, void *buf, size_t count, off_t offset, int flags,
+           struct glfs_stat *poststat) __THROW GFAPI_PUBLIC(glfs_pread, 6.0);
+
+ssize_t
+glfs_pwrite(glfs_fd_t *fd, const void *buf, size_t count, off_t offset,
+            int flags, struct glfs_stat *prestat,
+            struct glfs_stat *poststat) __THROW GFAPI_PUBLIC(glfs_pwrite, 6.0);
+
+int
+glfs_pread_async(glfs_fd_t *fd, void *buf, size_t count, off_t offset,
+                 int flags, glfs_io_cbk fn, void *data) __THROW
+    GFAPI_PUBLIC(glfs_pread_async, 6.0);
+
+int
+glfs_pwrite_async(glfs_fd_t *fd, const void *buf, int count, off_t offset,
+                  int flags, glfs_io_cbk fn, void *data) __THROW
+    GFAPI_PUBLIC(glfs_pwrite_async, 6.0);
 
 // glfs_p{read,write}v[_async]
 
-ssize_t glfs_preadv (glfs_fd_t *fd, const struct iovec *iov, int iovcnt,
-		     off_t offset, int flags) __THROW;
-ssize_t glfs_pwritev (glfs_fd_t *fd, const struct iovec *iov, int iovcnt,
-		      off_t offset, int flags) __THROW;
-int glfs_preadv_async (glfs_fd_t *fd, const struct iovec *iov,
-                       int count, off_t offset, int flags,
-                       glfs_io_cbk fn, void *data) __THROW;
-int glfs_pwritev_async (glfs_fd_t *fd, const struct iovec *iov,
-                        int count, off_t offset, int flags,
-                        glfs_io_cbk fn, void *data) __THROW;
+ssize_t
+glfs_preadv(glfs_fd_t *fd, const struct iovec *iov, int iovcnt, off_t offset,
+            int flags) __THROW GFAPI_PUBLIC(glfs_preadv, 3.4.0);
+
+ssize_t
+glfs_pwritev(glfs_fd_t *fd, const struct iovec *iov, int iovcnt, off_t offset,
+             int flags) __THROW GFAPI_PUBLIC(glfs_pwritev, 3.4.0);
+
+int
+glfs_preadv_async(glfs_fd_t *fd, const struct iovec *iov, int count,
+                  off_t offset, int flags, glfs_io_cbk fn, void *data) __THROW
+    GFAPI_PUBLIC(glfs_preadv_async, 6.0);
+
+int
+glfs_pwritev_async(glfs_fd_t *fd, const struct iovec *iov, int count,
+                   off_t offset, int flags, glfs_io_cbk fn, void *data) __THROW
+    GFAPI_PUBLIC(glfs_pwritev_async, 6.0);
+
+off_t
+glfs_lseek(glfs_fd_t *fd, off_t offset, int whence) __THROW
+    GFAPI_PUBLIC(glfs_lseek, 3.4.0);
+
+ssize_t
+glfs_copy_file_range(struct glfs_fd *glfd_in, off64_t *off_in,
+                     struct glfs_fd *glfd_out, off64_t *off_out, size_t len,
+                     unsigned int flags, struct glfs_stat *statbuf,
+                     struct glfs_stat *prestat,
+                     struct glfs_stat *poststat) __THROW
+    GFAPI_PUBLIC(glfs_copy_file_range, 6.0);
+
+int
+glfs_truncate(glfs_t *fs, const char *path, off_t length) __THROW
+    GFAPI_PUBLIC(glfs_truncate, 3.7.15);
+
+int
+glfs_ftruncate(glfs_fd_t *fd, off_t length, struct glfs_stat *prestat,
+               struct glfs_stat *poststat) __THROW
+    GFAPI_PUBLIC(glfs_ftruncate, 6.0);
+
+int
+glfs_ftruncate_async(glfs_fd_t *fd, off_t length, glfs_io_cbk fn,
+                     void *data) __THROW
+    GFAPI_PUBLIC(glfs_ftruncate_async, 6.0);
+
+int
+glfs_lstat(glfs_t *fs, const char *path, struct stat *buf) __THROW
+    GFAPI_PUBLIC(glfs_lstat, 3.4.0);
+
+int
+glfs_stat(glfs_t *fs, const char *path, struct stat *buf) __THROW
+    GFAPI_PUBLIC(glfs_stat, 3.4.0);
+
+int
+glfs_fstat(glfs_fd_t *fd, struct stat *buf) __THROW
+    GFAPI_PUBLIC(glfs_fstat, 3.4.0);
+
+int
+glfs_fsync(glfs_fd_t *fd, struct glfs_stat *prestat,
+           struct glfs_stat *poststat) __THROW GFAPI_PUBLIC(glfs_fsync, 6.0);
+
+int
+glfs_fsync_async(glfs_fd_t *fd, glfs_io_cbk fn, void *data) __THROW
+    GFAPI_PUBLIC(glfs_fsync_async, 6.0);
+
+int
+glfs_fdatasync(glfs_fd_t *fd, struct glfs_stat *prestat,
+               struct glfs_stat *poststat) __THROW
+    GFAPI_PUBLIC(glfs_fdatasync, 6.0);
+
+int
+glfs_fdatasync_async(glfs_fd_t *fd, glfs_io_cbk fn, void *data) __THROW
+    GFAPI_PUBLIC(glfs_fdatasync_async, 6.0);
+
+int
+glfs_access(glfs_t *fs, const char *path, int mode) __THROW
+    GFAPI_PUBLIC(glfs_access, 3.4.0);
+
+int
+glfs_symlink(glfs_t *fs, const char *oldpath, const char *newpath) __THROW
+    GFAPI_PUBLIC(glfs_symlink, 3.4.0);
+
+int
+glfs_readlink(glfs_t *fs, const char *path, char *buf, size_t bufsiz) __THROW
+    GFAPI_PUBLIC(glfs_readlink, 3.4.0);
+
+int
+glfs_mknod(glfs_t *fs, const char *path, mode_t mode, dev_t dev) __THROW
+    GFAPI_PUBLIC(glfs_mknod, 3.4.0);
+
+int
+glfs_mkdir(glfs_t *fs, const char *path, mode_t mode) __THROW
+    GFAPI_PUBLIC(glfs_mkdir, 3.4.0);
+
+int
+glfs_unlink(glfs_t *fs, const char *path) __THROW
+    GFAPI_PUBLIC(glfs_unlink, 3.4.0);
+
+int
+glfs_rmdir(glfs_t *fs, const char *path) __THROW
+    GFAPI_PUBLIC(glfs_rmdir, 3.4.0);
+
+int
+glfs_rename(glfs_t *fs, const char *oldpath, const char *newpath) __THROW
+    GFAPI_PUBLIC(glfs_rename, 3.4.0);
+
+int
+glfs_link(glfs_t *fs, const char *oldpath, const char *newpath) __THROW
+    GFAPI_PUBLIC(glfs_link, 3.4.0);
+
+glfs_fd_t *
+glfs_opendir(glfs_t *fs, const char *path) __THROW
+    GFAPI_PUBLIC(glfs_opendir, 3.4.0);
+
+/*
+ * @glfs_readdir_r and @glfs_readdirplus_r ARE thread safe AND re-entrant,
+ * but the interface has ambiguity about the size of @dirent to be allocated
+ * before calling the APIs. 512 byte buffer (for @dirent) is sufficient for
+ * all known systems which are tested againt glusterfs/gfapi, but may be
+ * insufficient in the future.
+ */
+
+int
+glfs_readdir_r(glfs_fd_t *fd, struct dirent *dirent,
+               struct dirent **result) __THROW
+    GFAPI_PUBLIC(glfs_readdir_r, 3.4.0);
+
+int
+glfs_readdirplus_r(glfs_fd_t *fd, struct stat *stat, struct dirent *dirent,
+                   struct dirent **result) __THROW
+    GFAPI_PUBLIC(glfs_readdirplus_r, 3.4.0);
+
+/*
+ * @glfs_readdir and @glfs_readdirplus are NEITHER thread safe NOR re-entrant
+ * when called on the same directory handle. However they ARE thread safe
+ * AND re-entrant when called on different directory handles (which may be
+ * referring to the same directory too.)
+ */
+
+struct dirent *
+glfs_readdir(glfs_fd_t *fd) __THROW GFAPI_PUBLIC(glfs_readdir, 3.5.0);
+
+struct dirent *
+glfs_readdirplus(glfs_fd_t *fd, struct stat *stat) __THROW
+    GFAPI_PUBLIC(glfs_readdirplus, 3.5.0);
+
+long
+glfs_telldir(glfs_fd_t *fd) __THROW GFAPI_PUBLIC(glfs_telldir, 3.4.0);
+
+void
+glfs_seekdir(glfs_fd_t *fd, long offset) __THROW
+    GFAPI_PUBLIC(glfs_seekdir, 3.4.0);
+
+int
+glfs_closedir(glfs_fd_t *fd) __THROW GFAPI_PUBLIC(glfs_closedir, 3.4.0);
+
+int
+glfs_statvfs(glfs_t *fs, const char *path, struct statvfs *buf) __THROW
+    GFAPI_PUBLIC(glfs_statvfs, 3.4.0);
+
+int
+glfs_chmod(glfs_t *fs, const char *path, mode_t mode) __THROW
+    GFAPI_PUBLIC(glfs_chmod, 3.4.0);
+
+int
+glfs_fchmod(glfs_fd_t *fd, mode_t mode) __THROW
+    GFAPI_PUBLIC(glfs_fchmod, 3.4.0);
+
+int
+glfs_chown(glfs_t *fs, const char *path, uid_t uid, gid_t gid) __THROW
+    GFAPI_PUBLIC(glfs_chown, 3.4.0);
 
+int
+glfs_lchown(glfs_t *fs, const char *path, uid_t uid, gid_t gid) __THROW
+    GFAPI_PUBLIC(glfs_lchown, 3.4.0);
 
-off_t glfs_lseek (glfs_fd_t *fd, off_t offset, int whence) __THROW;
+int
+glfs_fchown(glfs_fd_t *fd, uid_t uid, gid_t gid) __THROW
+    GFAPI_PUBLIC(glfs_fchown, 3.4.0);
 
-int glfs_truncate (glfs_t *fs, const char *path, off_t length) __THROW;
+int
+glfs_utimens(glfs_t *fs, const char *path,
+             const struct timespec times[2]) __THROW
+    GFAPI_PUBLIC(glfs_utimens, 3.4.0);
 
-int glfs_ftruncate (glfs_fd_t *fd, off_t length) __THROW;
-int glfs_ftruncate_async (glfs_fd_t *fd, off_t length, glfs_io_cbk fn,
-			  void *data) __THROW;
+int
+glfs_lutimens(glfs_t *fs, const char *path,
+              const struct timespec times[2]) __THROW
+    GFAPI_PUBLIC(glfs_lutimens, 3.4.0);
 
-int glfs_lstat (glfs_t *fs, const char *path, struct stat *buf) __THROW;
-int glfs_stat (glfs_t *fs, const char *path, struct stat *buf) __THROW;
-int glfs_fstat (glfs_fd_t *fd, struct stat *buf) __THROW;
+int
+glfs_futimens(glfs_fd_t *fd, const struct timespec times[2]) __THROW
+    GFAPI_PUBLIC(glfs_futimens, 3.4.0);
 
-int glfs_fsync (glfs_fd_t *fd) __THROW;
-int glfs_fsync_async (glfs_fd_t *fd, glfs_io_cbk fn, void *data) __THROW;
+ssize_t
+glfs_getxattr(glfs_t *fs, const char *path, const char *name, void *value,
+              size_t size) __THROW GFAPI_PUBLIC(glfs_getxattr, 3.4.0);
 
-int glfs_fdatasync (glfs_fd_t *fd) __THROW;
-int glfs_fdatasync_async (glfs_fd_t *fd, glfs_io_cbk fn, void *data) __THROW;
+ssize_t
+glfs_lgetxattr(glfs_t *fs, const char *path, const char *name, void *value,
+               size_t size) __THROW GFAPI_PUBLIC(glfs_lgetxattr, 3.4.0);
 
-int glfs_access (glfs_t *fs, const char *path, int mode) __THROW;
+ssize_t
+glfs_fgetxattr(glfs_fd_t *fd, const char *name, void *value,
+               size_t size) __THROW GFAPI_PUBLIC(glfs_fgetxattr, 3.4.0);
 
-int glfs_symlink (glfs_t *fs, const char *oldpath, const char *newpath) __THROW;
+ssize_t
+glfs_listxattr(glfs_t *fs, const char *path, void *value, size_t size) __THROW
+    GFAPI_PUBLIC(glfs_listxattr, 3.4.0);
 
-int glfs_readlink (glfs_t *fs, const char *path,
-                   char *buf, size_t bufsiz) __THROW;
+ssize_t
+glfs_llistxattr(glfs_t *fs, const char *path, void *value, size_t size) __THROW
+    GFAPI_PUBLIC(glfs_llistxattr, 3.4.0);
 
-int glfs_mknod (glfs_t *fs, const char *path, mode_t mode, dev_t dev) __THROW;
+ssize_t
+glfs_flistxattr(glfs_fd_t *fd, void *value, size_t size) __THROW
+    GFAPI_PUBLIC(glfs_flistxattr, 3.4.0);
 
-int glfs_mkdir (glfs_t *fs, const char *path, mode_t mode) __THROW;
+int
+glfs_setxattr(glfs_t *fs, const char *path, const char *name, const void *value,
+              size_t size, int flags) __THROW
+    GFAPI_PUBLIC(glfs_setxattr, 3.4.0);
 
-int glfs_unlink (glfs_t *fs, const char *path) __THROW;
+int
+glfs_lsetxattr(glfs_t *fs, const char *path, const char *name,
+               const void *value, size_t size, int flags) __THROW
+    GFAPI_PUBLIC(glfs_lsetxattr, 3.4.0);
 
-int glfs_rmdir (glfs_t *fs, const char *path) __THROW;
+int
+glfs_fsetxattr(glfs_fd_t *fd, const char *name, const void *value, size_t size,
+               int flags) __THROW GFAPI_PUBLIC(glfs_fsetxattr, 3.4.0);
 
-int glfs_rename (glfs_t *fs, const char *oldpath, const char *newpath) __THROW;
+int
+glfs_removexattr(glfs_t *fs, const char *path, const char *name) __THROW
+    GFAPI_PUBLIC(glfs_removexattr, 3.4.0);
 
-int glfs_link (glfs_t *fs, const char *oldpath, const char *newpath) __THROW;
+int
+glfs_lremovexattr(glfs_t *fs, const char *path, const char *name) __THROW
+    GFAPI_PUBLIC(glfs_lremovexattr, 3.4.0);
 
-glfs_fd_t *glfs_opendir (glfs_t *fs, const char *path) __THROW;
+int
+glfs_fremovexattr(glfs_fd_t *fd, const char *name) __THROW
+    GFAPI_PUBLIC(glfs_fremovexattr, 3.4.0);
+
+int
+glfs_fallocate(glfs_fd_t *fd, int keep_size, off_t offset, size_t len) __THROW
+    GFAPI_PUBLIC(glfs_fallocate, 3.5.0);
+
+int
+glfs_discard(glfs_fd_t *fd, off_t offset, size_t len) __THROW
+    GFAPI_PUBLIC(glfs_discard, 3.5.0);
+
+int
+glfs_discard_async(glfs_fd_t *fd, off_t length, size_t lent, glfs_io_cbk fn,
+                   void *data) __THROW GFAPI_PUBLIC(glfs_discard_async, 6.0);
+
+int
+glfs_zerofill(glfs_fd_t *fd, off_t offset, off_t len) __THROW
+    GFAPI_PUBLIC(glfs_zerofill, 3.5.0);
+
+int
+glfs_zerofill_async(glfs_fd_t *fd, off_t length, off_t len, glfs_io_cbk fn,
+                    void *data) __THROW GFAPI_PUBLIC(glfs_zerofill_async, 6.0);
+
+char *
+glfs_getcwd(glfs_t *fs, char *buf, size_t size) __THROW
+    GFAPI_PUBLIC(glfs_getcwd, 3.4.0);
+
+int
+glfs_chdir(glfs_t *fs, const char *path) __THROW
+    GFAPI_PUBLIC(glfs_chdir, 3.4.0);
+
+int
+glfs_fchdir(glfs_fd_t *fd) __THROW GFAPI_PUBLIC(glfs_fchdir, 3.4.0);
+
+char *
+glfs_realpath(glfs_t *fs, const char *path, char *resolved_path) __THROW
+    GFAPI_PUBLIC(glfs_realpath, 3.7.17);
 
 /*
- * @glfs_readdir_r and @glfs_readdirplus_r ARE thread safe AND re-entrant,
- * but the interface has ambiguity about the size of @dirent to be allocated
- * before calling the APIs. 512 byte buffer (for @dirent) is sufficient for
- * all known systems which are tested againt glusterfs/gfapi, but may be
- * insufficient in the future.
+ * @cmd and @flock are as specified in man fcntl(2).
+ */
+int
+glfs_posix_lock(glfs_fd_t *fd, int cmd, struct flock *flock) __THROW
+    GFAPI_PUBLIC(glfs_posix_lock, 3.4.0);
+
+/*
+  SYNOPSIS
+
+  glfs_file_lock: Request extended byte range lock on a file
+
+  DESCRIPTION
+
+  This function is capable of requesting either advisory or mandatory type
+  byte range locks on a file.
+
+  Note: To set a unique owner key for locks based on a particular file
+  descriptor, make use of glfs_fd_set_lkowner() api to do so before
+  requesting lock via this api. This owner key will be further consumed
+  by other incoming data modifying file operations via the same file
+  descriptor.
+
+  PARAMETERS
+
+  @fd: File descriptor
+
+  @cmd: As specified in man fcntl(2).
+
+  @flock: As specified in man fcntl(2).
+
+  @lk_mode: Required lock type from options available with the
+            enum glfs_lock_mode_t defined below.
+
+  RETURN VALUES
+
+  0   : Success. Lock has been granted.
+  -1  : Failure. @errno will be set indicating the type of failure.
+
  */
 
-int glfs_readdir_r (glfs_fd_t *fd, struct dirent *dirent,
-		    struct dirent **result) __THROW;
+/* Lock modes used by glfs_file_lock() */
+enum glfs_lock_mode { GLFS_LK_ADVISORY = 0, GLFS_LK_MANDATORY };
+typedef enum glfs_lock_mode glfs_lock_mode_t;
+
+int
+glfs_file_lock(glfs_fd_t *fd, int cmd, struct flock *flock,
+               glfs_lock_mode_t lk_mode) __THROW
+    GFAPI_PUBLIC(glfs_file_lock, 3.13.0);
 
-int glfs_readdirplus_r (glfs_fd_t *fd, struct stat *stat, struct dirent *dirent,
-			struct dirent **result) __THROW;
+glfs_fd_t *
+glfs_dup(glfs_fd_t *fd) __THROW GFAPI_PUBLIC(glfs_dup, 3.4.0);
+
+void
+glfs_free(void *ptr) __THROW GFAPI_PUBLIC(glfs_free, 3.7.16);
 
 /*
- * @glfs_readdir and @glfs_readdirplus are NEITHER thread safe NOR re-entrant
- * when called on the same directory handle. However they ARE thread safe
- * AND re-entrant when called on different directory handles (which may be
- * referring to the same directory too.)
+ * glfs_sysrq: send a system request to the @fs instance
+ *
+ * Different commands for @sysrq are possible, the defines for these are listed
+ * below the function definition.
+ *
+ * This function always returns success if the @sysrq is recognized. The return
+ * value does not way anythin about the result of the @sysrq execution. Not all
+ * @sysrq command will be able to return a success/failure status.
+ */
+int
+glfs_sysrq(glfs_t *fs, char sysrq) __THROW GFAPI_PUBLIC(glfs_sysrq, 3.10.0);
+
+#define GLFS_SYSRQ_HELP 'h' /* log a message with supported sysrq commands */
+#define GLFS_SYSRQ_STATEDUMP 's' /* create a statedump */
+
+/*
+ * Structure returned as part of xreaddirplus
+ */
+struct glfs_xreaddirp_stat;
+typedef struct glfs_xreaddirp_stat glfs_xreaddirp_stat_t;
+
+/* Request flags to be used in XREADDIRP operation */
+#define GFAPI_XREADDIRP_NULL                                                   \
+    0x00000000 /* by default, no stat will be fetched */
+#define GFAPI_XREADDIRP_STAT 0x00000001   /* Get stat */
+#define GFAPI_XREADDIRP_HANDLE 0x00000002 /* Get object handle */
+
+/*
+ * This stat structure returned gets freed as part of glfs_free(xstat)
+ */
+struct stat *
+glfs_xreaddirplus_get_stat(glfs_xreaddirp_stat_t *xstat) __THROW
+    GFAPI_PUBLIC(glfs_xreaddirplus_get_stat, 3.11.0);
+
+/*
+ * SYNOPSIS
+ *
+ * glfs_xreaddirplus_r: Extended Readirplus operation
+ *
+ * DESCRIPTION
+ *
+ * This API does readdirplus operation, but along with stat it can fetch other
+ * extra information like object handles etc for each of the dirents returned
+ * based on requested flags. On success it returns the set of flags successfully
+ * processed.
+ *
+ * Note that there are chances that some of the requested information may not be
+ * available or returned (for example if reached EOD). Ensure to validate the
+ * returned value to determine what flags have been successfully processed
+ * & set.
+ *
+ * PARAMETERS
+ *
+ * INPUT:
+ * @glfd: GFAPI file descriptor of the directory
+ * @flags: Flags determining xreaddirp_stat requested
+ *         Current available values are:
+ *              GFAPI_XREADDIRP_NULL
+ *              GFAPI_XREADDIRP_STAT
+ *              GFAPI_XREADDIRP_HANDLE
+ * @ext: Dirent struture to copy the values to
+ *       (though optional recommended to be allocated by application
+ *        esp., in multi-threaded environment)
+ *
+ * OUTPUT:
+ * @res: to store the next dirent value. If NULL and return value is '0',
+ *       it means it reached end of the directory.
+ * @xstat_p: Pointer to contain all the requested data returned
+ *           for that dirent. Application should make use of glfs_free() API
+ *           to free this pointer and the variables returned by
+ *           glfs_xreaddirplus_get_*() APIs.
+ *
+ * RETURN VALUE:
+ * >=0: SUCCESS (value contains the flags successfully processed)
+ *  -1: FAILURE
  */
+int
+glfs_xreaddirplus_r(glfs_fd_t *glfd, uint32_t flags,
+                    glfs_xreaddirp_stat_t **xstat_p, struct dirent *ext,
+                    struct dirent **res) __THROW
+    GFAPI_PUBLIC(glfs_xreaddirplus_r, 3.11.0);
+
+#define GFAPI_MAX_LOCK_OWNER_LEN 255
 
-struct dirent *glfs_readdir (glfs_fd_t *fd) __THROW;
+/*
+ *
+ * DESCRIPTION
+ *
+ * This API allows application to set lk_owner on a fd.
+ * A glfd can be associated with only single lk_owner. In case if there
+ * is need to set another lk_owner, applications can make use of
+ * 'glfs_dup' to get duplicate glfd and set new lk_owner on that second
+ * glfd.
+ *
+ * Also its not recommended to override or clear lk_owner value as the
+ * same shall be used to flush any outstanding locks while closing the fd.
+ *
+ * PARAMETERS
+ *
+ * INPUT:
+ * @glfd: GFAPI file descriptor
+ * @len: Size of lk_owner buffer. Max value can be GFAPI_MAX_LOCK_OWNER_LEN
+ * @data: lk_owner data buffer.
+ *
+ * OUTPUT:
+ * 0: SUCCESS
+ * -1: FAILURE
+ */
+int
+glfs_fd_set_lkowner(glfs_fd_t *glfd, void *data, int len) __THROW
+    GFAPI_PUBLIC(glfs_fd_set_lkowner, 3.10.7);
+
+/*
+ * Applications (currently NFS-Ganesha) can make use of this
+ * structure to read upcall notifications sent by server either
+ * by polling or registering a callback function.
+ *
+ * On success, applications need to check for 'reason' to decide
+ * if any upcall event is received.
+ *
+ * Currently supported upcall_events -
+ *      GLFS_UPCALL_INODE_INVALIDATE -
+ *              'event_arg' - glfs_upcall_inode
+ *
+ * After processing the event, applications need to free 'event_arg' with
+ * glfs_free().
+ *
+ * Also similar to I/Os, the application should ideally stop polling
+ * or unregister upcall_cbk function before calling glfs_fini(..).
+ * Hence making an assumption that 'fs' & ctx structures cannot be
+ * freed while in this routine.
+ */
+struct glfs_upcall;
+typedef struct glfs_upcall glfs_upcall_t;
+
+glfs_t *
+glfs_upcall_get_fs(glfs_upcall_t *arg) __THROW
+    GFAPI_PUBLIC(glfs_upcall_get_fs, 3.7.16);
+
+enum glfs_upcall_reason {
+    GLFS_UPCALL_EVENT_NULL = 0,
+    GLFS_UPCALL_INODE_INVALIDATE, /* invalidate cache entry */
+    GLFS_UPCALL_RECALL_LEASE,     /* recall lease */
+};
+typedef enum glfs_upcall_reason glfs_upcall_reason_t;
+
+glfs_upcall_reason_t
+glfs_upcall_get_reason(glfs_upcall_t *arg) __THROW
+    GFAPI_PUBLIC(glfs_upcall_get_reason, 3.7.16);
+
+/*
+ * Applications first need to make use of above API i.e,
+ * "glfs_upcall_get_reason" to determine which upcall event it has
+ * received. Post that below API - "glfs_upcall_get_event" should
+ * be used to get corresponding upcall event object.
+ *
+ * Below are the upcall_reason and corresponding upcall_event objects:
+ *      ==========================================================
+ *      glfs_upcall_reason           -    event_object
+ *      ==========================================================
+ *      GLFS_UPCALL_EVENT_NULL       -    NULL
+ *      GLFS_UPCALL_INODE_INVALIDATE -    struct glfs_upcall_inode
+ *      GLFS_UPCALL_RECALL_LEASE     -    struct glfs_upcall_lease
+ *
+ * After processing upcall event, glfs_free() should be called on the
+ * glfs_upcall.
+ */
+void *
+glfs_upcall_get_event(glfs_upcall_t *arg) __THROW
+    GFAPI_PUBLIC(glfs_upcall_get_event, 3.7.16);
+
+/*
+ * SYNOPSIS
+ *
+ * glfs_upcall_cbk: Upcall callback definition
+ *
+ * This is function type definition of the callback function pointer
+ * which has to be provided by the caller while registering for any
+ * upcall events.
+ *
+ * This function is called whenever any upcall which the application
+ * has registered for is received from the server.
+ *
+ * @up_arg: Upcall structure whose contents need to be interpreted by
+ * making use of glfs_upcall_* helper routines.
+ *
+ * @data: The same context pointer provided by the caller at the time of
+ * registering of upcall events. This may be used by the caller for any
+ * of its internal use while processing upcalls.
+ */
+typedef void (*glfs_upcall_cbk)(glfs_upcall_t *up_arg, void *data);
 
-struct dirent *glfs_readdirplus (glfs_fd_t *fd, struct stat *stat) __THROW;
+/*
+ * List of upcall events supported by gluster/gfapi
+ */
+#define GLFS_EVENT_INODE_INVALIDATE 0x00000001 /* invalidate cache entry */
+#define GLFS_EVENT_RECALL_LEASE 0x00000002     /* Recall lease */
+#define GLFS_EVENT_ANY 0xffffffff              /* for all the above events */
 
-long glfs_telldir (glfs_fd_t *fd) __THROW;
+/*
+ * SYNOPSIS
+ *
+ * glfs_upcall_register: Register for upcall events
+ *
+ * DESCRIPTION
+ *
+ * This function is used to register for various upcall events application
+ * is interested in and the callback function to be invoked when such
+ * events are triggered.
+ *
+ * Multiple calls of this routine shall override cbk function. That means
+ * only one cbk function can be used for all the upcall events registered
+ * and that shall be the one last updated.
+ *
+ * PARAMETERS:
+ *
+ * INPUT:
+ * @fs: The 'virtual mount' object
+ *
+ * @event_list: List of upcall events to be registered.
+ *              Current available values are:
+ *               - GLFS_EVENT_INODE_INVALIDATE
+ *               - GLFS_EVENT_RECALL_LEASE
+ *
+ * @cbk: The cbk routine to be invoked in case of any upcall received
+ * @data: Any opaque pointer provided by caller which shall be using while
+ * making cbk calls. This pointer may be used by caller for any of its
+ * internal use while processing upcalls. Can be NULL.
+ *
+ * RETURN VALUE:
+ * >0: SUCCESS (value contains the events successfully registered)
+ * -1: FAILURE
+ */
+int
+glfs_upcall_register(glfs_t *fs, uint32_t event_list, glfs_upcall_cbk cbk,
+                     void *data) __THROW
+    GFAPI_PUBLIC(glfs_upcall_register, 3.13.0);
 
-void glfs_seekdir (glfs_fd_t *fd, long offset) __THROW;
+/*
+ * SYNOPSIS
+ *
+ * glfs_upcall_unregister: Unregister for upcall events
+ *
+ * DESCRIPTION
+ *
+ * This function is used to unregister the upcall events application
+ * is not interested in. In case if the caller unregisters all the events
+ * it has registered for, it shall no more receive any upcall event.
+ *
+ * PARAMETERS:
+ *
+ * INPUT:
+ * @fs: The 'virtual mount' object
+ *
+ * @event_list: List of upcall events to be unregistered.
+ *              Current available values are:
+ *               - GLFS_EVENT_INODE_INVALIDATE
+ *               - GLFS_EVENT_RECALL_LEASE
+ * RETURN VALUE:
+ * >0: SUCCESS (value contains the events successfully unregistered)
+ * -1: FAILURE
+ */
+int
+glfs_upcall_unregister(glfs_t *fs, uint32_t event_list) __THROW
+    GFAPI_PUBLIC(glfs_upcall_unregister, 3.13.0);
+
+/* Lease Types */
+enum glfs_lease_types {
+    GLFS_LEASE_NONE = 0,
+    GLFS_RD_LEASE = 1,
+    GLFS_RW_LEASE = 2,
+};
+typedef enum glfs_lease_types glfs_lease_types_t;
+
+/* Lease cmds */
+enum glfs_lease_cmds {
+    GLFS_GET_LEASE = 1,
+    GLFS_SET_LEASE = 2,
+    GLFS_UNLK_LEASE = 3,
+};
+typedef enum glfs_lease_cmds glfs_lease_cmds_t;
+
+struct glfs_lease {
+    glfs_lease_cmds_t cmd;
+    glfs_lease_types_t lease_type;
+    glfs_leaseid_t lease_id;
+    unsigned int lease_flags;
+};
+typedef struct glfs_lease glfs_lease_t;
+
+typedef void (*glfs_recall_cbk)(glfs_lease_t lease, void *data);
 
-int glfs_closedir (glfs_fd_t *fd) __THROW;
+/*
+  SYNOPSIS
 
-int glfs_statvfs (glfs_t *fs, const char *path, struct statvfs *buf) __THROW;
+  glfs_lease: Takes a lease on a file.
 
-int glfs_chmod (glfs_t *fs, const char *path, mode_t mode) __THROW;
+  DESCRIPTION
 
-int glfs_fchmod (glfs_fd_t *fd, mode_t mode) __THROW;
+  This function takes lease on an open file.
 
-int glfs_chown (glfs_t *fs, const char *path, uid_t uid, gid_t gid) __THROW;
+  PARAMETERS
 
-int glfs_lchown (glfs_t *fs, const char *path, uid_t uid, gid_t gid) __THROW;
+  @glfd: The fd of the file on which lease should be taken,
+   this fd is returned by glfs_open/glfs_create.
 
-int glfs_fchown (glfs_fd_t *fd, uid_t uid, gid_t gid) __THROW;
+  @lease: Struct that defines the lease operation to be performed
+   on the file.
+      @lease.cmd - Can be one of the following values
+         GF_GET_LEASE:  Get the lease type currently present on the file,
+                        lease.lease_type will contain GF_RD_LEASE
+                        or GF_RW_LEASE or 0 if no leases.
+         GF_SET_LEASE:  Set the lease of given lease.lease_type on the file.
+         GF_UNLK_LEASE: Unlock the lease present on the given fd.
+                        Note that the every lease request should have
+                        a corresponding unlk_lease.
 
-int glfs_utimens (glfs_t *fs, const char *path,
-                  struct timespec times[2]) __THROW;
+      @lease.lease_type - Can be one of the following values
+         GF_RD_LEASE:   Read lease on a file, shared lease.
+         GF_RW_LEASE:   Read-Write lease on a file, exclusive lease.
 
-int glfs_lutimens (glfs_t *fs, const char *path,
-                   struct timespec times[2]) __THROW;
+      @lease.lease_id - A unique identification of lease, 128bits.
 
-int glfs_futimens (glfs_fd_t *fd, struct timespec times[2]) __THROW;
+  @fn: This is the function that is invoked when the lease has to be recalled
+  @data: It is a cookie, this pointer is returned as a part of recall
 
-ssize_t glfs_getxattr (glfs_t *fs, const char *path, const char *name,
-		       void *value, size_t size) __THROW;
+  fn and data field are stored as a part of glfs_fd, hence if there are multiple
+  glfs_lease calls, each of them updates the fn and data fields. glfs_recall_cbk
+  will be invoked with the last updated fn and data
 
-ssize_t glfs_lgetxattr (glfs_t *fs, const char *path, const char *name,
-			void *value, size_t size) __THROW;
+  RETURN VALUES
+  0:  Successful completion
+  <0: Failure. @errno will be set with the type of failure
+*/
 
-ssize_t glfs_fgetxattr (glfs_fd_t *fd, const char *name,
-			void *value, size_t size) __THROW;
+int
+glfs_lease(glfs_fd_t *glfd, glfs_lease_t *lease, glfs_recall_cbk fn,
+           void *data) __THROW GFAPI_PUBLIC(glfs_lease, 4.0.0);
 
-ssize_t glfs_listxattr (glfs_t *fs, const char *path,
-                        void *value, size_t size) __THROW;
+/*
+  SYNOPSIS
 
-ssize_t glfs_llistxattr (glfs_t *fs, const char *path, void *value,
-			 size_t size) __THROW;
+  glfs_fsetattr: Function to set attributes.
+  glfs_setattr: Function to set attributes
 
-ssize_t glfs_flistxattr (glfs_fd_t *fd, void *value, size_t size) __THROW;
+  DESCRIPTION
 
-int glfs_setxattr (glfs_t *fs, const char *path, const char *name,
-		   const void *value, size_t size, int flags) __THROW;
+  The functions are used to set attributes on the file.
 
-int glfs_lsetxattr (glfs_t *fs, const char *path, const char *name,
-		    const void *value, size_t size, int flags) __THROW;
+  PARAMETERS
 
-int glfs_fsetxattr (glfs_fd_t *fd, const char *name,
-		    const void *value, size_t size, int flags) __THROW;
+  @glfs_fsetattr
 
-int glfs_removexattr (glfs_t *fs, const char *path, const char *name) __THROW;
+     @glfd: The fd of the file for which the attributes are to be set,
+            this fd is returned by glfs_open/glfs_create.
 
-int glfs_lremovexattr (glfs_t *fs, const char *path, const char *name) __THROW;
+  @glfs_setattr
 
-int glfs_fremovexattr (glfs_fd_t *fd, const char *name) __THROW;
+         @fs: File object.
 
-int glfs_fallocate(glfs_fd_t *fd, int keep_size,
-                   off_t offset, size_t len) __THROW;
+         @path: The path of the file that is being operated on.
 
-int glfs_discard(glfs_fd_t *fd, off_t offset, size_t len) __THROW;
+         @follow: Flag used to resolve symlink.
 
 
-int glfs_discard_async (glfs_fd_t *fd, off_t length, size_t lent,
-			glfs_io_cbk fn, void *data) __THROW;
+  @stat: Struct that has information about the file.
 
-int glfs_zerofill(glfs_fd_t *fd, off_t offset, off_t len) __THROW;
+  @valid: This is the mask bit, that accepts GFAPI_SET_ATTR* masks.
+          Refer glfs.h to see the mask definitions.
 
-int glfs_zerofill_async (glfs_fd_t *fd, off_t length, off_t len,
-                        glfs_io_cbk fn, void *data) __THROW;
+  Both functions are similar in functionality, just that the
+  func setattr() uses file path whereas the func fsetattr()
+  uses the fd.
 
-char *glfs_getcwd (glfs_t *fs, char *buf, size_t size) __THROW;
+  RETURN VALUES
+  0:  Successful completion
+  <0: Failure. @errno will be set with the type of failure
 
-int glfs_chdir (glfs_t *fs, const char *path) __THROW;
+ */
 
-int glfs_fchdir (glfs_fd_t *fd) __THROW;
+int
+glfs_fsetattr(struct glfs_fd *glfd, struct glfs_stat *stat) __THROW
+    GFAPI_PUBLIC(glfs_fsetattr, 6.0);
 
-char *glfs_realpath (glfs_t *fs, const char *path, char *resolved_path) __THROW;
+int
+glfs_setattr(struct glfs *fs, const char *path, struct glfs_stat *stat,
+             int follow) __THROW GFAPI_PUBLIC(glfs_setattr, 6.0);
 
 /*
- * @cmd and @flock are as specified in man fcntl(2).
+  SYNOPSIS
+
+  glfs_set_statedump_path: Function to set statedump path.
+
+  DESCRIPTION
+
+  This function is used to set statedump directory
+
+  PARAMETERS
+
+  @fs: The 'virtual mount' object to be configured with the volume
+       specification file.
+
+  @path: statedump path. Should be a directory. But the API won't fail if the
+  directory doesn't exist yet, as one may create it later.
+
+  RETURN VALUES
+
+   0 : Success.
+  -1 : Failure. @errno will be set with the type of failure.
+
  */
-int glfs_posix_lock (glfs_fd_t *fd, int cmd, struct flock *flock) __THROW;
 
-glfs_fd_t *glfs_dup (glfs_fd_t *fd) __THROW;
+int
+glfs_set_statedump_path(struct glfs *fs, const char *path) __THROW
+    GFAPI_PUBLIC(glfs_set_statedump_path, 7.0);
 
 __END_DECLS
-
 #endif /* !_GLFS_H */
diff --git a/autogen.sh b/autogen.sh
index fef2f037426..c8cdc3f89fa 100755
--- a/autogen.sh
+++ b/autogen.sh
@@ -85,11 +85,7 @@ $TOOL --automake --copy --force
 echo Running ${AUTOCONF}...
 $AUTOCONF
 echo Running ${AUTOMAKE}...
-$AUTOMAKE --add-missing --copy --foreign
-
-# Run autogen in the argp-standalone sub-directory
-echo "Running autogen.sh in argp-standalone ..."
-( cd contrib/argp-standalone;./autogen.sh )
+$AUTOMAKE --add-missing --force-missing --copy
 
 # Instruct user on next steps
 echo
diff --git a/extras/checkpatch.pl b/build-aux/checkpatch.pl
index 02c2c2e02ee..17ae4e4d579 100755
--- a/extras/checkpatch.pl
+++ b/build-aux/checkpatch.pl
@@ -44,7 +44,7 @@ my $configuration_file = ".checkpatch.conf";
 my $max_line_length = 80;
 my $ignore_perl_version = 0;
 my $minimum_perl_version = 5.10.0;
-my $jenkins_url = $ENV{JENKINS_URL};
+my $gerrit_url = $ENV{GERRIT_URL};
 
 sub help {
     my ($exitcode) = @_;
@@ -57,7 +57,7 @@ Options:
   -q, --quiet                quiet
   --patch                    treat FILE as patchfile (default)
   --emacs                    emacs compile window format
-  --jenkins-url=STRING       provide URL patch was reviewed on
+  --gerrit-url=STRING        URL the patch was reviewed at
   --terse                    one line per report
   -f, --file                 treat FILE as regular source file
   --subjective, --strict     enable more subjective tests
@@ -121,7 +121,7 @@ GetOptions(
     'q|quiet+'  => \$quiet,
     'patch!'    => \$chk_patch,
     'emacs!'    => \$emacs,
-    'jenkins-url=s' => \$jenkins_url,
+    'gerrit-url=s' => \$gerrit_url,
     'terse!'    => \$terse,
     'f|file!'   => \$file,
     'subjective!'       => \$check,
@@ -612,15 +612,15 @@ sub top_of_glusterfs_tree {
         "geo-replication",
         "glusterfs-api.pc.in",
         "glusterfsd",
-        "glusterfs-hadoop",
         "glusterfs.spec.in",
+        "heal",
         "INSTALL",
         "libgfchangelog.pc.in",
         "libglusterfs",
         "MAINTAINERS",
         "Makefile.am",
         "NEWS",
-        "README",
+        "README.md",
         "rfc.sh",
         "rpc",
         "run-tests.sh",
@@ -1666,6 +1666,7 @@ sub process {
 
     our $clean = 1;
     my $signoff = 0;
+    my $subject_trailing_dot = 0;
     my $is_patch = 0;
 
     my $in_header_lines = 1;
@@ -1898,6 +1899,12 @@ sub process {
         next if ($realfile =~ /(checkpatch.pl)/);
         next if ($realfile =~ /\.(md|txt|doc|8|pdf|tex)$/);
 
+# Check that the subject does not have a trailing dot
+        if ($in_header_lines &&
+            $line =~ /^Subject: \[PATCH\] (.+)\.(\s*)$/) {
+                $subject_trailing_dot++;
+        }
+
 # Check the patch for a signoff:
         if ($line =~ /^\s*signed-off-by:/i) {
             $signoff++;
@@ -1943,11 +1950,11 @@ sub process {
                 }
             }
 
-            # Check if email is really Jenkins URL
+            # Check if email is really Gerrit URL
             if ($email =~ /^($url_tags)(.*)/) {
                 my $uri = $1;
                 my $url = $2;
-                if ($uri && $url !~ /$jenkins_url/) {
+                if ($uri && $url !~ /$gerrit_url/) {
                     ERROR("BAD_URL",
                           "Unrecognized url address: '$email'\n" . $herecurr);
                 }
@@ -2076,6 +2083,8 @@ sub process {
                  "Use of $flag is deprecated, please use \`$replacement->{$flag} instead.\n" . $herecurr) if ($replacement->{$flag});
         }
 
+# check we are in .spec file, then ignore this hunk
+        next if ($realfile eq "glusterfs.spec.in");
 
 # check we are in a valid source file if not then ignore this hunk
         next if ($realfile !~ /\.(h|c|pl|py|l|y|sh|in)$/);
@@ -2625,17 +2634,6 @@ sub process {
             }
         }
 
-# check for new typedefs, only function parameters and sparse annotations
-# make sense.
-        if ($line =~ /\btypedef\s/ &&
-            $line !~ /\btypedef\s+$Type\s*\(\s*\*?$Ident\s*\)\s*\(/ &&
-            $line !~ /\btypedef\s+$Type\s+$Ident\s*\(/ &&
-            $line !~ /\b$typeTypedefs\b/ &&
-            $line !~ /\b__bitwise(?:__|)\b/) {
-            WARN("NEW_TYPEDEFS",
-                 "do not add new typedefs\n" . $herecurr);
-        }
-
 # * goes on variable not on type
         # (char*[ const])
         while ($line =~ m{(\($NonptrType(\s*(?:$Modifier\b\s*|\*\s*)+)\))}g) {
@@ -2694,8 +2692,8 @@ sub process {
 
 # function brace can't be on same line, except for #defines of do while,
 # or if closed on same line
-        if (($line=~/$Type\s*$Ident\(.*\).*\s{/) and
-            !($line=~/\#\s*define.*do\s{/) and !($line=~/}/)) {
+        if (($line=~/$Type\s*$Ident\(.*\).*\s\{/) and
+            !($line=~/\#\s*define.*do\s\{/) and !($line=~/}/)) {
             ERROR("OPEN_BRACE",
                   "open brace '{' following function declarations go on the next line\n" . $herecurr);
         }
@@ -3096,12 +3094,12 @@ sub process {
 ##              }
 
 #need space before brace following if, while, etc
-        if (($line =~ /\(.*\){/ && $line !~ /\($Type\){/) ||
-            $line =~ /do{/) {
+        if (($line =~ /\(.*\)\{/ && $line !~ /\($Type\)\{/) ||
+            $line =~ /do\{/) {
             if (ERROR("SPACING",
                       "space required before the open brace '{'\n" . $herecurr) &&
                 $fix) {
-                $fixed[$linenr - 1] =~ s/^(\+.*(?:do|\))){/$1 {/;
+                $fixed[$linenr - 1] =~ s/^(\+.*(?:do|\)))\{/$1 {/;
             }
         }
 
@@ -3485,7 +3483,7 @@ sub process {
                 $dstat !~ /^for\s*$Constant$/ &&                                # for (...)
                 $dstat !~ /^for\s*$Constant\s+(?:$Ident|-?$Constant)$/ &&       # for (...) bar()
                 $dstat !~ /^do\s*{/ &&                                  # do {...
-                $dstat !~ /^\({/ &&                                             # ({...
+                $dstat !~ /^\(\{/ &&                                             # ({...
                 $ctx !~ /^.\s*#\s*define\s+TRACE_(?:SYSTEM|INCLUDE_FILE|INCLUDE_PATH)\b/)
             {
                 $ctx =~ s/\n*$//;
@@ -4232,6 +4230,10 @@ sub process {
         ERROR("NOT_UNIFIED_DIFF",
               "Does not appear to be a unified-diff format patch\n");
     }
+    if ($is_patch && $subject_trailing_dot != 0) {
+        ERROR("SUBJECT_TRAILING_DOT",
+              "The subject of the patch should not end with a dot.\n");
+    }
     if ($is_patch && $chk_signoff && $signoff == 0) {
         ERROR("MISSING_SIGN_OFF",
               "Missing Signed-off-by: line(s)\n");
diff --git a/build-aux/config.guess.dist b/build-aux/config.guess.dist
new file mode 100755
index 00000000000..881ba7a0438
--- /dev/null
+++ b/build-aux/config.guess.dist
@@ -0,0 +1,14 @@
+#!/bin/sh
+#
+# This script is intentionally left empty. Distributions that package GlusterFS
+# may want to to replace it with an updated copy from the automake project.
+#
+
+cat << EOM
+It is not expected to execute this script. When you are building from a
+released tarball (generated with 'make dist'), you are expected to pass
+--build=... and --host=... to ./configure or replace this config.guess script
+in the sources with an updated version.
+EOM
+
+exit 0
diff --git a/build-aux/config.sub.dist b/build-aux/config.sub.dist
new file mode 100755
index 00000000000..c5a0dbad282
--- /dev/null
+++ b/build-aux/config.sub.dist
@@ -0,0 +1,14 @@
+#!/bin/sh
+#
+# This script is intentionally left empty. Distributions that package GlusterFS
+# may want to to replace it with an updated copy from the automake project.
+#
+
+cat << EOM
+It is not expected to execute this script. When you are building from a
+released tarball (generated with 'make dist'), you are expected to pass
+--build=... and --host=... to ./configure or replace this config.sub script in
+the sources with an updated version.
+EOM
+
+exit 0
diff --git a/build-aux/pkg-version b/build-aux/pkg-version
index 83d4a5f9136..17ceab70c03 100755
--- a/build-aux/pkg-version
+++ b/build-aux/pkg-version
@@ -1,9 +1,13 @@
-#!/bin/sh
+#!/bin/bash
 
 # To override version/release from git,
 # create VERSION file containing text with version/release
 # eg. v3.4.0-1
-PKG_VERSION=`cat VERSION 2> /dev/null || git describe --tags --match "v[0-9]*"`
+
+# One thing to note, If one does 'git clone --depth N glusterfs.git',
+# the git describe command doesn't work. Hence you notice below that
+# we have added timestamp as version (YYYY.MM.DD) and release (HH.mmss)
+PKG_VERSION=`cat VERSION 2> /dev/null || git describe --tags --match "v[0-9]*" 2>/dev/null`
 
 get_version()
 {
@@ -18,7 +22,11 @@ get_version()
       sub(/^v/,"") ; print $1
     }'
 
-    echo $PKG_VERSION | awk "$AWK_VERSION" | tr -cd '[:alnum:].'
+    version=$(echo $PKG_VERSION | awk "$AWK_VERSION" | tr -cd '[:alnum:].')
+    if [ "x${version}" == "x" ] ; then
+       version=$(date +%Y.%m.%d | tr -d '\n')
+    fi
+    echo $version | tr -d '\n'
 }
 
 get_release()
@@ -37,7 +45,11 @@ get_release()
       else if (NF == 4) print $2, $3, "git" substr($4, 2)
     }'
 
-    echo $PKG_VERSION | awk "$AWK_RELEASE" | tr -cd '[:alnum:].'
+    release=$(echo $PKG_VERSION | awk "$AWK_RELEASE" | tr -cd '[:alnum:].')
+    if [ "x${release}" == "x" ] ; then
+       release=$(date +%H.%M%S | tr -d '\n')
+    fi
+    echo $release | tr -d '\n'
 }
 
 if test "x$1" = "x--full"; then
diff --git a/build-aux/xdrgen b/build-aux/xdrgen
deleted file mode 100755
index 0cefc9b4890..00000000000
--- a/build-aux/xdrgen
+++ /dev/null
@@ -1,104 +0,0 @@
-#!/bin/sh
-
-append_licence_header ()
-{
-    local src_file=$1;
-    local dst_file=$2;
-
-    cat >$dst_file <<EOF
-/*
-  Copyright (c) 2007-2014 Red Hat, Inc. <http://www.redhat.com>
-  This file is part of GlusterFS.
-
-  This file is licensed to you under your choice of the GNU Lesser
-  General Public License, version 3 or any later version (LGPLv3 or
-  later), or the GNU General Public License, version 2 (GPLv2), in all
-  cases as published by the Free Software Foundation.
-*/
-
-#include "compat.h"
-#include "xdr-common.h"
-#include "xdr-nfs3.h"
-
-#if defined(__GNUC__)
-#if __GNUC__ >= 4
-#if !defined(__clang__)
-#if !defined(__NetBSD__)
-#pragma GCC diagnostic ignored "-Wunused-but-set-variable"
-#pragma GCC diagnostic ignored "-Wunused-variable"
-#endif
-#else
-#pragma clang diagnostic ignored "-Wunused-variable"
-#pragma clang diagnostic ignored "-Wunused-value"
-#endif
-#endif
-#endif
-
-EOF
-
-    cat $src_file >> $dst_file;
-
-}
-
-gen_sources ()
-{
-    local xfile="$1";
-
-    local cfile="${1%.x}.c";
-    local hfile="${1%.x}.h";
-
-    local tmp_cfile="$1.c.tmp";
-    local tmp_hfile="$1.h.tmp";
-
-    rm -f $cfile;
-    rpcgen -c -o $cfile $xfile;
-    append_licence_header $cfile $tmp_cfile;
-    mv $tmp_cfile $cfile;
-    # remove unwanted temporary files (if any)
-    rm -f $tmp_cfile;
-    echo "Generated $cfile"
-    return
-}
-
-gen_headers ()
-{
-    local xfile="$1";
-
-    local cfile="${1%.x}.c";
-    local hfile="${1%.x}.h";
-
-    local tmp_cfile="$1.c.tmp";
-    local tmp_hfile="$1.h.tmp";
-
-    # no need for a temporary file here as there are no changes from glusterfs
-    rm -f $hfile;
-    rpcgen -h -o $hfile $xfile;
-    # the '#ifdef' part of file should be fixed
-    sed -e 's/-/_/g' $hfile > ${hfile}.new && mv ${hfile}.new $hfile;
-    # Gen header to temp file and append generated file
-    append_licence_header $hfile $tmp_hfile;
-    # now move the destination file to actual original file
-    mv $tmp_hfile $hfile;
-    rm -f $tmp_hfile;
-    echo "Generated $hfile";
-    return
-}
-
-main ()
-{
-    if [ $# -ne 2 ]; then
-        echo "wrong number of arguments given"
-        echo " $0 [header|source] <XDR-definition-file>.x"
-        exit 1;
-    fi
-
-    if [ $1 = "header" ]; then
-        gen_headers $2
-    fi
-
-    if [ $1 = "source" ]; then
-        gen_sources $2
-    fi
-}
-
-main "$@";
diff --git a/cli/src/Makefile.am b/cli/src/Makefile.am
index 434122b2a78..16063f27c7f 100644
--- a/cli/src/Makefile.am
+++ b/cli/src/Makefile.am
@@ -1,10 +1,11 @@
 sbin_PROGRAMS = gluster
 
-gluster_SOURCES = cli.c registry.c input.c cli-cmd.c cli-rl.c \
+gluster_SOURCES = cli.c registry.c input.c cli-cmd.c cli-rl.c cli-cmd-global.c \
 	 cli-cmd-volume.c cli-cmd-peer.c cli-rpc-ops.c cli-cmd-parser.c\
 	 cli-cmd-system.c cli-cmd-misc.c cli-xml-output.c cli-quotad-client.c cli-cmd-snapshot.c
 
 gluster_LDADD = $(top_builddir)/libglusterfs/src/libglusterfs.la $(GF_LDADD) \
+		$(top_builddir)/libglusterd/src/libglusterd.la \
 		$(RLLIBS) $(top_builddir)/rpc/xdr/src/libgfxdr.la \
 		$(top_builddir)/rpc/rpc-lib/src/libgfrpc.la \
 		$(XML_LIBS)
@@ -13,20 +14,24 @@ gluster_LDFLAGS = $(GF_LDFLAGS)
 noinst_HEADERS = cli.h cli-mem-types.h cli-cmd.h cli-quotad-client.h
 
 AM_CPPFLAGS = $(GF_CPPFLAGS) \
-	-I$(top_srcdir)/libglusterfs/src -I$(top_srcdir)/rpc/rpc-lib/src\
-	-I$(top_srcdir)/rpc/xdr/src\
+	-I$(top_srcdir)/libglusterfs/src -I$(top_srcdir)/rpc/rpc-lib/src \
+	-I$(top_srcdir)/rpc/xdr/src -I$(top_srcdir)/libglusterd/src \
+	-I$(top_builddir)/rpc/xdr/src \
 	-DDATADIR=\"$(localstatedir)\" \
 	-DCONFDIR=\"$(sysconfdir)/glusterfs\" \
-	-DGSYNCD_PREFIX=\"$(libexecdir)/glusterfs\"\
-	-DSYNCDAEMON_COMPILE=$(SYNCDAEMON_COMPILE) -DSBIN_DIR=\"$(sbindir)\"\
-	$(XML_CPPFLAGS)
+	-DGSYNCD_PREFIX=\"$(GLUSTERFS_LIBEXECDIR)\"\
+	-DGLFSHEAL_PREFIX=\"$(GLUSTERFS_LIBEXECDIR)\"\
+	-DSYNCDAEMON_COMPILE=$(SYNCDAEMON_COMPILE)
 
-AM_CFLAGS = -Wall $(GF_CFLAGS)
+AM_CFLAGS = -Wall $(GF_CFLAGS) $(XML_CFLAGS)
 
 CLEANFILES =
 
 $(top_builddir)/libglusterfs/src/libglusterfs.la:
 	$(MAKE) -C $(top_builddir)/libglusterfs/src/ all
 
+$(top_builddir)/libglusterd/src/libglusterd.la:
+	$(MAKE) -C $(top_builddir)/libglusterd/src/ all
+
 install-data-hook:
 	$(mkdir_p) $(DESTDIR)$(localstatedir)/run/gluster
diff --git a/cli/src/cli-cmd-global.c b/cli/src/cli-cmd-global.c
new file mode 100644
index 00000000000..2c9a5f01bb1
--- /dev/null
+++ b/cli/src/cli-cmd-global.c
@@ -0,0 +1,195 @@
+/*
+   Copyright (c) 2015 Red Hat, Inc. <http://www.redhat.com>
+   This file is part of GlusterFS.
+
+   This file is licensed to you under your choice of the GNU Lesser
+   General Public License, version 3 or any later version (LGPLv3 or
+   later), or the GNU General Public License, version 2 (GPLv2), in all
+   cases as published by the Free Software Foundation.
+*/
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <pthread.h>
+
+#include <sys/socket.h>
+#include <netdb.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+#include <netinet/in.h>
+
+#include "cli.h"
+#include "cli-cmd.h"
+#include "cli-mem-types.h"
+#include "cli1-xdr.h"
+#include <glusterfs/run.h>
+#include <glusterfs/syscall.h>
+#include <glusterfs/common-utils.h>
+
+int
+cli_cmd_global_help_cbk(struct cli_state *state, struct cli_cmd_word *in_word,
+                        const char **words, int wordcount);
+int
+cli_cmd_get_state_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                      const char **words, int wordcount);
+
+int
+cli_cmd_ganesha_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                    const char **words, int wordcount);
+
+struct cli_cmd global_cmds[] = {
+    {
+        "global help",
+        cli_cmd_global_help_cbk,
+        "list global commands",
+    },
+    {
+        "get-state [<daemon>] [[odir </path/to/output/dir/>] "
+        "[file <filename>]] [detail|volumeoptions]",
+        cli_cmd_get_state_cbk,
+        "Get local state representation of mentioned daemon",
+    },
+    {
+        "nfs-ganesha {enable| disable} ",
+        cli_cmd_ganesha_cbk,
+        "Enable/disable NFS-Ganesha support",
+    },
+    {NULL, NULL, NULL}};
+
+int
+cli_cmd_global_help_cbk(struct cli_state *state, struct cli_cmd_word *in_word,
+                        const char **words, int wordcount)
+{
+    struct cli_cmd *cmd = NULL;
+    struct cli_cmd *global_cmd = NULL;
+    int count = 0;
+
+    cmd = GF_MALLOC(sizeof(global_cmds), cli_mt_cli_cmd);
+    memcpy(cmd, global_cmds, sizeof(global_cmds));
+    count = (sizeof(global_cmds) / sizeof(struct cli_cmd));
+    cli_cmd_sort(cmd, count);
+
+    cli_out("\ngluster global commands");
+    cli_out("========================\n");
+    for (global_cmd = cmd; global_cmd->pattern; global_cmd++)
+        if (_gf_false == global_cmd->disable)
+            cli_out("%s - %s", global_cmd->pattern, global_cmd->desc);
+
+    cli_out("\n");
+    GF_FREE(cmd);
+    return 0;
+}
+
+int
+cli_cmd_global_register(struct cli_state *state)
+{
+    int ret = 0;
+    struct cli_cmd *cmd = NULL;
+    for (cmd = global_cmds; cmd->pattern; cmd++) {
+        ret = cli_cmd_register(&state->tree, cmd);
+        if (ret)
+            goto out;
+    }
+out:
+    return ret;
+}
+
+int
+cli_cmd_ganesha_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                    const char **words, int wordcount)
+
+{
+    int sent = 0;
+    int parse_error = 0;
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *options = NULL;
+    cli_local_t *local = NULL;
+    char *op_errstr = NULL;
+
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_GANESHA];
+
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame)
+        goto out;
+
+    ret = cli_cmd_ganesha_parse(state, words, wordcount, &options, &op_errstr);
+    if (ret) {
+        if (op_errstr) {
+            cli_err("%s", op_errstr);
+            GF_FREE(op_errstr);
+        } else
+            cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
+
+    CLI_LOCAL_INIT(local, words, frame, options);
+
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, options);
+    }
+
+out:
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Setting global option failed");
+    }
+
+    CLI_STACK_DESTROY(frame);
+    return ret;
+}
+
+int
+cli_cmd_get_state_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                      const char **words, int wordcount)
+{
+    int sent = 0;
+    int parse_error = 0;
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *options = NULL;
+    cli_local_t *local = NULL;
+    char *op_errstr = NULL;
+
+    ret = cli_cmd_get_state_parse(state, words, wordcount, &options,
+                                  &op_errstr);
+
+    if (ret) {
+        if (op_errstr) {
+            cli_err("%s", op_errstr);
+            cli_usage_out(word->pattern);
+            GF_FREE(op_errstr);
+        } else
+            cli_usage_out(word->pattern);
+
+        parse_error = 1;
+        goto out;
+    }
+
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
+
+    CLI_LOCAL_INIT(local, words, frame, options);
+
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_GET_STATE];
+    if (proc->fn)
+        ret = proc->fn(frame, THIS, options);
+out:
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Getting daemon state failed");
+    }
+
+    CLI_STACK_DESTROY(frame);
+
+    return ret;
+}
diff --git a/cli/src/cli-cmd-misc.c b/cli/src/cli-cmd-misc.c
index 566d7c978d9..e961d88da86 100644
--- a/cli/src/cli-cmd-misc.c
+++ b/cli/src/cli-cmd-misc.c
@@ -13,86 +13,105 @@
 #include <stdint.h>
 #include <pthread.h>
 
-#ifndef _CONFIG_H
-#define _CONFIG_H
-#include "config.h"
-#endif
-
 #include "cli.h"
 #include "cli-cmd.h"
 #include "cli-mem-types.h"
 #include "protocol-common.h"
 
-extern struct rpc_clnt *global_rpc;
-
-extern rpc_clnt_prog_t *cli_rpc_prog;
-
 extern struct cli_cmd volume_cmds[];
+extern struct cli_cmd bitrot_cmds[];
+extern struct cli_cmd quota_cmds[];
 extern struct cli_cmd cli_probe_cmds[];
 extern struct cli_cmd cli_log_cmds[];
 extern struct cli_cmd cli_system_cmds[];
 extern struct cli_cmd cli_bd_cmds[];
 extern struct cli_cmd snapshot_cmds[];
+extern struct cli_cmd global_cmds[];
 struct cli_cmd cli_misc_cmds[];
 
 int
-cli_cmd_quit_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                   const char **words, int wordcount)
+cli_cmd_quit_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                 const char **words, int wordcount)
 {
-        exit (0);
+    exit(0);
 }
 
-int
-cli_cmd_display_help (struct cli_state *state, struct cli_cmd_word *in_word,
-                      const char **words, int wordcount)
+static gf_boolean_t
+cli_is_help_command(const char *pattern)
 {
-        struct cli_cmd        *cmd[] = {volume_cmds, cli_probe_cmds,
-                                       cli_misc_cmds, snapshot_cmds,
-                                       NULL};
-        struct cli_cmd        *cmd_ind = NULL;
-        int                   i = 0;
-
-         /* cli_system_cmds commands for internal usage
-           they are not exposed
-         */
-        for (i=0; cmd[i]!=NULL; i++)
-                for (cmd_ind = cmd[i]; cmd_ind->pattern; cmd_ind++)
-                        if (_gf_false == cmd_ind->disable)
-                                cli_out ("%s - %s", cmd_ind->pattern,
-                                         cmd_ind->desc);
-
-        return 0;
+    /* FixFixFix
+     * This is not the best way to determine whether
+     * this is a help command
+     */
+    if (strstr(pattern, "help"))
+        return _gf_true;
+
+    return _gf_false;
 }
 
-struct cli_cmd cli_misc_cmds[] = {
-        { "quit",
-          cli_cmd_quit_cbk,
-          "quit"},
+int
+cli_cmd_display_help(struct cli_state *state, struct cli_cmd_word *in_word,
+                     const char **words, int wordcount)
+{
+    static struct cli_cmd *cmd[] = {
+        cli_misc_cmds, cli_probe_cmds, volume_cmds, bitrot_cmds,
+        quota_cmds,    snapshot_cmds,  global_cmds, NULL};
+    struct cli_cmd *cmd_ind = NULL;
+    int i = 0;
+    gf_boolean_t list_all = _gf_false;
+
+    /* cli_system_cmds commands for internal usage
+      they are not exposed
+    */
+
+    /* If "help all" */
+    if (wordcount == 2)
+        list_all = _gf_true;
+
+    for (i = 0; cmd[i] != NULL; i++) {
+        for (cmd_ind = cmd[i]; cmd_ind->pattern; cmd_ind++) {
+            if ((_gf_false == cmd_ind->disable) &&
+                cli_is_help_command(cmd_ind->pattern)) {
+                if (list_all && (cmd_ind->cbk)) {
+                    cmd_ind->cbk(state, in_word, words, wordcount);
+                } else {
+                    cli_out(" %-25s- %s", cmd_ind->pattern, cmd_ind->desc);
+                }
+            }
+        }
+    }
+
+    cli_out("\n");
+    return 0;
+}
 
-        { "help",
-           cli_cmd_display_help,
-           "display command options"},
+struct cli_cmd cli_help_cmds[] = {
+    {"help [all]", cli_cmd_display_help, "display help for command classes"},
 
-        { "exit",
-          cli_cmd_quit_cbk,
-          "exit"},
+    {NULL, NULL, NULL}};
 
-        { NULL, NULL, NULL }
-};
+struct cli_cmd cli_misc_cmds[] = {{"quit", cli_cmd_quit_cbk, "quit"},
+                                  {"exit", cli_cmd_quit_cbk, "exit"},
 
+                                  {NULL, NULL, NULL}};
 
 int
-cli_cmd_misc_register (struct cli_state *state)
+cli_cmd_misc_register(struct cli_state *state)
 {
-        int  ret = 0;
-        struct cli_cmd *cmd = NULL;
-
-        for (cmd = cli_misc_cmds; cmd->pattern; cmd++) {
-
-                ret = cli_cmd_register (&state->tree, cmd);
-                if (ret)
-                        goto out;
-        }
+    int ret = 0;
+    struct cli_cmd *cmd = NULL;
+
+    for (cmd = cli_misc_cmds; cmd->pattern; cmd++) {
+        ret = cli_cmd_register(&state->tree, cmd);
+        if (ret)
+            goto out;
+    }
+
+    for (cmd = cli_help_cmds; cmd->pattern; cmd++) {
+        ret = cli_cmd_register(&state->tree, cmd);
+        if (ret)
+            goto out;
+    }
 out:
-        return ret;
+    return ret;
 }
diff --git a/cli/src/cli-cmd-parser.c b/cli/src/cli-cmd-parser.c
index 28888ba656d..34620b4a31b 100644
--- a/cli/src/cli-cmd-parser.c
+++ b/cli/src/cli-cmd-parser.c
@@ -12,3193 +12,4306 @@
 #include <stdlib.h>
 #include <stdint.h>
 #include <pthread.h>
-
-#ifndef _CONFIG_H
-#define _CONFIG_H
-#include "config.h"
-#endif
+#include <fnmatch.h>
+#include <time.h>
 
 #include "cli.h"
 #include "cli-cmd.h"
 #include "cli-mem-types.h"
-#include "dict.h"
+#include <glusterfs/dict.h>
+#include <glusterfs/list.h>
 
 #include "protocol-common.h"
 #include "cli1-xdr.h"
 
 #define MAX_SNAP_DESCRIPTION_LEN 1024
 
-struct snap_config_opt_vals_ snap_confopt_vals[] = {
-        {.op_name        = "snap-max-hard-limit",
-         .question       = "Changing snapshot-max-hard-limit "
-                           "will lead to deletion of snapshots "
-                           "if they exceed the new limit.\n"
-                           "Do you want to continue?"
-        },
-        {.op_name        = "snap-max-soft-limit",
-         .question       = "Changing snapshot-max-soft-limit "
-                           "will lead to deletion of snapshots "
-                           "if they exceed the new limit.\n"
-                           "Do you want to continue?"
-        },
-        {.op_name        = "both",
-        .question        = "Changing snapshot-max-hard-limit & "
-                           "snapshot-max-soft-limit will lead to "
-                           "deletion of snapshots if they exceed "
-                           "the new limit.\nDo you want to continue?"
-        },
-        {.op_name        = NULL,
-        }
-};
+static struct snap_config_opt_vals_ snap_confopt_vals[] = {
+    {.op_name = "snap-max-hard-limit",
+     .question = "Changing snapshot-max-hard-limit "
+                 "will limit the creation of new snapshots "
+                 "if they exceed the new limit.\n"
+                 "Do you want to continue?"},
+    {.op_name = "snap-max-soft-limit",
+     .question = "If Auto-delete is enabled, snap-max-soft-limit will"
+                 " trigger deletion of oldest snapshot, on the "
+                 "creation of new snapshot, when the "
+                 "snap-max-soft-limit is reached.\n"
+                 "Do you want to change the snap-max-soft-limit?"},
+    {.op_name = "both",
+     .question = "Changing snapshot-max-hard-limit "
+                 "will limit the creation of new snapshots "
+                 "if they exceed the new snapshot-max-hard-limit.\n"
+                 "If Auto-delete is enabled, snap-max-soft-limit will"
+                 " trigger deletion of oldest snapshot, on the "
+                 "creation of new snapshot, when the "
+                 "snap-max-soft-limit is reached.\n"
+                 "Do you want to continue?"},
+    {
+        .op_name = NULL,
+    }};
 
 enum cli_snap_config_set_types {
-        GF_SNAP_CONFIG_SET_HARD = 0,
-        GF_SNAP_CONFIG_SET_SOFT = 1,
-        GF_SNAP_CONFIG_SET_BOTH = 2,
+    GF_SNAP_CONFIG_SET_HARD = 0,
+    GF_SNAP_CONFIG_SET_SOFT = 1,
+    GF_SNAP_CONFIG_SET_BOTH = 2,
 };
 typedef enum cli_snap_config_set_types cli_snap_config_set_types;
 
+typedef struct _cli_brick {
+    struct list_head list;
+    const char *name;
+    int32_t len;
+} cli_brick_t;
+
+int
+cli_cmd_validate_volume(char *volname);
+
 static const char *
-id_sel (void *wcon)
+id_sel(void *wcon)
 {
-        return (const char *)wcon;
+    return (const char *)wcon;
 }
 
 static char *
-str_getunamb (const char *tok, char **opwords)
+str_getunamb(const char *tok, char **opwords)
 {
-        return (char *)cli_getunamb (tok, (void **)opwords, id_sel);
+    return (char *)cli_getunamb(tok, (void **)opwords, id_sel);
 }
 
 int32_t
-cli_cmd_bricks_parse (const char **words, int wordcount, int brick_index,
-                      char **bricks, int *brick_count)
+cli_cmd_ta_brick_parse(const char **words, int wordcount, char **ta_brick)
 {
-        int                    ret = 0;
-        char             *tmp_list = NULL;
-        char    brick_list[120000] = {0,};
-        char                *space = " ";
-        char            *delimiter = NULL;
-        char            *host_name = NULL;
-        char        *free_list_ptr = NULL;
-        char               *tmpptr = NULL;
-        int                      j = 0;
-        int         brick_list_len = 0;
-        char             *tmp_host = NULL;
-
-        GF_ASSERT (words);
-        GF_ASSERT (wordcount);
-        GF_ASSERT (bricks);
-        GF_ASSERT (brick_index > 0);
-        GF_ASSERT (brick_index < wordcount);
-
-        strncpy (brick_list, space, strlen (space));
-        brick_list_len++;
-        while (brick_index < wordcount) {
-                if (validate_brick_name ((char *)words[brick_index])) {
-                        cli_err ("Wrong brick type: %s, use <HOSTNAME>:"
-                                 "<export-dir-abs-path>", words[brick_index]);
-                        ret = -1;
-                        goto out;
-                } else {
-                        delimiter = strrchr (words[brick_index], ':');
-                        ret = gf_canonicalize_path (delimiter + 1);
-                        if (ret)
-                                goto out;
-                }
-
-                if ((brick_list_len + strlen (words[brick_index]) + 1) > sizeof (brick_list)) {
-                        cli_err ("Total brick list is larger than a request. "
-                                 "Can take (brick_count %d)", *brick_count);
-                        ret = -1;
-                        goto out;
-                }
+    char *host_name = NULL;
+    char *tmp_host = NULL;
+    char *delimiter = NULL;
+    cli_brick_t *brick = NULL;
+    int ret = 0;
+
+    GF_ASSERT(words);
+    GF_ASSERT(wordcount);
+
+    if (validate_brick_name((char *)words[wordcount - 1])) {
+        cli_err(
+            "Wrong brick type: %s, use <HOSTNAME>:"
+            "<export-dir-abs-path>",
+            words[wordcount - 1]);
+        ret = -1;
+        goto out;
+    } else {
+        delimiter = strrchr(words[wordcount - 1], ':');
+        ret = gf_canonicalize_path(delimiter + 1);
+        if (ret)
+            goto out;
+    }
 
-                tmp_host = gf_strdup ((char *)words[brick_index]);
-                if (!tmp_host) {
-                        gf_log ("cli", GF_LOG_ERROR, "Out of memory");
-                        ret = -1;
-                        goto out;
-                }
-                get_host_name (tmp_host, &host_name);
-                if (!host_name) {
-                        ret = -1;
-                        gf_log("cli",GF_LOG_ERROR, "Unable to allocate "
-                               "memory");
-                        goto out;
-                }
+    tmp_host = gf_strdup((char *)words[wordcount - 1]);
+    if (!tmp_host) {
+        gf_log("cli", GF_LOG_ERROR, "Out of memory");
+        ret = -1;
+        goto out;
+    }
+    get_host_name(tmp_host, &host_name);
+    if (!host_name) {
+        ret = -1;
+        gf_log("cli", GF_LOG_ERROR,
+               "Unable to retrieve "
+               "hostname");
+        goto out;
+    }
+
+    if (!(strcmp(host_name, "localhost") && strcmp(host_name, "127.0.0.1") &&
+          strncmp(host_name, "0.", 2))) {
+        cli_err(
+            "Please provide a valid hostname/ip other "
+            "than localhost, 127.0.0.1 or loopback "
+            "address (0.0.0.0 to 0.255.255.255).");
+        ret = -1;
+        goto out;
+    }
+    if (!valid_internet_address(host_name, _gf_false, _gf_false)) {
+        cli_err(
+            "internet address '%s' does not conform to "
+            "standards",
+            host_name);
+    }
+
+    brick = GF_MALLOC(sizeof(cli_brick_t), gf_common_list_node);
+    if (brick == NULL) {
+        ret = -1;
+        gf_log("cli", GF_LOG_ERROR, "Out of memory");
+        goto out;
+    }
 
-                if (!(strcmp (host_name, "localhost") &&
-                      strcmp (host_name, "127.0.0.1") &&
-                      strncmp (host_name, "0.", 2))) {
-                        cli_err ("Please provide a valid hostname/ip other "
-                                 "than localhost, 127.0.0.1 or loopback "
-                                 "address (0.0.0.0 to 0.255.255.255).");
-                        ret = -1;
-                        GF_FREE (tmp_host);
-                        goto out;
-                }
-                if (!valid_internet_address (host_name, _gf_false)) {
-                        cli_err ("internet address '%s' does not conform to "
-                                 "standards", host_name);
-                }
-                GF_FREE (tmp_host);
-                tmp_list = gf_strdup (brick_list + 1);
-                if (free_list_ptr) {
-                        GF_FREE (free_list_ptr);
-                        free_list_ptr = NULL;
-                }
-                free_list_ptr = tmp_list;
-                j = 0;
-                while(j < *brick_count) {
-                        strtok_r (tmp_list, " ", &tmpptr);
-                        if (!(strcmp (tmp_list, words[brick_index]))) {
-                                ret = -1;
-                                cli_err ("Found duplicate"
-                                         " exports %s",words[brick_index]);
-                                goto out;
-                       }
-                       tmp_list = tmpptr;
-                       j++;
-                }
-                strcat (brick_list, words[brick_index]);
-                strcat (brick_list, " ");
-                brick_list_len += (strlen (words[brick_index]) + 1);
-                ++(*brick_count);
-                ++brick_index;
-        }
+    brick->name = words[wordcount - 1];
+    brick->len = strlen(words[wordcount - 1]);
+    *ta_brick = GF_MALLOC(brick->len + 3, gf_common_mt_char);
+    if (*ta_brick == NULL) {
+        ret = -1;
+        gf_log("cli", GF_LOG_ERROR, "Out of memory");
+        goto out;
+    }
 
-        *bricks = gf_strdup (brick_list);
-        if (!*bricks)
-                ret = -1;
+    strcat(*ta_brick, " ");
+    strcat(*ta_brick, brick->name);
+    strcat(*ta_brick, " ");
 out:
-        GF_FREE (free_list_ptr);
-        return ret;
+    if (tmp_host) {
+        GF_FREE(tmp_host);
+        tmp_host = NULL;
+    }
+    if (brick) {
+        GF_FREE(brick);
+        brick = NULL;
+    }
+
+    return ret;
 }
 
 int32_t
-cli_cmd_create_disperse_check(struct cli_state * state, int * disperse,
-                              int * redundancy, int count)
+cli_cmd_bricks_parse(const char **words, int wordcount, int brick_index,
+                     char **bricks, int *brick_count)
 {
-        int i = 0;
-        int tmp = 0;
-        gf_answer_t answer = GF_ANSWER_NO;
-        char question[128];
+    int ret = 0;
+    char *delimiter = NULL;
+    char *host_name = NULL;
+    char *tmp_host = NULL;
+    char *bricks_str = NULL;
+    int len = 0;
+    int brick_list_len = 1; /* For initial space */
+    struct list_head brick_list = {
+        0,
+    };
+    cli_brick_t *brick = NULL;
+
+    GF_ASSERT(words);
+    GF_ASSERT(wordcount);
+    GF_ASSERT(bricks);
+    GF_ASSERT(brick_index > 0);
+    GF_ASSERT(brick_index < wordcount);
+
+    INIT_LIST_HEAD(&brick_list);
+
+    while (brick_index < wordcount) {
+        if (validate_brick_name((char *)words[brick_index])) {
+            cli_err(
+                "Wrong brick type: %s, use <HOSTNAME>:"
+                "<export-dir-abs-path>",
+                words[brick_index]);
+            ret = -1;
+            goto out;
+        } else {
+            delimiter = strrchr(words[brick_index], ':');
+            ret = gf_canonicalize_path(delimiter + 1);
+            if (ret)
+                goto out;
+        }
 
-        const char * question1 = "There isn't an optimal redundancy value "
-                                 "for this configuration. Do you want to "
-                                 "create the volume with redundancy 1 ?";
+        tmp_host = gf_strdup((char *)words[brick_index]);
+        if (!tmp_host) {
+            gf_log("cli", GF_LOG_ERROR, "Out of memory");
+            ret = -1;
+            goto out;
+        }
+        get_host_name(tmp_host, &host_name);
+        if (!host_name) {
+            ret = -1;
+            gf_log("cli", GF_LOG_ERROR,
+                   "Unable to allocate "
+                   "memory");
+            GF_FREE(tmp_host);
+            goto out;
+        }
+
+        if (!(strcmp(host_name, "localhost") &&
+              strcmp(host_name, "127.0.0.1") && strncmp(host_name, "0.", 2))) {
+            cli_err(
+                "Please provide a valid hostname/ip other "
+                "than localhost, 127.0.0.1 or loopback "
+                "address (0.0.0.0 to 0.255.255.255).");
+            ret = -1;
+            GF_FREE(tmp_host);
+            goto out;
+        }
+        if (!valid_internet_address(host_name, _gf_false, _gf_false)) {
+            cli_err(
+                "internet address '%s' does not conform to "
+                "standards",
+                host_name);
+        }
+        GF_FREE(tmp_host);
+        list_for_each_entry(brick, &brick_list, list)
+        {
+            if (strcmp(brick->name, words[brick_index]) == 0) {
+                ret = -1;
+                cli_err("Found duplicate exports %s", words[brick_index]);
+                goto out;
+            }
+        }
 
-        const char * question2 = "The optimal redundancy for this "
-                                 "configuration is %d. Do you want to create "
-                                 "the volume with this value ?";
+        brick = GF_MALLOC(sizeof(cli_brick_t), gf_common_list_node);
+        if (brick == NULL) {
+            ret = -1;
+            gf_log("cli", GF_LOG_ERROR, "Out of memory");
+            goto out;
+        }
+        len = strlen(words[brick_index]);
+        brick->name = words[brick_index];
+        brick->len = len;
+        list_add_tail(&brick->list, &brick_list);
 
-        const char * question3 = "This configuration is not optimal on most "
-                                 "workloads. Do you want to use it ?";
+        brick_list_len += len + 1; /* Brick name + space */
+        ++(*brick_count);
+        ++brick_index;
+    }
 
-        if (*disperse <= 0) {
-                if (count < 3) {
-                        cli_err ("number of bricks must be greater "
-                                 "than 2");
+    /* If brick count is not valid exit here */
+    if (!*brick_count) {
+        cli_err("No bricks specified");
+        ret = -1;
+        goto out;
+    }
 
-                        return -1;
-                }
-                *disperse = count;
-        }
+    brick_list_len++; /* For terminating null char */
 
-        if (*redundancy == 0) {
-                tmp = *disperse - 1;
-                for (i = tmp / 2;
-                     (i > 0) && ((tmp & -tmp) != tmp);
-                     i--, tmp--);
+    bricks_str = GF_MALLOC(brick_list_len, gf_common_mt_char);
+    if (bricks_str == NULL) {
+        ret = -1;
+        gf_log("cli", GF_LOG_ERROR, "Out of memory");
+        goto out;
+    }
+    *bricks = bricks_str;
+    *bricks_str = ' ';
+    bricks_str++;
+    while (!list_empty(&brick_list)) {
+        brick = list_first_entry(&brick_list, cli_brick_t, list);
+        list_del_init(&brick->list);
+        memcpy(bricks_str, brick->name, brick->len);
+        bricks_str[brick->len] = ' ';
+        bricks_str += brick->len + 1;
+        GF_FREE(brick);
+    }
+    *bricks_str = 0;
 
-                if (i == 0) {
-                        answer = cli_cmd_get_confirmation(state, question1);
-                        if (answer == GF_ANSWER_NO)
-                                return -1;
+out:
+    while (!list_empty(&brick_list)) {
+        brick = list_first_entry(&brick_list, cli_brick_t, list);
+        list_del_init(&brick->list);
+        GF_FREE(brick);
+    }
 
-                        *redundancy = 1;
-                }
-                else
-                {
-                        *redundancy = *disperse - tmp;
-                        if (*redundancy > 1) {
-                                sprintf(question, question2, *redundancy);
-                                answer = cli_cmd_get_confirmation(state,
-                                                                  question);
-                                if (answer == GF_ANSWER_NO)
-                                        return -1;
-                        }
-                }
+    return ret;
+}
 
-                tmp = 0;
+int32_t
+cli_cmd_create_disperse_check(struct cli_state *state, int *disperse,
+                              int *redundancy, int *data, int count)
+{
+    int i = 0;
+    int tmp = 0;
+    gf_answer_t answer = GF_ANSWER_NO;
+    char question[128];
+
+    const char *question1 =
+        "There isn't an optimal redundancy value "
+        "for this configuration. Do you want to "
+        "create the volume with redundancy 1 ?";
+
+    const char *question2 =
+        "The optimal redundancy for this "
+        "configuration is %d. Do you want to create "
+        "the volume with this value ?";
+
+    const char *question3 =
+        "This configuration is not optimal on most "
+        "workloads. Do you want to use it ?";
+
+    const char *question4 =
+        "Redundancy for this configuration is %d. "
+        "Do you want to create "
+        "the volume with this value ?";
+
+    if (*data > 0) {
+        if (*disperse > 0 && *redundancy > 0) {
+            if (*disperse != (*data + *redundancy)) {
+                cli_err(
+                    "Disperse count(%d) should be equal "
+                    "to sum of disperse-data count(%d) and "
+                    "redundancy count(%d)",
+                    *disperse, *data, *redundancy);
+                return -1;
+            }
+        } else if (*redundancy > 0) {
+            *disperse = *data + *redundancy;
+        } else if (*disperse > 0) {
+            *redundancy = *disperse - *data;
+        } else {
+            if ((count - *data) >= *data) {
+                cli_err(
+                    "Please provide redundancy count "
+                    "along with disperse-data count");
+                return -1;
+            } else {
+                sprintf(question, question4, count - *data);
+                answer = cli_cmd_get_confirmation(state, question);
+                if (answer == GF_ANSWER_NO)
+                    return -1;
+                *redundancy = count - *data;
+                *disperse = count;
+            }
         }
-        else {
-                tmp = *disperse - *redundancy;
+    }
+
+    if (*disperse <= 0) {
+        if (count < 3) {
+            cli_err(
+                "number of bricks must be greater "
+                "than 2");
+
+            return -1;
         }
+        *disperse = count;
+    }
 
-        if (*redundancy > (*disperse - 1) / 2) {
-                cli_err ("redundancy must be less than %d for a "
-                         "disperse %d volume",
-                         (*disperse + 1) / 2, *disperse);
+    if (*redundancy == -1) {
+        tmp = *disperse - 1;
+        for (i = tmp / 2; (i > 0) && ((tmp & -tmp) != tmp); i--, tmp--)
+            ;
 
+        if (i == 0) {
+            answer = cli_cmd_get_confirmation(state, question1);
+            if (answer == GF_ANSWER_NO)
                 return -1;
-        }
 
-        if ((tmp & -tmp) != tmp) {
-                answer = cli_cmd_get_confirmation(state, question3);
+            *redundancy = 1;
+        } else {
+            *redundancy = *disperse - tmp;
+            if (*redundancy > 1) {
+                sprintf(question, question2, *redundancy);
+                answer = cli_cmd_get_confirmation(state, question);
                 if (answer == GF_ANSWER_NO)
-                        return -1;
+                    return -1;
+            }
         }
 
-        return 0;
-}
-
-int32_t
-cli_cmd_volume_create_parse (struct cli_state *state, const char **words,
-                             int wordcount, dict_t **options)
-{
-        dict_t  *dict = NULL;
-        char    *volname = NULL;
-        int     ret = -1;
-        gf1_cluster_type type = GF_CLUSTER_TYPE_NONE;
-        int     count = 1;
-        int     sub_count = 1;
-        int     brick_index = 0;
-        int     i = 0;
-        char    *trans_type = NULL;
-        int32_t index = 0;
-        char    *bricks = NULL;
-        int32_t brick_count = 0;
-        char    *opwords[] = { "replica", "stripe", "transport", "disperse",
-                               "redundancy", NULL };
-
-        char    *invalid_volnames[] = {"volume", "type", "subvolumes", "option",
-                                       "end-volume", "all", "volume_not_in_ring",
-                                       "description", "force",
-                                       "snap-max-hard-limit",
-                                       "snap-max-soft-limit", "auto-delete",
-                                       "activate-on-create", NULL};
-        char    *w = NULL;
-        char    *ptr = NULL;
-        int      op_count = 0;
-        int32_t  replica_count = 1;
-        int32_t  stripe_count = 1;
-        int32_t  disperse_count = -1;
-        int32_t  redundancy_count = 0;
-        gf_boolean_t is_force = _gf_false;
-        int wc = wordcount;
-
-        GF_ASSERT (words);
-        GF_ASSERT (options);
-
-        dict = dict_new ();
-
-        if (!dict)
-                goto out;
+        tmp = 0;
+    } else {
+        tmp = *disperse - *redundancy;
+    }
 
-        if (wordcount < 3)
-                goto out;
+    if ((*redundancy < 1) || (*redundancy > (*disperse - 1) / 2)) {
+        cli_err(
+            "redundancy must be greater than or equal to 1 and "
+            "less than %d for a disperse %d volume",
+            (*disperse + 1) / 2, *disperse);
 
-        volname = (char *)words[2];
+        return -1;
+    }
 
-        GF_ASSERT (volname);
+    if ((tmp & -tmp) != tmp) {
+        answer = cli_cmd_get_confirmation(state, question3);
+        if (answer == GF_ANSWER_NO)
+            return -1;
+    }
 
-        /* Validate the volume name here itself */
-        {
-                if (volname[0] == '-')
-                        goto out;
-
-                for (i = 0; invalid_volnames[i]; i++) {
-                        if (!strcmp (volname, invalid_volnames[i])) {
-                                cli_err ("\"%s\" cannot be the name of a volume.",
-                                         volname);
-                                goto out;
-                        }
-                }
+    return 0;
+}
 
-                if (strchr (volname, '/'))
-                        goto out;
+static int32_t
+cli_validate_disperse_volume(char *word, gf1_cluster_type type,
+                             const char **words, int32_t wordcount,
+                             int32_t index, int32_t *disperse_count,
+                             int32_t *redundancy_count, int32_t *data_count)
+{
+    int ret = -1;
+
+    switch (type) {
+        case GF_CLUSTER_TYPE_NONE:
+        case GF_CLUSTER_TYPE_DISPERSE:
+            if (strcmp(word, "disperse") == 0) {
+                if (*disperse_count >= 0) {
+                    cli_err("disperse option given twice");
+                    goto out;
+                }
+                if (wordcount < (index + 2)) {
+                    goto out;
+                }
+                ret = gf_string2int(words[index + 1], disperse_count);
+                if (ret == -1 && errno == EINVAL) {
+                    *disperse_count = 0;
+                    ret = 1;
+                } else if (ret == -1) {
+                    goto out;
+                } else {
+                    if (*disperse_count < 3) {
+                        cli_err(
+                            "disperse count must "
+                            "be greater than 2");
+                        goto out;
+                    }
+                    ret = 2;
+                }
+            } else if (strcmp(word, "disperse-data") == 0) {
+                if (*data_count >= 0) {
+                    cli_err("disperse-data option given twice");
+                    goto out;
+                }
+                if (wordcount < (index + 2)) {
+                    goto out;
+                }
+                ret = gf_string2int(words[index + 1], data_count);
+                if (ret == -1 || *data_count < 2) {
+                    cli_err("disperse-data must be greater than 1");
+                    goto out;
+                }
+                ret = 2;
+            } else if (strcmp(word, "redundancy") == 0) {
+                if (*redundancy_count >= 0) {
+                    cli_err("redundancy option given twice");
+                    goto out;
+                }
+                if (wordcount < (index + 2)) {
+                    goto out;
+                }
+                ret = gf_string2int(words[index + 1], redundancy_count);
+                if (ret == -1 || *redundancy_count < 1) {
+                    cli_err("redundancy must be greater than 0");
+                    goto out;
+                }
+                ret = 2;
+            }
+            break;
+        case GF_CLUSTER_TYPE_REPLICATE:
+            cli_err(
+                "replicated-dispersed volume is not "
+                "supported");
+            goto out;
+        default:
+            cli_err("Invalid type given");
+            break;
+    }
+out:
+    return ret;
+}
 
-                if (strlen (volname) > GD_VOLUME_NAME_MAX) {
-                        cli_err("Volume name exceeds %d characters.",
-                                GD_VOLUME_NAME_MAX);
-                        goto out;
-                }
+int32_t
+cli_validate_volname(const char *volname)
+{
+    int32_t ret = -1;
+    int32_t i = -1;
+    int volname_len;
+    static const char *const invalid_volnames[] = {"volume",
+                                                   "type",
+                                                   "subvolumes",
+                                                   "option",
+                                                   "end-volume",
+                                                   "all",
+                                                   "volume_not_in_ring",
+                                                   "description",
+                                                   "force",
+                                                   "snap-max-hard-limit",
+                                                   "snap-max-soft-limit",
+                                                   "auto-delete",
+                                                   "activate-on-create",
+                                                   NULL};
+
+    if (volname[0] == '-')
+        goto out;
 
-                for (i = 0; i < strlen (volname); i++)
-                        if (!isalnum (volname[i]) && (volname[i] != '_') &&
-                           (volname[i] != '-')) {
-                                cli_err ("Volume name should not contain \"%c\""
-                                         " character.\nVolume names can only"
-                                         "contain alphanumeric, '-' and '_' "
-                                         "characters.",volname[i]);
-                                goto out;
-                        }
+    for (i = 0; invalid_volnames[i]; i++) {
+        if (!strcmp(volname, invalid_volnames[i])) {
+            cli_err("\"%s\" cannot be the name of a volume.", volname);
+            goto out;
         }
+    }
 
-        if (wordcount < 4) {
-                ret = -1;
-                goto out;
-        }
+    if (strchr(volname, '/'))
+        goto out;
 
-        type = GF_CLUSTER_TYPE_NONE;
-        index = 3;
+    volname_len = strlen(volname);
+    if (volname_len > GD_VOLUME_NAME_MAX) {
+        cli_err("Volume name exceeds %d characters.", GD_VOLUME_NAME_MAX);
+        goto out;
+    }
+
+    for (i = 0; i < volname_len; i++) {
+        if (!isalnum(volname[i]) && (volname[i] != '_') &&
+            (volname[i] != '-')) {
+            cli_err(
+                "Volume name should not contain \"%c\""
+                " character.\nVolume names can only"
+                "contain alphanumeric, '-' and '_' "
+                "characters.",
+                volname[i]);
+            goto out;
+        }
+    }
+
+    ret = 0;
+out:
+    return ret;
+}
 
-        while (op_count < 3) {
-                ret = -1;
-                w = str_getunamb (words[index], opwords);
-                if (!w) {
-                        break;
-                } else if ((strcmp (w, "replica")) == 0) {
-                        switch (type) {
-                        case GF_CLUSTER_TYPE_STRIPE_REPLICATE:
-                        case GF_CLUSTER_TYPE_REPLICATE:
-                                cli_err ("replica option given twice");
-                                goto out;
-                        case GF_CLUSTER_TYPE_NONE:
-                                type = GF_CLUSTER_TYPE_REPLICATE;
-                                break;
-                        case GF_CLUSTER_TYPE_STRIPE:
-                                type = GF_CLUSTER_TYPE_STRIPE_REPLICATE;
-                                break;
-                        case GF_CLUSTER_TYPE_DISPERSE:
-                                cli_err ("replicated-dispersed volume is not "
-                                         "supported");
-                                goto out;
-                        }
+int32_t
+cli_cmd_volume_create_parse(struct cli_state *state, const char **words,
+                            int wordcount, dict_t **options, char **brick_list)
+{
+    dict_t *dict = NULL;
+    char *volname = NULL;
+    int ret = -1;
+    gf1_cluster_type type = GF_CLUSTER_TYPE_NONE;
+    int sub_count = 1;
+    int brick_index = 0;
+    char *trans_type = NULL;
+    int32_t index = 0;
+    char *bricks = NULL;
+    char *ta_brick = NULL;
+    int32_t brick_count = 0;
+    static char *opwords[] = {"replica",  "stripe",       "transport",
+                              "disperse", "redundancy",   "disperse-data",
+                              "arbiter",  "thin-arbiter", NULL};
+
+    char *w = NULL;
+    int op_count = 0;
+    int32_t replica_count = 1;
+    int32_t arbiter_count = 0;
+    int32_t thin_arbiter_count = 0;
+    int32_t stripe_count = 1;
+    int32_t disperse_count = -1;
+    int32_t redundancy_count = -1;
+    int32_t disperse_data_count = -1;
+    gf_boolean_t is_force = _gf_false;
+    int wc = wordcount;
+    gf_answer_t answer = GF_ANSWER_NO;
+    const char *question = NULL;
+
+    GF_ASSERT(words);
+    GF_ASSERT(options);
+
+    dict = dict_new();
+
+    if (!dict)
+        goto out;
 
-                        if (wordcount < (index+2)) {
-                                ret = -1;
-                                goto out;
-                        }
-                        replica_count = strtol (words[index+1], NULL, 0);
-                        if (replica_count < 2) {
-                                cli_err ("replica count should be greater"
-                                         " than 1");
-                                ret = -1;
-                                goto out;
-                        }
-                        ret = dict_set_int32 (dict, "replica-count", replica_count);
-                        if (ret)
-                                goto out;
-
-                        index += 2;
-
-                } else if ((strcmp (w, "stripe")) == 0) {
-                        switch (type) {
-                        case GF_CLUSTER_TYPE_STRIPE_REPLICATE:
-                        case GF_CLUSTER_TYPE_STRIPE:
-                                cli_err ("stripe option given twice");
-                                goto out;
-                        case GF_CLUSTER_TYPE_NONE:
-                                type = GF_CLUSTER_TYPE_STRIPE;
-                                break;
-                        case GF_CLUSTER_TYPE_REPLICATE:
-                                type = GF_CLUSTER_TYPE_STRIPE_REPLICATE;
-                                break;
-                        case GF_CLUSTER_TYPE_DISPERSE:
-                                cli_err ("striped-dispersed volume is not "
-                                         "supported");
-                                goto out;
-                        }
-                        if (wordcount < (index + 2)) {
-                                ret = -1;
-                                goto out;
-                        }
-                        stripe_count = strtol (words[index+1], NULL, 0);
-                        if (stripe_count < 2) {
-                                cli_err ("stripe count should be greater"
-                                         " than 1");
-                                ret = -1;
-                                goto out;
-                        }
-                        ret = dict_set_int32 (dict, "stripe-count", stripe_count);
-                        if (ret)
-                                goto out;
+    if (wordcount < 3)
+        goto out;
 
-                        index += 2;
+    volname = (char *)words[2];
 
-                } else if ((strcmp (w, "transport")) == 0) {
-                        if (trans_type) {
-                                cli_err ("'transport' option given more"
-                                         " than one time");
-                                goto out;
-                        }
-                        if ((strcasecmp (words[index+1], "tcp") == 0)) {
-                                trans_type = gf_strdup ("tcp");
-                        } else if ((strcasecmp (words[index+1], "rdma") == 0)) {
-                                trans_type = gf_strdup ("rdma");
-                        } else if ((strcasecmp (words[index+1], "tcp,rdma") == 0) ||
-                                   (strcasecmp (words[index+1], "rdma,tcp") == 0)) {
-                                trans_type = gf_strdup ("tcp,rdma");
-                        } else {
-                                gf_log ("", GF_LOG_ERROR, "incorrect transport"
-                                        " protocol specified");
-                                ret = -1;
-                                goto out;
-                        }
-                        index += 2;
-
-                } else if ((strcmp (w, "disperse")) == 0) {
-                        switch (type) {
-                        case GF_CLUSTER_TYPE_DISPERSE:
-                                if (disperse_count >= 0) {
-                                        cli_err ("disperse option given "
-                                                 "twice");
-                                        goto out;
-                                }
-                                break;
-                        case GF_CLUSTER_TYPE_NONE:
-                                type = GF_CLUSTER_TYPE_DISPERSE;
-                                break;
-                        case GF_CLUSTER_TYPE_STRIPE_REPLICATE:
-                                cli_err ("striped-replicated-dispersed volume "
-                                         "is not supported");
-                                goto out;
-                        case GF_CLUSTER_TYPE_STRIPE:
-                                cli_err ("striped-dispersed volume is not "
-                                         "supported");
-                                goto out;
-                        case GF_CLUSTER_TYPE_REPLICATE:
-                                cli_err ("replicated-dispersed volume is not "
-                                         "supported");
-                                goto out;
-                        }
+    GF_ASSERT(volname);
 
-                        if (wordcount >= (index+2)) {
-                                disperse_count = strtol (words[index + 1],
-                                                         &ptr, 0);
-                                if (*ptr != 0)
-                                        disperse_count = 0;
-                                else {
-                                        if (disperse_count < 3) {
-                                                cli_err ("disperse count must "
-                                                         "be greater than 2");
-                                                ret = -1;
-                                                goto out;
-                                        }
-                                        index++;
-                                }
-                        }
+    /* Validate the volume name here itself */
+    if (cli_validate_volname(volname) < 0)
+        goto out;
 
-                        index++;
-
-                } else if ((strcmp (w, "redundancy")) == 0) {
-                        switch (type) {
-                        case GF_CLUSTER_TYPE_NONE:
-                                type = GF_CLUSTER_TYPE_DISPERSE;
-                                break;
-                        case GF_CLUSTER_TYPE_DISPERSE:
-                                if (redundancy_count > 0) {
-                                        cli_err ("redundancy option given "
-                                                 "twice");
-                                        goto out;
-                                }
-                                break;
-                        case GF_CLUSTER_TYPE_STRIPE_REPLICATE:
-                                cli_err ("striped-replicated-dispersed volume "
-                                         "is not supported");
-                                goto out;
-                        case GF_CLUSTER_TYPE_STRIPE:
-                                cli_err ("striped-dispersed volume is not "
-                                         "supported");
-                                goto out;
-                        case GF_CLUSTER_TYPE_REPLICATE:
-                                cli_err ("replicated-dispersed volume is not "
-                                         "supported");
-                                goto out;
-                        }
+    if (wordcount < 4) {
+        ret = -1;
+        goto out;
+    }
 
-                        if (wordcount < (index+2)) {
-                                ret = -1;
-                                goto out;
-                        }
-                        redundancy_count = strtol (words[index+1], NULL, 0);
-                        if (redundancy_count < 1) {
-                                cli_err ("redundancy must be greater than 0");
-                                ret = -1;
-                                goto out;
-                        }
+    type = GF_CLUSTER_TYPE_NONE;
+    index = 3;
 
-                        index += 2;
+    while (op_count < 3) {
+        ret = -1;
+        w = str_getunamb(words[index], opwords);
+        if (!w) {
+            break;
+        } else if ((strcmp(w, "replica")) == 0) {
+            switch (type) {
+                case GF_CLUSTER_TYPE_STRIPE_REPLICATE:
+                case GF_CLUSTER_TYPE_REPLICATE:
+                    cli_err("replica option given twice");
+                    goto out;
+                case GF_CLUSTER_TYPE_NONE:
+                    type = GF_CLUSTER_TYPE_REPLICATE;
+                    break;
+                case GF_CLUSTER_TYPE_STRIPE:
+                    cli_err("stripe option not supported");
+                    goto out;
+                case GF_CLUSTER_TYPE_DISPERSE:
+                    cli_err(
+                        "replicated-dispersed volume is not "
+                        "supported");
+                    goto out;
+                default:
+                    cli_err("Invalid type given");
+                    goto out;
+            }
+
+            if (wordcount < (index + 2)) {
+                ret = -1;
+                goto out;
+            }
 
-                }              else {
-                        GF_ASSERT (!"opword mismatch");
+            replica_count = strtol(words[index + 1], NULL, 0);
+            if (replica_count < 2) {
+                cli_err(
+                    "replica count should be greater"
+                    " than 1");
+                ret = -1;
+                goto out;
+            }
+
+            index += 2;
+            if (words[index]) {
+                if (!strcmp(words[index], "arbiter")) {
+                    ret = gf_string2int(words[index + 1], &arbiter_count);
+                    if ((ret == -1) || (arbiter_count != 1)) {
+                        cli_err(
+                            "For arbiter "
+                            "configuration, "
+                            "replica count must be"
+                            " 2 and arbiter count "
+                            "must be 1. The 3rd "
+                            "brick of the replica "
+                            "will be the arbiter");
                         ret = -1;
                         goto out;
-                }
-                op_count++;
-        }
-
-        if (!trans_type)
-                trans_type = gf_strdup ("tcp");
-
-        /* reset the count value now */
-        count = 1;
-
-        if (index >= wordcount) {
+                    }
+                    ret = dict_set_int32(dict, "arbiter-count", arbiter_count);
+                    if (ret)
+                        goto out;
+                    index += 2;
+                } else if (!strcmp(words[index], "thin-arbiter")) {
+                    ret = gf_string2int(words[index + 1], &thin_arbiter_count);
+                    if ((ret == -1) || (thin_arbiter_count != 1) ||
+                        (replica_count != 2)) {
+                        cli_err(
+                            "For thin-arbiter "
+                            "configuration, "
+                            "replica count must be"
+                            " 2 and thin-arbiter count "
+                            "must be 1. The 3rd "
+                            "brick of the replica "
+                            "will be the thin-arbiter brick");
+                        ret = -1;
+                        goto out;
+                    }
+                    ret = dict_set_int32(dict, "thin-arbiter-count",
+                                         thin_arbiter_count);
+                    if (ret)
+                        goto out;
+                    index += 2;
+                }
+            }
+
+            /* Do this to keep glusterd happy with sending
+               "replica 3 arbiter 1" options to server */
+            if ((arbiter_count == 1) && (replica_count == 2))
+                replica_count += arbiter_count;
+
+            if (replica_count == 2 && thin_arbiter_count == 0) {
+                if (strcmp(words[wordcount - 1], "force")) {
+                    question =
+                        "Replica 2 volumes are prone"
+                        " to split-brain. Use "
+                        "Arbiter or Replica 3 to "
+                        "avoid this. See: "
+                        "http://docs.gluster.org/en/latest/"
+                        "Administrator%20Guide/"
+                        "Split%20brain%20and%20ways%20to%20deal%20with%20it/."
+                        "\nDo you still want to "
+                        "continue?\n";
+                    answer = cli_cmd_get_confirmation(state, question);
+                    if (GF_ANSWER_NO == answer) {
+                        gf_log("cli", GF_LOG_ERROR,
+                               "Volume create "
+                               "cancelled, exiting");
+                        ret = -1;
+                        goto out;
+                    }
+                }
+            }
+            ret = dict_set_int32(dict, "replica-count", replica_count);
+            if (ret)
+                goto out;
+
+        } else if ((strcmp(w, "stripe")) == 0) {
+            cli_err("stripe option not supported");
+            goto out;
+        } else if ((strcmp(w, "transport")) == 0) {
+            if (trans_type) {
+                cli_err(
+                    "'transport' option given more"
+                    " than one time");
+                goto out;
+            }
+            if ((strcasecmp(words[index + 1], "tcp") == 0)) {
+                trans_type = gf_strdup("tcp");
+            } else if ((strcasecmp(words[index + 1], "rdma") == 0)) {
+                trans_type = gf_strdup("rdma");
+            } else if ((strcasecmp(words[index + 1], "tcp,rdma") == 0) ||
+                       (strcasecmp(words[index + 1], "rdma,tcp") == 0)) {
+                trans_type = gf_strdup("tcp,rdma");
+            } else {
+                gf_log("", GF_LOG_ERROR,
+                       "incorrect transport"
+                       " protocol specified");
                 ret = -1;
                 goto out;
+            }
+            index += 2;
+
+        } else if ((strcmp(w, "disperse") == 0) ||
+                   (strcmp(w, "redundancy") == 0) ||
+                   (strcmp(w, "disperse-data") == 0)) {
+            ret = cli_validate_disperse_volume(
+                w, type, words, wordcount, index, &disperse_count,
+                &redundancy_count, &disperse_data_count);
+            if (ret < 0)
+                goto out;
+            index += ret;
+            type = GF_CLUSTER_TYPE_DISPERSE;
+        } else if ((strcmp(w, "arbiter") == 0)) {
+            cli_err(
+                "arbiter option must be preceded by replica "
+                "option.");
+            ret = -1;
+            goto out;
+        } else if ((strcmp(w, "thin-arbiter") == 0)) {
+            cli_err(
+                "thin-arbiter option must be preceded by replica "
+                "option.");
+            ret = -1;
+            goto out;
+        } else {
+            GF_ASSERT(!"opword mismatch");
+            ret = -1;
+            goto out;
         }
+        op_count++;
+    }
 
-        brick_index = index;
+    if (!trans_type)
+        trans_type = gf_strdup("tcp");
 
-        if (strcmp (words[wordcount - 1], "force") == 0) {
-                is_force = _gf_true;
-                wc = wordcount - 1;
-        }
+    if (index >= wordcount) {
+        ret = -1;
+        goto out;
+    }
 
-        ret = cli_cmd_bricks_parse (words, wc, brick_index, &bricks,
-                                    &brick_count);
-        if (ret)
-                goto out;
+    brick_index = index;
 
-        /* If brick-count is not valid when replica or stripe is
-           given, exit here */
-        if (!brick_count) {
-                cli_err ("No bricks specified");
-                ret = -1;
-                goto out;
-        }
+    if (strcmp(words[wordcount - 1], "force") == 0) {
+        is_force = _gf_true;
+        wc = wordcount - 1;
+    }
 
-        if (type == GF_CLUSTER_TYPE_DISPERSE) {
-                ret = cli_cmd_create_disperse_check(state, &disperse_count,
-                                                    &redundancy_count,
-                                                    brick_count);
-                if (!ret)
-                        ret = dict_set_int32 (dict, "disperse-count",
-                                              disperse_count);
-                if (!ret)
-                        ret = dict_set_int32 (dict, "redundancy-count",
-                                              redundancy_count);
-                if (ret)
-                        goto out;
+    // Exclude the thin-arbiter-brick i.e. last brick in the bricks list
+    if (thin_arbiter_count == 1) {
+        ret = cli_cmd_bricks_parse(words, wc - 1, brick_index, &bricks,
+                                   &brick_count);
+        if (ret)
+            goto out;
 
-                sub_count = disperse_count;
-        } else
-                sub_count = stripe_count * replica_count;
-
-        if (brick_count % sub_count) {
-                if (type == GF_CLUSTER_TYPE_STRIPE)
-                        cli_err ("number of bricks is not a multiple of "
-                                 "stripe count");
-                else if (type == GF_CLUSTER_TYPE_REPLICATE)
-                        cli_err ("number of bricks is not a multiple of "
-                                 "replica count");
-                else if (type == GF_CLUSTER_TYPE_DISPERSE)
-                        cli_err ("number of bricks is not a multiple of "
-                                 "disperse count");
-                else
-                        cli_err ("number of bricks given doesn't match "
-                                 "required count");
+        ret = cli_cmd_ta_brick_parse(words, wc, &ta_brick);
 
-                ret = -1;
-                goto out;
-        }
+    } else {
+        ret = cli_cmd_bricks_parse(words, wc, brick_index, &bricks,
+                                   &brick_count);
+    }
 
-        /* Everything is parsed fine. start setting info in dict */
-        ret = dict_set_str (dict, "volname", volname);
-        if (ret)
-                goto out;
+    if (ret)
+        goto out;
 
-        ret = dict_set_int32 (dict, "type", type);
+    if (type == GF_CLUSTER_TYPE_DISPERSE) {
+        ret = cli_cmd_create_disperse_check(state, &disperse_count,
+                                            &redundancy_count,
+                                            &disperse_data_count, brick_count);
+        if (!ret)
+            ret = dict_set_int32(dict, "disperse-count", disperse_count);
+        if (!ret)
+            ret = dict_set_int32(dict, "redundancy-count", redundancy_count);
         if (ret)
-                goto out;
+            goto out;
+
+        sub_count = disperse_count;
+    } else
+        sub_count = stripe_count * replica_count;
+
+    if (brick_count % sub_count) {
+        if (type == GF_CLUSTER_TYPE_STRIPE)
+            cli_err(
+                "number of bricks is not a multiple of "
+                "stripe count");
+        else if (type == GF_CLUSTER_TYPE_REPLICATE)
+            cli_err(
+                "number of bricks is not a multiple of "
+                "replica count");
+        else if (type == GF_CLUSTER_TYPE_DISPERSE)
+            cli_err(
+                "number of bricks is not a multiple of "
+                "disperse count");
+        else
+            cli_err(
+                "number of bricks given doesn't match "
+                "required count");
 
-        ret = dict_set_dynstr (dict, "transport", trans_type);
-        if (ret)
-                goto out;
-        trans_type = NULL;
+        ret = -1;
+        goto out;
+    }
 
-        ret = dict_set_dynstr (dict, "bricks", bricks);
-        if (ret)
-                goto out;
+    /* Everything is parsed fine. start setting info in dict */
+    ret = dict_set_str(dict, "volname", volname);
+    if (ret)
+        goto out;
 
-        ret = dict_set_int32 (dict, "count", brick_count);
-        if (ret)
-                goto out;
+    ret = dict_set_int32(dict, "type", type);
+    if (ret)
+        goto out;
+
+    ret = dict_set_dynstr(dict, "transport", trans_type);
+    if (ret)
+        goto out;
+    trans_type = NULL;
 
-        ret = dict_set_int32 (dict, "force", is_force);
+    ret = dict_set_dynstr(dict, "bricks", bricks);
+    if (ret)
+        goto out;
+
+    if (thin_arbiter_count == 1) {
+        ret = dict_set_dynstr(dict, "ta-brick", ta_brick);
         if (ret)
-                goto out;
+            goto out;
+    }
 
-        *options = dict;
+    ret = dict_set_int32(dict, "count", brick_count);
+    if (ret)
+        goto out;
+
+    ret = dict_set_int32(dict, "force", is_force);
+    if (ret)
+        goto out;
 
+    *options = dict;
+    *brick_list = bricks;
 out:
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to parse create volume CLI");
-                if (dict)
-                        dict_destroy (dict);
-        }
+    if (ret) {
+        GF_FREE(bricks);
+        GF_FREE(ta_brick);
+        gf_log("cli", GF_LOG_ERROR, "Unable to parse create volume CLI");
+        if (dict)
+            dict_unref(dict);
+    }
 
-        GF_FREE (trans_type);
+    GF_FREE(trans_type);
 
-        return ret;
+    return ret;
 }
 
 int32_t
-cli_cmd_volume_reset_parse (const char **words, int wordcount, dict_t **options)
+cli_cmd_volume_reset_parse(const char **words, int wordcount, dict_t **options)
 {
-        dict_t  *dict = NULL;
-        char    *volname = NULL;
-        int     ret = -1;
+    dict_t *dict = NULL;
+    char *volname = NULL;
+    int ret = -1;
 
-        GF_ASSERT (words);
-        GF_ASSERT (options);
+    GF_ASSERT(words);
+    GF_ASSERT(options);
 
-        dict = dict_new ();
+    dict = dict_new();
 
-        if (!dict)
-                goto out;
+    if (!dict)
+        goto out;
 
-        if (wordcount < 3)
-                goto out;
+    if (wordcount < 3)
+        goto out;
 
-        if (wordcount > 5)
-                goto out;
+    if (wordcount > 5)
+        goto out;
 
-        volname = (char *)words[2];
+    volname = (char *)words[2];
 
-        if (!volname) {
-                ret = -1;
-                goto out;
-        }
+    if (!volname) {
+        ret = -1;
+        goto out;
+    }
 
-        ret = dict_set_str (dict, "volname", volname);
+    ret = dict_set_str(dict, "volname", volname);
+    if (ret)
+        goto out;
+
+    if (wordcount == 3) {
+        ret = dict_set_str(dict, "key", "all");
         if (ret)
-                goto out;
+            goto out;
+    }
 
-        if (wordcount == 3) {
-                ret = dict_set_str (dict, "key", "all");
-                if (ret)
-                        goto out;
+    if (wordcount >= 4) {
+        if (!strcmp("force", (char *)words[3])) {
+            ret = dict_set_int32(dict, "force", 1);
+            if (ret)
+                goto out;
+            ret = dict_set_str(dict, "key", "all");
+            if (ret)
+                goto out;
+        } else {
+            ret = dict_set_str(dict, "key", (char *)words[3]);
+            if (ret)
+                goto out;
         }
+    }
 
-        if (wordcount >= 4) {
-                if (!strcmp ("force", (char*)words[3])) {
-                        ret = dict_set_int32 (dict, "force", 1);
-                        if (ret)
-                                goto out;
-                        ret = dict_set_str (dict, "key", "all");
-                        if (ret)
-                                goto out;
-                } else {
-                        ret = dict_set_str (dict, "key", (char *)words[3]);
-                        if (ret)
-                                goto out;
-                }
+    if (wordcount == 5) {
+        if (strcmp("force", (char *)words[4])) {
+            ret = -1;
+            goto out;
+        } else {
+            ret = dict_set_int32(dict, "force", 1);
+            if (ret)
+                goto out;
         }
+    }
+
+    *options = dict;
+
+out:
+    if (ret && dict) {
+        dict_unref(dict);
+    }
+
+    return ret;
+}
 
-        if (wordcount == 5) {
-                if (strcmp ("force", (char*)words[4])) {
+int32_t
+cli_cmd_get_state_parse(struct cli_state *state, const char **words,
+                        int wordcount, dict_t **options, char **op_errstr)
+{
+    dict_t *dict = NULL;
+    int ret = -1;
+    char *odir = NULL;
+    char *filename = NULL;
+    char *daemon_name = NULL;
+    int count = 0;
+    uint32_t cmd = 0;
+
+    GF_VALIDATE_OR_GOTO("cli", options, out);
+    GF_VALIDATE_OR_GOTO("cli", words, out);
+
+    dict = dict_new();
+    if (!dict)
+        goto out;
+
+    if (wordcount < 1 || wordcount > 7) {
+        *op_errstr = gf_strdup(
+            "Problem parsing arguments."
+            " Check usage.");
+        goto out;
+    }
+
+    if (wordcount >= 1) {
+        gf_asprintf(&daemon_name, "%s", "glusterd");
+
+        for (count = 1; count < wordcount; count++) {
+            if (strcmp(words[count], "odir") == 0 ||
+                strcmp(words[count], "file") == 0) {
+                if (strcmp(words[count], "odir") == 0) {
+                    if (++count < wordcount) {
+                        odir = (char *)words[count];
+                        continue;
+                    } else {
+                        ret = -1;
+                        goto out;
+                    }
+                } else if (strcmp(words[count], "file") == 0) {
+                    if (++count < wordcount) {
+                        filename = (char *)words[count];
+                        continue;
+                    } else {
+                        ret = -1;
+                        goto out;
+                    }
+                }
+            } else {
+                if (count > 1) {
+                    if (count == wordcount - 1) {
+                        if (strcmp(words[count], "detail") == 0) {
+                            cmd = GF_CLI_GET_STATE_DETAIL;
+                            continue;
+                        } else if (strcmp(words[count], "volumeoptions") == 0) {
+                            cmd = GF_CLI_GET_STATE_VOLOPTS;
+                            continue;
+                        }
+                    } else {
+                        *op_errstr = gf_strdup(
+                            "Problem"
+                            " parsing arguments. "
+                            "Check usage.");
                         ret = -1;
                         goto out;
+                    }
+                }
+                if (strcmp(words[count], "glusterd") == 0) {
+                    continue;
                 } else {
-                        ret = dict_set_int32 (dict, "force", 1);
-                        if (ret)
-                                goto out;
+                    if (count == wordcount - 1) {
+                        if (strcmp(words[count], "detail") == 0) {
+                            cmd = GF_CLI_GET_STATE_DETAIL;
+                            continue;
+                        } else if (strcmp(words[count], "volumeoptions") == 0) {
+                            cmd = GF_CLI_GET_STATE_VOLOPTS;
+                            continue;
+                        }
+                    }
+
+                    *op_errstr = gf_strdup(
+                        "glusterd is "
+                        "the only supported daemon.");
+                    ret = -1;
+                    goto out;
                 }
+            }
         }
 
-        *options = dict;
+        ret = dict_set_dynstr(dict, "daemon", daemon_name);
+        if (ret) {
+            *op_errstr = gf_strdup(
+                "Command failed. Please check "
+                " log file for more details.");
+            gf_log(THIS->name, GF_LOG_ERROR,
+                   "Setting daemon name to dictionary failed");
+            goto out;
+        }
+        daemon_name = NULL;
+
+        if (odir) {
+            ret = dict_set_str(dict, "odir", odir);
+            if (ret) {
+                *op_errstr = gf_strdup(
+                    "Command failed. Please"
+                    " check log file for"
+                    " more details.");
+                gf_log(THIS->name, GF_LOG_ERROR,
+                       "Setting output directory to"
+                       "dictionary failed");
+                goto out;
+            }
+        }
+
+        if (filename) {
+            ret = dict_set_str(dict, "filename", filename);
+            if (ret) {
+                *op_errstr = gf_strdup(
+                    "Command failed. Please"
+                    " check log file for"
+                    " more  details.");
+                gf_log(THIS->name, GF_LOG_ERROR,
+                       "Setting filename to dictionary failed");
+                goto out;
+            }
+        }
+
+        if (cmd) {
+            ret = dict_set_uint32(dict, "getstate-cmd", cmd);
+            if (ret) {
+                *op_errstr = gf_strdup(
+                    "Command failed. Please"
+                    " check log file for"
+                    " more details.");
+                gf_log(THIS->name, GF_LOG_ERROR,
+                       "Setting "
+                       "get-state command type to dictionary "
+                       "failed");
+                goto out;
+            }
+        }
+    }
 
 out:
-        if (ret && dict) {
-                dict_destroy (dict);
-        }
+    if (dict)
+        *options = dict;
+
+    if (ret && dict)
+        dict_unref(dict);
 
-                return ret;
+    GF_FREE(daemon_name);
+
+    return ret;
 }
 
 int32_t
-cli_cmd_quota_parse (const char **words, int wordcount, dict_t **options)
+cli_cmd_inode_quota_parse(const char **words, int wordcount, dict_t **options)
 {
-        dict_t          *dict    = NULL;
-        char            *volname = NULL;
-        int              ret     = -1;
-        int              i       = 0;
-        char             key[20] = {0, };
-        uint64_t         value   = 0;
-        gf_quota_type    type    = GF_QUOTA_OPTION_TYPE_NONE;
-        char           *opwords[] = { "enable", "disable", "limit-usage",
-                                      "remove", "list", "alert-time",
-                                      "soft-timeout", "hard-timeout",
-                                      "default-soft-limit", NULL};
-        char            *w       = NULL;
-        uint32_t         time    = 0;
-        double           percent = 0;
-
-        GF_ASSERT (words);
-        GF_ASSERT (options);
-
-        dict = dict_new ();
-        if (!dict)
-                goto out;
+    dict_t *dict = NULL;
+    char *volname = NULL;
+    int ret = -1;
 
-        if (wordcount < 4)
-                goto out;
+    GF_ASSERT(words);
+    GF_ASSERT(options);
 
-        volname = (char *)words[2];
-        if (!volname) {
-                ret = -1;
-                goto out;
-        }
+    dict = dict_new();
+    if (!dict) {
+        gf_log("cli", GF_LOG_ERROR, "dict_new failed");
+        goto out;
+    }
 
-        /* Validate the volume name here itself */
-        {
-                if (volname[0] == '-')
-                        goto out;
+    if (wordcount != 4)
+        goto out;
 
-                if (!strcmp (volname, "all")) {
-                        cli_err ("\"all\" cannot be the name of a volume.");
-                        goto out;
-                }
+    volname = (char *)words[2];
+    if (!volname) {
+        ret = -1;
+        goto out;
+    }
 
-                if (strchr (volname, '/'))
-                        goto out;
+    /* Validate the volume name here itself */
+    if (cli_validate_volname(volname) < 0)
+        goto out;
 
-                if (strlen (volname) > GD_VOLUME_NAME_MAX) {
-                        cli_err("Volname can not exceed %d characters.",
-                                GD_VOLUME_NAME_MAX);
-                        goto out;
-                }
+    ret = dict_set_str(dict, "volname", volname);
+    if (ret < 0)
+        goto out;
+
+    if (strcmp(words[3], "enable") != 0) {
+        cli_out("Invalid quota option : %s", words[3]);
+        ret = -1;
+        goto out;
+    }
+
+    ret = dict_set_int32(dict, "type", GF_QUOTA_OPTION_TYPE_ENABLE_OBJECTS);
+    if (ret < 0)
+        goto out;
 
-                for (i = 0; i < strlen (volname); i++)
-                       if (!isalnum (volname[i]) && (volname[i] != '_') &&
-                          (volname[i] != '-')) {
-                                cli_err ("Volume name should not contain \"%c\""
-                                         " character.\nVolume names can only"
-                                         "contain alphanumeric, '-' and '_' "
-                                         "characters.",volname[i]);
-                                goto out;
-                       }
+    *options = dict;
+out:
+    if (ret < 0) {
+        if (dict)
+            dict_unref(dict);
+    }
+
+    return ret;
+}
+
+int32_t
+cli_cmd_quota_parse(const char **words, int wordcount, dict_t **options)
+{
+    dict_t *dict = NULL;
+    char *volname = NULL;
+    int ret = -1;
+    int i = -1;
+    char key[20] = {
+        0,
+    };
+    int64_t value = 0;
+    gf_quota_type type = GF_QUOTA_OPTION_TYPE_NONE;
+    static char *opwords[] = {"enable",
+                              "disable",
+                              "limit-usage",
+                              "remove",
+                              "list",
+                              "alert-time",
+                              "soft-timeout",
+                              "hard-timeout",
+                              "default-soft-limit",
+                              "limit-objects",
+                              "list-objects",
+                              "remove-objects",
+                              NULL};
+    char *w = NULL;
+    uint32_t time = 0;
+    double percent = 0;
+    char *end_ptr = NULL;
+    int64_t limit = 0;
+
+    GF_ASSERT(words);
+    GF_ASSERT(options);
+
+    dict = dict_new();
+    if (!dict) {
+        gf_log("cli", GF_LOG_ERROR, "dict_new failed");
+        goto out;
+    }
+
+    if (wordcount < 4) {
+        if ((wordcount == 3) && !(strcmp(words[2], "help"))) {
+            ret = 1;
         }
+        goto out;
+    }
 
-        ret = dict_set_str (dict, "volname", volname);
-        if (ret < 0)
-                goto out;
+    volname = (char *)words[2];
+    if (!volname) {
+        ret = -1;
+        goto out;
+    }
 
-        w = str_getunamb (words[3], opwords);
-        if (!w) {
-                cli_out ("Invalid quota option : %s", words[3]);
-                ret = - 1;
-                goto out;
+    /* Validate the volume name here itself */
+    if (cli_validate_volname(volname) < 0)
+        goto out;
+
+    ret = dict_set_str(dict, "volname", volname);
+    if (ret < 0)
+        goto out;
+
+    w = str_getunamb(words[3], opwords);
+    if (!w) {
+        cli_out("Invalid quota option : %s", words[3]);
+        ret = -1;
+        goto out;
+    }
+
+    if (strcmp(w, "enable") == 0) {
+        if (wordcount == 4) {
+            type = GF_QUOTA_OPTION_TYPE_ENABLE;
+            ret = 0;
+            goto set_type;
+        } else {
+            ret = -1;
+            goto out;
         }
+    }
 
-        if (strcmp (w, "enable") == 0) {
-                if (wordcount == 4) {
-                        type = GF_QUOTA_OPTION_TYPE_ENABLE;
-                        ret = 0;
-                        goto set_type;
-                } else {
-                        ret = -1;
-                        goto out;
-                }
+    if (strcmp(w, "disable") == 0) {
+        if (wordcount == 4) {
+            type = GF_QUOTA_OPTION_TYPE_DISABLE;
+            ret = 0;
+            goto set_type;
+        } else {
+            ret = -1;
+            goto out;
         }
+    }
 
-        if (strcmp (w, "disable") == 0) {
-                if (wordcount == 4) {
-                        type = GF_QUOTA_OPTION_TYPE_DISABLE;
-                        ret = 0;
-                        goto set_type;
-                } else {
-                        ret = -1;
-                        goto out;
-                }
+    if (strcmp(w, "limit-usage") == 0) {
+        type = GF_QUOTA_OPTION_TYPE_LIMIT_USAGE;
+    } else if (strcmp(w, "limit-objects") == 0) {
+        type = GF_QUOTA_OPTION_TYPE_LIMIT_OBJECTS;
+    }
+
+    if (type == GF_QUOTA_OPTION_TYPE_LIMIT_USAGE ||
+        type == GF_QUOTA_OPTION_TYPE_LIMIT_OBJECTS) {
+        if (wordcount < 6 || wordcount > 7) {
+            ret = -1;
+            goto out;
         }
 
-        if (strcmp (w, "limit-usage") == 0) {
+        if (words[4][0] != '/') {
+            cli_err("Please enter absolute path");
+            ret = -1;
+            goto out;
+        }
+        ret = dict_set_str(dict, "path", (char *)words[4]);
+        if (ret)
+            goto out;
+
+        if (!words[5]) {
+            cli_err("Please enter the limit value to be set");
+            ret = -1;
+            goto out;
+        }
+
+        if (type == GF_QUOTA_OPTION_TYPE_LIMIT_USAGE) {
+            ret = gf_string2bytesize_int64(words[5], &value);
+            if (ret != 0 || value <= 0) {
+                if (errno == ERANGE || value <= 0) {
+                    ret = -1;
+                    cli_err(
+                        "Please enter an integer "
+                        "value in the range of "
+                        "(1 - %" PRId64 ")",
+                        INT64_MAX);
+                } else
+                    cli_err(
+                        "Please enter a correct "
+                        "value");
+                goto out;
+            }
+        } else {
+            errno = 0;
+            limit = strtol(words[5], &end_ptr, 10);
+            if (errno == ERANGE || errno == EINVAL || limit <= 0 ||
+                strcmp(end_ptr, "") != 0) {
+                ret = -1;
+                cli_err(
+                    "Please enter an integer value in "
+                    "the range 1 - %" PRId64,
+                    INT64_MAX);
+                goto out;
+            }
+        }
 
-                if (wordcount < 6 || wordcount > 7) {
-                        ret = -1;
-                        goto out;
-                }
+        ret = dict_set_str(dict, "hard-limit", (char *)words[5]);
+        if (ret < 0)
+            goto out;
 
-                type = GF_QUOTA_OPTION_TYPE_LIMIT_USAGE;
+        if (wordcount == 7) {
+            ret = gf_string2percent(words[6], &percent);
+            if (ret != 0 || percent > 100) {
+                ret = -1;
+                cli_err(
+                    "Please enter a correct value "
+                    "in the range of 0 to 100");
+                goto out;
+            }
 
-                if (words[4][0] != '/') {
-                        cli_err ("Please enter absolute path");
-                        ret = -1;
-                        goto out;
-                }
-                ret = dict_set_str (dict, "path", (char *) words[4]);
-                if (ret)
-                        goto out;
+            ret = dict_set_str(dict, "soft-limit", (char *)words[6]);
+            if (ret < 0)
+                goto out;
+        }
 
-                if (!words[5]) {
-                        cli_err ("Please enter the limit value to be set");
-                        ret = -1;
-                        goto out;
-                }
+        goto set_type;
+    }
 
-                ret = gf_string2bytesize_uint64 (words[5], &value);
-                if (ret != 0) {
-                        if (errno == ERANGE)
-                                cli_err ("Value too large: %s", words[5]);
-                        else
-                                cli_err ("Please enter a correct value");
-                        goto out;
-                }
+    if (strcmp(w, "remove") == 0) {
+        if (wordcount != 5) {
+            ret = -1;
+            goto out;
+        }
 
-                ret  = dict_set_str (dict, "hard-limit", (char *) words[5]);
-                if (ret < 0)
-                        goto out;
+        type = GF_QUOTA_OPTION_TYPE_REMOVE;
 
-                if (wordcount == 7) {
+        if (words[4][0] != '/') {
+            cli_err("Please enter absolute path");
+            ret = -1;
+            goto out;
+        }
 
-                        ret = gf_string2percent (words[6], &percent);
-                        if (ret != 0) {
-                                cli_err ("Please enter a correct value");
-                                goto out;
-                        }
+        ret = dict_set_str(dict, "path", (char *)words[4]);
+        if (ret < 0)
+            goto out;
+        goto set_type;
+    }
 
-                        ret = dict_set_str (dict, "soft-limit",
-                                            (char *) words[6]);
-                        if (ret < 0)
-                                goto out;
-                }
+    if (strcmp(w, "remove-objects") == 0) {
+        if (wordcount != 5) {
+            ret = -1;
+            goto out;
+        }
 
-                goto set_type;
+        type = GF_QUOTA_OPTION_TYPE_REMOVE_OBJECTS;
+
+        if (words[4][0] != '/') {
+            cli_err("Please enter absolute path");
+            ret = -1;
+            goto out;
         }
-        if (strcmp (w, "remove") == 0) {
-                if (wordcount != 5) {
-                        ret = -1;
-                        goto out;
-                }
 
-                type = GF_QUOTA_OPTION_TYPE_REMOVE;
+        ret = dict_set_str(dict, "path", (char *)words[4]);
+        if (ret < 0)
+            goto out;
+        goto set_type;
+    }
 
-                if (words[4][0] != '/') {
-                        cli_err ("Please enter absolute path");
-                        ret = -1;
-                        goto out;
-                }
+    if (strcmp(w, "list") == 0) {
+        type = GF_QUOTA_OPTION_TYPE_LIST;
 
-                ret = dict_set_str (dict, "path", (char *) words[4]);
-                if (ret < 0)
-                        goto out;
-                goto set_type;
+        if (words[4] && words[4][0] != '/') {
+            cli_err("Please enter absolute path");
+            ret = -1;
+            goto out;
         }
 
-        if (strcmp (w, "list") == 0) {
-                if (wordcount < 4) {
-                        ret = -1;
-                        goto out;
-                }
+        i = 4;
+        while (i < wordcount) {
+            snprintf(key, 20, "path%d", i - 4);
 
-                type = GF_QUOTA_OPTION_TYPE_LIST;
+            ret = dict_set_str(dict, key, (char *)words[i++]);
+            if (ret < 0)
+                goto out;
+        }
 
-                i = 4;
-                while (i < wordcount) {
-                        snprintf (key, 20, "path%d", i-4);
+        ret = dict_set_int32(dict, "count", i - 4);
+        if (ret < 0)
+            goto out;
 
-                        ret = dict_set_str (dict, key, (char *) words [i++]);
-                        if (ret < 0)
-                                goto out;
-                }
+        goto set_type;
+    }
 
-                ret = dict_set_int32 (dict, "count", i - 4);
-                if (ret < 0)
-                        goto out;
+    if (strcmp(w, "list-objects") == 0) {
+        type = GF_QUOTA_OPTION_TYPE_LIST_OBJECTS;
 
-                goto set_type;
+        i = 4;
+        while (i < wordcount) {
+            snprintf(key, 20, "path%d", i - 4);
+
+            ret = dict_set_str(dict, key, (char *)words[i++]);
+            if (ret < 0) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Failed to set "
+                       "quota patch in request dictionary");
+                goto out;
+            }
         }
 
+        ret = dict_set_int32(dict, "count", i - 4);
+        if (ret < 0) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to set quota "
+                   "limit count in request dictionary");
+            goto out;
+        }
 
-        if (strcmp (w, "alert-time") == 0) {
-                if (wordcount != 5) {
-                        ret = -1;
-                        goto out;
-                }
-                type = GF_QUOTA_OPTION_TYPE_ALERT_TIME;
+        goto set_type;
+    }
 
-                ret = gf_string2time (words[4], &time);
-                if (ret) {
-                        cli_err ("Invalid argument %s. Please enter a valid "
-                                 "string", words[4]);
-                        goto out;
-                }
+    if (strcmp(w, "alert-time") == 0) {
+        if (wordcount != 5) {
+            ret = -1;
+            goto out;
+        }
+        type = GF_QUOTA_OPTION_TYPE_ALERT_TIME;
 
-                ret = dict_set_str (dict, "value", (char *)words[4]);
-                if (ret < 0)
-                        goto out;
-                goto set_type;
+        ret = gf_string2time(words[4], &time);
+        if (ret) {
+            cli_err(
+                "Invalid argument %s. Please enter a valid "
+                "string",
+                words[4]);
+            goto out;
         }
-        if (strcmp (w, "soft-timeout") == 0) {
-                if (wordcount != 5) {
-                        ret = -1;
-                        goto out;
-                }
-                type = GF_QUOTA_OPTION_TYPE_SOFT_TIMEOUT;
 
-                ret = gf_string2time (words[4], &time);
-                if (ret) {
-                        cli_err ("Invalid argument %s. Please enter a valid "
-                                 "string", words[4]);
-                        goto out;
-                }
+        ret = dict_set_str(dict, "value", (char *)words[4]);
+        if (ret < 0)
+            goto out;
+        goto set_type;
+    }
 
-                ret = dict_set_str (dict, "value", (char *)words[4]);
-                if (ret < 0)
-                        goto out;
-                goto set_type;
+    if (strcmp(w, "soft-timeout") == 0) {
+        if (wordcount != 5) {
+            ret = -1;
+            goto out;
         }
-        if (strcmp (w, "hard-timeout") == 0) {
-                if(wordcount != 5) {
-                        ret = -1;
-                        goto out;
-                }
-                type = GF_QUOTA_OPTION_TYPE_HARD_TIMEOUT;
+        type = GF_QUOTA_OPTION_TYPE_SOFT_TIMEOUT;
 
-                ret = gf_string2time (words[4], &time);
-                if (ret) {
-                        cli_err ("Invalid argument %s. Please enter a valid "
-                                 "string", words[4]);
-                        goto out;
-                }
+        ret = gf_string2time(words[4], &time);
+        if (ret) {
+            cli_err(
+                "Invalid argument %s. Please enter a valid "
+                "string",
+                words[4]);
+            goto out;
+        }
 
-                ret = dict_set_str (dict, "value", (char *)words[4]);
-                if (ret < 0)
-                        goto out;
-                goto set_type;
+        ret = dict_set_str(dict, "value", (char *)words[4]);
+        if (ret < 0)
+            goto out;
+        goto set_type;
+    }
+
+    if (strcmp(w, "hard-timeout") == 0) {
+        if (wordcount != 5) {
+            ret = -1;
+            goto out;
         }
-        if (strcmp (w, "default-soft-limit") == 0) {
-                if(wordcount != 5) {
-                        ret = -1;
-                        goto out;
-                }
-                type = GF_QUOTA_OPTION_TYPE_DEFAULT_SOFT_LIMIT;
+        type = GF_QUOTA_OPTION_TYPE_HARD_TIMEOUT;
 
-                ret = dict_set_str (dict, "value", (char *)words[4]);
-                if (ret < 0)
-                        goto out;
-                goto set_type;
-        } else {
-                GF_ASSERT (!"opword mismatch");
+        ret = gf_string2time(words[4], &time);
+        if (ret) {
+            cli_err(
+                "Invalid argument %s. Please enter a valid "
+                "string",
+                words[4]);
+            goto out;
         }
 
-set_type:
-        ret = dict_set_int32 (dict, "type", type);
+        ret = dict_set_str(dict, "value", (char *)words[4]);
         if (ret < 0)
-                goto out;
+            goto out;
+        goto set_type;
+    }
+    if (strcmp(w, "default-soft-limit") == 0) {
+        if (wordcount != 5) {
+            ret = -1;
+            goto out;
+        }
+        type = GF_QUOTA_OPTION_TYPE_DEFAULT_SOFT_LIMIT;
 
-        *options = dict;
+        ret = dict_set_str(dict, "value", (char *)words[4]);
+        if (ret < 0)
+            goto out;
+        goto set_type;
+    } else {
+        GF_ASSERT(!"opword mismatch");
+    }
+
+set_type:
+    ret = dict_set_int32(dict, "type", type);
+    if (ret < 0)
+        goto out;
+
+    *options = dict;
 out:
-        if (ret < 0) {
-                if (dict)
-                        dict_destroy (dict);
-        }
+    if (ret < 0) {
+        if (dict)
+            dict_unref(dict);
+    }
 
-        return ret;
+    return ret;
 }
 
-static inline gf_boolean_t
-cli_is_key_spl (char *key)
+static gf_boolean_t
+cli_is_key_spl(char *key)
 {
-        return (strcmp (key, "group") == 0);
+    return (strcmp(key, "group") == 0);
 }
 
-static int
-cli_add_key_group (dict_t *dict, char *key, char *value, char **op_errstr)
+static int32_t
+cli_add_key_group_value(dict_t *dict, const char *name, const char *value,
+                        int32_t id, char **op_errstr)
 {
-        int             ret = -1;
-        int             opt_count = 0;
-        char            iter_key[1024] = {0,};
-        char            iter_val[1024] = {0,};
-        char            *saveptr = NULL;
-        char            *tok_key = NULL;
-        char            *tok_val = NULL;
-        char            *dkey = NULL;
-        char            *dval = NULL;
-        char            *tagpath = NULL;
-        char            *buf = NULL;
-        char            line[PATH_MAX + 256] = {0,};
-        char            errstr[2048] = "";
-        FILE            *fp = NULL;
-
-        ret = gf_asprintf (&tagpath, "%s/groups/%s",
-                           GLUSTERD_DEFAULT_WORKDIR, value);
-        if (ret == -1) {
-                tagpath = NULL;
-                goto out;
-        }
+    char *key = NULL;
+    char *data = NULL;
+    int32_t ret = -1;
 
-        fp = fopen (tagpath, "r");
-        if (!fp) {
-                ret = -1;
-                snprintf(errstr, sizeof(errstr), "Unable to open file '%s'."
-                         " Error: %s", tagpath, strerror (errno));
-                if (op_errstr)
-                        *op_errstr = gf_strdup(errstr);
-                goto out;
-        }
+    ret = gf_asprintf(&key, "%s%d", name, id);
+    if (ret < 0) {
+        goto out;
+    }
+    data = gf_strdup(value);
+    if (data == NULL) {
+        gf_log(THIS->name, GF_LOG_ERROR, "Failed to allocate memory for data");
+        ret = -1;
+        goto out;
+    }
 
-        opt_count = 0;
-        buf = line;
-        while (fscanf (fp, "%s", buf) != EOF) {
+    ret = dict_set_dynstr(dict, key, data);
+    if (ret == 0) {
+        data = NULL;
+    }
 
-                opt_count++;
-                tok_key = strtok_r (line, "=", &saveptr);
-                tok_val = strtok_r (NULL, "=", &saveptr);
-                if (!tok_key || !tok_val) {
-                        ret = -1;
-                        snprintf(errstr, sizeof(errstr), "'%s' file format "
-                                 "not valid.", tagpath);
-                        if (op_errstr)
-                                *op_errstr = gf_strdup(errstr);
-                        goto out;
-                }
+out:
+    GF_FREE(key);
+    GF_FREE(data);
 
-                snprintf (iter_key, sizeof (iter_key), "key%d", opt_count);
-                dkey = gf_strdup (tok_key);
-                ret = dict_set_dynstr (dict, iter_key, dkey);
-                if (ret)
-                        goto out;
-                dkey = NULL;
+    if ((ret != 0) && (op_errstr != NULL)) {
+        *op_errstr = gf_strdup("Failed to allocate memory");
+    }
 
-                snprintf (iter_val, sizeof (iter_val), "value%d", opt_count);
-                dval = gf_strdup (tok_val);
-                ret = dict_set_dynstr (dict, iter_val, dval);
-                if (ret)
-                        goto out;
-                dval = NULL;
+    return ret;
+}
 
-        }
+static int
+cli_add_key_group(dict_t *dict, char *key, char *value, char **op_errstr)
+{
+    int ret = -1;
+    int opt_count = 0;
+    char *saveptr = NULL;
+    char *tok_key = NULL;
+    char *tok_val = NULL;
+    char *tagpath = NULL;
+    char line[PATH_MAX + 256] = {
+        0,
+    };
+    FILE *fp = NULL;
+
+    ret = gf_asprintf(&tagpath, "%s/groups/%s", GLUSTERD_DEFAULT_WORKDIR,
+                      value);
+    if (ret == -1) {
+        tagpath = NULL;
+        goto out;
+    }
 
-        if (!opt_count) {
-                ret = -1;
-                snprintf(errstr, sizeof(errstr), "'%s' file format "
-                         "not valid.", tagpath);
-                if (op_errstr)
-                        *op_errstr = gf_strdup(errstr);
-                goto out;
+    fp = fopen(tagpath, "r");
+    if (!fp) {
+        ret = -1;
+        if (op_errstr) {
+            gf_asprintf(op_errstr,
+                        "Unable to open file '%s'. "
+                        "Error: %s",
+                        tagpath, strerror(errno));
+        }
+        goto out;
+    }
+
+    opt_count = 0;
+    while (fgets(line, sizeof(line), fp) != NULL) {
+        if (strlen(line) >= sizeof(line) - 1) {
+            ret = -1;
+            if (op_errstr != NULL) {
+                *op_errstr = gf_strdup("Line too long");
+            }
+            goto out;
+        }
+
+        /* Treat line that start with "#" as comments */
+        if ('#' == line[0])
+            continue;
+
+        opt_count++;
+        tok_key = strtok_r(line, "=", &saveptr);
+        tok_val = strtok_r(NULL, "\r\n", &saveptr);
+        if (!tok_key || !tok_val) {
+            ret = -1;
+            if (op_errstr) {
+                gf_asprintf(op_errstr,
+                            "'%s' file format "
+                            "not valid.",
+                            tagpath);
+            }
+            goto out;
+        }
+
+        ret = cli_add_key_group_value(dict, "key", tok_key, opt_count,
+                                      op_errstr);
+        if (ret != 0) {
+            goto out;
+        }
+        ret = cli_add_key_group_value(dict, "value", tok_val, opt_count,
+                                      op_errstr);
+        if (ret != 0) {
+            goto out;
+        }
+    }
+
+    if (!opt_count) {
+        ret = -1;
+        if (op_errstr) {
+            gf_asprintf(op_errstr, "'%s' file format not valid.", tagpath);
         }
-        ret = dict_set_int32 (dict, "count", opt_count);
+        goto out;
+    }
+    ret = dict_set_int32(dict, "count", opt_count);
 out:
 
-        GF_FREE (tagpath);
+    GF_FREE(tagpath);
 
-        if (ret) {
-                GF_FREE (dkey);
-                GF_FREE (dval);
-        }
-
-        if (fp)
-                fclose (fp);
+    if (fp)
+        fclose(fp);
 
-        return ret;
+    return ret;
 }
 
 int32_t
-cli_cmd_volume_set_parse (const char **words, int wordcount, dict_t **options,
-                          char **op_errstr)
+cli_cmd_volume_set_parse(struct cli_state *state, const char **words,
+                         int wordcount, dict_t **options, char **op_errstr)
 {
-        dict_t                  *dict = NULL;
-        char                    *volname = NULL;
-        int                     ret = -1;
-        int                     count = 0;
-        char                    *key = NULL;
-        char                    *value = NULL;
-        int                     i = 0;
-        char                    str[50] = {0,};
+    dict_t *dict = NULL;
+    char *volname = NULL;
+    int ret = -1;
+    int count = 0;
+    char *key = NULL;
+    char *value = NULL;
+    int i = 0;
+    char str[50] = {
+        0,
+    };
+    const char *question = NULL;
+    gf_answer_t answer = GF_ANSWER_NO;
+
+    GF_ASSERT(words);
+    GF_ASSERT(options);
+
+    dict = dict_new();
+
+    if (!dict)
+        goto out;
 
-        GF_ASSERT (words);
-        GF_ASSERT (options);
+    if (wordcount < 3)
+        goto out;
 
-        dict = dict_new ();
+    volname = (char *)words[2];
 
-        if (!dict)
-                goto out;
+    GF_ASSERT(volname);
 
-        if (wordcount < 3)
-                goto out;
+    ret = dict_set_str(dict, "volname", volname);
 
-        volname = (char *)words[2];
+    if (ret)
+        goto out;
 
-        GF_ASSERT (volname);
+    if (!strcmp(volname, "all")) {
+        ret = dict_set_str(dict, "globalname", "All");
+        if (ret) {
+            gf_log(THIS->name, GF_LOG_ERROR, "dict set on global key failed.");
+            goto out;
+        }
 
-        ret = dict_set_str (dict, "volname", volname);
+        ret = dict_set_int32(dict, "hold_global_locks", _gf_true);
+        if (ret) {
+            gf_log(THIS->name, GF_LOG_ERROR, "dict set on global key failed.");
+            goto out;
+        }
+    }
 
+    if ((!strcmp(volname, "help") || !strcmp(volname, "help-xml")) &&
+        wordcount == 3) {
+        ret = dict_set_str(dict, volname, volname);
         if (ret)
-                goto out;
+            goto out;
 
-        if ((!strcmp (volname, "help") || !strcmp (volname, "help-xml"))
-            && wordcount == 3 ) {
-                ret = dict_set_str (dict, volname, volname);
-                if (ret)
-                        goto out;
-
-        } else if (wordcount < 5) {
-                ret = -1;
-                goto out;
-
-        } else if (wordcount == 5  && cli_is_key_spl ((char *)words[3])) {
-                key = (char *) words[3];
-                value = (char *) words[4];
-                if ( !key || !value) {
-                        ret = -1;
-                        goto out;
-                }
+    } else if (wordcount < 5) {
+        ret = -1;
+        goto out;
 
-                ret = gf_strip_whitespace (value, strlen (value));
-                if (ret == -1)
-                        goto out;
+    } else if (wordcount == 5 && cli_is_key_spl((char *)words[3])) {
+        key = (char *)words[3];
+        value = (char *)words[4];
+        if (!key || !value) {
+            ret = -1;
+            goto out;
+        }
 
-                if (strlen (value) == 0) {
-                        ret = -1;
-                        goto out;
-                }
+        ret = gf_strip_whitespace(value, strlen(value));
+        if (ret == -1)
+            goto out;
 
-                ret = cli_add_key_group (dict, key, value, op_errstr);
-                if (ret == 0)
-                        *options = dict;
-                goto out;
+        if (strlen(value) == 0) {
+            ret = -1;
+            goto out;
         }
 
-        for (i = 3; i < wordcount; i+=2) {
+        ret = cli_add_key_group(dict, key, value, op_errstr);
+        if (ret == 0)
+            *options = dict;
+        goto out;
+    }
 
-                key = (char *) words[i];
-                value = (char *) words[i+1];
+    for (i = 3; i < wordcount; i += 2) {
+        key = (char *)words[i];
+        value = (char *)words[i + 1];
 
-                if ( !key || !value) {
-                        ret = -1;
-                        goto out;
-                }
+        if (!key || !value) {
+            ret = -1;
+            goto out;
+        }
 
-                count++;
+        count++;
 
-                ret = gf_strip_whitespace (value, strlen (value));
-                if (ret == -1)
-                        goto out;
+        if (fnmatch("user.*", key, FNM_NOESCAPE) != 0) {
+            ret = gf_strip_whitespace(value, strlen(value));
+            if (ret == -1)
+                goto out;
+        }
 
-                if (strlen (value) == 0) {
-                        ret = -1;
-                        goto out;
-                }
+        if (strlen(value) == 0) {
+            ret = -1;
+            goto out;
+        }
 
-                if (cli_is_key_spl (key)) {
-                        ret = -1;
-                        goto out;
-                }
+        if (cli_is_key_spl(key)) {
+            ret = -1;
+            goto out;
+        }
 
-                sprintf (str, "key%d", count);
-                ret = dict_set_str (dict, str, key);
-                if (ret)
-                        goto out;
+        sprintf(str, "key%d", count);
+        ret = dict_set_str(dict, str, key);
+        if (ret)
+            goto out;
 
-                sprintf (str, "value%d", count);
-                ret = dict_set_str (dict, str, value);
+        sprintf(str, "value%d", count);
+        ret = dict_set_str(dict, str, value);
 
-                if (ret)
-                        goto out;
+        if (ret)
+            goto out;
+
+        if ((!strcmp(key, "cluster.enable-shared-storage")) &&
+            (!strcmp(value, "disable"))) {
+            question =
+                "Disabling cluster.enable-shared-storage "
+                "will delete the shared storage volume"
+                "(gluster_shared_storage), which is used "
+                "by snapshot scheduler, geo-replication "
+                "and NFS-Ganesha. Do you still want to "
+                "continue?";
+            answer = cli_cmd_get_confirmation(state, question);
+            if (GF_ANSWER_NO == answer) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Operation "
+                       "cancelled, exiting");
+                *op_errstr = gf_strdup("Aborted by user.");
+                ret = -1;
+                goto out;
+            }
+        }
+        if ((!strcmp(key, "nfs.disable")) && (!strcmp(value, "off"))) {
+            question =
+                "Gluster NFS is being deprecated in favor "
+                "of NFS-Ganesha Enter \"yes\" to continue "
+                "using Gluster NFS";
+            answer = cli_cmd_get_confirmation(state, question);
+            if (GF_ANSWER_NO == answer) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Operation "
+                       "cancelled, exiting");
+                *op_errstr = gf_strdup("Aborted by user.");
+                ret = -1;
+                goto out;
+            }
         }
+    }
 
-        ret = dict_set_int32 (dict, "count", wordcount-3);
+    ret = dict_set_int32(dict, "count", wordcount - 3);
 
-        if (ret)
-                goto out;
+    if (ret)
+        goto out;
 
-        *options = dict;
+    *options = dict;
 
 out:
-        if (ret)
-                dict_destroy (dict);
+    if (ret && dict)
+        dict_unref(dict);
 
-        return ret;
+    return ret;
 }
 
 int32_t
-cli_cmd_volume_add_brick_parse (const char **words, int wordcount,
-                                dict_t **options)
+cli_cmd_volume_add_brick_parse(struct cli_state *state, const char **words,
+                               int wordcount, dict_t **options, int *ret_type)
 {
-        dict_t  *dict = NULL;
-        char    *volname = NULL;
-        int     ret = -1;
-        int     brick_count = 0, brick_index = 0;
-        char    *bricks = NULL;
-        char    *opwords_cl[] = { "replica", "stripe", NULL };
-        gf1_cluster_type type = GF_CLUSTER_TYPE_NONE;
-        int     count = 1;
-        char    *w = NULL;
-        int     index;
-        gf_boolean_t is_force = _gf_false;
-        int wc = wordcount;
-
-        GF_ASSERT (words);
-        GF_ASSERT (options);
-
-        dict = dict_new ();
-
-        if (!dict)
-                goto out;
+    dict_t *dict = NULL;
+    char *volname = NULL;
+    int ret = -1;
+    int brick_count = 0, brick_index = 0;
+    char *bricks = NULL;
+    static char *opwords_cl[] = {"replica", "stripe", NULL};
+    gf1_cluster_type type = GF_CLUSTER_TYPE_NONE;
+    int count = 1;
+    int arbiter_count = 0;
+    char *w = NULL;
+    int index;
+    gf_boolean_t is_force = _gf_false;
+    int wc = wordcount;
+    gf_answer_t answer = GF_ANSWER_NO;
+    const char *question = NULL;
+
+    GF_ASSERT(words);
+    GF_ASSERT(options);
+
+    dict = dict_new();
+
+    if (!dict)
+        goto out;
 
-        if (wordcount < 3)
-                goto out;
+    if (wordcount < 3)
+        goto out;
 
-        volname = (char *)words[2];
+    volname = (char *)words[2];
 
-        GF_ASSERT (volname);
+    GF_ASSERT(volname);
 
-        ret = dict_set_str (dict, "volname", volname);
+    ret = dict_set_str(dict, "volname", volname);
 
-        if (ret)
-                goto out;
+    if (ret)
+        goto out;
 
-        if (wordcount < 4) {
+    if (wordcount < 4) {
+        ret = -1;
+        goto out;
+    }
+    if (wordcount < 6) {
+        /* seems no options are given, go directly to the parse_brick */
+        brick_index = 3;
+        type = GF_CLUSTER_TYPE_NONE;
+        goto parse_bricks;
+    }
+
+    w = str_getunamb(words[3], opwords_cl);
+    if (!w) {
+        type = GF_CLUSTER_TYPE_NONE;
+        index = 3;
+    } else if ((strcmp(w, "replica")) == 0) {
+        type = GF_CLUSTER_TYPE_REPLICATE;
+        count = strtol(words[4], NULL, 0);
+        if (!count || (count < 2)) {
+            cli_err("replica count should be greater than 1");
+            ret = -1;
+            goto out;
+        }
+        ret = dict_set_int32(dict, "replica-count", count);
+        if (ret)
+            goto out;
+        index = 5;
+        if (words[index] && !strcmp(words[index], "arbiter")) {
+            arbiter_count = strtol(words[6], NULL, 0);
+            if (arbiter_count != 1 || count != 3) {
+                cli_err(
+                    "For arbiter configuration, replica "
+                    "count must be 3 and arbiter count "
+                    "must be 1. The 3rd brick of the "
+                    "replica will be the arbiter");
                 ret = -1;
                 goto out;
-        }
-        if (wordcount < 6) {
-                /* seems no options are given, go directly to the parse_brick */
-                brick_index = 3;
-                type = GF_CLUSTER_TYPE_NONE;
-                goto parse_bricks;
+            }
+            ret = dict_set_int32(dict, "arbiter-count", arbiter_count);
+            if (ret)
+                goto out;
+            index = 7;
         }
 
-        w = str_getunamb (words[3], opwords_cl);
-        if (!w) {
-                type = GF_CLUSTER_TYPE_NONE;
-                index = 3;
-        } else if ((strcmp (w, "replica")) == 0) {
-                type = GF_CLUSTER_TYPE_REPLICATE;
-                if (wordcount < 5) {
-                        ret = -1;
-                        goto out;
-                }
-                count = strtol (words[4], NULL, 0);
-                if (!count || (count < 2)) {
-                        cli_err ("replica count should be greater than 1");
-                        ret = -1;
-                        goto out;
-                }
-                ret = dict_set_int32 (dict, "replica-count", count);
-                if (ret)
-                        goto out;
-                index = 5;
-        } else if ((strcmp (w, "stripe")) == 0) {
-                type = GF_CLUSTER_TYPE_STRIPE;
-                if (wordcount < 5) {
-                        ret = -1;
-                        goto out;
-                }
-                count = strtol (words[4], NULL, 0);
-                if (!count || (count < 2)) {
-                        cli_err ("stripe count should be greater than 1");
-                        ret = -1;
-                        goto out;
+        if (count == 2) {
+            if (strcmp(words[wordcount - 1], "force")) {
+                question =
+                    "Replica 2 volumes are prone to "
+                    "split-brain. Use Arbiter or "
+                    "Replica 3 to avoid this. See: "
+                    "http://docs.gluster.org/en/latest/Administrator%20Guide/"
+                    "Split%20brain%20and%20ways%20to%20deal%20with%20it/."
+                    "\nDo you still want to continue?\n";
+                answer = cli_cmd_get_confirmation(state, question);
+                if (GF_ANSWER_NO == answer) {
+                    gf_log("cli", GF_LOG_ERROR,
+                           "Add brick"
+                           " cancelled, exiting");
+                    ret = -1;
+                    goto out;
                 }
-                ret = dict_set_int32 (dict, "stripe-count", count);
-                if (ret)
-                        goto out;
-                index = 5;
-        } else {
-                GF_ASSERT (!"opword mismatch");
-                ret = -1;
-                goto out;
+            }
         }
+    } else if ((strcmp(w, "stripe")) == 0) {
+        cli_err("stripe option not supported");
+        goto out;
+    } else {
+        GF_ASSERT(!"opword mismatch");
+        ret = -1;
+        goto out;
+    }
 
-        brick_index = index;
+    brick_index = index;
 
 parse_bricks:
 
-        if (strcmp (words[wordcount - 1], "force") == 0) {
-                is_force = _gf_true;
-                wc = wordcount - 1;
-        }
+    if (strcmp(words[wordcount - 1], "force") == 0) {
+        is_force = _gf_true;
+        wc = wordcount - 1;
+    }
 
-        ret = cli_cmd_bricks_parse (words, wc, brick_index, &bricks,
-                                    &brick_count);
-        if (ret)
-                goto out;
+    ret = cli_cmd_bricks_parse(words, wc, brick_index, &bricks, &brick_count);
+    if (ret)
+        goto out;
 
-        ret = dict_set_dynstr (dict, "bricks", bricks);
-        if (ret)
-                goto out;
+    ret = dict_set_dynstr(dict, "bricks", bricks);
+    if (ret)
+        goto out;
 
-        ret = dict_set_int32 (dict, "count", brick_count);
+    ret = dict_set_int32(dict, "count", brick_count);
 
-        if (ret)
-                goto out;
+    if (ret)
+        goto out;
 
-        ret = dict_set_int32 (dict, "force", is_force);
-        if (ret)
-                goto out;
+    ret = dict_set_int32(dict, "force", is_force);
+    if (ret)
+        goto out;
 
-        *options = dict;
+    *options = dict;
 
 out:
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to parse add-brick CLI");
-                if (dict)
-                        dict_destroy (dict);
-        }
+    if (ret_type)
+        *ret_type = type;
 
-        return ret;
-}
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to parse add-brick CLI");
+        if (dict)
+            dict_unref(dict);
+    }
 
+    return ret;
+}
 
 int32_t
-cli_cmd_volume_remove_brick_parse (const char **words, int wordcount,
-                                   dict_t **options, int *question)
+cli_cmd_volume_remove_brick_parse(struct cli_state *state, const char **words,
+                                  int wordcount, dict_t **options,
+                                  int *question, int *brick_count,
+                                  int32_t *comm)
 {
-        dict_t  *dict = NULL;
-        char    *volname = NULL;
-        char    *delimiter = NULL;
-        int     ret = -1;
-        char    key[50];
-        int     brick_count = 0, brick_index = 0;
-        int32_t tmp_index = 0;
-        int32_t j = 0;
-        char    *tmp_brick = NULL;
-        char    *tmp_brick1 = NULL;
-        char    *type_opword[] = { "replica", NULL };
-        char    *opwords[] = { "start", "commit", "stop", "status",
-                               "force", NULL };
-        char    *w = NULL;
-        int32_t  command = GF_OP_CMD_NONE;
-        long     count = 0;
-
-        GF_ASSERT (words);
-        GF_ASSERT (options);
-
-        if (wordcount < 5)
-                goto out;
+    dict_t *dict = NULL;
+    char *volname = NULL;
+    char *delimiter = NULL;
+    int ret = -1;
+    char key[50];
+    int brick_index = 0;
+    int32_t tmp_index = 0;
+    int32_t j = 0;
+    char *tmp_brick = NULL;
+    char *tmp_brick1 = NULL;
+    static char *type_opword[] = {"replica", NULL};
+    static char *opwords[] = {"start",  "commit", "stop",
+                              "status", "force",  NULL};
+    char *w = NULL;
+    int32_t command = GF_OP_CMD_NONE;
+    long count = 0;
+    gf_answer_t answer = GF_ANSWER_NO;
+    const char *ques = NULL;
+
+    GF_ASSERT(words);
+    GF_ASSERT(options);
+
+    if (wordcount < 5)
+        goto out;
 
-        dict = dict_new ();
-        if (!dict)
-                goto out;
+    dict = dict_new();
+    if (!dict)
+        goto out;
 
-        volname = (char *)words[2];
+    volname = (char *)words[2];
 
-        GF_ASSERT (volname);
+    GF_ASSERT(volname);
 
-        ret = dict_set_str (dict, "volname", volname);
-        if (ret)
-                goto out;
+    ret = dict_set_str(dict, "volname", volname);
+    if (ret)
+        goto out;
 
-        brick_index = 3;
-        w = str_getunamb (words[3], type_opword);
-        if (w && !strcmp ("replica", w)) {
-                if (wordcount < 6) {
-                        ret = -1;
-                        goto out;
-                }
-                count = strtol (words[4], NULL, 0);
-                if (count < 1) {
-                        cli_err ("replica count should be greater than 0 in "
-                                 "case of remove-brick");
-                        ret = -1;
-                        goto out;
+    brick_index = 3;
+    w = str_getunamb(words[3], type_opword);
+    if (w && !strcmp("replica", w)) {
+        if (wordcount < 6) {
+            ret = -1;
+            goto out;
+        }
+        count = strtol(words[4], NULL, 0);
+        if (count < 1) {
+            cli_err(
+                "replica count should be greater than 0 in "
+                "case of remove-brick");
+            ret = -1;
+            goto out;
+        }
+
+        if (count == 2) {
+            if (strcmp(words[wordcount - 1], "force")) {
+                ques =
+                    "Replica 2 volumes are prone to "
+                    "split-brain. Use Arbiter or Replica 3 "
+                    "to avoid this. See: "
+                    "http://docs.gluster.org/en/latest/Administrator%20Guide/"
+                    "Split%20brain%20and%20ways%20to%20deal%20with%20it/."
+                    "\nDo you still want to continue?\n";
+                answer = cli_cmd_get_confirmation(state, ques);
+                if (GF_ANSWER_NO == answer) {
+                    gf_log("cli", GF_LOG_ERROR,
+                           "Remove "
+                           "brick cancelled, exiting");
+                    ret = -1;
+                    goto out;
                 }
-
-                ret = dict_set_int32 (dict, "replica-count", count);
-                if (ret)
-                        goto out;
-                brick_index = 5;
-        } else if (w) {
-                GF_ASSERT (!"opword mismatch");
+            }
         }
 
-        w = str_getunamb (words[wordcount - 1], opwords);
-        if (!w) {
-                ret = -1;
-                goto out;
+        ret = dict_set_int32(dict, "replica-count", count);
+        if (ret)
+            goto out;
+        brick_index = 5;
+    } else if (w) {
+        GF_ASSERT(!"opword mismatch");
+    }
+
+    w = str_getunamb(words[wordcount - 1], opwords);
+    if (!w) {
+        ret = -1;
+        goto out;
+    } else {
+        /* handled this option */
+        wordcount--;
+        if (!strcmp("start", w)) {
+            command = GF_OP_CMD_START;
+            if (question)
+                *question = 1;
+        } else if (!strcmp("commit", w)) {
+            command = GF_OP_CMD_COMMIT;
+        } else if (!strcmp("stop", w)) {
+            command = GF_OP_CMD_STOP;
+        } else if (!strcmp("status", w)) {
+            command = GF_OP_CMD_STATUS;
+        } else if (!strcmp("force", w)) {
+            command = GF_OP_CMD_COMMIT_FORCE;
+            if (question)
+                *question = 1;
         } else {
-                /* handled this option */
-                wordcount--;
-                if (!strcmp ("start", w)) {
-                        command = GF_OP_CMD_START;
-                } else if (!strcmp ("commit", w)) {
-                        command = GF_OP_CMD_COMMIT;
-                        if (question)
-                                *question = 1;
-                } else if (!strcmp ("stop", w)) {
-                        command = GF_OP_CMD_STOP;
-                } else if (!strcmp ("status", w)) {
-                        command = GF_OP_CMD_STATUS;
-                } else if (!strcmp ("force", w)) {
-                        command = GF_OP_CMD_COMMIT_FORCE;
-                        if (question)
-                                *question = 1;
-                } else {
-                        GF_ASSERT (!"opword mismatch");
-                        ret = -1;
-                        goto out;
-                }
+            GF_ASSERT(!"opword mismatch");
+            ret = -1;
+            goto out;
         }
+    }
 
-        ret = dict_set_int32 (dict, "command", command);
-        if (ret)
-                gf_log ("cli", GF_LOG_INFO, "failed to set 'command' %d",
-                        command);
+    ret = dict_set_int32(dict, "command", command);
+    if (ret)
+        gf_log("cli", GF_LOG_INFO, "failed to set 'command' %d", command);
 
+    tmp_index = brick_index;
+    tmp_brick = GF_MALLOC(2048 * sizeof(*tmp_brick), gf_common_mt_char);
 
-        tmp_index = brick_index;
-        tmp_brick = GF_MALLOC(2048 * sizeof(*tmp_brick), gf_common_mt_char);
+    if (!tmp_brick) {
+        gf_log("", GF_LOG_ERROR,
+               "cli_cmd_volume_remove_brick_parse: "
+               "Unable to get memory");
+        ret = -1;
+        goto out;
+    }
 
-        if (!tmp_brick) {
-                gf_log ("",GF_LOG_ERROR,"cli_cmd_volume_remove_brick_parse: "
-                        "Unable to get memory");
+    tmp_brick1 = GF_MALLOC(2048 * sizeof(*tmp_brick1), gf_common_mt_char);
+
+    if (!tmp_brick1) {
+        gf_log("", GF_LOG_ERROR,
+               "cli_cmd_volume_remove_brick_parse: "
+               "Unable to get memory");
+        ret = -1;
+        goto out;
+    }
+
+    while (brick_index < wordcount) {
+        if (validate_brick_name((char *)words[brick_index])) {
+            cli_err(
+                "wrong brick type: %s, use <HOSTNAME>:"
+                "<export-dir-abs-path>",
+                words[brick_index]);
+            ret = -1;
+            goto out;
+        } else {
+            delimiter = strrchr(words[brick_index], ':');
+            ret = gf_canonicalize_path(delimiter + 1);
+            if (ret)
+                goto out;
+        }
+
+        j = tmp_index;
+        strcpy(tmp_brick, words[brick_index]);
+        while (j < brick_index) {
+            strcpy(tmp_brick1, words[j]);
+            if (!(strcmp(tmp_brick, tmp_brick1))) {
+                gf_log("", GF_LOG_ERROR,
+                       "Duplicate bricks"
+                       " found %s",
+                       words[brick_index]);
+                cli_err("Duplicate bricks found %s", words[brick_index]);
                 ret = -1;
                 goto out;
+            }
+            j++;
         }
+        snprintf(key, 50, "brick%d", ++(*brick_count));
+        ret = dict_set_str(dict, key, (char *)words[brick_index++]);
 
-        tmp_brick1 = GF_MALLOC(2048 * sizeof(*tmp_brick1), gf_common_mt_char);
+        if (ret)
+            goto out;
+    }
 
-        if (!tmp_brick1) {
-                gf_log ("",GF_LOG_ERROR,"cli_cmd_volume_remove_brick_parse: "
-                        "Unable to get memory");
-                ret = -1;
-                goto out;
-        }
+    if (command != GF_OP_CMD_STATUS && command != GF_OP_CMD_STOP) {
+        ret = dict_set_int32(dict, "count", *brick_count);
+        if (ret)
+            goto out;
+    }
 
-        while (brick_index < wordcount) {
-                if (validate_brick_name ((char *)words[brick_index])) {
-                        cli_err ("wrong brick type: %s, use <HOSTNAME>:"
-                                 "<export-dir-abs-path>", words[brick_index]);
-                        ret = -1;
-                        goto out;
-                } else {
-                        delimiter = strrchr(words[brick_index], ':');
-                        ret = gf_canonicalize_path (delimiter + 1);
-                        if (ret)
-                                goto out;
-                }
+    *options = dict;
 
-                j = tmp_index;
-                strcpy(tmp_brick, words[brick_index]);
-                while ( j < brick_index) {
-                        strcpy(tmp_brick1, words[j]);
-                        if (!(strcmp (tmp_brick, tmp_brick1))) {
-                                gf_log("",GF_LOG_ERROR, "Duplicate bricks"
-                                       " found %s", words[brick_index]);
-                                cli_err("Duplicate bricks found %s",
-                                        words[brick_index]);
-                                ret = -1;
-                                goto out;
-                        }
-                        j++;
-                }
-                snprintf (key, 50, "brick%d", ++brick_count);
-                ret = dict_set_str (dict, key, (char *)words[brick_index++]);
+out:
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to parse remove-brick CLI");
+        if (dict)
+            dict_unref(dict);
+    }
 
-                if (ret)
-                        goto out;
-        }
+    GF_FREE(tmp_brick);
+    GF_FREE(tmp_brick1);
 
-        ret = dict_set_int32 (dict, "count", brick_count);
+    *comm = command;
+
+    return ret;
+}
+
+int32_t
+cli_cmd_brick_op_validate_bricks(const char **words, dict_t *dict, int src,
+                                 int dst)
+{
+    int ret = -1;
+    char *delimiter = NULL;
+
+    if (validate_brick_name((char *)words[src])) {
+        cli_err(
+            "wrong brick type: %s, use "
+            "<HOSTNAME>:<export-dir-abs-path>",
+            words[3]);
+        ret = -1;
+        goto out;
+    } else {
+        delimiter = strrchr((char *)words[src], '/');
+        ret = gf_canonicalize_path(delimiter);
         if (ret)
-                goto out;
+            goto out;
+    }
 
-        *options = dict;
+    ret = dict_set_str(dict, "src-brick", (char *)words[src]);
+    if (ret)
+        goto out;
 
-out:
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to parse remove-brick CLI");
-                if (dict)
-                        dict_destroy (dict);
-        }
+    if (dst == -1) {
+        ret = 0;
+        goto out;
+    }
 
-        GF_FREE (tmp_brick);
-        GF_FREE (tmp_brick1);
+    if (validate_brick_name((char *)words[dst])) {
+        cli_err(
+            "wrong brick type: %s, use "
+            "<HOSTNAME>:<export-dir-abs-path>",
+            words[dst]);
+        ret = -1;
+        goto out;
+    } else {
+        delimiter = strrchr((char *)words[dst], '/');
+        ret = gf_canonicalize_path(delimiter);
+        if (ret)
+            goto out;
+    }
 
-        return ret;
+    ret = dict_set_str(dict, "dst-brick", (char *)words[dst]);
+    if (ret)
+        goto out;
+    ret = 0;
+out:
+    return ret;
 }
 
-
 int32_t
-cli_cmd_volume_replace_brick_parse (const char **words, int wordcount,
-                                   dict_t **options)
+cli_cmd_volume_reset_brick_parse(const char **words, int wordcount,
+                                 dict_t **options)
 {
-        dict_t  *dict = NULL;
-        char    *volname = NULL;
-        int     ret = -1;
-        int     op_index = 0;
-        char    *delimiter = NULL;
-        gf1_cli_replace_op replace_op = GF_REPLACE_OP_NONE;
-        char    *opwords[] = { "start", "commit", "pause", "abort", "status",
-                                NULL };
-        char    *w = NULL;
-        gf_boolean_t is_force = _gf_false;
-
-        GF_ASSERT (words);
-        GF_ASSERT (options);
-
-        dict = dict_new ();
-
-        if (!dict)
-                goto out;
+    int ret = -1;
+    char *volname = NULL;
+    dict_t *dict = NULL;
 
-        if (wordcount < 3)
-                goto out;
+    if (wordcount < 5 || wordcount > 7)
+        goto out;
 
-        volname = (char *)words[2];
+    dict = dict_new();
 
-        GF_ASSERT (volname);
+    if (!dict)
+        goto out;
 
-        ret = dict_set_str (dict, "volname", volname);
+    volname = (char *)words[2];
 
-        if (ret)
-                goto out;
+    ret = dict_set_str(dict, "volname", volname);
+    if (ret)
+        goto out;
 
-        if (wordcount < 4) {
-                ret = -1;
-                goto out;
+    if (wordcount == 5) {
+        if (strcmp(words[4], "start")) {
+            cli_err(
+                "Invalid option '%s' for reset-brick. Please "
+                "enter valid reset-brick command",
+                words[4]);
+            ret = -1;
+            goto out;
         }
 
-        if (validate_brick_name ((char *)words[3])) {
-                cli_err ("wrong brick type: %s, use "
-                         "<HOSTNAME>:<export-dir-abs-path>", words[3]);
-                ret = -1;
-                goto out;
-        } else {
-                delimiter = strrchr ((char *)words[3], ':');
-                ret = gf_canonicalize_path (delimiter + 1);
-                if (ret)
-                        goto out;
-        }
-        ret = dict_set_str (dict, "src-brick", (char *)words[3]);
+        ret = cli_cmd_brick_op_validate_bricks(words, dict, 3, -1);
+        if (ret)
+            goto out;
 
+        ret = dict_set_str(dict, "operation", "GF_RESET_OP_START");
         if (ret)
-                goto out;
+            goto out;
+    } else if (wordcount == 6) {
+        if (strcmp(words[5], "commit")) {
+            cli_err(
+                "Invalid option '%s' for reset-brick. Please "
+                "enter valid reset-brick command",
+                words[5]);
+            ret = -1;
+            goto out;
+        }
+
+        ret = cli_cmd_brick_op_validate_bricks(words, dict, 3, 4);
+        if (ret)
+            goto out;
 
-        if (wordcount < 5) {
-                ret = -1;
-                goto out;
-        }
+        ret = dict_set_str(dict, "operation", "GF_RESET_OP_COMMIT");
+        if (ret)
+            goto out;
+    } else if (wordcount == 7) {
+        if (strcmp(words[5], "commit") || strcmp(words[6], "force")) {
+            cli_err(
+                "Invalid option '%s %s' for reset-brick. Please "
+                "enter valid reset-brick command",
+                words[5], words[6]);
+            ret = -1;
+            goto out;
+        }
+
+        ret = cli_cmd_brick_op_validate_bricks(words, dict, 3, 4);
+        if (ret)
+            goto out;
 
-        if (validate_brick_name ((char *)words[4])) {
-                cli_err ("wrong brick type: %s, use "
-                         "<HOSTNAME>:<export-dir-abs-path>", words[4]);
-                ret = -1;
-                goto out;
-        } else {
-                delimiter = strrchr ((char *)words[4], ':');
-                ret = gf_canonicalize_path (delimiter + 1);
-                if (ret)
-                        goto out;
-        }
+        ret = dict_set_str(dict, "operation", "GF_RESET_OP_COMMIT_FORCE");
+        if (ret)
+            goto out;
+    }
 
+    *options = dict;
 
-        ret = dict_set_str (dict, "dst-brick", (char *)words[4]);
+out:
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to parse reset-brick CLI");
+        if (dict)
+            dict_unref(dict);
+    }
 
-        if (ret)
-                goto out;
+    return ret;
+}
 
-        op_index = 5;
-        if ((wordcount < (op_index + 1))) {
-                ret = -1;
-                goto out;
-        }
+int32_t
+cli_cmd_volume_replace_brick_parse(const char **words, int wordcount,
+                                   dict_t **options)
+{
+    int ret = -1;
+    char *volname = NULL;
+    dict_t *dict = NULL;
 
-        w = str_getunamb (words[op_index], opwords);
+    GF_ASSERT(words);
+    GF_ASSERT(options);
 
-        if (!w) {
-        } else if (!strcmp ("start", w)) {
-                replace_op = GF_REPLACE_OP_START;
-        } else if (!strcmp ("commit", w)) {
-                replace_op = GF_REPLACE_OP_COMMIT;
-        } else if (!strcmp ("pause", w)) {
-                replace_op = GF_REPLACE_OP_PAUSE;
-        } else if (!strcmp ("abort", w)) {
-                replace_op = GF_REPLACE_OP_ABORT;
-        } else if (!strcmp ("status", w)) {
-                replace_op = GF_REPLACE_OP_STATUS;
-        } else
-                GF_ASSERT (!"opword mismatch");
-
-        /* commit force option */
-
-        op_index = 6;
-
-        if (wordcount > (op_index + 1)) {
-                ret = -1;
-                goto out;
-        }
+    if (wordcount != 7) {
+        ret = -1;
+        goto out;
+    }
 
-        if (wordcount == (op_index + 1)) {
-                if ((replace_op != GF_REPLACE_OP_COMMIT) &&
-                    (replace_op != GF_REPLACE_OP_START)) {
-                        ret = -1;
-                        goto out;
-                }
-                if (!strcmp ("force", words[op_index])) {
-                        if (replace_op == GF_REPLACE_OP_COMMIT)
-                                replace_op = GF_REPLACE_OP_COMMIT_FORCE;
+    dict = dict_new();
 
-                        else if (replace_op == GF_REPLACE_OP_START)
-                                is_force = _gf_true;
-                }
-        }
+    if (!dict) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to allocate dictionary");
+        goto out;
+    }
 
-        if (replace_op == GF_REPLACE_OP_NONE) {
-                ret = -1;
-                goto out;
-        }
+    volname = (char *)words[2];
 
-        ret = dict_set_int32 (dict, "operation", (int32_t) replace_op);
+    GF_ASSERT(volname);
 
-        if (ret)
-                goto out;
+    ret = dict_set_str(dict, "volname", volname);
+    if (ret)
+        goto out;
 
-        ret = dict_set_int32 (dict, "force", is_force);
-        if (ret)
-                goto out;
+    ret = cli_cmd_brick_op_validate_bricks(words, dict, 3, 4);
+    if (ret)
+        goto out;
 
-        *options = dict;
+    /* commit force option */
+    if (strcmp("commit", words[5]) || strcmp("force", words[6])) {
+        cli_err(
+            "Invalid option '%s' '%s' for replace-brick. Please "
+            "enter valid replace-brick command",
+            words[5], words[6]);
+        ret = -1;
+        goto out;
+    }
+
+    ret = dict_set_str(dict, "operation", "GF_REPLACE_OP_COMMIT_FORCE");
+    if (ret)
+        goto out;
+
+    *options = dict;
 
 out:
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to parse replace-brick CLI");
-                if (dict)
-                        dict_destroy (dict);
-        }
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to parse reset-brick CLI");
+        if (dict)
+            dict_unref(dict);
+    }
 
-        return ret;
+    return ret;
 }
 
 int32_t
-cli_cmd_log_filename_parse (const char **words, int wordcount, dict_t **options)
+cli_cmd_log_filename_parse(const char **words, int wordcount, dict_t **options)
 {
-        dict_t  *dict = NULL;
-        char    *volname = NULL;
-        char    *str = NULL;
-        int     ret = -1;
-        char    *delimiter = NULL;
+    dict_t *dict = NULL;
+    char *volname = NULL;
+    char *str = NULL;
+    int ret = -1;
+    char *delimiter = NULL;
 
-        GF_ASSERT (words);
-        GF_ASSERT (options);
+    GF_ASSERT(words);
+    GF_ASSERT(options);
 
-        dict = dict_new ();
-        if (!dict)
-                goto out;
+    dict = dict_new();
+    if (!dict)
+        goto out;
 
-        volname = (char *)words[3];
-        GF_ASSERT (volname);
+    volname = (char *)words[3];
+    GF_ASSERT(volname);
 
-        ret = dict_set_str (dict, "volname", volname);
-        if (ret)
-                goto out;
+    ret = dict_set_str(dict, "volname", volname);
+    if (ret)
+        goto out;
 
-        str = (char *)words[4];
-        if (strchr (str, ':')) {
-                delimiter = strchr (words[4], ':');
-                if (!delimiter || delimiter == words[4]
-                    || *(delimiter+1) != '/') {
-                        cli_err ("wrong brick type: %s, use <HOSTNAME>:"
-                                 "<export-dir-abs-path>", words[4]);
-                        ret = -1;
-                        goto out;
-                } else {
-                        ret = gf_canonicalize_path (delimiter + 1);
-                        if (ret)
-                                goto out;
-                }
-                ret = dict_set_str (dict, "brick", str);
-                if (ret)
-                        goto out;
-                /* Path */
-                str = (char *)words[5];
-                ret = dict_set_str (dict, "path", str);
-                if (ret)
-                        goto out;
+    str = (char *)words[4];
+    if (strchr(str, ':')) {
+        delimiter = strchr(words[4], ':');
+        if (!delimiter || delimiter == words[4] || *(delimiter + 1) != '/') {
+            cli_err(
+                "wrong brick type: %s, use <HOSTNAME>:"
+                "<export-dir-abs-path>",
+                words[4]);
+            ret = -1;
+            goto out;
         } else {
-                ret = dict_set_str (dict, "path", str);
-                if (ret)
-                        goto out;
+            ret = gf_canonicalize_path(delimiter + 1);
+            if (ret)
+                goto out;
         }
+        ret = dict_set_str(dict, "brick", str);
+        if (ret)
+            goto out;
+        /* Path */
+        str = (char *)words[5];
+        ret = dict_set_str(dict, "path", str);
+        if (ret)
+            goto out;
+    } else {
+        ret = dict_set_str(dict, "path", str);
+        if (ret)
+            goto out;
+    }
 
-        *options = dict;
+    *options = dict;
 
 out:
-        if (ret && dict)
-                dict_destroy (dict);
+    if (ret && dict)
+        dict_unref(dict);
 
-        return ret;
+    return ret;
 }
 
 int32_t
-cli_cmd_log_level_parse (const char **words, int worcount, dict_t **options)
+cli_cmd_log_level_parse(const char **words, int worcount, dict_t **options)
 {
-        dict_t *dict            = NULL;
-        int     ret             = -1;
-
-        GF_ASSERT (words);
-        GF_ASSERT (options);
-
-        /*
-         * loglevel command format:
-         *  > volume log level <VOL> <XLATOR[*]> <LOGLEVEL>
-         *  > volume log level colon-o posix WARNING
-         *  > volume log level colon-o replicate* DEBUG
-         *  > volume log level coon-o * TRACE
-         */
-
-        GF_ASSERT ((strncmp(words[0], "volume", 6) == 0));
-        GF_ASSERT ((strncmp(words[1], "log", 3) == 0));
-        GF_ASSERT ((strncmp(words[2], "level", 5) == 0));
-
-        ret = glusterd_check_log_level(words[5]);
-        if (ret == -1) {
-                cli_err("Invalid log level [%s] specified", words[5]);
-                cli_err("Valid values for loglevel: (DEBUG|WARNING|ERROR"
-                        "|CRITICAL|NONE|TRACE)");
-                goto out;
-        }
+    dict_t *dict = NULL;
+    int ret = -1;
+
+    GF_ASSERT(words);
+    GF_ASSERT(options);
+
+    /*
+     * loglevel command format:
+     *  > volume log level <VOL> <XLATOR[*]> <LOGLEVEL>
+     *  > volume log level colon-o posix WARNING
+     *  > volume log level colon-o replicate* DEBUG
+     *  > volume log level coon-o * TRACE
+     */
+
+    GF_ASSERT((strncmp(words[0], "volume", 6) == 0));
+    GF_ASSERT((strncmp(words[1], "log", 3) == 0));
+    GF_ASSERT((strncmp(words[2], "level", 5) == 0));
+
+    ret = glusterd_check_log_level(words[5]);
+    if (ret == -1) {
+        cli_err("Invalid log level [%s] specified", words[5]);
+        cli_err(
+            "Valid values for loglevel: (DEBUG|WARNING|ERROR"
+            "|CRITICAL|NONE|TRACE)");
+        goto out;
+    }
 
-        dict = dict_new ();
-        if (!dict)
-                goto out;
+    dict = dict_new();
+    if (!dict)
+        goto out;
 
-        GF_ASSERT(words[3]);
-        GF_ASSERT(words[4]);
+    GF_ASSERT(words[3]);
+    GF_ASSERT(words[4]);
 
-        ret = dict_set_str (dict, "volname", (char *)words[3]);
-        if (ret)
-                goto out;
+    ret = dict_set_str(dict, "volname", (char *)words[3]);
+    if (ret)
+        goto out;
 
-        ret = dict_set_str (dict, "xlator", (char *)words[4]);
-        if (ret)
-                goto out;
+    ret = dict_set_str(dict, "xlator", (char *)words[4]);
+    if (ret)
+        goto out;
 
-        ret = dict_set_str (dict, "loglevel", (char *)words[5]);
-        if (ret)
-                goto out;
+    ret = dict_set_str(dict, "loglevel", (char *)words[5]);
+    if (ret)
+        goto out;
 
-        *options = dict;
+    *options = dict;
 
- out:
-        if (ret && dict)
-                dict_destroy (dict);
+out:
+    if (ret && dict)
+        dict_unref(dict);
 
-        return ret;
+    return ret;
 }
 
 int32_t
-cli_cmd_log_locate_parse (const char **words, int wordcount, dict_t **options)
+cli_cmd_log_locate_parse(const char **words, int wordcount, dict_t **options)
 {
-        dict_t  *dict = NULL;
-        char    *volname = NULL;
-        char    *str = NULL;
-        int     ret = -1;
-        char    *delimiter = NULL;
+    dict_t *dict = NULL;
+    char *volname = NULL;
+    char *str = NULL;
+    int ret = -1;
+    char *delimiter = NULL;
 
-        GF_ASSERT (words);
-        GF_ASSERT (options);
+    GF_ASSERT(words);
+    GF_ASSERT(options);
 
-        dict = dict_new ();
-        if (!dict)
-                goto out;
+    dict = dict_new();
+    if (!dict)
+        goto out;
 
-        volname = (char *)words[3];
-        GF_ASSERT (volname);
+    volname = (char *)words[3];
+    GF_ASSERT(volname);
 
-        ret = dict_set_str (dict, "volname", volname);
-        if (ret)
-                goto out;
+    ret = dict_set_str(dict, "volname", volname);
+    if (ret)
+        goto out;
 
-        if (words[4]) {
-                delimiter = strchr (words[4], ':');
-                if (!delimiter || delimiter == words[4]
-                    || *(delimiter+1) != '/') {
-                        cli_err ("wrong brick type: %s, use <HOSTNAME>:"
-                                 "<export-dir-abs-path>", words[4]);
-                        ret = -1;
-                        goto out;
-                } else {
-                        ret = gf_canonicalize_path (delimiter + 1);
-                        if (ret)
-                                goto out;
-                }
-                str = (char *)words[4];
-                ret = dict_set_str (dict, "brick", str);
-                if (ret)
-                        goto out;
+    if (words[4]) {
+        delimiter = strchr(words[4], ':');
+        if (!delimiter || delimiter == words[4] || *(delimiter + 1) != '/') {
+            cli_err(
+                "wrong brick type: %s, use <HOSTNAME>:"
+                "<export-dir-abs-path>",
+                words[4]);
+            ret = -1;
+            goto out;
+        } else {
+            ret = gf_canonicalize_path(delimiter + 1);
+            if (ret)
+                goto out;
         }
+        str = (char *)words[4];
+        ret = dict_set_str(dict, "brick", str);
+        if (ret)
+            goto out;
+    }
 
-        *options = dict;
+    *options = dict;
 
 out:
-        if (ret && dict)
-                dict_destroy (dict);
+    if (ret && dict)
+        dict_unref(dict);
 
-        return ret;
+    return ret;
 }
 
 int32_t
-cli_cmd_log_rotate_parse (const char **words, int wordcount, dict_t **options)
+cli_cmd_log_rotate_parse(const char **words, int wordcount, dict_t **options)
 {
-        dict_t  *dict = NULL;
-        char    *volname = NULL;
-        char    *str = NULL;
-        int     ret = -1;
-        char    *delimiter = NULL;
+    dict_t *dict = NULL;
+    char *volname = NULL;
+    char *str = NULL;
+    int ret = -1;
+    char *delimiter = NULL;
 
-        GF_ASSERT (words);
-        GF_ASSERT (options);
+    GF_ASSERT(words);
+    GF_ASSERT(options);
 
-        dict = dict_new ();
-        if (!dict)
-                goto out;
+    dict = dict_new();
+    if (!dict)
+        goto out;
 
-        if (strcmp ("rotate", words[3]) == 0)
-                volname = (char *)words[2];
-        else if (strcmp ("rotate", words[2]) == 0)
-                volname = (char *)words[3];
-        GF_ASSERT (volname);
+    if (strcmp("rotate", words[3]) == 0)
+        volname = (char *)words[2];
+    GF_ASSERT(volname);
 
-        ret = dict_set_str (dict, "volname", volname);
-        if (ret)
-                goto out;
+    ret = dict_set_str(dict, "volname", volname);
+    if (ret)
+        goto out;
 
-        if (words[4]) {
-                delimiter = strchr (words[4], ':');
-                if (!delimiter || delimiter == words[4]
-                    || *(delimiter+1) != '/') {
-                        cli_err ("wrong brick type: %s, use <HOSTNAME>:"
-                                 "<export-dir-abs-path>", words[4]);
-                        ret = -1;
-                        goto out;
-                } else {
-                        ret = gf_canonicalize_path (delimiter + 1);
-                        if (ret)
-                                goto out;
-                }
-                str = (char *)words[4];
-                ret = dict_set_str (dict, "brick", str);
-                if (ret)
-                        goto out;
+    if (words[4]) {
+        delimiter = strchr(words[4], ':');
+        if (!delimiter || delimiter == words[4] || *(delimiter + 1) != '/') {
+            cli_err(
+                "wrong brick type: %s, use <HOSTNAME>:"
+                "<export-dir-abs-path>",
+                words[4]);
+            ret = -1;
+            goto out;
+        } else {
+            ret = gf_canonicalize_path(delimiter + 1);
+            if (ret)
+                goto out;
         }
+        str = (char *)words[4];
+        ret = dict_set_str(dict, "brick", str);
+        if (ret)
+            goto out;
+    }
 
-        *options = dict;
+    *options = dict;
 
 out:
-        if (ret && dict)
-                dict_destroy (dict);
+    if (ret && dict)
+        dict_unref(dict);
 
-        return ret;
+    return ret;
 }
 
 static gf_boolean_t
-gsyncd_url_check (const char *w)
+gsyncd_url_check(const char *w)
 {
-        return !!strpbrk (w, ":/");
+    return !!strpbrk(w, ":/");
 }
 
 static gf_boolean_t
-gsyncd_glob_check (const char *w)
+valid_slave_gsyncd_url(const char *w)
 {
-        return !!strpbrk (w, "*?[");
+    if (strstr(w, ":::"))
+        return _gf_false;
+    else if (strstr(w, "::"))
+        return _gf_true;
+    else
+        return _gf_false;
 }
 
-static int
-config_parse (const char **words, int wordcount, dict_t *dict,
-              unsigned cmdi, unsigned glob)
+static gf_boolean_t
+gsyncd_glob_check(const char *w)
 {
-        int32_t            ret     = -1;
-        int32_t            i       = -1;
-        char               *append_str = NULL;
-        size_t             append_len = 0;
-        char               *subop = NULL;
+    return !!strpbrk(w, "*?[");
+}
 
-        switch ((wordcount - 1) - cmdi) {
+static int
+config_parse(const char **words, int wordcount, dict_t *dict, unsigned cmdi,
+             unsigned glob)
+{
+    int32_t ret = -1;
+    int32_t i = -1;
+    char *append_str = NULL;
+    size_t append_len = 0;
+    char *subop = NULL;
+    char *ret_chkpt = NULL;
+    struct tm checkpoint_time;
+    char chkpt_buf[20] = "";
+
+    switch ((wordcount - 1) - cmdi) {
         case 0:
-                subop = gf_strdup ("get-all");
-                break;
+            subop = gf_strdup("get-all");
+            break;
         case 1:
-                if (words[cmdi + 1][0] == '!') {
-                        (words[cmdi + 1])++;
-                        if (gf_asprintf (&subop, "del%s",
-                                         glob ? "-glob" : "") == -1)
-                                subop = NULL;
-                } else
-                        subop = gf_strdup ("get");
+            if (words[cmdi + 1][0] == '!') {
+                (words[cmdi + 1])++;
+                if (gf_asprintf(&subop, "del%s", glob ? "-glob" : "") == -1)
+                    subop = NULL;
+            } else
+                subop = gf_strdup("get");
 
-                ret = dict_set_str (dict, "op_name", ((char *)words[cmdi + 1]));
-                if (ret < 0)
-                        goto out;
-                break;
+            ret = dict_set_str(dict, "op_name", ((char *)words[cmdi + 1]));
+            if (ret < 0)
+                goto out;
+            break;
         default:
-                if (gf_asprintf (&subop, "set%s", glob ? "-glob" : "") == -1)
-                        subop = NULL;
+            if (gf_asprintf(&subop, "set%s", glob ? "-glob" : "") == -1)
+                subop = NULL;
 
-                ret = dict_set_str (dict, "op_name", ((char *)words[cmdi + 1]));
-                if (ret < 0)
-                        goto out;
+            ret = dict_set_str(dict, "op_name", ((char *)words[cmdi + 1]));
+            if (ret < 0)
+                goto out;
 
-                /* join the varargs by spaces to get the op_value */
+            /* join the varargs by spaces to get the op_value */
 
-                for (i = cmdi + 2; i < wordcount; i++)
-                        append_len += (strlen (words[i]) + 1);
-                /* trailing strcat will add two bytes, make space for that */
-                append_len++;
+            for (i = cmdi + 2; i < wordcount; i++)
+                append_len += (strlen(words[i]) + 1);
+            /* trailing strcat will add two bytes, make space for that */
+            append_len++;
 
-                append_str = GF_CALLOC (1, append_len, cli_mt_append_str);
-                if (!append_str) {
-                        ret = -1;
-                        goto out;
-                }
+            /* strcat is used on this allocation and hence expected to be
+             * initiatlized to 0. So GF_CALLOC is used.
+             */
+            append_str = GF_CALLOC(1, append_len, cli_mt_append_str);
+            if (!append_str) {
+                ret = -1;
+                goto out;
+            }
+
+            for (i = cmdi + 2; i < wordcount; i++) {
+                strcat(append_str, words[i]);
+                strcat(append_str, " ");
+            }
+            append_str[append_len - 2] = '\0';
+            /* "checkpoint now" is special: we resolve that "now" */
+            if ((strcmp(words[cmdi + 1], "checkpoint") == 0) &&
+                (strcmp(append_str, "now") == 0)) {
+                struct timeval tv = {
+                    0,
+                };
+
+                ret = gettimeofday(&tv, NULL);
+                if (ret == -1)
+                    goto out;
 
-                for (i = cmdi + 2; i < wordcount; i++) {
-                        strcat (append_str, words[i]);
-                        strcat (append_str, " ");
-                }
-                append_str[append_len - 2] = '\0';
-                /* "checkpoint now" is special: we resolve that "now" */
-                if ((strcmp (words[cmdi + 1], "checkpoint") == 0) &&
-                    (strcmp (append_str, "now") == 0)) {
-                        struct timeval tv = {0,};
-
-                        ret = gettimeofday (&tv, NULL);
-                        if (ret == -1)
-                                goto out;
-
-                        GF_FREE (append_str);
-                        append_str = GF_CALLOC (1, 300, cli_mt_append_str);
-                        if (!append_str) {
-                                ret = -1;
-                                goto out;
-                        }
-                        snprintf (append_str, 300, "now:%" GF_PRI_SECOND ".%06"GF_PRI_SUSECONDS,
-                                  tv.tv_sec, tv.tv_usec);
+                GF_FREE(append_str);
+                append_str = GF_MALLOC(300, cli_mt_append_str);
+                if (!append_str) {
+                    ret = -1;
+                    goto out;
+                }
+                snprintf(append_str, 300, "%" GF_PRI_SECOND, tv.tv_sec);
+            } else if ((strcmp(words[cmdi + 1], "checkpoint") == 0) &&
+                       (strcmp(append_str, "now") != 0)) {
+                memset(&checkpoint_time, 0, sizeof(struct tm));
+                ret_chkpt = strptime(append_str, "%Y-%m-%d %H:%M:%S",
+                                     &checkpoint_time);
+
+                if (ret_chkpt == NULL || *ret_chkpt != '\0') {
+                    ret = -1;
+                    cli_err(
+                        "Invalid Checkpoint label. Use format "
+                        "\"Y-m-d H:M:S\", Example: 2016-10-25 15:30:45");
+                    goto out;
+                }
+                GF_FREE(append_str);
+                append_str = GF_MALLOC(300, cli_mt_append_str);
+                if (!append_str) {
+                    ret = -1;
+                    goto out;
                 }
+                strftime(chkpt_buf, sizeof(chkpt_buf), "%s", &checkpoint_time);
+                snprintf(append_str, 300, "%s", chkpt_buf);
+            }
 
-                ret = dict_set_dynstr (dict, "op_value", append_str);
-        }
+            ret = dict_set_dynstr(dict, "op_value", append_str);
+            if (ret != 0) {
+                goto out;
+            }
+            append_str = NULL;
+    }
 
-        ret = -1;
-        if (subop) {
-                ret = dict_set_dynstr (dict, "subop", subop);
-                if (!ret)
-                      subop = NULL;
-        }
+    ret = -1;
+    if (subop) {
+        ret = dict_set_dynstr(dict, "subop", subop);
+        if (!ret)
+            subop = NULL;
+    }
 
 out:
-        if (ret && append_str)
-                GF_FREE (append_str);
+    GF_FREE(append_str);
+    GF_FREE(subop);
 
-        GF_FREE (subop);
-
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 
+/* ssh_port_parse: Parses and validates when ssh_port is given.
+ *                 ssh_index refers to index of ssh_port and
+ *                 type refers to either push-pem or no-verify
+ */
+
 static int32_t
-force_push_pem_parse (const char **words, int wordcount,
-                      dict_t *dict, unsigned *cmdi)
+parse_ssh_port(const char **words, int wordcount, dict_t *dict, unsigned *cmdi,
+               int ssh_index, char *type)
 {
-        int32_t            ret     = 0;
-
-        if (!strcmp ((char *)words[wordcount-1], "force")) {
-                if ((strcmp ((char *)words[wordcount-2], "start")) &&
-                    (strcmp ((char *)words[wordcount-2], "stop")) &&
-                    (strcmp ((char *)words[wordcount-2], "create")) &&
-                    (strcmp ((char *)words[wordcount-2], "push-pem")) &&
-                    (strcmp ((char *)words[wordcount-2], "pause")) &&
-                    (strcmp ((char *)words[wordcount-2], "resume"))) {
-                        ret = -1;
-                        goto out;
-                }
-                ret = dict_set_uint32 (dict, "force",
-                                       _gf_true);
-                if (ret)
-                        goto out;
-                (*cmdi)++;
+    int ret = 0;
+    char *end_ptr = NULL;
+    int64_t limit = 0;
 
-                if (!strcmp ((char *)words[wordcount-2], "push-pem")) {
-                        if (strcmp ((char *)words[wordcount-3], "create")) {
-                                ret = -1;
-                                goto out;
-                        }
-                        ret = dict_set_int32 (dict, "push_pem", 1);
-                        if (ret)
-                                goto out;
-                        (*cmdi)++;
-                }
-        } else if (!strcmp ((char *)words[wordcount-1], "push-pem")) {
-                if (strcmp ((char *)words[wordcount-2], "create")) {
-                        ret = -1;
-                        goto out;
-                }
-                ret = dict_set_int32 (dict, "push_pem", 1);
-                if (ret)
-                        goto out;
-                (*cmdi)++;
+    if (!strcmp((char *)words[ssh_index], "ssh-port")) {
+        if (strcmp((char *)words[ssh_index - 1], "create")) {
+            ret = -1;
+            goto out;
         }
+        (*cmdi)++;
+        limit = strtol(words[ssh_index + 1], &end_ptr, 10);
+        if (errno == ERANGE || errno == EINVAL || limit <= 0 ||
+            strcmp(end_ptr, "") != 0) {
+            ret = -1;
+            cli_err("Please enter an integer value for ssh_port ");
+            goto out;
+        }
+
+        ret = dict_set_int32(dict, "ssh_port", limit);
+        if (ret)
+            goto out;
+        (*cmdi)++;
+    } else if (strcmp((char *)words[ssh_index + 1], "create")) {
+        ret = -1;
+        goto out;
+    }
+
+    ret = dict_set_int32(dict, type, 1);
+    if (ret)
+        goto out;
+    (*cmdi)++;
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    return ret;
 }
 
-
-int32_t
-cli_cmd_gsync_set_parse (const char **words, int wordcount, dict_t **options)
+static int32_t
+force_push_pem_no_verify_parse(const char **words, int wordcount, dict_t *dict,
+                               unsigned *cmdi)
 {
-        int32_t            ret     = -1;
-        dict_t             *dict   = NULL;
-        gf1_cli_gsync_set  type    = GF_GSYNC_OPTION_TYPE_NONE;
-        int                i       = 0;
-        unsigned           masteri = 0;
-        unsigned           slavei  = 0;
-        unsigned           glob    = 0;
-        unsigned           cmdi    = 0;
-        char               *opwords[] = { "create", "status", "start", "stop",
-                                          "config", "force", "delete",
-                                          "push-pem", "detail", "pause",
-                                          "resume", NULL };
-        char               *w = NULL;
-
-        GF_ASSERT (words);
-        GF_ASSERT (options);
-
-        dict = dict_new ();
-        if (!dict)
-                goto out;
-
-        /* new syntax:
-         *
-         * volume geo-replication $m $s create [push-pem] [force]
-         * volume geo-replication [$m [$s]] status [detail]
-         * volume geo-replication [$m] $s config [[!]$opt [$val]]
-         * volume geo-replication $m $s start|stop [force]
-         * volume geo-replication $m $s delete
-         * volume geo-replication $m $s pause [force]
-         * volume geo-replication $m $s resume [force]
-         */
+    int32_t ret = 0;
+
+    if (!strcmp((char *)words[wordcount - 1], "force")) {
+        if ((strcmp((char *)words[wordcount - 2], "start")) &&
+            (strcmp((char *)words[wordcount - 2], "stop")) &&
+            (strcmp((char *)words[wordcount - 2], "create")) &&
+            (strcmp((char *)words[wordcount - 2], "no-verify")) &&
+            (strcmp((char *)words[wordcount - 2], "push-pem")) &&
+            (strcmp((char *)words[wordcount - 2], "pause")) &&
+            (strcmp((char *)words[wordcount - 2], "resume"))) {
+            ret = -1;
+            goto out;
+        }
+        ret = dict_set_int32n(dict, "force", SLEN("force"), 1);
+        if (ret)
+            goto out;
+        (*cmdi)++;
 
-        if (wordcount < 3)
+        if (!strcmp((char *)words[wordcount - 2], "push-pem")) {
+            ret = parse_ssh_port(words, wordcount, dict, cmdi, wordcount - 4,
+                                 "push_pem");
+            if (ret)
                 goto out;
-
-        for (i = 2; i <= 3 && i < wordcount - 1; i++) {
-                if (gsyncd_glob_check (words[i]))
-                        glob = i;
-                if (gsyncd_url_check (words[i])) {
-                        slavei = i;
-                        break;
-                }
-        }
-
-        if (glob && !slavei)
-                /* glob is allowed only for config, thus it implies there is a
-                 * slave argument; but that might have not been recognized on
-                 * the first scan as it's url characteristics has been covered
-                 * by the glob syntax.
-                 *
-                 * In this case, the slave is perforce the last glob-word -- the
-                 * upcoming one is neither glob, nor url, so it's definitely not
-                 * the slave.
-                 */
-                slavei = glob;
-        if (slavei) {
-                cmdi = slavei + 1;
-                if (slavei == 3)
-                        masteri = 2;
-        } else if (i <= 3) {
-                if (!strcmp ((char *)words[wordcount-1], "detail")) {
-                        /* For status detail it is mandatory to provide
-                         * both master and slave */
-                        ret = -1;
-                        goto out;
-                }
-
-                /* no $s, can only be status cmd
-                 * (with either a single $m before it or nothing)
-                 * -- these conditions imply that i <= 3 after
-                 * the iteration and that i is the successor of
-                 * the (0 or 1 length) sequence of $m-s.
-                 */
-                cmdi = i;
-                if (i == 3)
-                        masteri = 2;
-        } else
+        } else if (!strcmp((char *)words[wordcount - 2], "no-verify")) {
+            ret = parse_ssh_port(words, wordcount, dict, cmdi, wordcount - 4,
+                                 "no_verify");
+            if (ret)
                 goto out;
+        }
+    } else if (!strcmp((char *)words[wordcount - 1], "push-pem")) {
+        ret = parse_ssh_port(words, wordcount, dict, cmdi, wordcount - 3,
+                             "push_pem");
+        if (ret)
+            goto out;
+    } else if (!strcmp((char *)words[wordcount - 1], "no-verify")) {
+        ret = parse_ssh_port(words, wordcount, dict, cmdi, wordcount - 3,
+                             "no_verify");
+        if (ret)
+            goto out;
+    }
 
-        /* now check if input really complies syntax
-         * (in a somewhat redundant way, in favor
-         * transparent soundness)
-         */
-
-        if (masteri && gsyncd_url_check (words[masteri]))
-                goto out;
-        if (slavei && !glob && !gsyncd_url_check (words[slavei]))
-                goto out;
+out:
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
+}
 
-        w = str_getunamb (words[cmdi], opwords);
-        if (!w)
-                goto out;
+int32_t
+cli_cmd_gsync_set_parse(struct cli_state *state, const char **words,
+                        int wordcount, dict_t **options, char **errstr)
+{
+    int32_t ret = -1;
+    dict_t *dict = NULL;
+    gf1_cli_gsync_set type = GF_GSYNC_OPTION_TYPE_NONE;
+    int i = 0;
+    unsigned masteri = 0;
+    unsigned slavei = 0;
+    unsigned glob = 0;
+    unsigned cmdi = 0;
+    static char *opwords[] = {"create",    "status",   "start",  "stop",
+                              "config",    "force",    "delete", "ssh-port",
+                              "no-verify", "push-pem", "detail", "pause",
+                              "resume",    NULL};
+    char *w = NULL;
+    char *save_ptr = NULL;
+    char *slave_temp = NULL;
+    char *token = NULL;
+    gf_answer_t answer = GF_ANSWER_NO;
+    const char *question = NULL;
+
+    GF_ASSERT(words);
+    GF_ASSERT(options);
+
+    dict = dict_new();
+    if (!dict)
+        goto out;
 
-        if (strcmp (w, "create") == 0) {
-                type = GF_GSYNC_OPTION_TYPE_CREATE;
+    /* new syntax:
+     *
+     * volume geo-replication $m $s create [[ssh-port n] [[no-verify] |
+     * [push-pem]]] [force] volume geo-replication [$m [$s]] status [detail]
+     * volume geo-replication [$m] $s config [[!]$opt [$val]]
+     * volume geo-replication $m $s start|stop [force]
+     * volume geo-replication $m $s delete [reset-sync-time]
+     * volume geo-replication $m $s pause [force]
+     * volume geo-replication $m $s resume [force]
+     */
+
+    if (wordcount < 3)
+        goto out;
 
-                if (!masteri || !slavei)
-                        goto out;
-        } else if (strcmp (w, "status") == 0) {
-                type = GF_GSYNC_OPTION_TYPE_STATUS;
+    for (i = 2; i <= 3 && i < wordcount - 1; i++) {
+        if (gsyncd_glob_check(words[i]))
+            glob = i;
+        if (gsyncd_url_check(words[i])) {
+            slavei = i;
+            break;
+        }
+    }
 
-                if (slavei && !masteri)
-                        goto out;
-        } else if (strcmp (w, "config") == 0) {
-                type = GF_GSYNC_OPTION_TYPE_CONFIG;
+    if (glob && !slavei)
+        /* glob is allowed only for config, thus it implies there is a
+         * slave argument; but that might have not been recognized on
+         * the first scan as it's url characteristics has been covered
+         * by the glob syntax.
+         *
+         * In this case, the slave is perforce the last glob-word -- the
+         * upcoming one is neither glob, nor url, so it's definitely not
+         * the slave.
+         */
+        slavei = glob;
+    if (slavei) {
+        cmdi = slavei + 1;
+        if (slavei == 3)
+            masteri = 2;
+    } else if (i <= 4) {
+        if (strtail("detail", (char *)words[wordcount - 1])) {
+            cmdi = wordcount - 2;
+            if (i == 4)
+                masteri = 2;
+        } else {
+            /* no $s, can only be status cmd
+             * (with either a single $m before it or nothing)
+             * -- these conditions imply that i <= 3 after
+             * the iteration and that i is the successor of
+             * the (0 or 1 length) sequence of $m-s.
+             */
+            cmdi = i;
+            if (i == 3)
+                masteri = 2;
+        }
+    } else
+        goto out;
 
-                if (!slavei)
-                        goto out;
-        } else if (strcmp (w, "start") == 0) {
-                type = GF_GSYNC_OPTION_TYPE_START;
+    /* now check if input really complies syntax
+     * (in a somewhat redundant way, in favor
+     * transparent soundness)
+     */
 
-                if (!masteri || !slavei)
-                        goto out;
-        } else if (strcmp (w, "stop") == 0) {
-                type = GF_GSYNC_OPTION_TYPE_STOP;
+    if (masteri && gsyncd_url_check(words[masteri]))
+        goto out;
 
-                if (!masteri || !slavei)
-                        goto out;
-        } else if (strcmp (w, "delete") == 0) {
-                type = GF_GSYNC_OPTION_TYPE_DELETE;
+    if (slavei && !glob && !valid_slave_gsyncd_url(words[slavei])) {
+        gf_asprintf(errstr, "Invalid slave url: %s", words[slavei]);
+        goto out;
+    }
 
-                if (!masteri || !slavei)
-                        goto out;
-        } else if (strcmp (w, "pause") == 0) {
-                type = GF_GSYNC_OPTION_TYPE_PAUSE;
+    w = str_getunamb(words[cmdi], opwords);
+    if (!w)
+        goto out;
 
-                if (!masteri || !slavei)
-                        goto out;
-        } else if (strcmp (w, "resume") == 0) {
-                type = GF_GSYNC_OPTION_TYPE_RESUME;
+    if (strcmp(w, "create") == 0) {
+        type = GF_GSYNC_OPTION_TYPE_CREATE;
+
+        if (!masteri || !slavei)
+            goto out;
+    } else if (strcmp(w, "status") == 0) {
+        type = GF_GSYNC_OPTION_TYPE_STATUS;
+
+        if (slavei && !masteri)
+            goto out;
+    } else if (strcmp(w, "config") == 0) {
+        type = GF_GSYNC_OPTION_TYPE_CONFIG;
+
+        if (!slavei)
+            goto out;
+    } else if (strcmp(w, "start") == 0) {
+        type = GF_GSYNC_OPTION_TYPE_START;
+
+        if (!masteri || !slavei)
+            goto out;
+    } else if (strcmp(w, "stop") == 0) {
+        type = GF_GSYNC_OPTION_TYPE_STOP;
+
+        if (!masteri || !slavei)
+            goto out;
+    } else if (strcmp(w, "delete") == 0) {
+        type = GF_GSYNC_OPTION_TYPE_DELETE;
+
+        if (!masteri || !slavei)
+            goto out;
+    } else if (strcmp(w, "pause") == 0) {
+        type = GF_GSYNC_OPTION_TYPE_PAUSE;
+
+        if (!masteri || !slavei)
+            goto out;
+    } else if (strcmp(w, "resume") == 0) {
+        type = GF_GSYNC_OPTION_TYPE_RESUME;
+
+        if (!masteri || !slavei)
+            goto out;
+    } else
+        GF_ASSERT(!"opword mismatch");
+
+    ret = force_push_pem_no_verify_parse(words, wordcount, dict, &cmdi);
+    if (ret)
+        goto out;
 
-                if (!masteri || !slavei)
-                        goto out;
-        } else
-                GF_ASSERT (!"opword mismatch");
+    if (strtail("detail", (char *)words[wordcount - 1])) {
+        if (!strtail("status", (char *)words[wordcount - 2])) {
+            ret = -1;
+            goto out;
+        }
 
-        ret = force_push_pem_parse (words, wordcount, dict, &cmdi);
+        ret = dict_set_uint32(dict, "status-detail", _gf_true);
         if (ret)
-                goto out;
+            goto out;
+        cmdi++;
+    }
 
-        if (!strcmp ((char *)words[wordcount-1], "detail")) {
-                if (strcmp ((char *)words[wordcount-2], "status")) {
-                        ret = -1;
-                        goto out;
-                }
-                if (!slavei || !masteri) {
-                        ret = -1;
-                        goto out;
-                }
-                ret = dict_set_uint32 (dict, "status-detail", _gf_true);
-                if (ret)
-                        goto out;
-                cmdi++;
+    if (type == GF_GSYNC_OPTION_TYPE_DELETE &&
+        !strcmp((char *)words[wordcount - 1], "reset-sync-time")) {
+        if (strcmp((char *)words[wordcount - 2], "delete")) {
+            ret = -1;
+            goto out;
         }
+        ret = dict_set_uint32(dict, "reset-sync-time", _gf_true);
+        if (ret)
+            goto out;
+        cmdi++;
+    }
 
-        if (type != GF_GSYNC_OPTION_TYPE_CONFIG &&
-            (cmdi < wordcount - 1 || glob))
-                goto out;
+    if (type != GF_GSYNC_OPTION_TYPE_CONFIG && (cmdi < wordcount - 1 || glob)) {
+        ret = -1;
+        goto out;
+    }
 
-        /* If got so far, input is valid, assemble the message */
+    /* If got so far, input is valid, assemble the message */
 
-        ret = 0;
+    ret = 0;
 
-        if (masteri) {
-                ret = dict_set_str (dict, "master", (char *)words[masteri]);
-                if (!ret)
-                        ret = dict_set_str (dict, "volname",
-                                            (char *)words[masteri]);
-        }
-        if (!ret && slavei)
-                ret = dict_set_str (dict, "slave", (char *)words[slavei]);
+    if (masteri) {
+        ret = dict_set_str(dict, "master", (char *)words[masteri]);
         if (!ret)
-                ret = dict_set_int32 (dict, "type", type);
-        if (!ret && type == GF_GSYNC_OPTION_TYPE_CONFIG)
-                ret = config_parse (words, wordcount, dict, cmdi, glob);
+            ret = dict_set_str(dict, "volname", (char *)words[masteri]);
+    }
+    if (!ret && slavei) {
+        /* If geo-rep is created with root user using the syntax
+         * gluster vol geo-rep <mastervol> root@<slavehost> ...
+         * pass down only <slavehost> else pass as it is.
+         */
+        slave_temp = gf_strdup(words[slavei]);
+        if (slave_temp == NULL) {
+            ret = -1;
+            goto out;
+        }
+        token = strtok_r(slave_temp, "@", &save_ptr);
+        if (token && !strcmp(token, "root")) {
+            ret = dict_set_str(dict, "slave", (char *)words[slavei] + 5);
+        } else {
+            ret = dict_set_str(dict, "slave", (char *)words[slavei]);
+        }
+    }
+    if (!ret)
+        ret = dict_set_int32(dict, "type", type);
+    if (!ret && type == GF_GSYNC_OPTION_TYPE_CONFIG) {
+        if (!strcmp((char *)words[wordcount - 2], "ignore-deletes") &&
+            !strcmp((char *)words[wordcount - 1], "true")) {
+            question =
+                "There exists ~15 seconds delay for the option to take"
+                " effect from stime of the corresponding brick. Please"
+                " check the log for the time, the option is effective."
+                " Proceed";
+
+            answer = cli_cmd_get_confirmation(state, question);
+
+            if (GF_ANSWER_NO == answer) {
+                gf_log("cli", GF_LOG_INFO,
+                       "Operation "
+                       "cancelled, exiting");
+                *errstr = gf_strdup("Aborted by user.");
+                ret = -1;
+                goto out;
+            }
+        }
 
-out:
-        if (ret) {
-                if (dict)
-                        dict_destroy (dict);
-        } else
-                *options = dict;
+        ret = config_parse(words, wordcount, dict, cmdi, glob);
+    }
 
+out:
+    if (slave_temp)
+        GF_FREE(slave_temp);
+    if (ret && dict)
+        dict_unref(dict);
+    else
+        *options = dict;
 
-        return ret;
+    return ret;
 }
 
 int32_t
-cli_cmd_volume_profile_parse (const char **words, int wordcount,
-                              dict_t **options)
+cli_cmd_volume_profile_parse(const char **words, int wordcount,
+                             dict_t **options)
 {
-        dict_t              *dict       = NULL;
-        char                *volname    = NULL;
-        int                 ret         = -1;
-        gf1_cli_stats_op    op          = GF_CLI_STATS_NONE;
-        gf1_cli_info_op     info_op     = GF_CLI_INFO_NONE;
-        gf_boolean_t        is_peek     = _gf_false;
+    dict_t *dict = NULL;
+    char *volname = NULL;
+    int ret = -1;
+    gf1_cli_stats_op op = GF_CLI_STATS_NONE;
+    gf1_cli_info_op info_op = GF_CLI_INFO_NONE;
+    gf_boolean_t is_peek = _gf_false;
 
-        char      *opwords[] = { "start", "stop", "info", NULL };
-        char      *w = NULL;
+    static char *opwords[] = {"start", "stop", "info", NULL};
+    char *w = NULL;
 
-        GF_ASSERT (words);
-        GF_ASSERT (options);
+    GF_ASSERT(words);
+    GF_ASSERT(options);
 
-        dict = dict_new ();
-        if (!dict)
-                goto out;
+    dict = dict_new();
+    if (!dict)
+        goto out;
 
-        if (wordcount < 4)
-                goto out;
+    if (wordcount < 4)
+        goto out;
 
-        volname = (char *)words[2];
+    volname = (char *)words[2];
 
-        ret = dict_set_str (dict, "volname", volname);
-        if (ret)
-                goto out;
-
-        w = str_getunamb (words[3], opwords);
-        if (!w) {
-                ret = -1;
-                goto out;
-        }
+    ret = dict_set_str(dict, "volname", volname);
+    if (ret)
+        goto out;
 
-        if ((strcmp (w, "start") == 0 || strcmp (w, "stop") == 0) &&
-            wordcount > 5)
-                goto out;
+    w = str_getunamb(words[3], opwords);
+    if (!w) {
+        ret = -1;
+        goto out;
+    }
 
-        if (strcmp (w, "info") == 0 && wordcount > 7)
-                goto out;
+    if ((strcmp(w, "start") == 0 || strcmp(w, "stop") == 0) && wordcount > 5) {
+        ret = -1;
+        goto out;
+    }
 
-        if (strcmp (w, "start") == 0) {
-                op = GF_CLI_STATS_START;
-        } else if (strcmp (w, "stop") == 0) {
-                op = GF_CLI_STATS_STOP;
-        } else if (strcmp (w, "info") == 0) {
-                op = GF_CLI_STATS_INFO;
-                info_op = GF_CLI_INFO_ALL;
-                if (wordcount > 4) {
-                        if (strcmp (words[4], "incremental") == 0) {
-                                info_op = GF_CLI_INFO_INCREMENTAL;
-                                if (wordcount > 5 &&
-                                    strcmp (words[5], "peek") == 0) {
-                                        is_peek = _gf_true;
-                                }
-                        } else if (strcmp (words[4], "cumulative") == 0) {
-                                info_op = GF_CLI_INFO_CUMULATIVE;
-                        } else if (strcmp (words[4], "clear") == 0) {
-                                info_op = GF_CLI_INFO_CLEAR;
-                        } else if (strcmp (words[4], "peek") == 0) {
-                                is_peek = _gf_true;
-                        }
-                }
-        } else
-                GF_ASSERT (!"opword mismatch");
+    if (strcmp(w, "info") == 0 && wordcount > 7) {
+        ret = -1;
+        goto out;
+    }
+
+    if (strcmp(w, "start") == 0) {
+        op = GF_CLI_STATS_START;
+    } else if (strcmp(w, "stop") == 0) {
+        op = GF_CLI_STATS_STOP;
+    } else if (strcmp(w, "info") == 0) {
+        op = GF_CLI_STATS_INFO;
+        info_op = GF_CLI_INFO_ALL;
+        if (wordcount > 4) {
+            if (strcmp(words[4], "incremental") == 0) {
+                info_op = GF_CLI_INFO_INCREMENTAL;
+                if (wordcount > 5 && strcmp(words[5], "peek") == 0) {
+                    is_peek = _gf_true;
+                }
+            } else if (strcmp(words[4], "cumulative") == 0) {
+                info_op = GF_CLI_INFO_CUMULATIVE;
+            } else if (strcmp(words[4], "clear") == 0) {
+                info_op = GF_CLI_INFO_CLEAR;
+            } else if (strcmp(words[4], "peek") == 0) {
+                is_peek = _gf_true;
+            }
+        }
+    } else
+        GF_ASSERT(!"opword mismatch");
+
+    ret = dict_set_int32(dict, "op", (int32_t)op);
+    if (ret)
+        goto out;
 
-        ret = dict_set_int32 (dict, "op", (int32_t)op);
-        if (ret)
-                goto out;
+    ret = dict_set_int32(dict, "info-op", (int32_t)info_op);
+    if (ret)
+        goto out;
 
-        ret = dict_set_int32 (dict, "info-op", (int32_t)info_op);
-        if (ret)
-                goto out;
+    ret = dict_set_int32(dict, "peek", is_peek);
+    if (ret)
+        goto out;
 
-        ret = dict_set_int32 (dict, "peek", is_peek);
+    if (!strcmp(words[wordcount - 1], "nfs")) {
+        ret = dict_set_int32(dict, "nfs", _gf_true);
         if (ret)
-                goto out;
+            goto out;
+    }
 
-        if (!strcmp (words[wordcount - 1], "nfs")) {
-                ret = dict_set_int32 (dict, "nfs", _gf_true);
-                if (ret)
-                        goto out;
-        }
-
-        *options = dict;
+    *options = dict;
 out:
-        if (ret && dict)
-                dict_destroy (dict);
-        return ret;
+    if (ret && dict)
+        dict_unref(dict);
+    return ret;
 }
 
 int32_t
-cli_cmd_volume_top_parse (const char **words, int wordcount,
-                              dict_t **options)
+cli_cmd_volume_top_parse(const char **words, int wordcount, dict_t **options)
 {
-        dict_t  *dict           = NULL;
-        char    *volname        = NULL;
-        char    *value          = NULL;
-        char    *key            = NULL;
-        int      ret            = -1;
-        gf1_cli_stats_op op = GF_CLI_STATS_NONE;
-        gf1_cli_top_op    top_op = GF_CLI_TOP_NONE;
-        int32_t  list_cnt       = -1;
-        int      index          = 0;
-        int      perf           = 0;
-        uint32_t  blk_size      = 0;
-        uint32_t  count         = 0;
-        gf_boolean_t nfs        = _gf_false;
-        char    *delimiter      = NULL;
-        char    *opwords[]      = { "open", "read", "write", "opendir",
-                                    "readdir", "read-perf", "write-perf",
-                                    "clear", NULL };
-        char    *w = NULL;
-
-        GF_ASSERT (words);
-        GF_ASSERT (options);
-
-        dict = dict_new ();
-        if (!dict)
-                goto out;
+    dict_t *dict = NULL;
+    char *volname = NULL;
+    char *value = NULL;
+    char *key = NULL;
+    int ret = -1;
+    gf1_cli_stats_op op = GF_CLI_STATS_NONE;
+    gf1_cli_top_op top_op = GF_CLI_TOP_NONE;
+    int32_t list_cnt = -1;
+    int index = 0;
+    int perf = 0;
+    int32_t blk_size = 0;
+    int count = 0;
+    gf_boolean_t nfs = _gf_false;
+    char *delimiter = NULL;
+    static char *opwords[] = {"open",       "read",    "write",
+                              "opendir",    "readdir", "read-perf",
+                              "write-perf", "clear",   NULL};
+    char *w = NULL;
+
+    GF_ASSERT(words);
+    GF_ASSERT(options);
+
+    dict = dict_new();
+    if (!dict)
+        goto out;
 
-        if (wordcount < 4)
-                goto out;
+    if (wordcount < 4)
+        goto out;
 
-        volname = (char *)words[2];
+    volname = (char *)words[2];
 
-        ret = dict_set_str (dict, "volname", volname);
-        if (ret)
-                goto out;
+    ret = dict_set_str(dict, "volname", volname);
+    if (ret)
+        goto out;
 
-        op = GF_CLI_STATS_TOP;
-        ret = dict_set_int32 (dict, "op", (int32_t)op);
-        if (ret)
-                goto out;
+    op = GF_CLI_STATS_TOP;
+    ret = dict_set_int32(dict, "op", (int32_t)op);
+    if (ret)
+        goto out;
 
-        w = str_getunamb (words[3], opwords);
-        if (!w) {
-                ret = -1;
-                goto out;
+    w = str_getunamb(words[3], opwords);
+    if (!w) {
+        ret = -1;
+        goto out;
+    }
+    if (strcmp(w, "open") == 0) {
+        top_op = GF_CLI_TOP_OPEN;
+    } else if (strcmp(w, "read") == 0) {
+        top_op = GF_CLI_TOP_READ;
+    } else if (strcmp(w, "write") == 0) {
+        top_op = GF_CLI_TOP_WRITE;
+    } else if (strcmp(w, "opendir") == 0) {
+        top_op = GF_CLI_TOP_OPENDIR;
+    } else if (strcmp(w, "readdir") == 0) {
+        top_op = GF_CLI_TOP_READDIR;
+    } else if (strcmp(w, "read-perf") == 0) {
+        top_op = GF_CLI_TOP_READ_PERF;
+        perf = 1;
+    } else if (strcmp(w, "write-perf") == 0) {
+        top_op = GF_CLI_TOP_WRITE_PERF;
+        perf = 1;
+    } else if (strcmp(w, "clear") == 0) {
+        ret = dict_set_int32(dict, "clear-stats", 1);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, "Could not set clear-stats in dict");
+            goto out;
         }
-        if (strcmp (w, "open") == 0) {
-                top_op = GF_CLI_TOP_OPEN;
-        } else if (strcmp (w, "read") == 0) {
-                top_op = GF_CLI_TOP_READ;
-        } else if (strcmp (w, "write") == 0) {
-                top_op = GF_CLI_TOP_WRITE;
-        } else if (strcmp (w, "opendir") == 0) {
-                top_op = GF_CLI_TOP_OPENDIR;
-        } else if (strcmp (w, "readdir") == 0) {
-                top_op = GF_CLI_TOP_READDIR;
-        } else if (strcmp (w, "read-perf") == 0) {
-                top_op = GF_CLI_TOP_READ_PERF;
-                perf = 1;
-        } else if (strcmp (w, "write-perf") == 0) {
-                top_op = GF_CLI_TOP_WRITE_PERF;
-                perf = 1;
-        } else if (strcmp (w, "clear") == 0) {
-                ret = dict_set_int32 (dict, "clear-stats", 1);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Could not set clear-stats in dict");
-                        goto out;
-                }
-        } else
-                GF_ASSERT (!"opword mismatch");
-        ret = dict_set_int32 (dict, "top-op", (int32_t)top_op);
+    } else
+        GF_ASSERT(!"opword mismatch");
+    ret = dict_set_int32(dict, "top-op", (int32_t)top_op);
+    if (ret)
+        goto out;
+
+    if ((wordcount > 4) && !strcmp(words[4], "nfs")) {
+        nfs = _gf_true;
+        ret = dict_set_int32(dict, "nfs", nfs);
         if (ret)
+            goto out;
+        index = 5;
+    } else {
+        index = 4;
+    }
+
+    for (; index < wordcount; index += 2) {
+        key = (char *)words[index];
+        value = (char *)words[index + 1];
+
+        if (!key || !value) {
+            ret = -1;
+            goto out;
+        }
+        if (!strcmp(key, "brick")) {
+            delimiter = strchr(value, ':');
+            if (!delimiter || delimiter == value || *(delimiter + 1) != '/') {
+                cli_err(
+                    "wrong brick type: %s, use <HOSTNAME>:"
+                    "<export-dir-abs-path>",
+                    value);
+                ret = -1;
                 goto out;
-
-        if ((wordcount > 4) && !strcmp (words[4], "nfs")) {
-                nfs = _gf_true;
-                ret = dict_set_int32 (dict, "nfs", nfs);
+            } else {
+                ret = gf_canonicalize_path(delimiter + 1);
                 if (ret)
-                        goto out;
-                index = 5;
-        } else {
-                index = 4;
-        }
-
-        for (; index < wordcount; index+=2) {
-
-                key = (char *) words[index];
-                value = (char *) words[index+1];
-
-                if ( key && !value ) {
-                        ret = -1;
-                        goto out;
-                }
-                if (!strcmp (key, "brick")) {
-                        delimiter = strchr (value, ':');
-                        if (!delimiter || delimiter == value
-                            || *(delimiter+1) != '/') {
-                                cli_err ("wrong brick type: %s, use <HOSTNAME>:"
-                                         "<export-dir-abs-path>", value);
-                                ret = -1;
-                                goto out;
-                        } else {
-                                ret = gf_canonicalize_path (delimiter + 1);
-                                if (ret)
-                                        goto out;
-                        }
-                        ret = dict_set_str (dict, "brick", value);
-
-                } else if (!strcmp (key, "list-cnt")) {
-                        ret = gf_is_str_int (value);
-                        if (!ret)
-                                list_cnt = atoi (value);
-                        if (ret || (list_cnt < 0) || (list_cnt > 100)) {
-                                cli_err ("list-cnt should be between 0 to 100");
-                                ret = -1;
-                                goto out;
-                        }
-                } else if (perf && !nfs && !strcmp (key, "bs")) {
-                        ret = gf_is_str_int (value);
-                        if (!ret)
-                                blk_size = atoi (value);
-                        if (ret || (blk_size <= 0)) {
-                                if (blk_size < 0)
-                                        cli_err ("block size is an invalid"
-                                                 " number");
-                                else
-                                        cli_err ("block size should be an "
-                                                 "integer greater than zero");
-                                ret = -1;
-                                goto out;
-                        }
-                        ret = dict_set_uint32 (dict, "blk-size", blk_size);
-                } else if (perf && !nfs && !strcmp (key, "count")) {
-                        ret = gf_is_str_int (value);
-                        if (!ret)
-                                count = atoi(value);
-                        if (ret || (count <= 0)) {
-                                if (count < 0)
-                                        cli_err ("count is an invalid number");
-                                else
-                                        cli_err ("count should be an integer "
-                                                 "greater than zero");
-
-                                ret = -1;
-                                goto out;
-                        }
-                        ret = dict_set_uint32 (dict, "blk-cnt", count);
-                } else {
-                        ret = -1;
-                        goto out;
-                }
-                if (ret) {
-                        gf_log ("", GF_LOG_WARNING, "Dict set failed for "
-                                "key %s", key);
-                        goto out;
-                }
-        }
-        if (list_cnt == -1)
-                list_cnt = 100;
-        ret = dict_set_int32 (dict, "list-cnt", list_cnt);
-        if (ret) {
-                gf_log ("", GF_LOG_WARNING, "Dict set failed for list_cnt");
+                    goto out;
+            }
+            ret = dict_set_str(dict, "brick", value);
+
+        } else if (!strcmp(key, "list-cnt")) {
+            ret = gf_is_str_int(value);
+            if (!ret)
+                list_cnt = atoi(value);
+            if (ret || (list_cnt < 0) || (list_cnt > 100)) {
+                cli_err("list-cnt should be between 0 to 100");
+                ret = -1;
                 goto out;
-        }
-
-        if ((blk_size > 0) ^ (count > 0)) {
-                cli_err ("Need to give both 'bs' and 'count'");
+            }
+        } else if (perf && !nfs && !strcmp(key, "bs")) {
+            ret = gf_is_str_int(value);
+            if (!ret)
+                blk_size = atoi(value);
+            if (ret || (blk_size <= 0)) {
+                if (blk_size < 0)
+                    cli_err(
+                        "block size is an invalid"
+                        " number");
+                else
+                    cli_err(
+                        "block size should be an "
+                        "integer greater than zero");
                 ret = -1;
                 goto out;
-        } else if (((uint64_t)blk_size * count) > (10 * GF_UNIT_GB)) {
-                cli_err ("'bs * count' value %"PRIu64" is greater than "
-                         "maximum allowed value of 10GB",
-                         ((uint64_t)blk_size * count));
+            }
+            ret = dict_set_uint32(dict, "blk-size", (uint32_t)blk_size);
+        } else if (perf && !nfs && !strcmp(key, "count")) {
+            ret = gf_is_str_int(value);
+            if (!ret)
+                count = atoi(value);
+            if (ret || (count <= 0)) {
+                if (count < 0)
+                    cli_err("count is an invalid number");
+                else
+                    cli_err(
+                        "count should be an integer "
+                        "greater than zero");
+
                 ret = -1;
                 goto out;
+            }
+            ret = dict_set_uint32(dict, "blk-cnt", count);
+        } else {
+            ret = -1;
+            goto out;
         }
+        if (ret) {
+            gf_log("", GF_LOG_WARNING,
+                   "Dict set failed for "
+                   "key %s",
+                   key);
+            goto out;
+        }
+    }
+    if (list_cnt == -1)
+        list_cnt = 100;
+    ret = dict_set_int32(dict, "list-cnt", list_cnt);
+    if (ret) {
+        gf_log("", GF_LOG_WARNING, "Dict set failed for list_cnt");
+        goto out;
+    }
 
-        *options = dict;
+    if ((blk_size > 0) ^ (count > 0)) {
+        cli_err("Need to give both 'bs' and 'count'");
+        ret = -1;
+        goto out;
+    } else if (((uint64_t)blk_size * count) > (10 * GF_UNIT_GB)) {
+        cli_err("'bs * count' value %" PRIu64
+                " is greater than "
+                "maximum allowed value of 10GB",
+                ((uint64_t)blk_size * count));
+        ret = -1;
+        goto out;
+    }
+
+    *options = dict;
 out:
-        if (ret && dict)
-                dict_destroy (dict);
-        return ret;
+    if (ret && dict)
+        dict_unref(dict);
+    return ret;
 }
 
 uint32_t
-cli_cmd_get_statusop (const char *arg)
+cli_cmd_get_statusop(const char *arg)
 {
-        int        i         = 0;
-        uint32_t   ret       = GF_CLI_STATUS_NONE;
-        char      *w         = NULL;
-        char      *opwords[] = {"detail", "mem", "clients", "fd",
-                                "inode", "callpool", "tasks", NULL};
-        struct {
-                char      *opname;
-                uint32_t   opcode;
-        } optable[] = {
-                { "detail",   GF_CLI_STATUS_DETAIL   },
-                { "mem",      GF_CLI_STATUS_MEM      },
-                { "clients",  GF_CLI_STATUS_CLIENTS  },
-                { "fd",       GF_CLI_STATUS_FD       },
-                { "inode",    GF_CLI_STATUS_INODE    },
-                { "callpool", GF_CLI_STATUS_CALLPOOL },
-                { "tasks",    GF_CLI_STATUS_TASKS    },
-                { NULL }
-        };
-
-        w = str_getunamb (arg, opwords);
-        if (!w) {
-                gf_log ("cli", GF_LOG_DEBUG,
-                        "Not a status op  %s", arg);
-                goto out;
-        }
+    int i = 0;
+    uint32_t ret = GF_CLI_STATUS_NONE;
+    char *w = NULL;
+    static char *opwords[] = {"detail",   "mem",   "clients",     "fd", "inode",
+                              "callpool", "tasks", "client-list", NULL};
+    static struct {
+        char *opname;
+        uint32_t opcode;
+    } optable[] = {{"detail", GF_CLI_STATUS_DETAIL},
+                   {"mem", GF_CLI_STATUS_MEM},
+                   {"clients", GF_CLI_STATUS_CLIENTS},
+                   {"fd", GF_CLI_STATUS_FD},
+                   {"inode", GF_CLI_STATUS_INODE},
+                   {"callpool", GF_CLI_STATUS_CALLPOOL},
+                   {"tasks", GF_CLI_STATUS_TASKS},
+                   {"client-list", GF_CLI_STATUS_CLIENT_LIST},
+                   {NULL}};
+
+    w = str_getunamb(arg, opwords);
+    if (!w) {
+        gf_log("cli", GF_LOG_DEBUG, "Not a status op  %s", arg);
+        goto out;
+    }
 
-        for (i = 0; optable[i].opname; i++) {
-                if (!strcmp (w, optable[i].opname)) {
-                        ret = optable[i].opcode;
-                        break;
-                }
+    for (i = 0; optable[i].opname; i++) {
+        if (!strcmp(w, optable[i].opname)) {
+            ret = optable[i].opcode;
+            break;
         }
+    }
 
- out:
-        return ret;
+out:
+    return ret;
 }
 
 int
-cli_cmd_volume_status_parse (const char **words, int wordcount,
-                             dict_t **options)
+cli_cmd_volume_status_parse(const char **words, int wordcount, dict_t **options)
 {
-        dict_t    *dict            = NULL;
-        int        ret             = -1;
-        uint32_t   cmd             = 0;
+    dict_t *dict = NULL;
+    int ret = -1;
+    uint32_t cmd = 0;
 
-        GF_ASSERT (options);
+    GF_ASSERT(options);
 
-        dict = dict_new ();
-        if (!dict)
-                goto out;
-
-        switch (wordcount) {
+    dict = dict_new();
+    if (!dict)
+        goto out;
 
+    switch (wordcount) {
         case 2:
+            cmd = GF_CLI_STATUS_ALL;
+            ret = 0;
+            break;
+
+        case 3:
+            if (!strcmp(words[2], "all")) {
                 cmd = GF_CLI_STATUS_ALL;
                 ret = 0;
-                break;
 
-        case 3:
-                if (!strcmp (words[2], "all")) {
-                        cmd = GF_CLI_STATUS_ALL;
-                        ret = 0;
+            } else {
+                cmd = GF_CLI_STATUS_VOL;
+                ret = dict_set_str(dict, "volname", (char *)words[2]);
+            }
 
-                } else {
-                        cmd = GF_CLI_STATUS_VOL;
-                        ret = dict_set_str (dict, "volname", (char *)words[2]);
-                }
-
-                break;
+            break;
 
         case 4:
-                cmd = cli_cmd_get_statusop (words[3]);
-
-                if (!strcmp (words[2], "all")) {
-                        if (cmd == GF_CLI_STATUS_NONE) {
-                                cli_err ("%s is not a valid status option",
-                                         words[3]);
-                                ret = -1;
-                                goto out;
-                        }
-                        cmd |= GF_CLI_STATUS_ALL;
-                        ret  = 0;
-
-                } else {
-                        ret = dict_set_str (dict, "volname",
-                                            (char *)words[2]);
-                        if (ret)
-                                goto out;
-
-                        if (cmd == GF_CLI_STATUS_NONE) {
-                                if (!strcmp (words[3], "nfs")) {
-                                        cmd |= GF_CLI_STATUS_NFS;
-                                } else if (!strcmp (words[3], "shd")) {
-                                        cmd |= GF_CLI_STATUS_SHD;
-                                } else if (!strcmp (words[3], "quotad")) {
-                                        cmd |= GF_CLI_STATUS_QUOTAD;
-                                } else if (!strcmp (words[3], "snapd")) {
-                                        cmd |= GF_CLI_STATUS_SNAPD;
-                                } else {
-                                        cmd = GF_CLI_STATUS_BRICK;
-                                        ret = dict_set_str (dict, "brick",
-                                                            (char *)words[3]);
-                                }
-
-                        } else {
-                                cmd |= GF_CLI_STATUS_VOL;
-                                ret  = 0;
-                        }
-                }
-
-                break;
+            cmd = cli_cmd_get_statusop(words[3]);
 
-        case 5:
-                if (!strcmp (words[2], "all")) {
-                        cli_err ("Cannot specify brick/nfs for \"all\"");
-                        ret = -1;
-                        goto out;
-                }
-
-                cmd = cli_cmd_get_statusop (words[4]);
+            if (!strcmp(words[2], "all")) {
                 if (cmd == GF_CLI_STATUS_NONE) {
-                        cli_err ("%s is not a valid status option",
-                                 words[4]);
-                        ret = -1;
-                        goto out;
+                    cli_err("%s is not a valid status option", words[3]);
+                    ret = -1;
+                    goto out;
                 }
+                cmd |= GF_CLI_STATUS_ALL;
+                ret = 0;
 
-
-                ret = dict_set_str (dict, "volname", (char *)words[2]);
+            } else {
+                ret = dict_set_str(dict, "volname", (char *)words[2]);
                 if (ret)
-                        goto out;
+                    goto out;
 
-                if (!strcmp (words[3], "nfs")) {
-                        if (cmd == GF_CLI_STATUS_FD ||
-                            cmd == GF_CLI_STATUS_DETAIL ||
-                            cmd == GF_CLI_STATUS_TASKS) {
-                                cli_err ("Detail/FD/Tasks status not available"
-                                         " for NFS Servers");
-                                ret = -1;
-                                goto out;
-                        }
+                if (cmd == GF_CLI_STATUS_NONE) {
+                    if (!strcmp(words[3], "nfs")) {
                         cmd |= GF_CLI_STATUS_NFS;
-                } else if (!strcmp (words[3], "shd")){
-                        if (cmd == GF_CLI_STATUS_FD ||
-                            cmd == GF_CLI_STATUS_CLIENTS ||
-                            cmd == GF_CLI_STATUS_DETAIL ||
-                            cmd == GF_CLI_STATUS_TASKS) {
-                                cli_err ("Detail/FD/Clients/Tasks status not "
-                                         "available for Self-heal Daemons");
-                                ret = -1;
-                                goto out;
-                        }
+                    } else if (!strcmp(words[3], "shd")) {
                         cmd |= GF_CLI_STATUS_SHD;
-                } else if (!strcmp (words[3], "quotad")) {
-                        if (cmd == GF_CLI_STATUS_FD ||
-                            cmd == GF_CLI_STATUS_CLIENTS ||
-                            cmd == GF_CLI_STATUS_DETAIL ||
-                            cmd == GF_CLI_STATUS_INODE) {
-                                cli_err ("Detail/FD/Clients/Inode status not "
-                                         "available for Quota Daemon");
-                                ret = -1;
-                                goto out;
-                        }
+                    } else if (!strcmp(words[3], "quotad")) {
                         cmd |= GF_CLI_STATUS_QUOTAD;
-                } else if  (!strcmp (words[3], "snapd")) {
-                        if (cmd == GF_CLI_STATUS_FD ||
-                            cmd == GF_CLI_STATUS_CLIENTS ||
-                            cmd == GF_CLI_STATUS_DETAIL ||
-                            cmd == GF_CLI_STATUS_INODE) {
-                                cli_err ("Detail/FD/Clients/Inode status not "
-                                         "available for snap daemon");
-                                ret = -1;
-                                goto out;
-                        }
+                    } else if (!strcmp(words[3], "snapd")) {
                         cmd |= GF_CLI_STATUS_SNAPD;
+                    } else if (!strcmp(words[3], "bitd")) {
+                        cmd |= GF_CLI_STATUS_BITD;
+                    } else if (!strcmp(words[3], "scrub")) {
+                        cmd |= GF_CLI_STATUS_SCRUB;
+                    } else {
+                        cmd = GF_CLI_STATUS_BRICK;
+                        ret = dict_set_str(dict, "brick", (char *)words[3]);
+                    }
+
                 } else {
-                        if (cmd == GF_CLI_STATUS_TASKS) {
-                                cli_err ("Tasks status not available for "
-                                         "bricks");
-                                ret = -1;
-                                goto out;
-                        }
-                        cmd |= GF_CLI_STATUS_BRICK;
-                        ret = dict_set_str (dict, "brick", (char *)words[3]);
+                    cmd |= GF_CLI_STATUS_VOL;
+                    ret = 0;
                 }
-                break;
+            }
 
-        default:
-                goto out;
-        }
+            break;
 
-        if (ret)
+        case 5:
+            if (!strcmp(words[2], "all")) {
+                cli_err("Cannot specify brick/nfs for \"all\"");
+                ret = -1;
                 goto out;
+            }
 
-        ret = dict_set_int32 (dict, "cmd", cmd);
-        if (ret)
+            cmd = cli_cmd_get_statusop(words[4]);
+            if (cmd == GF_CLI_STATUS_NONE) {
+                cli_err("%s is not a valid status option", words[4]);
+                ret = -1;
                 goto out;
+            }
+
+            ret = dict_set_str(dict, "volname", (char *)words[2]);
+            if (ret)
+                goto out;
+
+            if (!strcmp(words[3], "nfs")) {
+                if (cmd == GF_CLI_STATUS_FD || cmd == GF_CLI_STATUS_DETAIL ||
+                    cmd == GF_CLI_STATUS_TASKS) {
+                    cli_err(
+                        "Detail/FD/Tasks status not available"
+                        " for NFS Servers");
+                    ret = -1;
+                    goto out;
+                }
+                cmd |= GF_CLI_STATUS_NFS;
+            } else if (!strcmp(words[3], "shd")) {
+                if (cmd == GF_CLI_STATUS_FD || cmd == GF_CLI_STATUS_CLIENTS ||
+                    cmd == GF_CLI_STATUS_DETAIL || cmd == GF_CLI_STATUS_TASKS) {
+                    cli_err(
+                        "Detail/FD/Clients/Tasks status not "
+                        "available for Self-heal Daemons");
+                    ret = -1;
+                    goto out;
+                }
+                cmd |= GF_CLI_STATUS_SHD;
+            } else if (!strcmp(words[3], "quotad")) {
+                if (cmd == GF_CLI_STATUS_FD || cmd == GF_CLI_STATUS_CLIENTS ||
+                    cmd == GF_CLI_STATUS_DETAIL || cmd == GF_CLI_STATUS_INODE) {
+                    cli_err(
+                        "Detail/FD/Clients/Inode status not "
+                        "available for Quota Daemon");
+                    ret = -1;
+                    goto out;
+                }
+                cmd |= GF_CLI_STATUS_QUOTAD;
+            } else if (!strcmp(words[3], "snapd")) {
+                if (cmd == GF_CLI_STATUS_FD || cmd == GF_CLI_STATUS_CLIENTS ||
+                    cmd == GF_CLI_STATUS_DETAIL || cmd == GF_CLI_STATUS_INODE) {
+                    cli_err(
+                        "Detail/FD/Clients/Inode status not "
+                        "available for snap daemon");
+                    ret = -1;
+                    goto out;
+                }
+                cmd |= GF_CLI_STATUS_SNAPD;
+            } else {
+                if (cmd == GF_CLI_STATUS_TASKS) {
+                    cli_err(
+                        "Tasks status not available for "
+                        "bricks");
+                    ret = -1;
+                    goto out;
+                }
+                cmd |= GF_CLI_STATUS_BRICK;
+                ret = dict_set_str(dict, "brick", (char *)words[3]);
+            }
+            break;
 
-        *options = dict;
+        default:
+            goto out;
+    }
+
+    if (ret)
+        goto out;
+
+    ret = dict_set_int32(dict, "cmd", cmd);
+    if (ret)
+        goto out;
 
- out:
-        if (ret && dict)
-                dict_destroy (dict);
+    *options = dict;
 
-        return ret;
+out:
+    if (ret && dict)
+        dict_unref(dict);
+
+    return ret;
 }
 
 gf_boolean_t
-cli_cmd_validate_dumpoption (const char *arg, char **option)
+cli_cmd_validate_dumpoption(const char *arg, char **option)
 {
-        char    *opwords[] = {"all", "nfs", "mem", "iobuf", "callpool", "priv",
-                              "fd", "inode", "history", "inodectx", "fdctx",
-                              "quotad", NULL};
-        char    *w = NULL;
-
-        w = str_getunamb (arg, opwords);
-        if (!w) {
-                gf_log ("cli", GF_LOG_DEBUG, "Unknown statedump option  %s",
-                        arg);
-                return _gf_false;
-        }
-        *option = w;
-        return _gf_true;
+    static char *opwords[] = {"all",   "nfs",    "mem",   "iobuf",   "callpool",
+                              "priv",  "fd",     "inode", "history", "inodectx",
+                              "fdctx", "quotad", NULL};
+    char *w = NULL;
+
+    w = str_getunamb(arg, opwords);
+    if (!w) {
+        gf_log("cli", GF_LOG_DEBUG, "Unknown statedump option  %s", arg);
+        return _gf_false;
+    }
+    *option = w;
+    return _gf_true;
 }
 
 int
-cli_cmd_volume_statedump_options_parse (const char **words, int wordcount,
-                                        dict_t **options)
+cli_cmd_volume_statedump_options_parse(const char **words, int wordcount,
+                                       dict_t **options)
 {
-        int     ret = 0;
-        int     i = 0;
-        dict_t  *dict = NULL;
-        int     option_cnt = 0;
-        char    *option = NULL;
-        char    option_str[100] = {0,};
-
-        for (i = 3; i < wordcount; i++, option_cnt++) {
-                if (!cli_cmd_validate_dumpoption (words[i], &option)) {
-                        ret = -1;
-                        goto out;
-                }
-                strncat (option_str, option, strlen (option));
-                strncat (option_str, " ", 1);
+    int ret = 0;
+    int i = 0;
+    dict_t *dict = NULL;
+    int option_cnt = 0;
+    char *option = NULL;
+    char *option_str = NULL;
+    char *tmp_str = NULL;
+    char *tmp = NULL;
+    char *ip_addr = NULL;
+    char *pid = NULL;
+
+    if ((wordcount >= 5) && ((strcmp(words[3], "client")) == 0)) {
+        tmp = gf_strdup(words[4]);
+        if (!tmp) {
+            ret = -1;
+            goto out;
+        }
+        ip_addr = strtok(tmp, ":");
+        pid = strtok(NULL, ":");
+        if (valid_internet_address(ip_addr, _gf_true, _gf_false) && pid &&
+            gf_valid_pid(pid, strlen(pid))) {
+            ret = gf_asprintf(&option_str, "%s %s %s", words[3], ip_addr, pid);
+            if (ret < 0) {
+                goto out;
+            }
+            option_cnt = 3;
+        } else {
+            ret = -1;
+            goto out;
         }
-        if((strstr (option_str, "nfs")) && strstr (option_str, "quotad")) {
+    } else {
+        for (i = 3; i < wordcount; i++, option_cnt++) {
+            if (!cli_cmd_validate_dumpoption(words[i], &option)) {
                 ret = -1;
                 goto out;
+            }
+            tmp_str = option_str;
+            option_str = NULL;
+            ret = gf_asprintf(&option_str, "%s%s ", tmp_str ? tmp_str : "",
+                              option);
+            GF_FREE(tmp_str);
+            if (ret < 0) {
+                goto out;
+            }
         }
+        if (option_str && (strstr(option_str, "nfs")) &&
+            strstr(option_str, "quotad")) {
+            ret = -1;
+            goto out;
+        }
+    }
 
-        dict = dict_new ();
-        if (!dict)
-                goto out;
+    dict = dict_new();
+    if (!dict) {
+        ret = -1;
+        goto out;
+    }
 
-        ret = dict_set_dynstr (dict, "options", gf_strdup (option_str));
-        if (ret)
-                goto out;
+    /* dynamic string in dict is freed up when dict is freed up, and hence
+    if option_str is NULL pass in an duplicate empty string to the same */
+    ret = dict_set_dynstr(dict, "options",
+                          (option_str ? option_str : gf_strdup("")));
+    if (ret)
+        goto out;
+    option_str = NULL;
 
-        ret = dict_set_int32 (dict, "option_cnt", option_cnt);
-        if (ret)
-                goto out;
+    ret = dict_set_int32(dict, "option_cnt", option_cnt);
+    if (ret)
+        goto out;
 
-        *options = dict;
+    *options = dict;
 out:
-        if (ret && dict)
-                dict_destroy (dict);
-        if (ret)
-                gf_log ("cli", GF_LOG_ERROR, "Error parsing dumpoptions");
-        return ret;
+    GF_FREE(tmp);
+    GF_FREE(option_str);
+    if (ret && dict)
+        dict_unref(dict);
+    if (ret)
+        gf_log("cli", GF_LOG_ERROR, "Error parsing dumpoptions");
+    return ret;
 }
 
 int
-cli_cmd_volume_clrlks_opts_parse (const char **words, int wordcount,
-                                  dict_t **options)
+cli_cmd_volume_clrlks_opts_parse(const char **words, int wordcount,
+                                 dict_t **options)
 {
-        int     ret = -1;
-        int     i = 0;
-        dict_t  *dict = NULL;
-        char    *kind_opts[4] = {"blocked", "granted", "all", NULL};
-        char    *types[4] = {"inode", "entry", "posix", NULL};
-        char    *free_ptr = NULL;
-
-        dict = dict_new ();
-        if (!dict)
-                goto out;
+    int ret = -1;
+    int i = 0;
+    dict_t *dict = NULL;
+    char *kind_opts[4] = {"blocked", "granted", "all", NULL};
+    char *types[4] = {"inode", "entry", "posix", NULL};
+    char *free_ptr = NULL;
+
+    dict = dict_new();
+    if (!dict)
+        goto out;
 
-        if (strcmp (words[4], "kind"))
-                goto out;
+    if (strcmp(words[4], "kind"))
+        goto out;
 
-        for (i = 0; kind_opts[i]; i++) {
-               if (!strcmp (words[5], kind_opts[i])) {
-                       free_ptr = gf_strdup (words[5]);
-                       ret = dict_set_dynstr (dict, "kind", free_ptr);
-                       if (ret)
-                               goto out;
-                       free_ptr = NULL;
-                       break;
-               }
-        }
-        if (i == 3)
+    for (i = 0; kind_opts[i]; i++) {
+        if (!strcmp(words[5], kind_opts[i])) {
+            free_ptr = gf_strdup(words[5]);
+            ret = dict_set_dynstr(dict, "kind", free_ptr);
+            if (ret)
                 goto out;
+            free_ptr = NULL;
+            break;
+        }
+    }
+    if (i == 3)
+        goto out;
 
-        ret = -1;
-        for (i = 0; types[i]; i++) {
-               if (!strcmp (words[6], types[i])) {
-                       free_ptr = gf_strdup (words[6]);
-                       ret = dict_set_dynstr (dict, "type", free_ptr);
-                       if (ret)
-                               goto out;
-                       free_ptr = NULL;
-                       break;
-               }
-        }
-        if (i == 3)
+    ret = -1;
+    for (i = 0; types[i]; i++) {
+        if (!strcmp(words[6], types[i])) {
+            free_ptr = gf_strdup(words[6]);
+            ret = dict_set_dynstr(dict, "type", free_ptr);
+            if (ret)
                 goto out;
-
-        if (wordcount == 8) {
-                free_ptr = gf_strdup (words[7]);
-                ret = dict_set_dynstr (dict, "opts", free_ptr);
-                if (ret)
-                        goto out;
-                free_ptr = NULL;
+            free_ptr = NULL;
+            break;
         }
+    }
+    if (i == 3)
+        goto out;
 
-        ret = 0;
-        *options = dict;
+    if (wordcount == 8) {
+        free_ptr = gf_strdup(words[7]);
+        ret = dict_set_dynstr(dict, "opts", free_ptr);
+        if (ret)
+            goto out;
+        free_ptr = NULL;
+    }
+
+    ret = 0;
+    *options = dict;
 out:
-       if (ret) {
-               GF_FREE (free_ptr);
-               dict_unref (dict);
-       }
+    if (ret) {
+        GF_FREE(free_ptr);
+        dict_unref(dict);
+    }
 
-       return ret;
+    return ret;
 }
 
 static int
-extract_hostname_path_from_token (const char *tmp_words, char **hostname,
-                                  char **path)
+extract_hostname_path_from_token(const char *tmp_words, char **hostname,
+                                 char **path)
 {
-        int ret = 0;
-        char *delimiter = NULL;
-        char *tmp_host = NULL;
-        char *host_name = NULL;
-        char *words = NULL;
+    int ret = 0;
+    char *delimiter = NULL;
+    char *tmp_host = NULL;
+    char *host_name = NULL;
+    char *words = NULL;
+    int str_len = 0;
+    *hostname = NULL;
+    *path = NULL;
+
+    str_len = strlen(tmp_words) + 1;
+    words = GF_MALLOC(str_len, gf_common_mt_char);
+    if (!words) {
+        ret = -1;
+        goto out;
+    }
 
-        *hostname = NULL;
-        *path = NULL;
+    snprintf(words, str_len, "%s", tmp_words);
 
-        words = GF_CALLOC (1, strlen (tmp_words) + 1, gf_common_mt_char);
-        if (!words){
+    if (validate_brick_name(words)) {
+        cli_err(
+            "Wrong brick type: %s, use <HOSTNAME>:"
+            "<export-dir-abs-path>",
+            words);
+        ret = -1;
+        goto out;
+    } else {
+        delimiter = strrchr(words, ':');
+        ret = gf_canonicalize_path(delimiter + 1);
+        if (ret) {
+            goto out;
+        } else {
+            str_len = strlen(delimiter + 1) + 1;
+            *path = GF_MALLOC(str_len, gf_common_mt_char);
+            if (!*path) {
                 ret = -1;
                 goto out;
+            }
+            snprintf(*path, str_len, "%s", delimiter + 1);
         }
+    }
 
-        strncpy (words, tmp_words, strlen (tmp_words) + 1);
+    tmp_host = gf_strdup(words);
+    if (!tmp_host) {
+        gf_log("cli", GF_LOG_ERROR, "Out of memory");
+        ret = -1;
+        goto out;
+    }
+    get_host_name(tmp_host, &host_name);
+    if (!host_name) {
+        ret = -1;
+        gf_log("cli", GF_LOG_ERROR,
+               "Unable to allocate "
+               "memory");
+        goto out;
+    }
+    if (!(strcmp(host_name, "localhost") && strcmp(host_name, "127.0.0.1") &&
+          strncmp(host_name, "0.", 2))) {
+        cli_err(
+            "Please provide a valid hostname/ip other "
+            "than localhost, 127.0.0.1 or loopback "
+            "address (0.0.0.0 to 0.255.255.255).");
+        ret = -1;
+        goto out;
+    }
+    if (!valid_internet_address(host_name, _gf_false, _gf_false)) {
+        cli_err(
+            "internet address '%s' does not conform to "
+            "standards",
+            host_name);
+        ret = -1;
+        goto out;
+    }
 
-        if (validate_brick_name (words)) {
-                cli_err ("Wrong brick type: %s, use <HOSTNAME>:"
-                        "<export-dir-abs-path>", words);
-                ret = -1;
-                goto out;
-        } else {
-                delimiter = strrchr (words, ':');
-                ret = gf_canonicalize_path (delimiter + 1);
-                if (ret) {
-                        goto out;
-                } else {
-                        *path = GF_CALLOC (1, strlen (delimiter+1) +1,
-                                           gf_common_mt_char);
-                        if (!*path) {
-                           ret = -1;
-                                goto out;
+    str_len = strlen(host_name) + 1;
+    *hostname = GF_MALLOC(str_len, gf_common_mt_char);
+    if (!*hostname) {
+        ret = -1;
+        goto out;
+    }
+    snprintf(*hostname, str_len, "%s", host_name);
+    ret = 0;
 
-                        }
-                        strncpy (*path, delimiter +1,
-                                 strlen(delimiter + 1) + 1);
-                }
-        }
+out:
+    GF_FREE(words);
+    GF_FREE(tmp_host);
+    return ret;
+}
 
-        tmp_host = gf_strdup (words);
-        if (!tmp_host) {
-                gf_log ("cli", GF_LOG_ERROR, "Out of memory");
-                ret = -1;
-                goto out;
-        }
-        get_host_name (tmp_host, &host_name);
-        if (!host_name) {
-                ret = -1;
-                gf_log("cli",GF_LOG_ERROR, "Unable to allocate "
-                        "memory");
-                goto out;
-        }
-        if (!(strcmp (host_name, "localhost") &&
-            strcmp (host_name, "127.0.0.1") &&
-            strncmp (host_name, "0.", 2))) {
-                cli_err ("Please provide a valid hostname/ip other "
-                         "than localhost, 127.0.0.1 or loopback "
-                         "address (0.0.0.0 to 0.255.255.255).");
-                ret = -1;
-                goto out;
-        }
-        if (!valid_internet_address (host_name, _gf_false)) {
-                cli_err ("internet address '%s' does not conform to "
-                          "standards", host_name);
-                ret = -1;
-                goto out;
-        }
+static int
+set_hostname_path_in_dict(const char *token, dict_t *dict, int heal_op)
+{
+    char *hostname = NULL;
+    char *path = NULL;
+    int ret = 0;
 
-        *hostname = GF_CALLOC (1, strlen (host_name) + 1,
-                                       gf_common_mt_char);
-        if (!*hostname) {
-                ret = -1;
-                goto out;
-        }
-        strncpy (*hostname, host_name, strlen (host_name) + 1);
-        ret = 0;
+    ret = extract_hostname_path_from_token(token, &hostname, &path);
+    if (ret)
+        goto out;
+
+    switch (heal_op) {
+        case GF_SHD_OP_SBRAIN_HEAL_FROM_BRICK:
+            ret = dict_set_dynstr(dict, "heal-source-hostname", hostname);
+            if (ret)
+                goto out;
+            hostname = NULL;
+            ret = dict_set_dynstr(dict, "heal-source-brickpath", path);
+            if (ret) {
+                goto out;
+            }
+            path = NULL;
+            break;
+        case GF_SHD_OP_STATISTICS_HEAL_COUNT_PER_REPLICA:
+            ret = dict_set_dynstr(dict, "per-replica-cmd-hostname", hostname);
+            if (ret)
+                goto out;
+            hostname = NULL;
+            ret = dict_set_dynstr(dict, "per-replica-cmd-path", path);
+            if (ret) {
+                goto out;
+            }
+            path = NULL;
+            break;
+        default:
+            ret = -1;
+            break;
+    }
 
 out:
-        GF_FREE (words);
-        GF_FREE (tmp_host);
-        return ret;
+    GF_FREE(hostname);
+    GF_FREE(path);
+    return ret;
 }
 
+static int
+heal_command_type_get(const char *command)
+{
+    int i = 0;
+    /* subcommands are set as NULL */
+    char *heal_cmds[GF_SHD_OP_HEAL_DISABLE + 1] = {
+        [GF_SHD_OP_INVALID] = NULL,
+        [GF_SHD_OP_HEAL_INDEX] = NULL,
+        [GF_SHD_OP_HEAL_FULL] = "full",
+        [GF_SHD_OP_INDEX_SUMMARY] = "info",
+        [GF_SHD_OP_SPLIT_BRAIN_FILES] = NULL,
+        [GF_SHD_OP_STATISTICS] = "statistics",
+        [GF_SHD_OP_STATISTICS_HEAL_COUNT] = NULL,
+        [GF_SHD_OP_STATISTICS_HEAL_COUNT_PER_REPLICA] = NULL,
+        [GF_SHD_OP_SBRAIN_HEAL_FROM_BIGGER_FILE] = NULL,
+        [GF_SHD_OP_SBRAIN_HEAL_FROM_BRICK] = NULL,
+        [GF_SHD_OP_HEAL_ENABLE] = "enable",
+        [GF_SHD_OP_HEAL_DISABLE] = "disable",
+    };
+
+    for (i = 0; i <= GF_SHD_OP_HEAL_DISABLE; i++) {
+        if (heal_cmds[i] && (strcmp(heal_cmds[i], command) == 0))
+            return i;
+    }
+
+    return GF_SHD_OP_INVALID;
+}
 
 int
-cli_cmd_volume_heal_options_parse (const char **words, int wordcount,
-                                   dict_t **options)
+cli_cmd_volume_heal_options_parse(const char **words, int wordcount,
+                                  dict_t **options)
 {
-        int     ret = 0;
-        dict_t  *dict = NULL;
-        char    *hostname = NULL;
-        char    *path = NULL;
+    int ret = 0;
+    dict_t *dict = NULL;
+    gf_xl_afr_op_t op = GF_SHD_OP_INVALID;
 
-        dict = dict_new ();
-        if (!dict)
-                goto out;
+    dict = dict_new();
+    if (!dict) {
+        gf_log(THIS->name, GF_LOG_ERROR, "Failed to create the dict");
+        ret = -1;
+        goto out;
+    }
 
-        ret = dict_set_str (dict, "volname", (char *) words[2]);
-        if (ret) {
-                gf_log (THIS->name, GF_LOG_ERROR, "failed to set volname");
-                goto out;
+    ret = dict_set_str(dict, "volname", (char *)words[2]);
+    if (ret) {
+        gf_log(THIS->name, GF_LOG_ERROR, "failed to set volname");
+        goto out;
+    }
+
+    if (wordcount == 3) {
+        ret = dict_set_int32(dict, "heal-op", GF_SHD_OP_HEAL_INDEX);
+        goto done;
+    }
+
+    if (wordcount == 4) {
+        op = heal_command_type_get(words[3]);
+        if (op == GF_SHD_OP_INVALID) {
+            ret = -1;
+            goto out;
+        }
+
+        ret = dict_set_int32(dict, "heal-op", op);
+        goto done;
+    }
+
+    if (wordcount == 5) {
+        if (strcmp(words[3], "info") && strcmp(words[3], "statistics") &&
+            strcmp(words[3], "granular-entry-heal")) {
+            ret = -1;
+            goto out;
         }
 
-        if (wordcount == 3) {
-                ret = dict_set_int32 (dict, "heal-op", GF_AFR_OP_HEAL_INDEX);
+        if (!strcmp(words[3], "info")) {
+            if (!strcmp(words[4], "split-brain")) {
+                ret = dict_set_int32(dict, "heal-op",
+                                     GF_SHD_OP_SPLIT_BRAIN_FILES);
                 goto done;
+            }
+            if (!strcmp(words[4], "summary")) {
+                ret = dict_set_int32(dict, "heal-op", GF_SHD_OP_HEAL_SUMMARY);
+                goto done;
+            }
         }
 
-        if (wordcount == 4) {
-                if (!strcmp (words[3], "full")) {
-                        ret = dict_set_int32 (dict, "heal-op",
-                                              GF_AFR_OP_HEAL_FULL);
-                        goto done;
-                } else if (!strcmp (words[3], "statistics")) {
-                        ret = dict_set_int32 (dict, "heal-op",
-                                              GF_AFR_OP_STATISTICS);
-                        goto done;
-
-                } else if (!strcmp (words[3], "info")) {
-                        ret = dict_set_int32 (dict, "heal-op",
-                                              GF_AFR_OP_INDEX_SUMMARY);
-                        goto done;
-                } else {
-                        ret = -1;
-                        goto out;
-                }
+        if (!strcmp(words[3], "statistics")) {
+            if (!strcmp(words[4], "heal-count")) {
+                ret = dict_set_int32(dict, "heal-op",
+                                     GF_SHD_OP_STATISTICS_HEAL_COUNT);
+                goto done;
+            }
         }
-        if (wordcount == 5) {
-                if (strcmp (words[3], "info") &&
-                    strcmp (words[3], "statistics")) {
-                        ret = -1;
-                        goto out;
-                }
 
-                if (!strcmp (words[3], "info")) {
-                        if (!strcmp (words[4], "healed")) {
-                                ret = dict_set_int32 (dict, "heal-op",
-                                                      GF_AFR_OP_HEALED_FILES);
-                                goto done;
-                        }
-                        if (!strcmp (words[4], "heal-failed")) {
-                                ret = dict_set_int32 (dict, "heal-op",
-                                                   GF_AFR_OP_HEAL_FAILED_FILES);
-                                goto done;
-                        }
-                        if (!strcmp (words[4], "split-brain")) {
-                                ret = dict_set_int32 (dict, "heal-op",
-                                                   GF_AFR_OP_SPLIT_BRAIN_FILES);
-                                goto done;
-                        }
-                }
+        if (!strcmp(words[3], "granular-entry-heal")) {
+            if (!strcmp(words[4], "enable")) {
+                ret = dict_set_int32(dict, "heal-op",
+                                     GF_SHD_OP_GRANULAR_ENTRY_HEAL_ENABLE);
+                goto done;
+            } else if (!strcmp(words[4], "disable")) {
+                ret = dict_set_int32(dict, "heal-op",
+                                     GF_SHD_OP_GRANULAR_ENTRY_HEAL_DISABLE);
+                goto done;
+            }
+        }
 
-                if (!strcmp (words[3], "statistics")) {
-                        if (!strcmp (words[4], "heal-count")) {
-                                ret = dict_set_int32 (dict, "heal-op",
-                                               GF_AFR_OP_STATISTICS_HEAL_COUNT);
-                                goto done;
-                        }
-                }
-                ret = -1;
+        ret = -1;
+        goto out;
+    }
+    if (wordcount == 6) {
+        if (strcmp(words[3], "split-brain")) {
+            ret = -1;
+            goto out;
+        }
+        if (!strcmp(words[4], "bigger-file")) {
+            ret = dict_set_int32(dict, "heal-op",
+                                 GF_SHD_OP_SBRAIN_HEAL_FROM_BIGGER_FILE);
+            if (ret)
+                goto out;
+            ret = dict_set_str(dict, "file", (char *)words[5]);
+            if (ret)
                 goto out;
+            goto done;
         }
-        if (wordcount == 7) {
-                if (!strcmp (words[3], "statistics")
-                    && !strcmp (words[4], "heal-count")
-                    && !strcmp (words[5], "replica")) {
-
-                        ret = dict_set_int32 (dict, "heal-op",
-                                   GF_AFR_OP_STATISTICS_HEAL_COUNT_PER_REPLICA);
-                        if (ret)
-                                goto out;
-                        ret = extract_hostname_path_from_token (words[6],
-                                                              &hostname, &path);
-                        if (ret)
-                                goto out;
-                        ret = dict_set_dynstr (dict, "per-replica-cmd-hostname",
-                                               hostname);
-                        if (ret)
-                                goto out;
-                        ret = dict_set_dynstr (dict, "per-replica-cmd-path",
-                                               path);
-                        if (ret)
-                                goto out;
-                        else
-                                goto done;
-
-                }
+        if (!strcmp(words[4], "latest-mtime")) {
+            ret = dict_set_int32(dict, "heal-op",
+                                 GF_SHD_OP_SBRAIN_HEAL_FROM_LATEST_MTIME);
+            if (ret)
+                goto out;
+            ret = dict_set_str(dict, "file", (char *)words[5]);
+            if (ret)
+                goto out;
+            goto done;
+        }
+        if (!strcmp(words[4], "source-brick")) {
+            ret = dict_set_int32(dict, "heal-op",
+                                 GF_SHD_OP_SBRAIN_HEAL_FROM_BRICK);
+            if (ret)
+                goto out;
+            ret = set_hostname_path_in_dict(words[5], dict,
+                                            GF_SHD_OP_SBRAIN_HEAL_FROM_BRICK);
+            if (ret)
+                goto out;
+            goto done;
         }
         ret = -1;
         goto out;
+    }
+    if (wordcount == 7) {
+        if (!strcmp(words[3], "statistics") &&
+            !strcmp(words[4], "heal-count") && !strcmp(words[5], "replica")) {
+            ret = dict_set_int32(dict, "heal-op",
+                                 GF_SHD_OP_STATISTICS_HEAL_COUNT_PER_REPLICA);
+            if (ret)
+                goto out;
+            ret = set_hostname_path_in_dict(
+                words[6], dict, GF_SHD_OP_STATISTICS_HEAL_COUNT_PER_REPLICA);
+            if (ret)
+                goto out;
+            goto done;
+        }
+        if (!strcmp(words[3], "split-brain") &&
+            !strcmp(words[4], "source-brick")) {
+            ret = dict_set_int32(dict, "heal-op",
+                                 GF_SHD_OP_SBRAIN_HEAL_FROM_BRICK);
+            ret = set_hostname_path_in_dict(words[5], dict,
+                                            GF_SHD_OP_SBRAIN_HEAL_FROM_BRICK);
+            if (ret)
+                goto out;
+            ret = dict_set_str(dict, "file", (char *)words[6]);
+            if (ret)
+                goto out;
+            goto done;
+        }
+    }
+    ret = -1;
+    goto out;
 done:
-        *options = dict;
+    *options = dict;
 out:
-        if (ret && dict) {
-                dict_unref (dict);
-                *options = NULL;
-        }
+    if (ret && dict) {
+        dict_unref(dict);
+        *options = NULL;
+    }
 
-        return ret;
+    return ret;
 }
 
 int
-cli_cmd_volume_defrag_parse (const char **words, int wordcount,
-                             dict_t **options)
+cli_cmd_volume_defrag_parse(const char **words, int wordcount, dict_t **options)
 {
-        dict_t                 *dict = NULL;
-        int                      ret = -1;
-        char                *option  = NULL;
-        char                *volname = NULL;
-        char                *command = NULL;
-        gf_cli_defrag_type       cmd = 0;
-
-        GF_ASSERT (words);
-        GF_ASSERT (options);
-
-        dict = dict_new ();
-        if (!dict)
-                goto out;
-
-        if (!((wordcount == 4) || (wordcount == 5)))
-                goto out;
-
-        if (wordcount == 4) {
-                if (strcmp (words[3], "start") && strcmp (words[3], "stop") &&
-                    strcmp (words[3], "status"))
-                            goto out;
-        } else {
-                if (strcmp (words[3], "fix-layout") &&
-                    strcmp (words[3], "start"))
-                        goto out;
-        }
-
-        volname = (char *) words[2];
-
-        if (wordcount == 4) {
-                command = (char *) words[3];
-        }
-        if (wordcount == 5) {
-               if ((strcmp (words[3], "fix-layout") ||
-                    strcmp (words[4], "start")) &&
-                    (strcmp (words[3], "start") ||
-                    strcmp (words[4], "force"))) {
-                        ret = -1;
-                        goto out;
-                }
-                command = (char *) words[3];
-                option = (char *) words[4];
-        }
+    dict_t *dict = NULL;
+    int ret = -1;
+    char *option = NULL;
+    char *volname = NULL;
+    char *command = NULL;
+    gf_cli_defrag_type cmd = 0;
+
+    GF_ASSERT(words);
+    GF_ASSERT(options);
+
+    dict = dict_new();
+    if (!dict)
+        goto out;
 
-        if (strcmp (command, "start") == 0) {
-                cmd = GF_DEFRAG_CMD_START;
-                if (option && strcmp (option, "force") == 0) {
-                                cmd = GF_DEFRAG_CMD_START_FORCE;
-                        }
-                goto done;
-        }
+    if (!((wordcount == 4) || (wordcount == 5)))
+        goto out;
 
-        if (strcmp (command, "fix-layout") == 0) {
-                cmd = GF_DEFRAG_CMD_START_LAYOUT_FIX;
-                goto done;
-        }
-        if (strcmp (command, "stop") == 0) {
-                cmd = GF_DEFRAG_CMD_STOP;
-                goto done;
-        }
-        if (strcmp (command, "status") == 0) {
-                cmd = GF_DEFRAG_CMD_STATUS;
-        }
+    if (wordcount == 4) {
+        if (strcmp(words[3], "start") && strcmp(words[3], "stop") &&
+            strcmp(words[3], "status"))
+            goto out;
+    } else {
+        if (strcmp(words[3], "fix-layout") && strcmp(words[3], "start"))
+            goto out;
+    }
+
+    volname = (char *)words[2];
+
+    if (wordcount == 4) {
+        command = (char *)words[3];
+    }
+    if (wordcount == 5) {
+        if ((strcmp(words[3], "fix-layout") || strcmp(words[4], "start")) &&
+            (strcmp(words[3], "start") || strcmp(words[4], "force"))) {
+            ret = -1;
+            goto out;
+        }
+        command = (char *)words[3];
+        option = (char *)words[4];
+    }
+
+    if (strcmp(command, "start") == 0) {
+        cmd = GF_DEFRAG_CMD_START;
+        if (option && strcmp(option, "force") == 0) {
+            cmd = GF_DEFRAG_CMD_START_FORCE;
+        }
+        goto done;
+    }
+
+    if (strcmp(command, "fix-layout") == 0) {
+        cmd = GF_DEFRAG_CMD_START_LAYOUT_FIX;
+        goto done;
+    }
+    if (strcmp(command, "stop") == 0) {
+        cmd = GF_DEFRAG_CMD_STOP;
+        goto done;
+    }
+    if (strcmp(command, "status") == 0) {
+        cmd = GF_DEFRAG_CMD_STATUS;
+    }
 
 done:
-        ret = dict_set_str (dict, "volname", volname);
+    ret = dict_set_str(dict, "volname", volname);
 
-        if (ret) {
-                gf_log (THIS->name, GF_LOG_ERROR, "failed to set dict");
-                goto out;
-        }
+    if (ret) {
+        gf_log(THIS->name, GF_LOG_ERROR, "failed to set dict");
+        goto out;
+    }
 
-        ret = dict_set_int32 (dict, "rebalance-command", (int32_t) cmd);
+    ret = dict_set_int32(dict, "rebalance-command", (int32_t)cmd);
 
-        if (ret) {
-                gf_log (THIS->name, GF_LOG_ERROR, "failed to set dict");
-                goto out;
-        }
+    if (ret) {
+        gf_log(THIS->name, GF_LOG_ERROR, "failed to set dict");
+        goto out;
+    }
 
-        *options = dict;
+    *options = dict;
 
 out:
-        if (ret && dict)
-                dict_destroy (dict);
+    if (ret && dict)
+        dict_unref(dict);
 
-        return ret;
+    return ret;
 }
 
 int32_t
-cli_snap_create_desc_parse (dict_t *dict, const char **words,
-                            size_t wordcount, int32_t desc_opt_loc)
+cli_snap_create_desc_parse(dict_t *dict, const char **words, size_t wordcount,
+                           int32_t desc_opt_loc)
 {
-        int32_t        ret      = -1;
-        char          *desc     = NULL;
-        int32_t        desc_len = 0;
+    int32_t ret = -1;
+    char *desc = NULL;
+    int32_t desc_len = 0;
+    int len;
 
-        desc = GF_CALLOC (MAX_SNAP_DESCRIPTION_LEN + 1, sizeof(char),
-                          gf_common_mt_char);
-        if (!desc) {
-                ret = -1;
-                goto out;
-        }
+    desc = GF_MALLOC(MAX_SNAP_DESCRIPTION_LEN + 1, gf_common_mt_char);
+    if (!desc) {
+        ret = -1;
+        goto out;
+    }
+
+    len = strlen(words[desc_opt_loc]);
+    if (len >= MAX_SNAP_DESCRIPTION_LEN) {
+        cli_out(
+            "snapshot create: description truncated: "
+            "Description provided is longer than 1024 characters");
+        desc_len = MAX_SNAP_DESCRIPTION_LEN;
+    } else {
+        desc_len = len;
+    }
+
+    snprintf(desc, desc_len + 1, "%s", words[desc_opt_loc]);
+    /* Calculating the size of the description as given by the user */
+
+    ret = dict_set_dynstr(dict, "description", desc);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Unable to save snap "
+               "description");
+        goto out;
+    }
 
+    ret = 0;
+out:
+    if (ret && desc)
+        GF_FREE(desc);
 
-        if (strlen (words[desc_opt_loc]) >= MAX_SNAP_DESCRIPTION_LEN) {
-                cli_out ("snapshot create: description truncated: "
-                         "Description provided is longer than 1024 characters");
-                desc_len = MAX_SNAP_DESCRIPTION_LEN;
-        } else {
-                desc_len = strlen (words[desc_opt_loc]);
-        }
+    return ret;
+}
 
-        strncpy (desc, words[desc_opt_loc], desc_len);
-        desc[desc_len] = '\0';
-        /* Calculating the size of the description as given by the user */
+/* Function to check whether the Volume name is repeated */
+int
+cli_check_if_volname_repeated(const char **words, unsigned int start_index,
+                              uint64_t cur_index)
+{
+    uint64_t i = -1;
+    int ret = 0;
 
-        ret = dict_set_dynstr (dict, "description", desc);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to save snap "
-                        "description");
-                goto out;
-        }
+    GF_ASSERT(words);
 
-        ret = 0;
+    for (i = start_index; i < cur_index; i++) {
+        if (strcmp(words[i], words[cur_index]) == 0) {
+            ret = -1;
+            goto out;
+        }
+    }
 out:
-        if (ret && desc)
-                GF_FREE (desc);
-
-        return ret;
+    return ret;
 }
 
-/* Function to check whether the Volume name is repeated */
+/* snapshot clone <clonename> <snapname>
+ * @arg-0, dict     : Request Dictionary to be sent to server side.
+ * @arg-1, words    : Contains individual words of CLI command.
+ * @arg-2, wordcount: Contains number of words present in the CLI command.
+ *
+ * return value : -1 on failure
+ *                 0 on success
+ */
 int
-cli_check_if_volname_repeated (const char **words, unsigned int start_index,
-                           uint64_t cur_index) {
-        uint64_t        i       =       -1;
-        int             ret     =        0;
+cli_snap_clone_parse(dict_t *dict, const char **words, int wordcount)
+{
+    uint64_t i = 0;
+    int ret = -1;
+    char *clonename = NULL;
+    unsigned int cmdi = 2;
+    /* cmdi is command index, here cmdi is "2" (gluster snapshot clone)*/
+
+    GF_ASSERT(words);
+    GF_ASSERT(dict);
+
+    if (wordcount == cmdi + 1) {
+        cli_err("Invalid Syntax.");
+        gf_log("cli", GF_LOG_ERROR,
+               "Invalid number of  words for snap clone command");
+        goto out;
+    }
 
-        GF_ASSERT (words);
+    if (strlen(words[cmdi]) >= GLUSTERD_MAX_SNAP_NAME) {
+        cli_err(
+            "snapshot clone: failed: clonename cannot exceed "
+            "255 characters.");
+        gf_log("cli", GF_LOG_ERROR, "Clone name too long");
+
+        goto out;
+    }
+
+    clonename = (char *)words[cmdi];
+    for (i = 0; i < strlen(clonename); i++) {
+        /* Following volume name convention */
+        if (!isalnum(clonename[i]) &&
+            (clonename[i] != '_' && (clonename[i] != '-'))) {
+            /* TODO : Is this message enough?? */
+            cli_err(
+                "Clonename can contain only alphanumeric, "
+                "\"-\" and \"_\" characters");
+            goto out;
+        }
+    }
+
+    ret = dict_set_int32(dict, "volcount", 1);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not save volcount");
+        goto out;
+    }
+
+    ret = dict_set_str(dict, "clonename", (char *)words[cmdi]);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Could not save clone "
+               "name(%s)",
+               (char *)words[cmdi]);
+        goto out;
+    }
+
+    /* Filling snap name in the dictionary */
+    ret = dict_set_str(dict, "snapname", (char *)words[cmdi + 1]);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Could not "
+               "save snap name(%s)",
+               (char *)words[cmdi + 1]);
+        goto out;
+    }
+
+    ret = 0;
 
-        for (i = start_index ; i < cur_index ; i++) {
-                if (strcmp (words[i], words[cur_index]) == 0) {
-                        ret = -1;
-                        goto out;
-                }
-        }
 out:
-        return ret;
+    return ret;
 }
 
 /* snapshot create <snapname> <vol-name(s)> [description <description>]
@@ -3211,159 +4324,181 @@ out:
  *                 0 on success
  */
 int
-cli_snap_create_parse (dict_t *dict, const char **words, int wordcount) {
-        uint64_t        i               =       0;
-        int             ret             =       -1;
-        uint64_t        volcount        =       0;
-        char            key[PATH_MAX]   =       "";
-        char            *snapname       =       NULL;
-        unsigned int    cmdi            =       2;
-        int             flags           =       0;
-        /* cmdi is command index, here cmdi is "2" (gluster snapshot create)*/
-
-        GF_ASSERT (words);
-        GF_ASSERT (dict);
-
-        if (wordcount <= cmdi + 1) {
-                cli_err ("Invalid Syntax.");
-                gf_log ("cli", GF_LOG_ERROR,
-                        "Too less words for snap create command");
-                goto out;
-        }
+cli_snap_create_parse(dict_t *dict, const char **words, int wordcount)
+{
+    uint64_t i = 0;
+    int ret = -1;
+    uint64_t volcount = 0;
+    char key[PATH_MAX] = "";
+    char *snapname = NULL;
+    unsigned int cmdi = 2;
+    int flags = 0;
+    /* cmdi is command index, here cmdi is "2" (gluster snapshot create)*/
+
+    GF_ASSERT(words);
+    GF_ASSERT(dict);
+
+    if (wordcount <= cmdi + 1) {
+        cli_err("Invalid Syntax.");
+        gf_log("cli", GF_LOG_ERROR, "Too less words for snap create command");
+        goto out;
+    }
 
-        if (strlen(words[cmdi]) >= GLUSTERD_MAX_SNAP_NAME) {
-                cli_err ("snapshot create: failed: snapname cannot exceed "
-                         "255 characters.");
-                gf_log ("cli", GF_LOG_ERROR, "Snapname too long");
+    if (strlen(words[cmdi]) >= GLUSTERD_MAX_SNAP_NAME) {
+        cli_err(
+            "snapshot create: failed: snapname cannot exceed "
+            "255 characters.");
+        gf_log("cli", GF_LOG_ERROR, "Snapname too long");
 
-                goto out;
+        goto out;
+    }
+
+    snapname = (char *)words[cmdi];
+    for (i = 0; i < strlen(snapname); i++) {
+        /* Following volume name convention */
+        if (!isalnum(snapname[i]) &&
+            (snapname[i] != '_' && (snapname[i] != '-'))) {
+            /* TODO : Is this message enough?? */
+            cli_err(
+                "Snapname can contain only alphanumeric, "
+                "\"-\" and \"_\" characters");
+            goto out;
+        }
+    }
+
+    ret = dict_set_str(dict, "snapname", (char *)words[cmdi]);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Could not save snap "
+               "name(%s)",
+               (char *)words[cmdi]);
+        goto out;
+    }
+
+    /* Filling volume name in the dictionary */
+    for (i = cmdi + 1;
+         i < wordcount && (strcmp(words[i], "description")) != 0 &&
+         (strcmp(words[i], "force") != 0) &&
+         (strcmp(words[i], "no-timestamp") != 0);
+         i++) {
+        volcount++;
+        /* volume index starts from 1 */
+        ret = snprintf(key, sizeof(key), "volname%" PRIu64, volcount);
+        if (ret < 0) {
+            goto out;
         }
 
-        snapname = (char *) words[cmdi];
-        for (i = 0 ; i < strlen (snapname); i++) {
-                /* Following volume name convention */
-                if (!isalnum (snapname[i]) && (snapname[i] != '_'
-                                           && (snapname[i] != '-'))) {
-                        /* TODO : Is this message enough?? */
-                        cli_err ("Snapname can contain only alphanumeric, "
-                                 "\"-\" and \"_\" characters");
-                        goto out;
-                }
+        ret = dict_set_str(dict, key, (char *)words[i]);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Could not "
+                   "save volume name(%s)",
+                   (char *)words[i]);
+            goto out;
         }
 
-        ret = dict_set_str (dict, "snapname", (char *)words[cmdi]);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not save snap "
-                        "name(%s)", (char *)words[cmdi]);
-                goto out;
+        if (i >= cmdi + 2) {
+            ret = -1;
+            cli_err(
+                "Creating multiple volume snapshot is not "
+                "supported as of now");
+            goto out;
         }
+        /* TODO : remove this above condition check once
+         * multiple volume snapshot is supported */
+    }
 
-        /* Filling volume name in the dictionary */
-        for (i = cmdi + 1 ; i < wordcount
-                            && (strcmp (words[i], "description")) != 0
-                            && (strcmp (words[i], "force") != 0); i++) {
-                volcount++;
-                /* volume index starts from 1 */
-                ret = snprintf (key, sizeof (key), "volname%"PRIu64, volcount);
-                if (ret < 0) {
-                        goto out;
-                }
+    if (volcount == 0) {
+        ret = -1;
+        cli_err("Please provide the volume name");
+        gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+        goto out;
+    }
 
-                ret = dict_set_str (dict, key, (char *)words[i]);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Could not "
-                                "save volume name(%s)", (char *)words[i]);
-                        goto out;
-                }
+    ret = dict_set_int32(dict, "volcount", volcount);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not save volcount");
+        goto out;
+    }
+
+    /* Verify how we got out of "for" loop,
+     * if it is by reaching wordcount limit then goto "out",
+     * because we need not parse for "description","force" and
+     * "no-timestamp" after this.
+     */
+    if (i == wordcount) {
+        goto out;
+    }
 
-                if (i >= cmdi + 2) {
-                        ret = -1;
-                        cli_err("Creating multiple volume snapshot is not "
-                                "supported as of now");
-                        goto out;
-                }
-                /* TODO : remove this above condition check once
-                 * multiple volume snapshot is supported */
+    if (strcmp(words[i], "no-timestamp") == 0) {
+        ret = dict_set_int32n(dict, "no-timestamp", SLEN("no-timestamp"), 1);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Could not save "
+                   "time-stamp option");
         }
+        if (i == (wordcount - 1))
+            goto out;
+        i++;
+    }
 
-        if (volcount == 0) {
-                ret = -1;
-                cli_err ("Please provide the volume name");
-                gf_log ("cli", GF_LOG_ERROR, "Invalid Syntax");
-                goto out;
+    if ((strcmp(words[i], "description")) == 0) {
+        ++i;
+        if (i > (wordcount - 1)) {
+            ret = -1;
+            cli_err("Please provide a description");
+            gf_log("cli", GF_LOG_ERROR, "Description not provided");
+            goto out;
         }
 
-        ret = dict_set_int32 (dict, "volcount", volcount);
+        ret = cli_snap_create_desc_parse(dict, words, wordcount, i);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not save volcount");
-                goto out;
-        }
-
-        /* Verify how we got out of "for" loop,
-         * if it is by reaching wordcount limit then goto "out",
-         * because we need not parse for "description" and "force"
-         * after this.
+            gf_log("cli", GF_LOG_ERROR,
+                   "Could not save snap "
+                   "description");
+            goto out;
+        }
+
+        if (i == (wordcount - 1))
+            goto out;
+        i++;
+        /* point the index to next word.
+         * As description might be follwed by force option.
+         * Before that, check if wordcount limit is reached
          */
-        if (i == wordcount) {
-                goto out;
-        }
-
-        if ((strcmp (words[i], "description")) == 0) {
-                ++i;
-                if (i > (wordcount - 1)) {
-                        ret = -1;
-                        cli_err ("Please provide a description");
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Description not provided");
-                        goto out;
-                }
+    }
 
-                ret = cli_snap_create_desc_parse(dict, words, wordcount, i);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Could not save snap "
-                                "description");
-                        goto out;
-                }
-
-                if (i == (wordcount - 1))
-                        goto out;
-                i++;
-                /* point the index to next word.
-                 * As description might be follwed by force option.
-                 * Before that, check if wordcount limit is reached
-                 */
-        }
+    if (strcmp(words[i], "force") == 0) {
+        flags = GF_CLI_FLAG_OP_FORCE;
 
-        if (strcmp (words[i], "force") == 0) {
-                flags = GF_CLI_FLAG_OP_FORCE;
-
-        } else {
-                ret = -1;
-                cli_err ("Invalid Syntax.");
-                gf_log ("cli", GF_LOG_ERROR, "Invalid Syntax");
-                goto out;
-        }
+    } else {
+        ret = -1;
+        cli_err("Invalid Syntax.");
+        gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+        goto out;
+    }
 
-        /* Check if the command has anything after "force" keyword */
-        if (++i < wordcount) {
-                ret = -1;
-                gf_log ("cli", GF_LOG_ERROR, "Invalid Syntax");
-                goto out;
-        }
+    /* Check if the command has anything after "force" keyword */
+    if (++i < wordcount) {
+        ret = -1;
+        gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+        goto out;
+    }
 
-        ret = 0;
+    ret = 0;
 
 out:
-        if(ret == 0) {
-                /*Adding force flag in either of the case i.e force set
-                 * or unset*/
-                ret = dict_set_int32 (dict, "flags", flags);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Could not save "
-                                "snap force option");
-                }
+    if (ret == 0) {
+        /*Adding force flag in either of the case i.e force set
+         * or unset*/
+        ret = dict_set_int32(dict, "flags", flags);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Could not save "
+                   "snap force option");
         }
-        return ret;
+    }
+    return ret;
 }
 
 /* snapshot list [volname]
@@ -3375,30 +4510,30 @@ out:
  *                 0 on success
  */
 int
-cli_snap_list_parse (dict_t *dict, const char **words, int wordcount) {
-        int             ret     =       -1;
+cli_snap_list_parse(dict_t *dict, const char **words, int wordcount)
+{
+    int ret = -1;
 
-        GF_ASSERT (words);
-        GF_ASSERT (dict);
+    GF_ASSERT(words);
+    GF_ASSERT(dict);
 
-        if (wordcount < 2 || wordcount > 3) {
-                gf_log ("cli", GF_LOG_ERROR, "Invalid Syntax");
-                goto out;
-        }
+    if (wordcount < 2 || wordcount > 3) {
+        gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+        goto out;
+    }
 
-        if (wordcount == 2) {
-                ret = 0;
-                goto out;
-        }
+    if (wordcount == 2) {
+        ret = 0;
+        goto out;
+    }
 
-        ret = dict_set_str (dict, "volname", (char *)words[2]);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR,
-                        "Failed to save volname in dictionary");
-                goto out;
-        }
+    ret = dict_set_str(dict, "volname", (char *)words[2]);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to save volname in dictionary");
+        goto out;
+    }
 out:
-        return ret;
+    return ret;
 }
 
 /* snapshot info [(snapname |  volume <volname>)]
@@ -3410,87 +4545,88 @@ out:
  *                 0 on success
  */
 int
-cli_snap_info_parse (dict_t *dict, const char **words, int wordcount)
+cli_snap_info_parse(dict_t *dict, const char **words, int wordcount)
 {
+    int ret = -1;
+    int32_t cmd = GF_SNAP_INFO_TYPE_ALL;
+    unsigned int cmdi = 2;
+    /* cmdi is command index, here cmdi is "2" (gluster snapshot info)*/
 
-        int             ret             =       -1;
-        int32_t         cmd             =       GF_SNAP_INFO_TYPE_ALL;
-        unsigned int    cmdi            =        2;
-        /* cmdi is command index, here cmdi is "2" (gluster snapshot info)*/
+    GF_ASSERT(words);
+    GF_ASSERT(dict);
 
-        GF_ASSERT (words);
-        GF_ASSERT (dict);
+    if (wordcount > 4 || wordcount < cmdi) {
+        gf_log("cli", GF_LOG_ERROR, "Invalid syntax");
+        goto out;
+    }
 
-        if (wordcount > 4 || wordcount < cmdi) {
-                gf_log ("cli", GF_LOG_ERROR, "Invalid syntax");
-                goto out;
-        }
+    if (wordcount == cmdi) {
+        ret = 0;
+        goto out;
+    }
 
-        if (wordcount == cmdi) {
-                ret = 0;
-                goto out;
+    /* If 3rd word is not "volume", then it must
+     * be snapname.
+     */
+    if (strcmp(words[cmdi], "volume") != 0) {
+        ret = dict_set_str(dict, "snapname", (char *)words[cmdi]);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Unable to save "
+                   "snapname %s",
+                   words[cmdi]);
+            goto out;
         }
 
-        /* If 3rd word is not "volume", then it must
-         * be snapname.
+        /* Once snap name is parsed, if we encounter any other
+         * word then fail it. Invalid Syntax.
+         * example : snapshot info <snapname> word
          */
-        if (strcmp (words[cmdi], "volume") != 0) {
-                ret = dict_set_str (dict, "snapname",
-                                   (char *)words[cmdi]);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Unable to save "
-                                "snapname %s", words[cmdi]);
-                        goto out;
-                }
-
-                /* Once snap name is parsed, if we encounter any other
-                 * word then fail it. Invalid Syntax.
-                 * example : snapshot info <snapname> word
-                 */
-                if ((cmdi + 1) != wordcount) {
-                        ret = -1;
-                        gf_log ("cli", GF_LOG_ERROR, "Invalid Syntax");
-                        goto out;
-                }
-
-                cmd = GF_SNAP_INFO_TYPE_SNAP;
-                ret = 0;
-                goto out;
-                /* No need to continue the parsing once we
-                 * get the snapname
-                 */
+        if ((cmdi + 1) != wordcount) {
+            ret = -1;
+            gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+            goto out;
         }
 
-        /* If 3rd word is "volume", then check if next word
-         * is present. As, "snapshot info volume" is an
-         * invalid command.
+        cmd = GF_SNAP_INFO_TYPE_SNAP;
+        ret = 0;
+        goto out;
+        /* No need to continue the parsing once we
+         * get the snapname
          */
-        if ((cmdi + 1) == wordcount) {
-                ret = -1;
-                gf_log ("cli", GF_LOG_ERROR, "Invalid Syntax");
-                goto out;
-        }
+    }
 
-        ret = dict_set_str (dict, "volname", (char *)words[wordcount - 1]);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not save "
-                        "volume name %s", words[wordcount - 1]);
-                goto out;
-        }
-        cmd = GF_SNAP_INFO_TYPE_VOL;
+    /* If 3rd word is "volume", then check if next word
+     * is present. As, "snapshot info volume" is an
+     * invalid command.
+     */
+    if ((cmdi + 1) == wordcount) {
+        ret = -1;
+        gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+        goto out;
+    }
+
+    ret = dict_set_str(dict, "volname", (char *)words[wordcount - 1]);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Could not save "
+               "volume name %s",
+               words[wordcount - 1]);
+        goto out;
+    }
+    cmd = GF_SNAP_INFO_TYPE_VOL;
 out:
-        if (ret == 0) {
-                ret = dict_set_int32 (dict, "cmd", cmd);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Could not save "
-                                "type of snapshot info");
-                }
+    if (ret == 0) {
+        ret = dict_set_int32(dict, "sub-cmd", cmd);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Could not save "
+                   "type of snapshot info");
         }
-        return ret;
+    }
+    return ret;
 }
 
-
-
 /* snapshot restore <snapname>
  * @arg-0, dict     : Request Dictionary to be sent to server side.
  * @arg-1, words    : Contains individual words of CLI command.
@@ -3500,42 +4636,43 @@ out:
  *                 0 on success
  */
 int
-cli_snap_restore_parse (dict_t *dict, const char **words, int wordcount,
-                        struct cli_state *state)
+cli_snap_restore_parse(dict_t *dict, const char **words, int wordcount,
+                       struct cli_state *state)
 {
+    int ret = -1;
+    const char *question = NULL;
+    gf_answer_t answer = GF_ANSWER_NO;
 
-        int             ret             =       -1;
-        const char      *question       =       NULL;
-        gf_answer_t     answer          =       GF_ANSWER_NO;
-
-        GF_ASSERT (words);
-        GF_ASSERT (dict);
-
-        if (wordcount != 3) {
-                gf_log ("cli", GF_LOG_ERROR, "Invalid Syntax");
-                goto out;
-        }
+    GF_ASSERT(words);
+    GF_ASSERT(dict);
 
-        ret = dict_set_str (dict, "snapname", (char *)words[2]);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to save snap-name %s",
-                        words[2]);
-                goto out;
-        }
-
-        question = "Restore operation will replace the "
-                   "original volume with the snapshotted volume. "
-                   "Do you still want to continue?";
+    if (wordcount != 3) {
+        gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+        goto out;
+    }
 
-        answer = cli_cmd_get_confirmation (state, question);
-        if (GF_ANSWER_NO == answer) {
-                ret = 1;
-                gf_log ("cli", GF_LOG_ERROR, "User cancelled a snapshot "
-                        "restore operation for snap %s", (char *)words[2]);
-                goto out;
-        }
+    ret = dict_set_str(dict, "snapname", (char *)words[2]);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to save snap-name %s", words[2]);
+        goto out;
+    }
+
+    question =
+        "Restore operation will replace the "
+        "original volume with the snapshotted volume. "
+        "Do you still want to continue?";
+
+    answer = cli_cmd_get_confirmation(state, question);
+    if (GF_ANSWER_NO == answer) {
+        ret = 1;
+        gf_log("cli", GF_LOG_ERROR,
+               "User cancelled a snapshot "
+               "restore operation for snap %s",
+               (char *)words[2]);
+        goto out;
+    }
 out:
-        return ret;
+    return ret;
 }
 
 /* snapshot activate <snapname> [force]
@@ -3547,43 +4684,41 @@ out:
  *                 0 on success
  */
 int
-cli_snap_activate_parse (dict_t *dict, const char **words, int wordcount)
+cli_snap_activate_parse(dict_t *dict, const char **words, int wordcount)
 {
+    int ret = -1;
+    int flags = 0;
 
-        int ret = -1;
-        int flags = 0;
+    GF_ASSERT(words);
+    GF_ASSERT(dict);
 
-        GF_ASSERT (words);
-        GF_ASSERT (dict);
-
-        if ((wordcount < 3) || (wordcount > 4)) {
-                gf_log ("cli", GF_LOG_ERROR, "Invalid Syntax");
-                goto out;
-        }
+    if ((wordcount < 3) || (wordcount > 4)) {
+        gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+        goto out;
+    }
 
-        ret = dict_set_str (dict, "snapname", (char *)words[2]);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to save snap-name %s",
-                        words[2]);
-                goto out;
-        }
+    ret = dict_set_str(dict, "snapname", (char *)words[2]);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to save snap-name %s", words[2]);
+        goto out;
+    }
 
-        if (wordcount == 4) {
-                if (!strcmp("force", (char *)words[3])) {
-                        flags = GF_CLI_FLAG_OP_FORCE;
-                } else {
-                        gf_log ("cli", GF_LOG_ERROR, "Invalid option");
-                        ret = -1;
-                        goto out;
-                }
-        }
-        ret = dict_set_int32 (dict, "flags", flags);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to save force option");
-                goto out;
-        }
+    if (wordcount == 4) {
+        if (!strcmp("force", (char *)words[3])) {
+            flags = GF_CLI_FLAG_OP_FORCE;
+        } else {
+            gf_log("cli", GF_LOG_ERROR, "Invalid option");
+            ret = -1;
+            goto out;
+        }
+    }
+    ret = dict_set_int32(dict, "flags", flags);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to save force option");
+        goto out;
+    }
 out:
-        return ret;
+    return ret;
 }
 
 /* snapshot deactivate <snapname>
@@ -3596,42 +4731,41 @@ out:
  *                 1 if user cancelled the request
  */
 int
-cli_snap_deactivate_parse (dict_t *dict, const char **words, int wordcount,
-                        struct cli_state *state)
+cli_snap_deactivate_parse(dict_t *dict, const char **words, int wordcount,
+                          struct cli_state *state)
 {
+    int ret = -1;
+    gf_answer_t answer = GF_ANSWER_NO;
+    const char *question =
+        "Deactivating snap will make its "
+        "data inaccessible. Do you want to "
+        "continue?";
+
+    GF_ASSERT(words);
+    GF_ASSERT(dict);
+
+    if ((wordcount != 3)) {
+        gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+        goto out;
+    }
 
-        int             ret             = -1;
-        gf_answer_t     answer          = GF_ANSWER_NO;
-        const char     *question        = "Deactivating snap will make its "
-                                          "data inaccessible. Do you want to "
-                                          "continue?";
-
-
-        GF_ASSERT (words);
-        GF_ASSERT (dict);
-
-        if ((wordcount != 3)) {
-                gf_log ("cli", GF_LOG_ERROR, "Invalid Syntax");
-                goto out;
-        }
-
-        ret = dict_set_str (dict, "snapname", (char *)words[2]);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to save snap-name %s",
-                        words[2]);
-                goto out;
-        }
-
-        answer = cli_cmd_get_confirmation (state, question);
-        if (GF_ANSWER_NO == answer) {
-                ret = 1;
-                gf_log ("cli", GF_LOG_DEBUG, "User cancelled "
-                        "snapshot deactivate operation");
-                goto out;
-        }
+    ret = dict_set_str(dict, "snapname", (char *)words[2]);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to save snap-name %s", words[2]);
+        goto out;
+    }
+
+    answer = cli_cmd_get_confirmation(state, question);
+    if (GF_ANSWER_NO == answer) {
+        ret = 1;
+        gf_log("cli", GF_LOG_DEBUG,
+               "User cancelled "
+               "snapshot deactivate operation");
+        goto out;
+    }
 
 out:
-        return ret;
+    return ret;
 }
 
 /* snapshot delete (all | snapname | volume <volname>)
@@ -3644,78 +4778,84 @@ out:
  *                 1 if user cancel the operation
  */
 int
-cli_snap_delete_parse (dict_t *dict, const char **words, int wordcount,
-                       struct cli_state *state) {
-
-        int             ret             =       -1;
-        const char      *question       =       NULL;
-        int32_t         cmd             =       -1;
-        unsigned int    cmdi            =       2;
-        gf_answer_t     answer          =       GF_ANSWER_NO;
-
-        GF_ASSERT (words);
-        GF_ASSERT (dict);
+cli_snap_delete_parse(dict_t *dict, const char **words, int wordcount,
+                      struct cli_state *state)
+{
+    int ret = -1;
+    const char *question = NULL;
+    int32_t cmd = -1;
+    unsigned int cmdi = 2;
+    gf_answer_t answer = GF_ANSWER_NO;
 
-        if (wordcount > 4 || wordcount <= cmdi) {
-                gf_log ("cli", GF_LOG_ERROR, "Invalid Syntax");
-                goto out;
-        }
+    GF_ASSERT(words);
+    GF_ASSERT(dict);
 
-        question = "Deleting snap will erase all the information about "
-                   "the snap. Do you still want to continue?";
+    if (wordcount > 4 || wordcount <= cmdi) {
+        gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+        goto out;
+    }
 
-        if (strcmp (words [cmdi], "all") == 0) {
-                ret = 0;
-                cmd = GF_SNAP_DELETE_TYPE_ALL;
-        } else if (strcmp (words [cmdi], "volume") == 0) {
-                if (++cmdi == wordcount) {
-                        ret = -1;
-                        gf_log ("cli", GF_LOG_ERROR, "Invalid Syntax");
-                        goto out;
-                }
+    question =
+        "Deleting snap will erase all the information about "
+        "the snap. Do you still want to continue?";
 
-                ret = dict_set_str (dict, "volname",
-                                    (char *)words[cmdi]);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Could not save "
-                                "volume name %s", words[wordcount - 1]);
-                        goto out;
-                }
-                cmd = GF_SNAP_DELETE_TYPE_VOL;
-        } else {
-                ret = dict_set_str (dict, "snapname", (char *)words[cmdi]);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Unable to save "
-                                "snapname %s", words[2]);
-                        goto out;
-                }
-                cmd = GF_SNAP_DELETE_TYPE_SNAP;
+    if (strcmp(words[cmdi], "all") == 0) {
+        ret = 0;
+        cmd = GF_SNAP_DELETE_TYPE_ALL;
+    } else if (strcmp(words[cmdi], "volume") == 0) {
+        if (++cmdi == wordcount) {
+            ret = -1;
+            gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+            goto out;
         }
 
-        if ((cmdi + 1) != wordcount) {
-                ret = -1;
-                gf_log ("cli", GF_LOG_ERROR, "Invalid Syntax");
-                goto out;
+        ret = dict_set_str(dict, "volname", (char *)words[cmdi]);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Could not save "
+                   "volume name %s",
+                   words[wordcount - 1]);
+            goto out;
+        }
+        cmd = GF_SNAP_DELETE_TYPE_VOL;
+    } else {
+        ret = dict_set_str(dict, "snapname", (char *)words[cmdi]);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Unable to save "
+                   "snapname %s",
+                   words[2]);
+            goto out;
         }
+        cmd = GF_SNAP_DELETE_TYPE_SNAP;
+    }
 
-        if (cmd == GF_SNAP_DELETE_TYPE_SNAP) {
-                answer = cli_cmd_get_confirmation (state, question);
-                if (GF_ANSWER_NO == answer) {
-                        ret = 1;
-                        gf_log ("cli", GF_LOG_DEBUG, "User cancelled "
-                                "snapshot delete operation for snap %s",
-                                (char *)words[2]);
-                        goto out;
-                }
-        }
+    if ((cmdi + 1) != wordcount) {
+        ret = -1;
+        gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+        goto out;
+    }
 
-        ret = dict_set_int32 (dict, "delete-cmd", cmd);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not save "
-                        "type of snapshot delete");
-        }
+    if (cmd == GF_SNAP_DELETE_TYPE_SNAP) {
+        answer = cli_cmd_get_confirmation(state, question);
+        if (GF_ANSWER_NO == answer) {
+            ret = 1;
+            gf_log("cli", GF_LOG_DEBUG,
+                   "User cancelled "
+                   "snapshot delete operation for snap %s",
+                   (char *)words[2]);
+            goto out;
+        }
+    }
+
+    ret = dict_set_int32(dict, "sub-cmd", cmd);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Could not save "
+               "type of snapshot delete");
+    }
 out:
-        return ret;
+    return ret;
 }
 
 /* snapshot status [(snapname | volume <volname>)]
@@ -3727,121 +4867,138 @@ out:
  *                 0 on success
  */
 int
-cli_snap_status_parse (dict_t *dict, const char **words, int wordcount)
+cli_snap_status_parse(dict_t *dict, const char **words, int wordcount)
 {
+    int ret = -1;
+    int32_t cmd = GF_SNAP_STATUS_TYPE_ALL;
+    unsigned int cmdi = 2;
+    /* cmdi is command index, here cmdi is "2" (gluster snapshot status)*/
 
-        int             ret  =        -1;
-        int32_t         cmd  =       GF_SNAP_STATUS_TYPE_ALL;
-        unsigned int    cmdi =        2;
-        /* cmdi is command index, here cmdi is "2" (gluster snapshot status)*/
+    GF_ASSERT(words);
+    GF_ASSERT(dict);
 
-        GF_ASSERT (words);
-        GF_ASSERT (dict);
+    if (wordcount > 4 || wordcount < cmdi) {
+        gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+        goto out;
+    }
 
-        if (wordcount > 4 || wordcount < cmdi) {
-                gf_log ("cli", GF_LOG_ERROR, "Invalid Syntax");
-                goto out;
-        }
+    if (wordcount == cmdi) {
+        ret = 0;
+        goto out;
+    }
 
-        if (wordcount == cmdi) {
-                ret = 0;
-                goto out;
+    /* if 3rd word is not "volume", then it must be "snapname"
+     */
+    if (strcmp(words[cmdi], "volume") != 0) {
+        ret = dict_set_str(dict, "snapname", (char *)words[cmdi]);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Count not save "
+                   "snap name %s",
+                   words[cmdi]);
+            goto out;
         }
 
-        /* if 3rd word is not "volume", then it must be "snapname"
-        */
-        if (strcmp (words[cmdi], "volume") != 0) {
-                ret = dict_set_str (dict, "snapname",
-                                   (char *)words[cmdi]);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Count not save "
-                                "snap name %s", words[cmdi]);
-                        goto out;
-                }
-
-                if ((cmdi + 1) != wordcount) {
-                        ret = -1;
-                        gf_log ("cli", GF_LOG_ERROR, "Invalid Syntax");
-                        goto out;
-                }
-
-                ret = 0;
-                cmd = GF_SNAP_STATUS_TYPE_SNAP;
-                goto out;
+        if ((cmdi + 1) != wordcount) {
+            ret = -1;
+            gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+            goto out;
         }
 
-        /* If 3rd word is "volume", then check if next word is present.
-         * As, "snapshot info volume" is an invalid command
-         */
-        if ((cmdi + 1) == wordcount) {
-                ret = -1;
-                gf_log ("cli", GF_LOG_ERROR, "Invalid Syntax");
-                goto out;
-        }
+        ret = 0;
+        cmd = GF_SNAP_STATUS_TYPE_SNAP;
+        goto out;
+    }
 
-        ret = dict_set_str (dict, "volname", (char *)words [wordcount - 1]);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Count not save "
-                        "volume name %s", words[wordcount - 1]);
-                goto out;
-        }
-        cmd = GF_SNAP_STATUS_TYPE_VOL;
+    /* If 3rd word is "volume", then check if next word is present.
+     * As, "snapshot info volume" is an invalid command
+     */
+    if ((cmdi + 1) == wordcount) {
+        ret = -1;
+        gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+        goto out;
+    }
+
+    ret = dict_set_str(dict, "volname", (char *)words[wordcount - 1]);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Count not save "
+               "volume name %s",
+               words[wordcount - 1]);
+        goto out;
+    }
+    cmd = GF_SNAP_STATUS_TYPE_VOL;
 
 out:
-        if (ret == 0) {
-                ret = dict_set_int32 (dict, "status-cmd", cmd);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Could not save cmd "
-                                "of snapshot status");
-                }
+    if (ret == 0) {
+        ret = dict_set_int32(dict, "sub-cmd", cmd);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Could not save cmd "
+                   "of snapshot status");
         }
+    }
 
-        return ret;
+    return ret;
 }
 
-
 /* return value:
  *      -1  in case of failure.
  *       0  in case of success.
  */
 int32_t
-cli_snap_config_limit_parse (const char **words, dict_t *dict,
-                             unsigned int wordcount, unsigned int index,
-                             char *key)
+cli_snap_config_limit_parse(const char **words, dict_t *dict,
+                            unsigned int wordcount, unsigned int index,
+                            char *key)
 {
-        int             ret             = -1;
-        int             limit           = 0;
-        char            *end_ptr        = NULL;
+    int ret = -1;
+    int limit = 0;
+    char *end_ptr = NULL;
 
-        GF_ASSERT (words);
-        GF_ASSERT (dict);
-        GF_ASSERT (key);
+    GF_ASSERT(words);
+    GF_ASSERT(dict);
+    GF_ASSERT(key);
 
-        if (index >= wordcount) {
-                ret = -1;
-                cli_err ("Please provide a value for %s.", key);
-                gf_log ("cli", GF_LOG_ERROR, "Value not provided for %s", key);
-                goto out;
-        }
+    if (index >= wordcount) {
+        ret = -1;
+        cli_err("Please provide a value for %s.", key);
+        gf_log("cli", GF_LOG_ERROR, "Value not provided for %s", key);
+        goto out;
+    }
 
-        limit = strtol (words[index], &end_ptr, 10);
+    limit = strtol(words[index], &end_ptr, 10);
 
-        if (limit <= 0 || strcmp (end_ptr, "") != 0) {
-                ret = -1;
-                cli_err("Please enter an integer value "
-                        "greater than zero for %s", key);
-                goto out;
-        }
+    if (limit <= 0 || strcmp(end_ptr, "") != 0) {
+        ret = -1;
+        cli_err(
+            "Please enter an integer value "
+            "greater than zero for %s",
+            key);
+        goto out;
+    }
+
+    ret = dict_set_int32(dict, key, limit);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Could not set "
+               "%s in dictionary",
+               key);
+        goto out;
+    }
 
-        ret = dict_set_int32 (dict, key, limit);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not set "
-                        "%s in dictionary", key);
-                goto out;
-        }
+    ret = dict_set_dynstr_with_alloc(dict, "globalname", "All");
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not set global key");
+        goto out;
+    }
+    ret = dict_set_int32(dict, "hold_global_locks", _gf_true);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not set global locks");
+        goto out;
+    }
 
 out:
-        return ret;
+    return ret;
 }
 
 /* function cli_snap_config_parse
@@ -3857,466 +5014,933 @@ out:
   NOTE : snap-max-soft-limit can only be set for system.
 */
 int32_t
-cli_snap_config_parse (const char **words, int wordcount, dict_t *dict,
-                       struct cli_state *state)
+cli_snap_config_parse(const char **words, int wordcount, dict_t *dict,
+                      struct cli_state *state)
 {
-        int                            ret                 = -1;
-        gf_answer_t                    answer              = GF_ANSWER_NO;
-        gf_boolean_t                   vol_presence        = _gf_false;
-        struct snap_config_opt_vals_  *conf_vals           = NULL;
-        int8_t                         hard_limit          = 0;
-        int8_t                         soft_limit          = 0;
-        int8_t                         config_type         = -1;
-        const char                    *question            = NULL;
-        unsigned int                   cmdi                = 2;
-        /* cmdi is command index, here cmdi is "2" (gluster snapshot config)*/
-
-        GF_ASSERT (words);
-        GF_ASSERT (dict);
-        GF_ASSERT (state);
-
-        if ((wordcount < 2) || (wordcount > 7)) {
-                gf_log ("cli", GF_LOG_ERROR,
-                        "Invalid wordcount(%d)", wordcount);
-                goto out;
-        }
+    int ret = -1;
+    gf_answer_t answer = GF_ANSWER_NO;
+    gf_boolean_t vol_presence = _gf_false;
+    struct snap_config_opt_vals_ *conf_vals = NULL;
+    int8_t hard_limit = 0;
+    int8_t soft_limit = 0;
+    int8_t config_type = -1;
+    const char *question = NULL;
+    unsigned int cmdi = 2;
+    /* cmdi is command index, here cmdi is "2" (gluster snapshot config)*/
+
+    GF_ASSERT(words);
+    GF_ASSERT(dict);
+    GF_ASSERT(state);
+
+    if ((wordcount < 2) || (wordcount > 7)) {
+        gf_log("cli", GF_LOG_ERROR, "Invalid wordcount(%d)", wordcount);
+        goto out;
+    }
 
-        if (wordcount == 2) {
-                config_type = GF_SNAP_CONFIG_DISPLAY;
-                ret = 0;
-                goto set;
+    if (wordcount == 2) {
+        config_type = GF_SNAP_CONFIG_DISPLAY;
+        ret = 0;
+        goto set;
+    }
+
+    /* auto-delete cannot be a volume name */
+    /* Check whether the 3rd word is volname */
+    if (strcmp(words[cmdi], "snap-max-hard-limit") != 0 &&
+        strcmp(words[cmdi], "snap-max-soft-limit") != 0 &&
+        strcmp(words[cmdi], "auto-delete") != 0 &&
+        strcmp(words[cmdi], "activate-on-create") != 0) {
+        ret = dict_set_str(dict, "volname", (char *)words[cmdi]);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, "Failed to set volname");
+            goto out;
         }
+        cmdi++;
+        vol_presence = _gf_true;
 
-        /* auto-delete cannot be a volume name */
-        /* Check whether the 3rd word is volname */
-        if (strcmp (words[cmdi], "snap-max-hard-limit") != 0
-             && strcmp (words[cmdi], "snap-max-soft-limit") != 0
-             && strcmp (words[cmdi], "auto-delete") != 0
-             && strcmp (words[cmdi], "activate-on-create") != 0) {
-                ret = dict_set_str (dict, "volname", (char *)words[cmdi]);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to set volname");
-                        goto out;
-                }
-                cmdi++;
-                vol_presence = _gf_true;
-
-                if (cmdi == wordcount) {
-                        config_type = GF_SNAP_CONFIG_DISPLAY;
-                        ret = 0;
-                        goto set;
-                }
+        if (cmdi == wordcount) {
+            config_type = GF_SNAP_CONFIG_DISPLAY;
+            ret = 0;
+            goto set;
         }
+    }
 
-        config_type = GF_SNAP_CONFIG_TYPE_SET;
-
-        if (strcmp (words[cmdi], "snap-max-hard-limit") == 0) {
-                ret = cli_snap_config_limit_parse (words, dict, wordcount,
-                                                ++cmdi, "snap-max-hard-limit");
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to parse snap "
-                                "config hard limit");
-                        goto out;
-                }
-                hard_limit = 1;
+    config_type = GF_SNAP_CONFIG_TYPE_SET;
 
-                if (++cmdi == wordcount) {
-                        ret = 0;
-                        goto set;
-                }
+    if (strcmp(words[cmdi], "snap-max-hard-limit") == 0) {
+        ret = cli_snap_config_limit_parse(words, dict, wordcount, ++cmdi,
+                                          "snap-max-hard-limit");
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to parse snap "
+                   "config hard limit");
+            goto out;
         }
+        hard_limit = 1;
 
-        if (strcmp (words[cmdi], "snap-max-soft-limit") == 0) {
-                if (vol_presence == 1) {
-                        ret = -1;
-                        cli_err ("Soft limit cannot be set to individual "
-                                  "volumes.");
-                        gf_log ("cli", GF_LOG_ERROR, "Soft limit cannot be "
-                                "set to volumes");
-                        goto out;
-                }
-
-                ret = cli_snap_config_limit_parse (words, dict, wordcount,
-                                                ++cmdi, "snap-max-soft-limit");
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to parse snap "
-                                "config soft limit");
-                        goto out;
-                }
-
-                if (++cmdi != wordcount) {
-                        ret = -1;
-                        gf_log ("cli", GF_LOG_ERROR, "Invalid Syntax");
-                        goto out;
-                }
-                soft_limit = 1;
+        if (++cmdi == wordcount) {
+            ret = 0;
+            goto set;
         }
+    }
 
-        if (hard_limit || soft_limit)
-                goto set;
+    if (strcmp(words[cmdi], "snap-max-soft-limit") == 0) {
+        if (vol_presence == 1) {
+            ret = -1;
+            cli_err(
+                "Soft limit cannot be set to individual "
+                "volumes.");
+            gf_log("cli", GF_LOG_ERROR,
+                   "Soft limit cannot be "
+                   "set to volumes");
+            goto out;
+        }
 
-        if (strcmp(words[cmdi], "auto-delete") == 0) {
-                if (vol_presence == 1) {
-                        ret = -1;
-                        cli_err ("As of now, auto-delete option cannot be set "
-                                "to volumes");
-                        gf_log ("cli", GF_LOG_ERROR, "auto-delete option "
-                                "cannot be set to volumes");
-                        goto out;
-                }
+        ret = cli_snap_config_limit_parse(words, dict, wordcount, ++cmdi,
+                                          "snap-max-soft-limit");
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to parse snap "
+                   "config soft limit");
+            goto out;
+        }
 
-                if (++cmdi >= wordcount) {
-                        ret = -1;
-                        gf_log ("cli", GF_LOG_ERROR, "Invalid Syntax");
-                        goto out;
-                }
+        if (++cmdi != wordcount) {
+            ret = -1;
+            gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+            goto out;
+        }
+        soft_limit = 1;
+    }
 
-                ret = dict_set_str (dict, "auto-delete", (char *)words[cmdi]);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to set "
-                                "value of auto-delete in request "
-                                "dictionary");
-                        goto out;
-                }
+    if (hard_limit || soft_limit)
+        goto set;
 
-                if (++cmdi != wordcount) {
-                        ret = -1;
-                        gf_log ("cli", GF_LOG_ERROR, "Invalid Syntax");
-                        goto out;
-                }
-        } else if (strcmp(words[cmdi], "activate-on-create") == 0) {
-                if (vol_presence == 1) {
-                        ret = -1;
-                        cli_err ("As of now, activate-on-create option "
-                                 "cannot be set to volumes");
-                        gf_log ("cli", GF_LOG_ERROR, "activate-on-create "
-                                "option cannot be set to volumes");
-                        goto out;
-                }
+    if (strcmp(words[cmdi], "auto-delete") == 0) {
+        if (vol_presence == 1) {
+            ret = -1;
+            cli_err(
+                "As of now, auto-delete option cannot be set "
+                "to volumes");
+            gf_log("cli", GF_LOG_ERROR,
+                   "auto-delete option "
+                   "cannot be set to volumes");
+            goto out;
+        }
 
-                if (++cmdi >= wordcount) {
-                        ret = -1;
-                        gf_log ("cli", GF_LOG_ERROR, "Invalid Syntax");
-                        goto out;
-                }
+        if (++cmdi >= wordcount) {
+            ret = -1;
+            gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+            goto out;
+        }
 
-                ret = dict_set_str (dict, "snap-activate-on-create",
-                                    (char *)words[cmdi]);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to set value "
-                                "of activate-on-create in request dictionary");
-                        goto out;
-                }
+        ret = dict_set_str(dict, "auto-delete", (char *)words[cmdi]);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to set "
+                   "value of auto-delete in request "
+                   "dictionary");
+            goto out;
+        }
+
+        if (++cmdi != wordcount) {
+            ret = -1;
+            gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+            goto out;
+        }
+    } else if (strcmp(words[cmdi], "activate-on-create") == 0) {
+        if (vol_presence == 1) {
+            ret = -1;
+            cli_err(
+                "As of now, activate-on-create option "
+                "cannot be set to volumes");
+            gf_log("cli", GF_LOG_ERROR,
+                   "activate-on-create "
+                   "option cannot be set to volumes");
+            goto out;
+        }
+
+        if (++cmdi >= wordcount) {
+            ret = -1;
+            gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+            goto out;
+        }
+
+        ret = dict_set_str(dict, "snap-activate-on-create",
+                           (char *)words[cmdi]);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to set value "
+                   "of activate-on-create in request dictionary");
+            goto out;
+        }
 
-                if (++cmdi != wordcount) {
-                        ret = -1;
-                        gf_log ("cli", GF_LOG_ERROR, "Invalid Syntax");
-                        goto out;
-                }
-        } else {
-                ret = -1;
-                gf_log ("cli", GF_LOG_ERROR, "Invalid Syntax");
-                goto out;
+        if (++cmdi != wordcount) {
+            ret = -1;
+            gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+            goto out;
         }
+    } else {
+        ret = -1;
+        gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+        goto out;
+    }
 
-        ret = 0; /* Success */
+    ret = 0; /* Success */
 
 set:
-        ret = dict_set_int32 (dict, "config-command", config_type);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to set "
-                        "config-command");
-                goto out;
-        }
+    ret = dict_set_int32(dict, "config-command", config_type);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Unable to set "
+               "config-command");
+        goto out;
+    }
 
-        if (config_type == GF_SNAP_CONFIG_TYPE_SET &&
-           (hard_limit || soft_limit)) {
-                conf_vals = snap_confopt_vals;
-                if (hard_limit && soft_limit) {
-                        question = conf_vals[GF_SNAP_CONFIG_SET_BOTH].question;
-                } else if (soft_limit) {
-                        question = conf_vals[GF_SNAP_CONFIG_SET_SOFT].question;
-                } else if (hard_limit) {
-                        question = conf_vals[GF_SNAP_CONFIG_SET_HARD].question;
-                }
+    if (config_type == GF_SNAP_CONFIG_TYPE_SET && (hard_limit || soft_limit)) {
+        conf_vals = snap_confopt_vals;
+        if (hard_limit && soft_limit) {
+            question = conf_vals[GF_SNAP_CONFIG_SET_BOTH].question;
+        } else if (soft_limit) {
+            question = conf_vals[GF_SNAP_CONFIG_SET_SOFT].question;
+        } else if (hard_limit) {
+            question = conf_vals[GF_SNAP_CONFIG_SET_HARD].question;
+        }
 
-                answer = cli_cmd_get_confirmation (state, question);
-                if (GF_ANSWER_NO == answer) {
-                        ret = 1;
-                        gf_log ("cli", GF_LOG_DEBUG, "User cancelled "
-                        "snapshot config operation");
-                }
+        answer = cli_cmd_get_confirmation(state, question);
+        if (GF_ANSWER_NO == answer) {
+            ret = 1;
+            gf_log("cli", GF_LOG_DEBUG,
+                   "User cancelled "
+                   "snapshot config operation");
         }
+    }
 
 out:
-        return ret;
+    return ret;
 }
 
 int
-validate_snapname (const char *snapname, char **opwords) {
-        int     ret     =       -1;
-        int     i       =       0;
+validate_op_name(const char *op, const char *opname, char **opwords)
+{
+    int ret = -1;
+    int i = 0;
 
-        GF_ASSERT (snapname);
-        GF_ASSERT (opwords);
+    GF_ASSERT(opname);
+    GF_ASSERT(opwords);
 
-        for (i = 0 ; opwords[i] != NULL; i++) {
-                if (strcmp (opwords[i], snapname) == 0) {
-                        cli_out ("\"%s\" cannot be a snapname", snapname);
-                        goto out;
-                }
+    for (i = 0; opwords[i] != NULL; i++) {
+        if (strcmp(opwords[i], opname) == 0) {
+            cli_out("\"%s\" cannot be a %s", opname, op);
+            goto out;
         }
-        ret = 0;
+    }
+    ret = 0;
 out:
-        return ret;
+    return ret;
 }
 
 int32_t
-cli_cmd_snapshot_parse (const char **words, int wordcount, dict_t **options,
-                        struct cli_state *state)
+cli_cmd_snapshot_parse(const char **words, int wordcount, dict_t **options,
+                       struct cli_state *state)
 {
-        int32_t            ret        = -1;
-        dict_t             *dict      = NULL;
-        gf1_cli_snapshot   type       = GF_SNAP_OPTION_TYPE_NONE;
-        char               *w         = NULL;
-        char               *opwords[] = {"create", "delete", "restore",
-                                        "activate", "deactivate", "list",
-                                        "status", "config", "info", NULL};
-        char               *invalid_snapnames[] = {"description", "force",
-                                                  "volume", "all", NULL};
-
-        GF_ASSERT (words);
-        GF_ASSERT (options);
-        GF_ASSERT (state);
-
-        dict = dict_new ();
-        if (!dict)
-                goto out;
+    int32_t ret = -1;
+    dict_t *dict = NULL;
+    gf1_cli_snapshot type = GF_SNAP_OPTION_TYPE_NONE;
+    char *w = NULL;
+    static char *opwords[] = {"create",     "delete", "restore", "activate",
+                              "deactivate", "list",   "status",  "config",
+                              "info",       "clone",  NULL};
+    static char *invalid_snapnames[] = {"description", "force", "volume", "all",
+                                        NULL};
+    static char *invalid_volnames[] = {"volume",
+                                       "type",
+                                       "subvolumes",
+                                       "option",
+                                       "end-volume",
+                                       "all",
+                                       "volume_not_in_ring",
+                                       "description",
+                                       "force",
+                                       "snap-max-hard-limit",
+                                       "snap-max-soft-limit",
+                                       "auto-delete",
+                                       "activate-on-create",
+                                       NULL};
 
-        /* Lowest wordcount possible */
-        if (wordcount < 2) {
-                gf_log ("", GF_LOG_ERROR,
-                        "Invalid command: Not enough arguments");
-                goto out;
-        }
+    GF_ASSERT(words);
+    GF_ASSERT(options);
+    GF_ASSERT(state);
 
-        w = str_getunamb (words[1], opwords);
-        if (!w) {
-                /* Checks if the operation is a valid operation */
-                gf_log ("", GF_LOG_ERROR, "Opword Mismatch");
-                goto out;
-        }
+    dict = dict_new();
+    if (!dict)
+        goto out;
 
-        if (!strcmp (w, "create")) {
-                type = GF_SNAP_OPTION_TYPE_CREATE;
-        } else if (!strcmp (w, "list")) {
-                type = GF_SNAP_OPTION_TYPE_LIST;
-        } else if (!strcmp (w, "info")) {
-                type = GF_SNAP_OPTION_TYPE_INFO;
-        } else if (!strcmp (w, "delete")) {
-                type = GF_SNAP_OPTION_TYPE_DELETE;
-        } else if (!strcmp (w, "config")) {
-                type = GF_SNAP_OPTION_TYPE_CONFIG;
-        } else if (!strcmp (w, "restore")) {
-                type = GF_SNAP_OPTION_TYPE_RESTORE;
-        } else if (!strcmp (w, "status")) {
-                type = GF_SNAP_OPTION_TYPE_STATUS;
-        } else if (!strcmp (w, "activate")) {
-                type = GF_SNAP_OPTION_TYPE_ACTIVATE;
-        } else if (!strcmp (w, "deactivate")) {
-                type = GF_SNAP_OPTION_TYPE_DEACTIVATE;
-        }
-
-        if (type != GF_SNAP_OPTION_TYPE_CONFIG &&
-            type != GF_SNAP_OPTION_TYPE_STATUS) {
-                ret = dict_set_int32 (dict, "hold_snap_locks", _gf_true);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Unable to set hold-snap-locks value "
-                                "as _gf_true");
-                        goto out;
-                }
-        }
+    /* Lowest wordcount possible */
+    if (wordcount < 2) {
+        gf_log("", GF_LOG_ERROR, "Invalid command: Not enough arguments");
+        goto out;
+    }
 
-        /* Following commands does not require volume locks  */
-        if (type == GF_SNAP_OPTION_TYPE_STATUS ||
-            type == GF_SNAP_OPTION_TYPE_ACTIVATE ||
-            type == GF_SNAP_OPTION_TYPE_DEACTIVATE) {
-                ret = dict_set_int32 (dict, "hold_vol_locks", _gf_false);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Setting volume lock "
-                                "flag failed");
-                        goto out;
-                }
+    w = str_getunamb(words[1], opwords);
+    if (!w) {
+        /* Checks if the operation is a valid operation */
+        gf_log("", GF_LOG_ERROR, "Opword Mismatch");
+        goto out;
+    }
+
+    if (!strcmp(w, "create")) {
+        type = GF_SNAP_OPTION_TYPE_CREATE;
+    } else if (!strcmp(w, "list")) {
+        type = GF_SNAP_OPTION_TYPE_LIST;
+    } else if (!strcmp(w, "info")) {
+        type = GF_SNAP_OPTION_TYPE_INFO;
+    } else if (!strcmp(w, "delete")) {
+        type = GF_SNAP_OPTION_TYPE_DELETE;
+    } else if (!strcmp(w, "config")) {
+        type = GF_SNAP_OPTION_TYPE_CONFIG;
+    } else if (!strcmp(w, "restore")) {
+        type = GF_SNAP_OPTION_TYPE_RESTORE;
+    } else if (!strcmp(w, "status")) {
+        type = GF_SNAP_OPTION_TYPE_STATUS;
+    } else if (!strcmp(w, "activate")) {
+        type = GF_SNAP_OPTION_TYPE_ACTIVATE;
+    } else if (!strcmp(w, "deactivate")) {
+        type = GF_SNAP_OPTION_TYPE_DEACTIVATE;
+    } else if (!strcmp(w, "clone")) {
+        type = GF_SNAP_OPTION_TYPE_CLONE;
+    }
+
+    if (type != GF_SNAP_OPTION_TYPE_CONFIG &&
+        type != GF_SNAP_OPTION_TYPE_STATUS) {
+        ret = dict_set_int32(dict, "hold_snap_locks", _gf_true);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Unable to set hold-snap-locks value "
+                   "as _gf_true");
+            goto out;
+        }
+    }
+
+    /* Following commands does not require volume locks  */
+    if (type == GF_SNAP_OPTION_TYPE_STATUS ||
+        type == GF_SNAP_OPTION_TYPE_ACTIVATE ||
+        type == GF_SNAP_OPTION_TYPE_DEACTIVATE) {
+        ret = dict_set_int32(dict, "hold_vol_locks", _gf_false);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Setting volume lock "
+                   "flag failed");
+            goto out;
         }
+    }
 
-        /* Check which op is intended */
-        switch (type) {
+    /* Check which op is intended */
+    switch (type) {
         case GF_SNAP_OPTION_TYPE_CREATE:
-                /* Syntax :
-                 * gluster snapshot create <snapname> <vol-name(s)>
-                 *                         [description <description>]
-                 *                         [force]
-                 */
-                /* In cases where the snapname is not given then
-                 * parsing fails & snapname cannot be "description",
-                 * "force" and "volume", that check is made here
-                 */
-                if (wordcount == 2){
-                        ret = -1;
-                        gf_log ("cli", GF_LOG_ERROR, "Invalid Syntax");
-                        goto out;
-                }
+            /* Syntax :
+             * gluster snapshot create <snapname> <vol-name(s)>
+             *                         [no-timestamp]
+             *                         [description <description>]
+             *                         [force]
+             */
+            /* In cases where the snapname is not given then
+             * parsing fails & snapname cannot be "description",
+             * "force" and "volume", that check is made here
+             */
+            if (wordcount == 2) {
+                ret = -1;
+                gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+                goto out;
+            }
 
-                ret = validate_snapname (words[2], invalid_snapnames);
-                if (ret) {
-                        goto out;
-                }
+            ret = validate_op_name("snapname", words[2], invalid_snapnames);
+            if (ret) {
+                goto out;
+            }
 
-                ret = cli_snap_create_parse (dict, words, wordcount);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "create command parsing failed.");
-                        goto out;
-                }
-                break;
+            ret = cli_snap_create_parse(dict, words, wordcount);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR, "create command parsing failed.");
+                goto out;
+            }
+            break;
+
+        case GF_SNAP_OPTION_TYPE_CLONE:
+            /* Syntax :
+             * gluster snapshot clone <clonename> <snapname>
+             */
+            /* In cases where the clonename is not given then
+             * parsing fails & snapname cannot be "description",
+             * "force" and "volume", that check is made here
+             */
+            if (wordcount == 2) {
+                ret = -1;
+                gf_log("cli", GF_LOG_ERROR, "Invalid Syntax");
+                goto out;
+            }
+
+            ret = validate_op_name("clonename", words[2], invalid_volnames);
+            if (ret) {
+                goto out;
+            }
+
+            ret = cli_snap_clone_parse(dict, words, wordcount);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR, "clone command parsing failed.");
+                goto out;
+            }
+            break;
 
         case GF_SNAP_OPTION_TYPE_INFO:
-                /* Syntax :
-                 * gluster snapshot info [(snapname] | [vol <volname>)]
-                 */
-                ret = cli_snap_info_parse (dict, words, wordcount);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to parse "
-                                "snapshot info command");
-                        goto out;
-                }
-                break;
+            /* Syntax :
+             * gluster snapshot info [(snapname] | [vol <volname>)]
+             */
+            ret = cli_snap_info_parse(dict, words, wordcount);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Failed to parse "
+                       "snapshot info command");
+                goto out;
+            }
+            break;
 
         case GF_SNAP_OPTION_TYPE_LIST:
-                /* Syntax :
-                 * gluster snaphsot list [volname]
-                 */
+            /* Syntax :
+             * gluster snaphsot list [volname]
+             */
 
-                ret = cli_snap_list_parse (dict, words, wordcount);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to parse "
-                                "snapshot list command");
-                        goto out;
-                }
-                break;
+            ret = cli_snap_list_parse(dict, words, wordcount);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Failed to parse "
+                       "snapshot list command");
+                goto out;
+            }
+            break;
 
         case GF_SNAP_OPTION_TYPE_DELETE:
-                /* Syntax :
-                 * snapshot delete (all | snapname | volume <volname>)
-                 */
-                ret = cli_snap_delete_parse (dict, words, wordcount, state);
-                if (ret) {
-                        /* A positive ret value means user cancelled
-                        * the command */
-                        if (ret < 0) {
-                                gf_log ("cli", GF_LOG_ERROR, "Failed to parse "
-                                        "snapshot delete command");
-                        }
-                        goto out;
+            /* Syntax :
+             * snapshot delete (all | snapname | volume <volname>)
+             */
+            ret = cli_snap_delete_parse(dict, words, wordcount, state);
+            if (ret) {
+                /* A positive ret value means user cancelled
+                 * the command */
+                if (ret < 0) {
+                    gf_log("cli", GF_LOG_ERROR,
+                           "Failed to parse "
+                           "snapshot delete command");
                 }
-                break;
+                goto out;
+            }
+            break;
 
         case GF_SNAP_OPTION_TYPE_CONFIG:
-                /* snapshot config [volname]  [snap-max-hard-limit <count>]
-                 *                            [snap-max-soft-limit <percent>] */
-                ret = cli_snap_config_parse (words, wordcount, dict, state);
-                if (ret) {
-                        if (ret < 0)
-                                gf_log ("cli", GF_LOG_ERROR,
-                                        "config command parsing failed.");
-                        goto out;
+            /* snapshot config [volname]  [snap-max-hard-limit <count>]
+             *                            [snap-max-soft-limit <percent>] */
+            ret = cli_snap_config_parse(words, wordcount, dict, state);
+            if (ret) {
+                if (ret < 0)
+                    gf_log("cli", GF_LOG_ERROR,
+                           "config command parsing failed.");
+                goto out;
+            }
+
+            ret = dict_set_int32(dict, "type", GF_SNAP_OPTION_TYPE_CONFIG);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Unable to set "
+                       "config type");
+                ret = -1;
+                goto out;
+            }
+            break;
+
+        case GF_SNAP_OPTION_TYPE_STATUS: {
+            /* Syntax :
+             * gluster snapshot status [(snapname |
+             *                         volume <volname>)]
+             */
+            ret = cli_snap_status_parse(dict, words, wordcount);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Failed to parse "
+                       "snapshot status command");
+                goto out;
+            }
+            break;
+        }
+
+        case GF_SNAP_OPTION_TYPE_RESTORE:
+            /* Syntax:
+             * snapshot restore <snapname>
+             */
+            ret = cli_snap_restore_parse(dict, words, wordcount, state);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Failed to parse "
+                       "restore command");
+                goto out;
+            }
+            break;
+
+        case GF_SNAP_OPTION_TYPE_ACTIVATE:
+            /* Syntax:
+             * snapshot activate <snapname> [force]
+             */
+            ret = cli_snap_activate_parse(dict, words, wordcount);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Failed to parse "
+                       "start command");
+                goto out;
+            }
+            break;
+        case GF_SNAP_OPTION_TYPE_DEACTIVATE:
+            /* Syntax:
+             * snapshot deactivate <snapname>
+             */
+            ret = cli_snap_deactivate_parse(dict, words, wordcount, state);
+            if (ret) {
+                /* A positive ret value means user cancelled
+                 * the command */
+                if (ret < 0) {
+                    gf_log("cli", GF_LOG_ERROR,
+                           "Failed to parse deactivate "
+                           "command");
                 }
+                goto out;
+            }
+            break;
+
+        default:
+            ret = -1;
+            gf_log("", GF_LOG_ERROR, "Opword Mismatch");
+            goto out;
+    }
+
+    ret = dict_set_int32(dict, "type", type);
+    if (ret) {
+        gf_log("", GF_LOG_ERROR, "Failed to set type.");
+        goto out;
+    }
+    /* If you got so far, input is valid */
+    ret = 0;
+out:
+    if (ret) {
+        if (dict)
+            dict_unref(dict);
+    } else
+        *options = dict;
+
+    return ret;
+}
+
+int
+cli_cmd_validate_volume(char *volname)
+{
+    int i = 0;
+    int ret = -1;
+    int volname_len;
+
+    if (volname[0] == '-')
+        return ret;
+
+    if (!strcmp(volname, "all")) {
+        cli_err("\"all\" cannot be the name of a volume.");
+        return ret;
+    }
+
+    if (strchr(volname, '/')) {
+        cli_err("Volume name should not contain \"/\" character.");
+        return ret;
+    }
+
+    volname_len = strlen(volname);
+    if (volname_len > GD_VOLUME_NAME_MAX) {
+        cli_err("Volname can not exceed %d characters.", GD_VOLUME_NAME_MAX);
+        return ret;
+    }
+
+    for (i = 0; i < volname_len; i++)
+        if (!isalnum(volname[i]) && (volname[i] != '_') &&
+            (volname[i] != '-')) {
+            cli_err(
+                "Volume name should not contain \"%c\""
+                " character.\nVolume names can only"
+                "contain alphanumeric, '-' and '_' "
+                "characters.",
+                volname[i]);
+            return ret;
+        }
+
+    ret = 0;
 
-                ret = dict_set_int32 (dict, "type", GF_SNAP_OPTION_TYPE_CONFIG);
+    return ret;
+}
+
+int32_t
+cli_cmd_bitrot_parse(const char **words, int wordcount, dict_t **options)
+{
+    int32_t ret = -1;
+    char *w = NULL;
+    char *volname = NULL;
+    static char *opwords[] = {"enable",          "disable", "scrub-throttle",
+                              "scrub-frequency", "scrub",   "signing-time",
+                              "signer-threads",  NULL};
+    static char *scrub_throt_values[] = {"lazy", "normal", "aggressive", NULL};
+    static char *scrub_freq_values[] = {
+        "hourly", "daily", "weekly", "biweekly", "monthly", "minute", NULL};
+    static char *scrub_values[] = {"pause", "resume", "status", "ondemand",
+                                   NULL};
+    dict_t *dict = NULL;
+    gf_bitrot_type type = GF_BITROT_OPTION_TYPE_NONE;
+    int32_t expiry_time = 0;
+    int32_t signer_th_count = 0;
+
+    GF_ASSERT(words);
+    GF_ASSERT(options);
+
+    /* Hack to print out bitrot help properly */
+    if ((wordcount == 3) && !(strcmp(words[2], "help"))) {
+        ret = 1;
+        return ret;
+    }
+
+    if (wordcount < 4 || wordcount > 5) {
+        gf_log("cli", GF_LOG_ERROR, "Invalid syntax");
+        goto out;
+    }
+
+    dict = dict_new();
+    if (!dict)
+        goto out;
+
+    volname = (char *)words[2];
+    if (!volname) {
+        ret = -1;
+        goto out;
+    }
+
+    ret = cli_cmd_validate_volume(volname);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to validate volume name");
+        goto out;
+    }
+
+    ret = dict_set_str(dict, "volname", volname);
+    if (ret) {
+        cli_out("Failed to set volume name in dictionary ");
+        goto out;
+    }
+
+    w = str_getunamb(words[3], opwords);
+    if (!w) {
+        cli_out("Invalid bit rot option : %s", words[3]);
+        ret = -1;
+        goto out;
+    }
+
+    if (strcmp(w, "enable") == 0) {
+        if (wordcount == 4) {
+            type = GF_BITROT_OPTION_TYPE_ENABLE;
+            ret = 0;
+            goto set_type;
+        } else {
+            ret = -1;
+            goto out;
+        }
+    }
+
+    if (strcmp(w, "disable") == 0) {
+        if (wordcount == 4) {
+            type = GF_BITROT_OPTION_TYPE_DISABLE;
+            ret = 0;
+            goto set_type;
+        } else {
+            ret = -1;
+            goto out;
+        }
+    }
+
+    if (!strcmp(w, "scrub-throttle")) {
+        if (!words[4]) {
+            cli_err(
+                "Missing scrub-throttle value for bitrot "
+                "option");
+            ret = -1;
+            goto out;
+        } else {
+            w = str_getunamb(words[4], scrub_throt_values);
+            if (!w) {
+                cli_err(
+                    "Invalid scrub-throttle option for "
+                    "bitrot");
+                ret = -1;
+                goto out;
+            } else {
+                type = GF_BITROT_OPTION_TYPE_SCRUB_THROTTLE;
+                ret = dict_set_str(dict, "scrub-throttle-value",
+                                   (char *)words[4]);
                 if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Unable to set "
-                                "config type");
-                        ret = -1;
-                        goto out;
+                    cli_out(
+                        "Failed to set scrub-throttle "
+                        "value in the dict");
+                    goto out;
                 }
-                break;
-
-        case GF_SNAP_OPTION_TYPE_STATUS:
-                {
-                        /* Syntax :
-                         * gluster snapshot status [(snapname |
-                         *                         volume <volname>)]
-                         */
-                        ret = cli_snap_status_parse (dict, words, wordcount);
-                        if (ret) {
-                                gf_log ("cli", GF_LOG_ERROR, "Failed to parse "
-                                        "snapshot status command");
-                                goto out;
-                        }
-                        break;
+                goto set_type;
+            }
+        }
+    }
+
+    if (!strcmp(words[3], "scrub-frequency")) {
+        if (!words[4]) {
+            cli_err("Missing scrub-frequency value");
+            ret = -1;
+            goto out;
+        } else {
+            w = str_getunamb(words[4], scrub_freq_values);
+            if (!w) {
+                cli_err("Invalid frequency option for bitrot");
+                ret = -1;
+                goto out;
+            } else {
+                type = GF_BITROT_OPTION_TYPE_SCRUB_FREQ;
+                ret = dict_set_str(dict, "scrub-frequency-value",
+                                   (char *)words[4]);
+                if (ret) {
+                    cli_out(
+                        "Failed to set dict for "
+                        "bitrot");
+                    goto out;
                 }
+                goto set_type;
+            }
+        }
+    }
 
-        case GF_SNAP_OPTION_TYPE_RESTORE:
-                /* Syntax:
-                 * snapshot restore <snapname>
-                 */
-                ret = cli_snap_restore_parse (dict, words, wordcount, state);
+    if (!strcmp(words[3], "scrub")) {
+        if (!words[4]) {
+            cli_err("Missing scrub value for bitrot option");
+            ret = -1;
+            goto out;
+        } else {
+            w = str_getunamb(words[4], scrub_values);
+            if (!w) {
+                cli_err("Invalid scrub option for bitrot");
+                ret = -1;
+                goto out;
+            } else {
+                if (strcmp(words[4], "status") == 0) {
+                    type = GF_BITROT_CMD_SCRUB_STATUS;
+                } else if (strcmp(words[4], "ondemand") == 0) {
+                    type = GF_BITROT_CMD_SCRUB_ONDEMAND;
+                } else {
+                    type = GF_BITROT_OPTION_TYPE_SCRUB;
+                }
+                ret = dict_set_str(dict, "scrub-value", (char *)words[4]);
                 if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to parse "
-                                "restore command");
-                        goto out;
+                    cli_out(
+                        "Failed to set dict for "
+                        "bitrot");
+                    goto out;
                 }
-                break;
-
-                case GF_SNAP_OPTION_TYPE_ACTIVATE:
-                        /* Syntax:
-                        * snapshot activate <snapname> [force]
-                        */
-                        ret = cli_snap_activate_parse (dict, words, wordcount);
-                        if (ret) {
-                                gf_log ("cli", GF_LOG_ERROR, "Failed to parse "
-                                        "start command");
-                                goto out;
-                        }
-                        break;
-                case GF_SNAP_OPTION_TYPE_DEACTIVATE:
-                        /* Syntax:
-                        * snapshot deactivate <snapname>
-                        */
-                        ret = cli_snap_deactivate_parse (dict, words, wordcount,
-                                state);
-                        if (ret) {
-                                /* A positive ret value means user cancelled
-                                 * the command */
-                                if (ret < 0) {
-                                        gf_log ("cli", GF_LOG_ERROR,
-                                                "Failed to parse deactivate "
-                                                "command");
-                                }
-                                goto out;
-                        }
-                        break;
+                goto set_type;
+            }
+        }
+    }
 
-        default:
-                gf_log ("", GF_LOG_ERROR, "Opword Mismatch");
+    if (!strcmp(words[3], "signing-time")) {
+        if (!words[4]) {
+            cli_err(
+                "Missing signing-time value for bitrot "
+                "option");
+            ret = -1;
+            goto out;
+        } else {
+            type = GF_BITROT_OPTION_TYPE_EXPIRY_TIME;
+
+            expiry_time = strtol(words[4], NULL, 0);
+            if (expiry_time < 1) {
+                cli_err(
+                    "Expiry time  value should not be less"
+                    " than 1");
+                ret = -1;
+                goto out;
+            }
+
+            ret = dict_set_uint32(dict, "expiry-time",
+                                  (unsigned int)expiry_time);
+            if (ret) {
+                cli_out("Failed to set dict for bitrot");
                 goto out;
+            }
+            goto set_type;
         }
+    } else if (!strcmp(words[3], "signer-threads")) {
+        if (!words[4]) {
+            cli_err(
+                "Missing signer-thread value for bitrot "
+                "option");
+            ret = -1;
+            goto out;
+        } else {
+            type = GF_BITROT_OPTION_TYPE_SIGNER_THREADS;
 
-        ret = dict_set_int32 (dict, "type", type);
-        if (ret) {
-                gf_log ("", GF_LOG_ERROR,
-                        "Failed to set type.");
+            signer_th_count = strtol(words[4], NULL, 0);
+            if (signer_th_count < 1) {
+                cli_err("signer-thread count should not be less than 1");
+                ret = -1;
                 goto out;
+            }
+
+            ret = dict_set_uint32(dict, "signer-threads",
+                                  (unsigned int)signer_th_count);
+            if (ret) {
+                cli_out("Failed to set dict for bitrot");
+                goto out;
+            }
+            goto set_type;
         }
-        /* If you got so far, input is valid */
-        ret = 0;
+    } else {
+        cli_err(
+            "Invalid option %s for bitrot. Please enter valid "
+            "bitrot option",
+            words[3]);
+        ret = -1;
+        goto out;
+    }
+set_type:
+    ret = dict_set_int32(dict, "type", type);
+    if (ret < 0)
+        goto out;
+
+    *options = dict;
+
 out:
-        if (ret) {
-                if (dict)
-                        dict_destroy (dict);
-        } else
-                *options = dict;
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to parse bitrot command");
+        if (dict)
+            dict_unref(dict);
+    }
 
-        return ret;
+    return ret;
+}
+
+/* Parsing global option for NFS-Ganesha config
+ *  gluster nfs-ganesha enable/disable */
+
+int32_t
+cli_cmd_ganesha_parse(struct cli_state *state, const char **words,
+                      int wordcount, dict_t **options, char **op_errstr)
+{
+    dict_t *dict = NULL;
+    int ret = -1;
+    char *key = NULL;
+    char *value = NULL;
+    char *w = NULL;
+    static char *opwords[] = {"enable", "disable", NULL};
+    const char *question = NULL;
+    gf_answer_t answer = GF_ANSWER_NO;
+
+    GF_ASSERT(words);
+    GF_ASSERT(options);
+
+    dict = dict_new();
+
+    if (!dict)
+        goto out;
+
+    if (wordcount != 2)
+        goto out;
+
+    key = (char *)words[0];
+    value = (char *)words[1];
+
+    if (!key || !value) {
+        cli_out("Usage : nfs-ganesha <enable/disable>");
+        ret = -1;
+        goto out;
+    }
+
+    ret = gf_strip_whitespace(value, strlen(value));
+    if (ret == -1)
+        goto out;
+
+    if (strcmp(key, "nfs-ganesha")) {
+        gf_asprintf(op_errstr,
+                    "Global option: error: ' %s '"
+                    "is not a valid global option.",
+                    key);
+        ret = -1;
+        goto out;
+    }
+
+    w = str_getunamb(value, opwords);
+    if (!w) {
+        cli_out(
+            "Invalid global option \n"
+            "Usage : nfs-ganesha <enable/disable>");
+        ret = -1;
+        goto out;
+    }
+
+    if (strcmp(value, "enable") == 0) {
+        question =
+            "Enabling NFS-Ganesha requires Gluster-NFS to be "
+            "disabled across the trusted pool. Do you "
+            "still want to continue?\n";
+    } else if (strcmp(value, "disable") == 0) {
+        question =
+            "Disabling NFS-Ganesha will tear down the entire "
+            "ganesha cluster across the trusted pool. Do you "
+            "still want to continue?\n";
+    } else {
+        ret = -1;
+        goto out;
+    }
+    answer = cli_cmd_get_confirmation(state, question);
+    if (GF_ANSWER_NO == answer) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Global operation "
+               "cancelled, exiting");
+        ret = -1;
+        goto out;
+    }
+    cli_out("This will take a few minutes to complete. Please wait ..");
+
+    ret = dict_set_str(dict, "key", key);
+    if (ret) {
+        gf_log(THIS->name, GF_LOG_ERROR, "dict set on key failed");
+        goto out;
+    }
+
+    ret = dict_set_str(dict, "value", value);
+    if (ret) {
+        gf_log(THIS->name, GF_LOG_ERROR, "dict set on value failed");
+        goto out;
+    }
+
+    ret = dict_set_str(dict, "globalname", "All");
+    if (ret) {
+        gf_log(THIS->name, GF_LOG_ERROR,
+               "dict set on global"
+               " key failed.");
+        goto out;
+    }
+
+    ret = dict_set_int32(dict, "hold_global_locks", _gf_true);
+    if (ret) {
+        gf_log(THIS->name, GF_LOG_ERROR,
+               "dict set on global key "
+               "failed.");
+        goto out;
+    }
+
+    *options = dict;
+out:
+    if (ret)
+        dict_unref(dict);
+
+    return ret;
 }
diff --git a/cli/src/cli-cmd-peer.c b/cli/src/cli-cmd-peer.c
index 551312411a0..084998701d8 100644
--- a/cli/src/cli-cmd-peer.c
+++ b/cli/src/cli-cmd-peer.c
@@ -13,288 +13,305 @@
 #include <stdint.h>
 #include <pthread.h>
 
-#ifndef _CONFIG_H
-#define _CONFIG_H
-#include "config.h"
-#endif
-
 #include "cli.h"
 #include "cli-cmd.h"
 #include "cli-mem-types.h"
 #include "cli1-xdr.h"
 #include "protocol-common.h"
+#include <glusterfs/events.h>
 
-extern struct rpc_clnt *global_rpc;
-
-extern rpc_clnt_prog_t *cli_rpc_prog;
-
-int cli_cmd_peer_help_cbk (struct cli_state *state, struct cli_cmd_word *in_word,
+int
+cli_cmd_peer_help_cbk(struct cli_state *state, struct cli_cmd_word *in_word,
                       const char **words, int wordcount);
 
 int
-cli_cmd_peer_probe_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                   const char **words, int wordcount)
+cli_cmd_peer_probe_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                       const char **words, int wordcount)
 {
-        int                     ret = -1;
-        rpc_clnt_procedure_t    *proc = NULL;
-        call_frame_t            *frame = NULL;
-        dict_t                  *dict = NULL;
-        int                     sent = 0;
-        int                     parse_error = 0;
-        cli_local_t             *local = NULL;
-
-        if (!(wordcount == 3)) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                goto out;
-        }
-
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_PROBE];
-
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
-
-        dict = dict_new ();
-        if (!dict)
-                goto out;
-
-        ret = dict_set_str (dict, "hostname", (char *)words[2]);
-        if (ret)
-                goto out;
-
-        ret = valid_internet_address ((char *) words[2], _gf_false);
-        if (ret == 1) {
-                ret = 0;
-        } else {
-                cli_out ("%s is an invalid address", words[2]);
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                ret = -1;
-                goto out;
-        }
-/*        if (words[3]) {
-                ret = dict_set_str (dict, "port", (char *)words[3]);
-                if (ret)
-                        goto out;
-        }
-*/
-
-        CLI_LOCAL_INIT (local, words, frame, dict);
-
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, dict);
-        }
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *dict = NULL;
+    int sent = 0;
+    int parse_error = 0;
+    cli_local_t *local = NULL;
+
+    if (!(wordcount == 3)) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
+
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_PROBE];
+
+    dict = dict_new();
+    if (!dict)
+        goto out;
+
+    ret = dict_set_str(dict, "hostname", (char *)words[2]);
+    if (ret)
+        goto out;
+
+    ret = valid_internet_address((char *)words[2], _gf_false, _gf_false);
+    if (ret == 1) {
+        ret = 0;
+    } else {
+        cli_out("%s is an invalid address", words[2]);
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        ret = -1;
+        goto out;
+    }
+    /*        if (words[3]) {
+                    ret = dict_set_str (dict, "port", (char *)words[3]);
+                    if (ret)
+                            goto out;
+            }
+    */
+
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
+
+    CLI_LOCAL_INIT(local, words, frame, dict);
+
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, dict);
+    }
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_out ("Peer probe failed");
-        }
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Peer probe failed");
+    }
 
-        CLI_STACK_DESTROY (frame);
+    CLI_STACK_DESTROY(frame);
 
-        return ret;
-}
+    if (ret == 0) {
+        gf_event(EVENT_PEER_ATTACH, "host=%s", (char *)words[2]);
+    }
 
+    return ret;
+}
 
 int
-cli_cmd_peer_deprobe_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                          const char **words, int wordcount)
+cli_cmd_peer_deprobe_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                         const char **words, int wordcount)
 {
-        int                   ret   = -1;
-        rpc_clnt_procedure_t *proc  = NULL;
-        call_frame_t         *frame = NULL;
-        dict_t               *dict  = NULL;
-        int                  flags = 0;
-        int                  sent = 0;
-        int                  parse_error = 0;
-        cli_local_t          *local = NULL;
-
-        if ((wordcount < 3) || (wordcount > 4)) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                goto out;
-        }
-
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_DEPROBE];
-
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
-
-        dict = dict_new ();
-
-        ret = dict_set_str (dict, "hostname", (char *)words[2]);
-        if (ret)
-                goto out;
-
-/*        if (words[3]) {
-                ret = dict_set_str (dict, "port", (char *)words[3]);
-                if (ret)
-                        goto out;
-        }
-*/
-        if (wordcount == 4) {
-                if (!strcmp("force", words[3]))
-                        flags |= GF_CLI_FLAG_OP_FORCE;
-                else {
-                        ret = -1;
-                        cli_usage_out (word->pattern);
-                        parse_error = 1;
-                        goto out;
-                }
-        }
-        ret = dict_set_int32 (dict, "flags", flags);
-        if (ret)
-                goto out;
-
-        CLI_LOCAL_INIT (local, words, frame, dict);
-
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, dict);
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *dict = NULL;
+    int flags = 0;
+    int sent = 0;
+    int parse_error = 0;
+    cli_local_t *local = NULL;
+    gf_answer_t answer = GF_ANSWER_NO;
+    const char *question = NULL;
+
+    if ((wordcount < 3) || (wordcount > 4)) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
+    question =
+        "All clients mounted through the peer which is getting detached need "
+        "to be remounted using one of the other active peers in the trusted "
+        "storage pool to ensure client gets notification on any changes done "
+        "on the gluster configuration and if the same has been done do you "
+        "want to proceed?";
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_DEPROBE];
+
+    dict = dict_new();
+
+    ret = dict_set_str(dict, "hostname", (char *)words[2]);
+    if (ret)
+        goto out;
+
+    /*        if (words[3]) {
+                    ret = dict_set_str (dict, "port", (char *)words[3]);
+                    if (ret)
+                            goto out;
+            }
+    */
+    if (wordcount == 4) {
+        if (!strcmp("force", words[3]))
+            flags |= GF_CLI_FLAG_OP_FORCE;
+        else {
+            ret = -1;
+            cli_usage_out(word->pattern);
+            parse_error = 1;
+            goto out;
         }
+    }
+    ret = dict_set_int32(dict, "flags", flags);
+    if (ret)
+        goto out;
+    answer = cli_cmd_get_confirmation(state, question);
+    if (GF_ANSWER_NO == answer) {
+        ret = 0;
+        goto out;
+    }
+
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
+
+    CLI_LOCAL_INIT(local, words, frame, dict);
+
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, dict);
+    }
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_out ("Peer detach failed");
-        }
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Peer detach failed");
+    }
 
-        CLI_STACK_DESTROY (frame);
+    CLI_STACK_DESTROY(frame);
 
-        return ret;
+    if (ret == 0) {
+        gf_event(EVENT_PEER_DETACH, "host=%s", (char *)words[2]);
+    }
+
+    return ret;
 }
 
 int
-cli_cmd_peer_status_cbk (struct cli_state *state, struct cli_cmd_word *word,
+cli_cmd_peer_status_cbk(struct cli_state *state, struct cli_cmd_word *word,
                         const char **words, int wordcount)
 {
-        int                     ret = -1;
-        rpc_clnt_procedure_t    *proc = NULL;
-        call_frame_t            *frame = NULL;
-        int                     sent = 0;
-        int                     parse_error = 0;
-
-        if (wordcount != 2) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                goto out;
-        }
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    int sent = 0;
+    int parse_error = 0;
 
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_LIST_FRIENDS];
+    if (wordcount != 2) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
 
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_LIST_FRIENDS];
 
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, (void *)GF_CLI_LIST_PEERS);
-        }
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame)
+        goto out;
+
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, (void *)GF_CLI_LIST_PEERS);
+    }
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_out ("Peer status failed");
-        }
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Peer status failed");
+    }
 
-        CLI_STACK_DESTROY (frame);
+    CLI_STACK_DESTROY(frame);
 
-        return ret;
+    return ret;
 }
 
 int
-cli_cmd_pool_list_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                       const char **words, int wordcount)
+cli_cmd_pool_list_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                      const char **words, int wordcount)
 {
-        int                     ret = -1;
-        rpc_clnt_procedure_t    *proc = NULL;
-        call_frame_t            *frame = NULL;
-        int                     sent = 0;
-        int                     parse_error = 0;
-
-        if (wordcount != 2) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                goto out;
-        }
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    int sent = 0;
+    int parse_error = 0;
 
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_LIST_FRIENDS];
+    if (wordcount != 2) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
 
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_LIST_FRIENDS];
 
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS,
-                                (void *)GF_CLI_LIST_POOL_NODES);
-        }
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame)
+        goto out;
+
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, (void *)GF_CLI_LIST_POOL_NODES);
+    }
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_err ("pool list: command execution failed");
-        }
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_err("pool list: command execution failed");
+    }
 
-        CLI_STACK_DESTROY (frame);
+    CLI_STACK_DESTROY(frame);
 
-        return ret;
+    return ret;
 }
 
 struct cli_cmd cli_probe_cmds[] = {
-        { "peer probe <HOSTNAME>",
-          cli_cmd_peer_probe_cbk,
-          "probe peer specified by <HOSTNAME>"},
+    {"peer probe { <HOSTNAME> | <IP-address> }", cli_cmd_peer_probe_cbk,
+     "probe peer specified by <HOSTNAME>"},
 
-        { "peer detach <HOSTNAME> [force]",
-          cli_cmd_peer_deprobe_cbk,
-          "detach peer specified by <HOSTNAME>"},
+    {"peer detach { <HOSTNAME> | <IP-address> } [force]",
+     cli_cmd_peer_deprobe_cbk, "detach peer specified by <HOSTNAME>"},
 
-        { "peer status",
-          cli_cmd_peer_status_cbk,
-          "list status of peers"},
+    {"peer status", cli_cmd_peer_status_cbk, "list status of peers"},
 
-	{ "peer help",
-           cli_cmd_peer_help_cbk,
-           "Help command for peer "},
+    {"peer help", cli_cmd_peer_help_cbk, "display help for peer commands"},
 
-        { "pool list",
-          cli_cmd_pool_list_cbk,
-          "list all the nodes in the pool (including localhost)"},
+    {"pool list", cli_cmd_pool_list_cbk,
+     "list all the nodes in the pool (including localhost)"},
 
-        { NULL, NULL, NULL }
-};
+    {NULL, NULL, NULL}};
 
 int
-cli_cmd_peer_help_cbk (struct cli_state *state, struct cli_cmd_word *in_word,
+cli_cmd_peer_help_cbk(struct cli_state *state, struct cli_cmd_word *in_word,
                       const char **words, int wordcount)
 {
-        struct cli_cmd        *cmd = NULL;
+    struct cli_cmd *cmd = NULL;
+    struct cli_cmd *probe_cmd = NULL;
+    int count = 0;
+
+    cli_out("\ngluster peer commands");
+    cli_out("======================\n");
 
+    cmd = GF_MALLOC(sizeof(cli_probe_cmds), cli_mt_cli_cmd);
+    memcpy(cmd, cli_probe_cmds, sizeof(cli_probe_cmds));
+    count = (sizeof(cli_probe_cmds) / sizeof(struct cli_cmd));
+    cli_cmd_sort(cmd, count);
 
+    for (probe_cmd = cmd; probe_cmd->pattern; probe_cmd++)
+        cli_out("%s - %s", probe_cmd->pattern, probe_cmd->desc);
 
-        for (cmd = cli_probe_cmds; cmd->pattern; cmd++)
-                cli_out ("%s - %s", cmd->pattern, cmd->desc);
+    GF_FREE(cmd);
 
-        return 0;
+    cli_out("\n");
+    return 0;
 }
 
 int
-cli_cmd_probe_register (struct cli_state *state)
+cli_cmd_probe_register(struct cli_state *state)
 {
-        int  ret = 0;
-        struct cli_cmd *cmd = NULL;
+    int ret = 0;
+    struct cli_cmd *cmd = NULL;
 
-        for (cmd = cli_probe_cmds; cmd->pattern; cmd++) {
-
-                ret = cli_cmd_register (&state->tree, cmd);
-                if (ret)
-                        goto out;
-        }
+    for (cmd = cli_probe_cmds; cmd->pattern; cmd++) {
+        ret = cli_cmd_register(&state->tree, cmd);
+        if (ret)
+            goto out;
+    }
 out:
-        return ret;
+    return ret;
 }
diff --git a/cli/src/cli-cmd-snapshot.c b/cli/src/cli-cmd-snapshot.c
index d03b5b99dd0..859d6b2e40d 100644
--- a/cli/src/cli-cmd-snapshot.c
+++ b/cli/src/cli-cmd-snapshot.c
@@ -13,144 +13,123 @@
 #include <stdint.h>
 #include <pthread.h>
 
-#ifndef _CONFIG_H
-#define _CONFIG_H
-#include "config.h"
-#endif
-
 #include "cli.h"
 #include "cli-cmd.h"
-
-extern rpc_clnt_prog_t *cli_rpc_prog;
+#include "cli-mem-types.h"
 
 int
-cli_cmd_snapshot_help_cbk (struct cli_state *state, struct cli_cmd_word *in_word,
-                           const char **words, int wordcount);
+cli_cmd_snapshot_help_cbk(struct cli_state *state, struct cli_cmd_word *in_word,
+                          const char **words, int wordcount);
 
 int
-cli_cmd_snapshot_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                      const char **words, int wordcount)
+cli_cmd_snapshot_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                     const char **words, int wordcount)
 {
-        int                      ret       = 0;
-        int                      parse_err = 0;
-        dict_t                  *options   = NULL;
-        rpc_clnt_procedure_t    *proc      = NULL;
-        call_frame_t            *frame     = NULL;
-        cli_local_t             *local     = NULL;
-
-        proc = &cli_rpc_prog->proctable [GLUSTER_CLI_SNAP];
-        if (proc == NULL) {
-               ret = -1;
-                goto out;
+    int ret = 0;
+    int parse_err = 0;
+    dict_t *options = NULL;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    cli_local_t *local = NULL;
+
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_SNAP];
+
+    /* Parses the command entered by the user */
+    ret = cli_cmd_snapshot_parse(words, wordcount, &options, state);
+    if (ret) {
+        if (ret < 0) {
+            cli_usage_out(word->pattern);
+            parse_err = 1;
+        } else {
+            /* User might have cancelled the snapshot operation */
+            ret = 0;
         }
+        goto out;
+    }
 
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (frame == NULL) {
-                ret = -1;
-                goto out;
-        }
-
-        /* Parses the command entered by the user */
-        ret = cli_cmd_snapshot_parse (words, wordcount, &options, state);
-        if (ret) {
-                if (ret < 0) {
-                        cli_usage_out (word->pattern);
-                        parse_err = 1;
-                } else {
-                        /* User might have cancelled the snapshot operation */
-                        ret = 0;
-                }
-                goto out;
-        }
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (frame == NULL) {
+        ret = -1;
+        goto out;
+    }
 
-        CLI_LOCAL_INIT (local, words, frame, options);
+    CLI_LOCAL_INIT(local, words, frame, options);
 
-        if (proc->fn)
-                ret = proc->fn (frame, THIS, options);
+    if (proc->fn)
+        ret = proc->fn(frame, THIS, options);
 
 out:
-        if (ret && parse_err == 0)
-                cli_out ("Snapshot command failed");
+    if (ret && parse_err == 0)
+        cli_out("Snapshot command failed");
 
-        CLI_STACK_DESTROY (frame);
+    CLI_STACK_DESTROY(frame);
 
-        return ret;
+    return ret;
 }
 
 struct cli_cmd snapshot_cmds[] = {
-        { "snapshot help",
-          cli_cmd_snapshot_help_cbk,
-          "display help for snapshot commands"
-        },
-        { "snapshot create <snapname> <volname(s)> [description <description>] [force]",
-          cli_cmd_snapshot_cbk,
-          "Snapshot Create."
-        },
-        { "snapshot restore <snapname>",
-          cli_cmd_snapshot_cbk,
-          "Snapshot Restore."
-        },
-        { "snapshot status [(snapname | volume <volname>)]",
-          cli_cmd_snapshot_cbk,
-          "Snapshot Status."
-        },
-        { "snapshot info [(snapname | volume <volname>)]",
-          cli_cmd_snapshot_cbk,
-          "Snapshot Info."
-        },
-        { "snapshot list [volname]",
-          cli_cmd_snapshot_cbk,
-          "Snapshot List."
-        },
-        {"snapshot config [volname] ([snap-max-hard-limit <count>] "
-         "[snap-max-soft-limit <percent>]) "
-         "| ([auto-delete <enable|disable>])"
-         "| ([activate-on-create <enable|disable>])",
-          cli_cmd_snapshot_cbk,
-          "Snapshot Config."
-        },
-        {"snapshot delete (all | snapname | volume <volname>)",
-          cli_cmd_snapshot_cbk,
-          "Snapshot Delete."
-        },
-        {"snapshot activate <snapname> [force]",
-          cli_cmd_snapshot_cbk,
-          "Activate snapshot volume."
-        },
-        {"snapshot deactivate <snapname>",
-          cli_cmd_snapshot_cbk,
-          "Deactivate snapshot volume."
-        },
-        { NULL, NULL, NULL }
-};
+    {"snapshot help", cli_cmd_snapshot_help_cbk,
+     "display help for snapshot commands"},
+    {"snapshot create <snapname> <volname> [no-timestamp] "
+     "[description <description>] [force]",
+     cli_cmd_snapshot_cbk, "Snapshot Create."},
+    {"snapshot clone <clonename> <snapname>", cli_cmd_snapshot_cbk,
+     "Snapshot Clone."},
+    {"snapshot restore <snapname>", cli_cmd_snapshot_cbk, "Snapshot Restore."},
+    {"snapshot status [(snapname | volume <volname>)]", cli_cmd_snapshot_cbk,
+     "Snapshot Status."},
+    {"snapshot info [(snapname | volume <volname>)]", cli_cmd_snapshot_cbk,
+     "Snapshot Info."},
+    {"snapshot list [volname]", cli_cmd_snapshot_cbk, "Snapshot List."},
+    {"snapshot config [volname] ([snap-max-hard-limit <count>] "
+     "[snap-max-soft-limit <percent>]) "
+     "| ([auto-delete <enable|disable>])"
+     "| ([activate-on-create <enable|disable>])",
+     cli_cmd_snapshot_cbk, "Snapshot Config."},
+    {"snapshot delete (all | snapname | volume <volname>)",
+     cli_cmd_snapshot_cbk, "Snapshot Delete."},
+    {"snapshot activate <snapname> [force]", cli_cmd_snapshot_cbk,
+     "Activate snapshot volume."},
+    {"snapshot deactivate <snapname>", cli_cmd_snapshot_cbk,
+     "Deactivate snapshot volume."},
+    {NULL, NULL, NULL}};
 
 int
-cli_cmd_snapshot_help_cbk (struct cli_state *state,
-                           struct cli_cmd_word *in_word,
-                           const char **words,
-                           int wordcount)
+cli_cmd_snapshot_help_cbk(struct cli_state *state, struct cli_cmd_word *in_word,
+                          const char **words, int wordcount)
 {
-        struct cli_cmd        *cmd = NULL;
+    struct cli_cmd *cmd = NULL;
+    struct cli_cmd *snap_cmd = NULL;
+    int count = 0;
 
-        for (cmd = snapshot_cmds; cmd->pattern; cmd++)
-                if (_gf_false == cmd->disable)
-                        cli_out ("%s - %s", cmd->pattern, cmd->desc);
+    cmd = GF_MALLOC(sizeof(snapshot_cmds), cli_mt_cli_cmd);
+    memcpy(cmd, snapshot_cmds, sizeof(snapshot_cmds));
+    count = (sizeof(snapshot_cmds) / sizeof(struct cli_cmd));
+    cli_cmd_sort(cmd, count);
 
-        return 0;
+    cli_out("\ngluster snapshot commands");
+    cli_out("=========================\n");
+
+    for (snap_cmd = cmd; snap_cmd->pattern; snap_cmd++)
+        if (_gf_false == snap_cmd->disable)
+            cli_out("%s - %s", snap_cmd->pattern, snap_cmd->desc);
+    cli_out("\n");
+
+    GF_FREE(cmd);
+    return 0;
 }
 
 int
-cli_cmd_snapshot_register (struct cli_state *state)
+cli_cmd_snapshot_register(struct cli_state *state)
 {
-        int  ret = 0;
-        struct cli_cmd *cmd = NULL;
-
-        for (cmd = snapshot_cmds; cmd->pattern; cmd++) {
-
-                ret = cli_cmd_register (&state->tree, cmd);
-                if (ret)
-                        goto out;
-        }
+    int ret = 0;
+    struct cli_cmd *cmd = NULL;
+
+    for (cmd = snapshot_cmds; cmd->pattern; cmd++) {
+        ret = cli_cmd_register(&state->tree, cmd);
+        if (ret)
+            goto out;
+    }
 out:
-        return ret;
+    return ret;
 }
diff --git a/cli/src/cli-cmd-system.c b/cli/src/cli-cmd-system.c
index 690a104183a..801e8f4efed 100644
--- a/cli/src/cli-cmd-system.c
+++ b/cli/src/cli-cmd-system.c
@@ -13,590 +13,612 @@
 #include <stdint.h>
 #include <pthread.h>
 
-#ifndef _CONFIG_H
-#define _CONFIG_H
-#include "config.h"
-#endif
-
 #include "cli.h"
 #include "cli-cmd.h"
 #include "cli-mem-types.h"
 #include "protocol-common.h"
 
+int
+cli_cmd_system_help_cbk(struct cli_state *state, struct cli_cmd_word *in_word,
+                        const char **words, int wordcount);
 
-extern struct rpc_clnt *global_rpc;
-
-extern rpc_clnt_prog_t *cli_rpc_prog;
-
-int cli_cmd_system_help_cbk (struct cli_state *state, struct cli_cmd_word *in_word,
-                             const char **words, int wordcount);
-
-int cli_cmd_copy_file_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                           const char **words, int wordcount);
+int
+cli_cmd_copy_file_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                      const char **words, int wordcount);
 
-int cli_cmd_sys_exec_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                          const char **words, int wordcount);
+int
+cli_cmd_sys_exec_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                     const char **words, int wordcount);
 
 int
-cli_cmd_getspec_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                     const char **words, int wordcount)
+cli_cmd_getspec_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                    const char **words, int wordcount)
 {
-        int                     ret = -1;
-        rpc_clnt_procedure_t    *proc = NULL;
-        call_frame_t            *frame = NULL;
-        dict_t                  *dict = NULL;
-
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
-
-        dict = dict_new ();
-        if (!dict)
-                goto out;
-
-        if (wordcount != 3) {
-                cli_usage_out (word->pattern);
-                goto out;
-        }
-
-        ret = dict_set_str (dict, "volid", (char *)words[2]);
-        if (ret)
-                goto out;
-
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_GETSPEC];
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, dict);
-        }
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *dict = NULL;
+
+    if (wordcount != 3) {
+        cli_usage_out(word->pattern);
+        goto out;
+    }
+
+    dict = dict_new();
+    if (!dict)
+        goto out;
+
+    ret = dict_set_str(dict, "volid", (char *)words[2]);
+    if (ret)
+        goto out;
+
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_GETSPEC];
+    if (proc->fn) {
+        frame = create_frame(THIS, THIS->ctx->pool);
+        if (!frame) {
+            ret = -1;
+            goto out;
+        }
+        ret = proc->fn(frame, THIS, dict);
+    }
 
 out:
-        if (!proc && ret) {
-                if (dict)
-                        dict_destroy (dict);
-                if (wordcount > 1)
-                        cli_out ("Fetching spec for volume %s failed",
-                                 (char *)words[2]);
-        }
+    if (!proc && ret) {
+        if (wordcount > 1)
+            cli_out("Fetching spec for volume %s failed", (char *)words[2]);
+    }
+
+    if (dict)
+        dict_unref(dict);
 
-        return ret;
+    CLI_STACK_DESTROY(frame);
+    return ret;
 }
 
 int
-cli_cmd_pmap_b2p_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                 const char **words, int wordcount)
+cli_cmd_pmap_b2p_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                     const char **words, int wordcount)
 {
-        int                     ret = -1;
-        rpc_clnt_procedure_t    *proc = NULL;
-        call_frame_t            *frame = NULL;
-        dict_t                  *dict = NULL;
-
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
-
-        dict = dict_new ();
-        if (!dict)
-                goto out;
-
-        if (wordcount != 4) {
-                cli_usage_out (word->pattern);
-                goto out;
-        }
-
-        ret = dict_set_str (dict, "brick", (char *)words[3]);
-        if (ret)
-                goto out;
-
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_PMAP_PORTBYBRICK];
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, dict);
-        }
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *dict = NULL;
+
+    if (wordcount != 4) {
+        cli_usage_out(word->pattern);
+        goto out;
+    }
+
+    dict = dict_new();
+    if (!dict)
+        goto out;
+
+    ret = dict_set_str(dict, "brick", (char *)words[3]);
+    if (ret)
+        goto out;
+
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_PMAP_PORTBYBRICK];
+    if (proc->fn) {
+        frame = create_frame(THIS, THIS->ctx->pool);
+        if (!frame) {
+            ret = -1;
+            goto out;
+        }
+        ret = proc->fn(frame, THIS, dict);
+    }
 
 out:
-        if (!proc && ret) {
-                if (dict)
-                        dict_destroy (dict);
-                if (wordcount > 1)
-                        cli_out ("Fetching spec for volume %s failed",
-                                 (char *)words[3]);
-        }
+    if (!proc && ret) {
+        if (wordcount > 1)
+            cli_out("Fetching spec for volume %s failed", (char *)words[3]);
+    }
+
+    if (dict)
+        dict_unref(dict);
 
-        return ret;
+    CLI_STACK_DESTROY(frame);
+    return ret;
 }
 
 int
-cli_cmd_fsm_log_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                     const char **words, int wordcount)
+cli_cmd_fsm_log_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                    const char **words, int wordcount)
 {
-        int                             ret = -1;
-        rpc_clnt_procedure_t            *proc = NULL;
-        call_frame_t                    *frame = NULL;
-        char                            *name = "";
-
-        if ((wordcount != 4) && (wordcount != 3)) {
-                cli_usage_out (word->pattern);
-                goto out;
-        }
-
-        if (wordcount == 4)
-                name = (char*)words[3];
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_FSM_LOG];
-        if (proc && proc->fn) {
-                frame = create_frame (THIS, THIS->ctx->pool);
-                if (!frame)
-                        goto out;
-                ret = proc->fn (frame, THIS, (void*)name);
-        }
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    char *name = "";
+
+    if ((wordcount != 4) && (wordcount != 3)) {
+        cli_usage_out(word->pattern);
+        goto out;
+    }
+
+    if (wordcount == 4)
+        name = (char *)words[3];
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_FSM_LOG];
+    if (proc && proc->fn) {
+        frame = create_frame(THIS, THIS->ctx->pool);
+        if (!frame)
+            goto out;
+        ret = proc->fn(frame, THIS, (void *)name);
+    }
 out:
-        return ret;
+    return ret;
 }
 
 int
-cli_cmd_getwd_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                   const char **words, int wordcount)
+cli_cmd_getwd_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                  const char **words, int wordcount)
 {
-        int                             ret = -1;
-        rpc_clnt_procedure_t            *proc = NULL;
-        call_frame_t                    *frame = NULL;
-
-        if (wordcount != 2) {
-                cli_usage_out (word->pattern);
-                goto out;
-        }
-
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_GETWD];
-        if (proc && proc->fn) {
-                frame = create_frame (THIS, THIS->ctx->pool);
-                if (!frame)
-                        goto out;
-                ret = proc->fn (frame, THIS, NULL);
-        }
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+
+    if (wordcount != 2) {
+        cli_usage_out(word->pattern);
+        goto out;
+    }
+
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_GETWD];
+    if (proc && proc->fn) {
+        frame = create_frame(THIS, THIS->ctx->pool);
+        if (!frame)
+            goto out;
+        ret = proc->fn(frame, THIS, NULL);
+    }
 out:
-        return ret;
+    return ret;
 }
 
 static dict_t *
-make_seq_dict (int argc, char **argv)
+make_seq_dict(int argc, char **argv)
 {
-        char index[] = "4294967296"; // 1<<32
-        int i        = 0;
-        int ret      = 0;
-        dict_t *dict = dict_new ();
-
-        if (!dict)
-                return NULL;
-
-        for (i = 0; i < argc; i++) {
-                snprintf(index, sizeof(index), "%d", i);
-                ret = dict_set_str (dict, index, argv[i]);
-                if (ret == -1)
-                        break;
-        }
-
-        if (ret) {
-                dict_destroy (dict);
-                dict = NULL;
-        }
-
-        return dict;
+    char index[] = "4294967296";  // 1<<32
+    int i = 0;
+    int len;
+    int ret = 0;
+    dict_t *dict = dict_new();
+
+    if (!dict)
+        return NULL;
+
+    for (i = 0; i < argc; i++) {
+        len = snprintf(index, sizeof(index), "%d", i);
+        ret = dict_set_strn(dict, index, len, argv[i]);
+        if (ret == -1)
+            break;
+    }
+
+    if (ret) {
+        dict_unref(dict);
+        dict = NULL;
+    }
+
+    return dict;
 }
 
 int
-cli_cmd_mount_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                   const char **words, int wordcount)
+cli_cmd_mount_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                  const char **words, int wordcount)
 {
-        rpc_clnt_procedure_t *proc = NULL;
-        call_frame_t *frame        = NULL;
-        int ret                    = -1;
-        dict_t *dict               = NULL;
-        void *dataa[]              = {NULL, NULL};
-
-        if (wordcount < 4) {
-                cli_usage_out (word->pattern);
-                goto out;
-        }
-
-        dict = make_seq_dict (wordcount - 3, (char **)words + 3);
-        if (!dict)
-                goto out;
-
-        dataa[0] = (void *)words[2];
-        dataa[1] = dict;
-
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_MOUNT];
-        if (proc && proc->fn) {
-                frame = create_frame (THIS, THIS->ctx->pool);
-                if (!frame)
-                        goto out;
-                ret = proc->fn (frame, THIS, dataa);
-        }
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    int ret = -1;
+    dict_t *dict = NULL;
+    void *dataa[] = {NULL, NULL};
+
+    if (wordcount < 4) {
+        cli_usage_out(word->pattern);
+        goto out;
+    }
+
+    dict = make_seq_dict(wordcount - 3, (char **)words + 3);
+    if (!dict)
+        goto out;
+
+    dataa[0] = (void *)words[2];
+    dataa[1] = dict;
+
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_MOUNT];
+    if (proc && proc->fn) {
+        frame = create_frame(THIS, THIS->ctx->pool);
+        if (!frame)
+            goto out;
+        ret = proc->fn(frame, THIS, dataa);
+    }
 
- out:
-        if (dict)
-                dict_unref (dict);
+out:
+    if (dict)
+        dict_unref(dict);
 
-        if (!proc && ret)
-                cli_out ("Mount command failed");
+    if (!proc && ret)
+        cli_out("Mount command failed");
 
-        return ret;
+    return ret;
 }
 
 int
-cli_cmd_umount_cbk (struct cli_state *state, struct cli_cmd_word *word,
+cli_cmd_umount_cbk(struct cli_state *state, struct cli_cmd_word *word,
                    const char **words, int wordcount)
 {
-        rpc_clnt_procedure_t *proc = NULL;
-        call_frame_t *frame        = NULL;
-        int ret                    = -1;
-        dict_t *dict               = NULL;
-
-        if (!(wordcount == 3 ||
-              (wordcount == 4 && strcmp (words[3], "lazy") == 0))) {
-                cli_usage_out (word->pattern);
-                goto out;
-        }
-
-        dict = dict_new ();
-        if (!dict)
-                goto out;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    int ret = -1;
+    dict_t *dict = NULL;
+
+    if (!(wordcount == 3 ||
+          (wordcount == 4 && strcmp(words[3], "lazy") == 0))) {
+        cli_usage_out(word->pattern);
+        goto out;
+    }
+
+    dict = dict_new();
+    if (!dict)
+        goto out;
+
+    ret = dict_set_str(dict, "path", (char *)words[2]);
+    if (ret != 0)
+        goto out;
+    ret = dict_set_int32(dict, "lazy", wordcount == 4);
+    if (ret != 0)
+        goto out;
+
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_UMOUNT];
+    if (proc && proc->fn) {
+        frame = create_frame(THIS, THIS->ctx->pool);
+        if (!frame) {
+            gf_log(THIS->name, GF_LOG_ERROR, "failed to create frame");
+            ret = -1;
+            goto out;
+        }
+        ret = proc->fn(frame, THIS, dict);
+    }
 
-        ret = dict_set_str (dict, "path", (char *)words[2]);
-        if (ret != 0)
-                goto out;
-        ret = dict_set_int32 (dict, "lazy", wordcount == 4);
-        if (ret != 0)
-                goto out;
-
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_UMOUNT];
-        if (proc && proc->fn) {
-                frame = create_frame (THIS, THIS->ctx->pool);
-                if (!frame)
-                        goto out;
-                ret = proc->fn (frame, THIS, dict);
-        }
-
- out:
-        if (dict)
-                dict_unref (dict);
+out:
+    if (dict)
+        dict_unref(dict);
 
-        if (!proc && ret)
-                cli_out ("Umount command failed");
+    if (!proc && ret)
+        cli_out("Umount command failed");
 
-        return ret;
+    return ret;
 }
 
 int
-cli_cmd_uuid_get_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                      const char **words, int wordcount)
+cli_cmd_uuid_get_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                     const char **words, int wordcount)
 {
-        int                     ret = -1;
-        int                     sent = 0;
-        int                     parse_error = 0;
-        dict_t                  *dict  = NULL;
-        rpc_clnt_procedure_t    *proc = NULL;
-        call_frame_t            *frame = NULL;
-        cli_local_t             *local = NULL;
-        xlator_t                *this  = NULL;
-
-        this = THIS;
-        if (wordcount != 3) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                goto out;
-        }
-
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_UUID_GET];
-        frame = create_frame (this, this->ctx->pool);
-        if (!frame)
-                goto out;
-
-        dict = dict_new ();
-        if (!dict)
-                goto out;
-
-        CLI_LOCAL_INIT (local, words, frame, dict);
-        if (proc->fn)
-                ret = proc->fn (frame, this, dict);
+    int ret = -1;
+    int sent = 0;
+    int parse_error = 0;
+    dict_t *dict = NULL;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    cli_local_t *local = NULL;
+    xlator_t *this = NULL;
+
+    this = THIS;
+    if (wordcount != 3) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
+
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_UUID_GET];
+    frame = create_frame(this, this->ctx->pool);
+    if (!frame)
+        goto out;
+
+    dict = dict_new();
+    if (!dict)
+        goto out;
+
+    CLI_LOCAL_INIT(local, words, frame, dict);
+    if (proc->fn)
+        ret = proc->fn(frame, this, dict);
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_out ("uuid get failed");
-        }
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("uuid get failed");
+    }
 
-        if (dict)
-                dict_unref (dict);
+    if (dict)
+        dict_unref(dict);
 
-        CLI_STACK_DESTROY (frame);
-        return ret;
+    CLI_STACK_DESTROY(frame);
+    return ret;
 }
 
 int
-cli_cmd_uuid_reset_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                        const char **words, int wordcount)
+cli_cmd_uuid_reset_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                       const char **words, int wordcount)
 {
-        int                     ret = -1;
-        rpc_clnt_procedure_t    *proc = NULL;
-        call_frame_t            *frame = NULL;
-        int                     sent = 0;
-        int                     parse_error = 0;
-        gf_answer_t             answer = GF_ANSWER_NO;
-        char                    *question = NULL;
-        cli_local_t             *local = NULL;
-        dict_t                  *dict  = NULL;
-        xlator_t                *this  = NULL;
-
-        question = "Resetting uuid changes the uuid of local glusterd. "
-                   "Do you want to continue?";
-
-        if (wordcount != 3) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                goto out;
-        }
-
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_UUID_RESET];
-
-        this = THIS;
-        frame = create_frame (this, this->ctx->pool);
-        if (!frame)
-                goto out;
-
-        dict = dict_new ();
-        if (!dict) {
-                ret = -1;
-                goto out;
-        }
-        CLI_LOCAL_INIT (local, words, frame, dict);
-        answer = cli_cmd_get_confirmation (state, question);
-
-        if (GF_ANSWER_NO == answer) {
-                ret = 0;
-                goto out;
-        }
-
-        //send NULL as argument since no dictionary is sent to glusterd
-        if (proc->fn) {
-                ret = proc->fn (frame, this, dict);
-        }
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    int sent = 0;
+    int parse_error = 0;
+    gf_answer_t answer = GF_ANSWER_NO;
+    char *question = NULL;
+    cli_local_t *local = NULL;
+    dict_t *dict = NULL;
+    xlator_t *this = NULL;
+
+    question =
+        "Resetting uuid changes the uuid of local glusterd. "
+        "Do you want to continue?";
+
+    if (wordcount != 3) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
+
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_UUID_RESET];
+
+    this = THIS;
+    frame = create_frame(this, this->ctx->pool);
+    if (!frame)
+        goto out;
+
+    dict = dict_new();
+    if (!dict) {
+        ret = -1;
+        goto out;
+    }
+    CLI_LOCAL_INIT(local, words, frame, dict);
+    answer = cli_cmd_get_confirmation(state, question);
+
+    if (GF_ANSWER_NO == answer) {
+        ret = 0;
+        goto out;
+    }
+
+    // send NULL as argument since no dictionary is sent to glusterd
+    if (proc->fn) {
+        ret = proc->fn(frame, this, dict);
+    }
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_out ("uuid reset failed");
-        }
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("uuid reset failed");
+    }
+
+    if (dict)
+        dict_unref(dict);
 
-        CLI_STACK_DESTROY (frame);
+    CLI_STACK_DESTROY(frame);
 
-        return ret;
+    return ret;
 }
 
-struct cli_cmd cli_system_cmds[] = {
-        { "system:: getspec <VOLNAME>",
-          cli_cmd_getspec_cbk,
-          "fetch the volume file for the volume <VOLNAME>"},
+static struct cli_cmd cli_system_cmds[] = {
+    {"system:: getspec <VOLNAME>", cli_cmd_getspec_cbk,
+     "fetch the volume file for the volume <VOLNAME>"},
 
-        { "system:: portmap brick2port <BRICK>",
-          cli_cmd_pmap_b2p_cbk,
-          "query which port <BRICK> listens on"},
+    {"system:: portmap brick2port <BRICK>", cli_cmd_pmap_b2p_cbk,
+     "query which port <BRICK> listens on"},
 
-        { "system:: fsm log [<peer-name>]",
-          cli_cmd_fsm_log_cbk,
-          "display fsm transitions"},
+    {"system:: fsm log [<peer-name>]", cli_cmd_fsm_log_cbk,
+     "display fsm transitions"},
 
-        { "system:: getwd",
-          cli_cmd_getwd_cbk,
-          "query glusterd work directory"},
+    {"system:: getwd", cli_cmd_getwd_cbk, "query glusterd work directory"},
 
-        { "system:: mount <label> <args...>",
-          cli_cmd_mount_cbk,
-          "request a mount"},
+    {"system:: mount <label> <args...>", cli_cmd_mount_cbk, "request a mount"},
 
-        { "system:: umount <path> [lazy]",
-          cli_cmd_umount_cbk,
-          "request an umount"},
+    {"system:: umount <path> [lazy]", cli_cmd_umount_cbk, "request an umount"},
 
-        { "system:: uuid get",
-          cli_cmd_uuid_get_cbk,
-          "get uuid of glusterd"},
+    {"system:: uuid get", cli_cmd_uuid_get_cbk, "get uuid of glusterd"},
 
-        { "system:: uuid reset",
-          cli_cmd_uuid_reset_cbk,
-          "reset the uuid of glusterd"},
+    {"system:: uuid reset", cli_cmd_uuid_reset_cbk,
+     "reset the uuid of glusterd"},
 
-        { "system:: help",
-           cli_cmd_system_help_cbk,
-           "display help for system commands"},
+    {"system:: help", cli_cmd_system_help_cbk,
+     "display help for system commands"},
 
-        { "system:: copy file [<filename>]",
-           cli_cmd_copy_file_cbk,
-           "Copy file from current node's $working_dir to "
-           "$working_dir of all cluster nodes"},
+    {"system:: copy file [<filename>]", cli_cmd_copy_file_cbk,
+     "Copy file from current node's $working_dir to "
+     "$working_dir of all cluster nodes"},
 
-        { "system:: execute <command> <args>",
-           cli_cmd_sys_exec_cbk,
-           "Execute the command on all the nodes "
-           "in the cluster and display their output."},
+    {"system:: execute <command> <args>", cli_cmd_sys_exec_cbk,
+     "Execute the command on all the nodes "
+     "in the cluster and display their output."},
 
-        { NULL, NULL, NULL }
-};
+    {NULL, NULL, NULL}};
 
 int
-cli_cmd_sys_exec_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                      const char **words, int wordcount)
+cli_cmd_sys_exec_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                     const char **words, int wordcount)
 {
-        char                   cmd_arg_name[PATH_MAX] = "";
-        char                  *command                = NULL;
-        char                  *saveptr                = NULL;
-        char                  *tmp                    = NULL;
-        int                    ret                    = -1;
-        int                    i                      = -1;
-        int                    cmd_args_count         = 0;
-        int                    in_cmd_args_count      = 0;
-        rpc_clnt_procedure_t  *proc                   = NULL;
-        call_frame_t          *frame                  = NULL;
-        dict_t                *dict                   = NULL;
-        cli_local_t           *local                  = NULL;
-
-        if (wordcount < 3) {
-                cli_usage_out (word->pattern);
-                goto out;
-        }
-
-        dict = dict_new ();
-        if (!dict)
-                goto out;
-
-        command = strtok_r ((char *)words[2], " ", &saveptr);
-        do {
-                tmp = strtok_r (NULL, " ", &saveptr);
-                if (tmp) {
-                        in_cmd_args_count++;
-                        memset (cmd_arg_name, '\0', sizeof(cmd_arg_name));
-                        snprintf (cmd_arg_name, sizeof(cmd_arg_name),
-                                  "cmd_arg_%d", in_cmd_args_count);
-                        ret = dict_set_str (dict, cmd_arg_name, tmp);
-                        if (ret) {
-                                gf_log ("", GF_LOG_ERROR, "Unable to set "
-                                        "%s in dict", cmd_arg_name);
-                                goto out;
-                        }
-                }
-        } while (tmp);
-
-        cmd_args_count = wordcount - 3;
-
-        ret = dict_set_str (dict, "command", command);
-        if (ret) {
-                gf_log ("", GF_LOG_ERROR, "Unable to set command in dict");
+    char cmd_arg_name[PATH_MAX] = "";
+    char *command = NULL;
+    char *saveptr = NULL;
+    char *tmp = NULL;
+    int ret = -1;
+    int i = -1;
+    int len;
+    int cmd_args_count = 0;
+    int in_cmd_args_count = 0;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *dict = NULL;
+    cli_local_t *local = NULL;
+
+    if ((wordcount < 3) || (words[2] == NULL)) {
+        cli_usage_out(word->pattern);
+        goto out;
+    }
+
+    command = strtok_r((char *)words[2], " ", &saveptr);
+    if (command == NULL) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to parse command");
+        goto out;
+    }
+
+    dict = dict_new();
+    if (!dict)
+        goto out;
+
+    do {
+        tmp = strtok_r(NULL, " ", &saveptr);
+        if (tmp) {
+            in_cmd_args_count++;
+            snprintf(cmd_arg_name, sizeof(cmd_arg_name), "cmd_arg_%d",
+                     in_cmd_args_count);
+            ret = dict_set_str(dict, cmd_arg_name, tmp);
+            if (ret) {
+                gf_log("", GF_LOG_ERROR,
+                       "Unable to set "
+                       "%s in dict",
+                       cmd_arg_name);
                 goto out;
+            }
         }
+    } while (tmp);
 
-        for (i=1; i <= cmd_args_count; i++) {
-                in_cmd_args_count++;
-                memset (cmd_arg_name, '\0', sizeof(cmd_arg_name));
-                snprintf (cmd_arg_name, sizeof(cmd_arg_name),
-                          "cmd_arg_%d", in_cmd_args_count);
-                ret = dict_set_str (dict, cmd_arg_name,
-                                    (char *)words[2+i]);
-                if (ret) {
-                        gf_log ("", GF_LOG_ERROR, "Unable to set %s in dict",
-                               cmd_arg_name);
-                        goto out;
-                }
-        }
+    cmd_args_count = wordcount - 3;
 
-        ret = dict_set_int32 (dict, "cmd_args_count", in_cmd_args_count);
-        if (ret) {
-                gf_log ("", GF_LOG_ERROR,
-                        "Unable to set cmd_args_count in dict");
-                goto out;
-        }
+    ret = dict_set_str(dict, "command", command);
+    if (ret) {
+        gf_log("", GF_LOG_ERROR, "Unable to set command in dict");
+        goto out;
+    }
 
-        ret = dict_set_str (dict, "volname", "N/A");
+    for (i = 1; i <= cmd_args_count; i++) {
+        in_cmd_args_count++;
+        len = snprintf(cmd_arg_name, sizeof(cmd_arg_name), "cmd_arg_%d",
+                       in_cmd_args_count);
+        ret = dict_set_strn(dict, cmd_arg_name, len, (char *)words[2 + i]);
         if (ret) {
-                gf_log ("", GF_LOG_ERROR, "Unable to set volname in dict");
-                goto out;
-        }
-
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_SYS_EXEC];
-        if (proc && proc->fn) {
-                frame = create_frame (THIS, THIS->ctx->pool);
-                if (!frame)
-                        goto out;
-                CLI_LOCAL_INIT (local, words, frame, dict);
-                ret = proc->fn (frame, THIS, (void*)dict);
-        }
+            gf_log("", GF_LOG_ERROR, "Unable to set %s in dict", cmd_arg_name);
+            goto out;
+        }
+    }
+
+    ret = dict_set_int32(dict, "cmd_args_count", in_cmd_args_count);
+    if (ret) {
+        gf_log("", GF_LOG_ERROR, "Unable to set cmd_args_count in dict");
+        goto out;
+    }
+
+    ret = dict_set_str(dict, "volname", "N/A");
+    if (ret) {
+        gf_log("", GF_LOG_ERROR, "Unable to set volname in dict");
+        goto out;
+    }
+
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_SYS_EXEC];
+    if (proc->fn) {
+        frame = create_frame(THIS, THIS->ctx->pool);
+        if (!frame) {
+            gf_log(THIS->name, GF_LOG_ERROR, "failed to create frame");
+            ret = -1;
+            goto out;
+        }
+        CLI_LOCAL_INIT(local, words, frame, dict);
+        ret = proc->fn(frame, THIS, (void *)dict);
+
+        /* proc->fn is processed synchronously, which means that the
+         * execution flow won't return here until the operation is
+         * fully processed, including any related callback. For this
+         * reason, it's safe to destroy the stack here, since no one
+         * can still be using it. Additionally, it's not easy to move
+         * the stack destroy to the callback executed after completion
+         * of the operation because there are multiple things than can
+         * fail even before having queued the callback, so we would
+         * still need to destroy the stack if proc->fn returns an
+         * error. */
+        CLI_STACK_DESTROY(frame);
+        dict = NULL;
+    }
 out:
-        return ret;
+    if (dict != NULL) {
+        dict_unref(dict);
+    }
+
+    return ret;
 }
 
 int
-cli_cmd_copy_file_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                       const char **words, int wordcount)
+cli_cmd_copy_file_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                      const char **words, int wordcount)
 {
-        int                    ret      = -1;
-        rpc_clnt_procedure_t  *proc     = NULL;
-        call_frame_t          *frame    = NULL;
-        char                  *filename = "";
-        dict_t                *dict     = NULL;
-        cli_local_t           *local    = NULL;
-
-        if (wordcount != 4) {
-                cli_usage_out (word->pattern);
-                goto out;
-        }
-
-        dict = dict_new ();
-        if (!dict)
-                goto out;
-
-        filename = (char*)words[3];
-        ret = dict_set_str (dict, "source", filename);
-        if (ret)
-                 gf_log ("", GF_LOG_ERROR, "Unable to set filename in dict");
-
-        ret = dict_set_str (dict, "volname", "N/A");
-        if (ret)
-                 gf_log ("", GF_LOG_ERROR, "Unable to set volname in dict");
-
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_COPY_FILE];
-        if (proc && proc->fn) {
-                frame = create_frame (THIS, THIS->ctx->pool);
-                if (!frame)
-                        goto out;
-                CLI_LOCAL_INIT (local, words, frame, dict);
-                ret = proc->fn (frame, THIS, (void*)dict);
-        }
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    char *filename = "";
+    dict_t *dict = NULL;
+    cli_local_t *local = NULL;
+
+    if (wordcount != 4) {
+        cli_usage_out(word->pattern);
+        goto out;
+    }
+
+    dict = dict_new();
+    if (!dict)
+        goto out;
+
+    filename = (char *)words[3];
+    ret = dict_set_str(dict, "source", filename);
+    if (ret)
+        gf_log("", GF_LOG_ERROR, "Unable to set filename in dict");
+
+    ret = dict_set_str(dict, "volname", "N/A");
+    if (ret)
+        gf_log("", GF_LOG_ERROR, "Unable to set volname in dict");
+
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_COPY_FILE];
+    if (proc && proc->fn) {
+        frame = create_frame(THIS, THIS->ctx->pool);
+        if (!frame) {
+            gf_log(THIS->name, GF_LOG_ERROR, "failed to create frame");
+            ret = -1;
+            goto out;
+        }
+        CLI_LOCAL_INIT(local, words, frame, dict);
+        ret = proc->fn(frame, THIS, (void *)dict);
+    }
 out:
-        return ret;
+    return ret;
 }
 
 int
-cli_cmd_system_help_cbk (struct cli_state *state, struct cli_cmd_word *in_word,
-                         const char **words, int wordcount)
+cli_cmd_system_help_cbk(struct cli_state *state, struct cli_cmd_word *in_word,
+                        const char **words, int wordcount)
 {
-        struct cli_cmd *cmd = NULL;
+    struct cli_cmd *cmd = NULL;
+    struct cli_cmd *system_cmd = NULL;
+    int count = 0;
 
-        for (cmd = cli_system_cmds; cmd->pattern; cmd++)
-                cli_out ("%s - %s", cmd->pattern, cmd->desc);
+    cmd = GF_MALLOC(sizeof(cli_system_cmds), cli_mt_cli_cmd);
+    memcpy(cmd, cli_system_cmds, sizeof(cli_system_cmds));
+    count = (sizeof(cli_system_cmds) / sizeof(struct cli_cmd));
+    cli_cmd_sort(cmd, count);
 
-        return 0;
+    for (system_cmd = cmd; system_cmd->pattern; system_cmd++)
+        cli_out("%s - %s", system_cmd->pattern, system_cmd->desc);
+
+    GF_FREE(cmd);
+    return 0;
 }
 
 int
-cli_cmd_system_register (struct cli_state *state)
+cli_cmd_system_register(struct cli_state *state)
 {
-        int  ret = 0;
-        struct cli_cmd *cmd = NULL;
-
-        for (cmd = cli_system_cmds; cmd->pattern; cmd++) {
+    int ret = 0;
+    struct cli_cmd *cmd = NULL;
 
-                ret = cli_cmd_register (&state->tree, cmd);
-                if (ret)
-                        goto out;
-        }
+    for (cmd = cli_system_cmds; cmd->pattern; cmd++) {
+        ret = cli_cmd_register(&state->tree, cmd);
+        if (ret)
+            goto out;
+    }
 out:
-        return ret;
+    return ret;
 }
diff --git a/cli/src/cli-cmd-volume.c b/cli/src/cli-cmd-volume.c
index 238c8673d75..f238851586e 100644
--- a/cli/src/cli-cmd-volume.c
+++ b/cli/src/cli-cmd-volume.c
@@ -19,2356 +19,3260 @@
 #include <sys/wait.h>
 #include <netinet/in.h>
 
-#ifndef _CONFIG_H
-#define _CONFIG_H
-#include "config.h"
-#endif
-
 #include "cli.h"
 #include "cli-cmd.h"
 #include "cli-mem-types.h"
 #include "cli1-xdr.h"
-#include "run.h"
-#include "syscall.h"
-#include "common-utils.h"
+#include <glusterfs/run.h>
+#include <glusterfs/syscall.h>
+#include <glusterfs/common-utils.h>
+#include <glusterfs/events.h>
 
-extern struct rpc_clnt *global_rpc;
-extern struct rpc_clnt *global_quotad_rpc;
-
-extern rpc_clnt_prog_t *cli_rpc_prog;
 extern rpc_clnt_prog_t cli_quotad_clnt;
 
-int
-cli_cmd_volume_help_cbk (struct cli_state *state, struct cli_cmd_word *in_word,
-                      const char **words, int wordcount);
+static int
+gf_asprintf_append(char **string_ptr, const char *format, ...);
 
 int
-cli_cmd_volume_info_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                         const char **words, int wordcount)
-{
-        int                             ret = -1;
-        rpc_clnt_procedure_t            *proc = NULL;
-        call_frame_t                    *frame = NULL;
-        cli_cmd_volume_get_ctx_t        ctx = {0,};
-        cli_local_t                     *local = NULL;
-        int                             sent = 0;
-        int                             parse_error = 0;
+cli_cmd_volume_help_cbk(struct cli_state *state, struct cli_cmd_word *in_word,
+                        const char **words, int wordcount);
 
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_GET_VOLUME];
+int
+cli_cmd_bitrot_help_cbk(struct cli_state *state, struct cli_cmd_word *in_word,
+                        const char **words, int wordcount);
 
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
+int
+cli_cmd_quota_help_cbk(struct cli_state *state, struct cli_cmd_word *in_word,
+                       const char **words, int wordcount);
 
-        if ((wordcount == 2)  || (wordcount == 3 &&
-                                  !strcmp (words[2], "all"))) {
-                ctx.flags = GF_CLI_GET_NEXT_VOLUME;
-                proc = &cli_rpc_prog->proctable[GLUSTER_CLI_GET_NEXT_VOLUME];
-        } else if (wordcount == 3) {
-                ctx.flags = GF_CLI_GET_VOLUME;
-                ctx.volname = (char *)words[2];
-                if (strlen (ctx.volname) > GD_VOLUME_NAME_MAX) {
-                        cli_out ("Invalid volume name");
-                        goto out;
-                }
-                proc = &cli_rpc_prog->proctable[GLUSTER_CLI_GET_VOLUME];
-        } else {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                return -1;
+int
+cli_cmd_volume_info_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                        const char **words, int wordcount)
+{
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    cli_cmd_volume_get_ctx_t ctx = {
+        0,
+    };
+    cli_local_t *local = NULL;
+    int sent = 0;
+    int parse_error = 0;
+
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_GET_VOLUME];
+
+    if ((wordcount == 2) || (wordcount == 3 && !strcmp(words[2], "all"))) {
+        ctx.flags = GF_CLI_GET_NEXT_VOLUME;
+        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_GET_NEXT_VOLUME];
+    } else if (wordcount == 3) {
+        ctx.flags = GF_CLI_GET_VOLUME;
+        ctx.volname = (char *)words[2];
+        if (strlen(ctx.volname) > GD_VOLUME_NAME_MAX) {
+            cli_out("Invalid volume name");
+            goto out;
         }
+        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_GET_VOLUME];
+    } else {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        return -1;
+    }
 
-        local = cli_local_get ();
+    local = cli_local_get();
 
-        if (!local)
-                goto out;
+    if (!local)
+        goto out;
 
-        local->get_vol.flags = ctx.flags;
-        if (ctx.volname)
-                local->get_vol.volname = gf_strdup (ctx.volname);
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame)
+        goto out;
 
-        frame->local = local;
+    local->get_vol.flags = ctx.flags;
+    if (ctx.volname)
+        local->get_vol.volname = gf_strdup(ctx.volname);
 
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, &ctx);
-        }
+    frame->local = local;
 
-out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_out ("Getting Volume information failed!");
-        }
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, &ctx);
+    }
 
-        CLI_STACK_DESTROY (frame);
+out:
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Getting Volume information failed!");
+    }
 
-        return ret;
+    CLI_STACK_DESTROY(frame);
 
+    return ret;
 }
 
 int
-cli_cmd_sync_volume_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                         const char **words, int wordcount)
+cli_cmd_sync_volume_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                        const char **words, int wordcount)
 {
-        int                     ret = -1;
-        rpc_clnt_procedure_t    *proc = NULL;
-        call_frame_t            *frame = NULL;
-        int                     sent = 0;
-        int                     parse_error = 0;
-        dict_t                  *dict = NULL;
-        cli_local_t             *local = NULL;
-        gf_answer_t             answer = GF_ANSWER_NO;
-        const char              *question = "Sync volume may make data "
-                                            "inaccessible while the sync "
-                                            "is in progress. Do you want "
-                                            "to continue?";
-
-        if ((wordcount < 3) || (wordcount > 4)) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                goto out;
-        }
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    int sent = 0;
+    int parse_error = 0;
+    dict_t *dict = NULL;
+    cli_local_t *local = NULL;
+    gf_answer_t answer = GF_ANSWER_NO;
+    const char *question =
+        "Sync volume may make data "
+        "inaccessible while the sync "
+        "is in progress. Do you want "
+        "to continue?";
+
+    if ((wordcount < 3) || (wordcount > 4)) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
 
-        dict = dict_new ();
-        if (!dict)
-                goto out;
+    dict = dict_new();
+    if (!dict)
+        goto out;
 
-        if ((wordcount == 3) || !strcmp(words[3], "all")) {
-                ret = dict_set_int32 (dict, "flags", (int32_t)
-                                      GF_CLI_SYNC_ALL);
-                if (ret) {
-                        gf_log (THIS->name, GF_LOG_ERROR, "failed to set"
-                                "flag");
-                        goto out;
-                }
-        } else {
-                ret = dict_set_str (dict, "volname", (char *) words[3]);
-                if (ret) {
-                        gf_log (THIS->name, GF_LOG_ERROR, "failed to set "
-                                "volume");
-                        goto out;
-                }
+    if ((wordcount == 3) || !strcmp(words[3], "all")) {
+        ret = dict_set_int32(dict, "flags", (int32_t)GF_CLI_SYNC_ALL);
+        if (ret) {
+            gf_log(THIS->name, GF_LOG_ERROR,
+                   "failed to set"
+                   "flag");
+            goto out;
         }
-
-        ret = dict_set_str (dict, "hostname", (char *) words[2]);
+    } else {
+        ret = dict_set_str(dict, "volname", (char *)words[3]);
         if (ret) {
-                gf_log (THIS->name, GF_LOG_ERROR, "failed to set hostname");
-                goto out;
+            gf_log(THIS->name, GF_LOG_ERROR,
+                   "failed to set "
+                   "volume");
+            goto out;
         }
+    }
 
-        if (!(state->mode & GLUSTER_MODE_SCRIPT)) {
-                answer = cli_cmd_get_confirmation (state, question);
-                if (GF_ANSWER_NO == answer) {
-                        ret = 0;
-                        goto out;
-                }
+    ret = dict_set_str(dict, "hostname", (char *)words[2]);
+    if (ret) {
+        gf_log(THIS->name, GF_LOG_ERROR, "failed to set hostname");
+        goto out;
+    }
+
+    if (!(state->mode & GLUSTER_MODE_SCRIPT)) {
+        answer = cli_cmd_get_confirmation(state, question);
+        if (GF_ANSWER_NO == answer) {
+            ret = 0;
+            goto out;
         }
+    }
 
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_SYNC_VOLUME];
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_SYNC_VOLUME];
 
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        gf_log(THIS->name, GF_LOG_ERROR, "failed to create frame");
+        ret = -1;
+        goto out;
+    }
 
-        CLI_LOCAL_INIT (local, words, frame, dict);
+    CLI_LOCAL_INIT(local, words, frame, dict);
 
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, dict);
-        }
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, dict);
+    }
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_out ("Volume sync failed");
-        }
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Volume sync failed");
+    }
 
-        CLI_STACK_DESTROY (frame);
+    CLI_STACK_DESTROY(frame);
 
-        return ret;
+    return ret;
 }
 
 int
-cli_cmd_volume_create_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                           const char **words, int wordcount)
+cli_cmd_volume_create_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                          const char **words, int wordcount)
 {
-        int                     ret = -1;
-        rpc_clnt_procedure_t    *proc = NULL;
-        call_frame_t            *frame = NULL;
-        dict_t                  *options = NULL;
-        int                     sent = 0;
-        int                     parse_error = 0;
-        char                    *brick_list = NULL;
-        int32_t                 brick_count = 0;
-        int32_t                 sub_count = 0;
-        int32_t                 type = GF_CLUSTER_TYPE_NONE;
-        cli_local_t             *local = NULL;
-        char                    *trans_type = NULL;
-
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_CREATE_VOLUME];
-
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *options = NULL;
+    int sent = 0;
+    int parse_error = 0;
+    cli_local_t *local = NULL;
+    char *trans_type = NULL;
+    char *bricks = NULL;
+
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_CREATE_VOLUME];
+
+    ret = cli_cmd_volume_create_parse(state, words, wordcount, &options,
+                                      &bricks);
+
+    if (ret) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
 
-        ret = cli_cmd_volume_create_parse (state, words, wordcount, &options);
+    ret = dict_get_str(options, "transport", &trans_type);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to get transport type");
+        goto out;
+    }
 
+    if (state->mode & GLUSTER_MODE_WIGNORE) {
+        ret = dict_set_int32(options, "force", _gf_true);
         if (ret) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                goto out;
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to set force "
+                   "option");
+            goto out;
         }
+    }
 
-        ret = dict_get_str (options, "transport", &trans_type);
-        if (ret) {
-                gf_log("cli", GF_LOG_ERROR, "Unable to get transport type");
-                goto out;
-        }
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
 
-        if (state->mode & GLUSTER_MODE_WIGNORE) {
-                ret = dict_set_int32 (options, "force", _gf_true);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to set force "
-                                "option");
-                        goto out;
-                }
-        }
+    CLI_LOCAL_INIT(local, words, frame, options);
 
-        CLI_LOCAL_INIT (local, words, frame, options);
-
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, options);
-        }
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, options);
+    }
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_out ("Volume create failed");
-        }
-
-        CLI_STACK_DESTROY (frame);
-
-        return ret;
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Volume create failed");
+    }
+
+    if (ret == 0) {
+        gf_event(EVENT_VOLUME_CREATE, "name=%s;bricks=%s", (char *)words[2],
+                 bricks);
+    }
+
+    CLI_STACK_DESTROY(frame);
+    return ret;
 }
 
-
 int
-cli_cmd_volume_delete_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                           const char **words, int wordcount)
+cli_cmd_volume_delete_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                          const char **words, int wordcount)
 {
-        int                     ret = -1;
-        rpc_clnt_procedure_t    *proc = NULL;
-        call_frame_t            *frame = NULL;
-        char                    *volname = NULL;
-        gf_answer_t             answer = GF_ANSWER_NO;
-        const char              *question = NULL;
-        int                     sent = 0;
-        int                     parse_error = 0;
-        cli_local_t             *local = NULL;
-        dict_t                  *dict = NULL;
-
-        question = "Deleting volume will erase all information about the volume. "
-                   "Do you want to continue?";
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_DELETE_VOLUME];
-
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
-
-        dict = dict_new ();
-        if (!dict)
-                goto out;
-
-        if (wordcount != 3) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                goto out;
-        }
-
-        answer = cli_cmd_get_confirmation (state, question);
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    char *volname = NULL;
+    gf_answer_t answer = GF_ANSWER_NO;
+    const char *question = NULL;
+    int sent = 0;
+    int parse_error = 0;
+    cli_local_t *local = NULL;
+    dict_t *dict = NULL;
+
+    question =
+        "Deleting volume will erase all information about the volume. "
+        "Do you want to continue?";
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_DELETE_VOLUME];
+
+    if (wordcount != 3) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
 
-        if (GF_ANSWER_NO == answer) {
-                ret = 0;
-                goto out;
-        }
+    volname = (char *)words[2];
 
-        volname = (char *)words[2];
+    dict = dict_new();
+    if (!dict)
+        goto out;
 
-        ret = dict_set_str (dict, "volname", volname);
+    ret = dict_set_str(dict, "volname", volname);
+    if (ret) {
+        gf_log(THIS->name, GF_LOG_WARNING, "dict set failed");
+        goto out;
+    }
+
+    if (!strcmp(volname, GLUSTER_SHARED_STORAGE)) {
+        question =
+            "Deleting the shared storage volume"
+            "(gluster_shared_storage), will affect features "
+            "like snapshot scheduler, geo-replication "
+            "and NFS-Ganesha. Do you still want to "
+            "continue?";
+    }
+
+    answer = cli_cmd_get_confirmation(state, question);
+    if (GF_ANSWER_NO == answer) {
+        ret = 0;
+        goto out;
+    }
 
-        if (ret) {
-                gf_log (THIS->name, GF_LOG_WARNING, "dict set failed");
-                goto out;
-        }
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
 
-        CLI_LOCAL_INIT (local, words, frame, dict);
+    CLI_LOCAL_INIT(local, words, frame, dict);
 
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, dict);
-        }
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, dict);
+    }
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_out ("Volume delete failed");
-        }
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Volume delete failed");
+    }
+
+    CLI_STACK_DESTROY(frame);
 
-        CLI_STACK_DESTROY (frame);
+    if (ret == 0 && GF_ANSWER_YES == answer) {
+        gf_event(EVENT_VOLUME_DELETE, "name=%s", (char *)words[2]);
+    }
 
-        return ret;
+    return ret;
 }
 
 int
-cli_cmd_volume_start_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                          const char **words, int wordcount)
+cli_cmd_volume_start_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                         const char **words, int wordcount)
 {
-        int                     ret = -1;
-        rpc_clnt_procedure_t    *proc = NULL;
-        call_frame_t            *frame = NULL;
-        int                     sent = 0;
-        int                     parse_error = 0;
-        dict_t                  *dict = NULL;
-        int                     flags = 0;
-        cli_local_t             *local = NULL;
-
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    int sent = 0;
+    int parse_error = 0;
+    dict_t *dict = NULL;
+    int flags = 0;
+    cli_local_t *local = NULL;
+
+    if (wordcount < 3 || wordcount > 4) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
 
-        if (wordcount < 3 || wordcount > 4) {
-               cli_usage_out (word->pattern);
-                parse_error = 1;
-               goto out;
-        }
+    if (!words[2])
+        goto out;
 
-        dict = dict_new ();
-        if (!dict) {
-                goto out;
+    if (wordcount == 4) {
+        if (!strcmp("force", words[3])) {
+            flags |= GF_CLI_FLAG_OP_FORCE;
+        } else {
+            ret = -1;
+            cli_usage_out(word->pattern);
+            parse_error = 1;
+            goto out;
         }
+    }
 
-        if (!words[2])
-                goto out;
+    dict = dict_new();
+    if (!dict) {
+        goto out;
+    }
 
-        ret = dict_set_str (dict, "volname", (char *)words[2]);
-        if (ret) {
-                gf_log (THIS->name, GF_LOG_ERROR, "dict set failed");
-                goto out;
-        }
+    ret = dict_set_str(dict, "volname", (char *)words[2]);
+    if (ret) {
+        gf_log(THIS->name, GF_LOG_ERROR, "dict set failed");
+        goto out;
+    }
 
-        if (wordcount == 4) {
-                if (!strcmp("force", words[3])) {
-                        flags |= GF_CLI_FLAG_OP_FORCE;
-                } else {
-                        ret = -1;
-                        cli_usage_out (word->pattern);
-                        parse_error = 1;
-                        goto out;
-                }
-        }
-        ret = dict_set_int32 (dict, "flags", flags);
-        if (ret) {
-                 gf_log (THIS->name, GF_LOG_ERROR,
-                         "dict set failed");
-                 goto out;
-        }
+    ret = dict_set_int32(dict, "flags", flags);
+    if (ret) {
+        gf_log(THIS->name, GF_LOG_ERROR, "dict set failed");
+        goto out;
+    }
 
-        if (ret < 0) {
-                gf_log (THIS->name, GF_LOG_ERROR,
-                        "failed to serialize dict");
-                goto out;
-        }
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_START_VOLUME];
 
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_START_VOLUME];
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
 
-        CLI_LOCAL_INIT (local, words, frame, dict);
+    CLI_LOCAL_INIT(local, words, frame, dict);
 
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, dict);
-        }
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, dict);
+    }
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_out ("Volume start failed");
-        }
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Volume start failed");
+    }
+
+    CLI_STACK_DESTROY(frame);
 
-        CLI_STACK_DESTROY (frame);
+    if (ret == 0) {
+        gf_event(EVENT_VOLUME_START, "name=%s;force=%d", (char *)words[2],
+                 (flags & GF_CLI_FLAG_OP_FORCE));
+    }
 
-        return ret;
+    return ret;
 }
 
 gf_answer_t
-cli_cmd_get_confirmation (struct cli_state *state, const char *question)
+cli_cmd_get_confirmation(struct cli_state *state, const char *question)
 {
-        char                    answer[5] = {'\0', };
-        char                    flush = '\0';
-        size_t			len;
+    char answer[5] = {
+        '\0',
+    };
+    int flush = '\0';
+    size_t len;
 
-        if (state->mode & GLUSTER_MODE_SCRIPT)
-                return GF_ANSWER_YES;
+    if (state->mode & GLUSTER_MODE_SCRIPT)
+        return GF_ANSWER_YES;
 
-	printf ("%s (y/n) ", question);
+    printf("%s (y/n) ", question);
 
-        if (fgets (answer, 4, stdin) == NULL) {
-                cli_out("gluster cli read error");
-                goto out;
-        }
+    if (fgets(answer, 4, stdin) == NULL) {
+        cli_out("gluster cli read error");
+        goto out;
+    }
 
-	len = strlen (answer);
+    len = strlen(answer);
 
-	if (len && answer [len - 1] == '\n'){
-		answer [--len] = '\0';
-	} else {
-		do{
-			flush = getchar ();
-		}while (flush != '\n');
-	}
+    if (len && answer[len - 1] == '\n') {
+        answer[--len] = '\0';
+    } else {
+        do {
+            flush = getchar();
+        } while (flush != '\n');
+    }
 
-	if (len > 3)
-		goto out;
+    if (len > 3)
+        goto out;
 
-	if (!strcasecmp (answer, "y") || !strcasecmp (answer, "yes"))
-		return GF_ANSWER_YES;
+    if (!strcasecmp(answer, "y") || !strcasecmp(answer, "yes"))
+        return GF_ANSWER_YES;
 
-	else if (!strcasecmp (answer, "n") || !strcasecmp (answer, "no"))
-		return GF_ANSWER_NO;
+    else if (!strcasecmp(answer, "n") || !strcasecmp(answer, "no"))
+        return GF_ANSWER_NO;
 
 out:
-	cli_out ("Invalid input, please enter y/n");
+    cli_out("Invalid input, please enter y/n");
 
-	return GF_ANSWER_NO;
+    return GF_ANSWER_NO;
 }
 
 int
-cli_cmd_volume_stop_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                           const char **words, int wordcount)
+cli_cmd_volume_stop_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                        const char **words, int wordcount)
 {
-        int                     ret = -1;
-        rpc_clnt_procedure_t    *proc = NULL;
-        call_frame_t            *frame = NULL;
-        int                     flags   = 0;
-        gf_answer_t             answer = GF_ANSWER_NO;
-        int                     sent = 0;
-        int                     parse_error = 0;
-        dict_t                  *dict = NULL;
-        char                    *volname = NULL;
-        cli_local_t             *local = NULL;
-
-        const char *question = "Stopping volume will make its data inaccessible. "
-                               "Do you want to continue?";
-
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
-
-        if (wordcount < 3 || wordcount > 4) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                goto out;
-        }
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    int flags = 0;
+    gf_answer_t answer = GF_ANSWER_NO;
+    int sent = 0;
+    int parse_error = 0;
+    dict_t *dict = NULL;
+    char *volname = NULL;
+    cli_local_t *local = NULL;
+
+    const char *question =
+        "Stopping volume will make its data inaccessible. "
+        "Do you want to continue?";
+
+    if (wordcount < 3 || wordcount > 4) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
 
-        volname = (char*) words[2];
+    volname = (char *)words[2];
 
-        dict = dict_new ();
-        ret = dict_set_str (dict, "volname", volname);
-        if (ret) {
-                gf_log (THIS->name, GF_LOG_ERROR, "dict set failed");
-                goto out;
-        }
-
-        if (wordcount == 4) {
-                if (!strcmp("force", words[3])) {
-                        flags |= GF_CLI_FLAG_OP_FORCE;
-                } else {
-                        ret = -1;
-                        cli_usage_out (word->pattern);
-                        parse_error = 1;
-                        goto out;
-                }
-        }
-        ret = dict_set_int32 (dict, "flags", flags);
-        if (ret) {
-                gf_log (THIS->name, GF_LOG_ERROR,
-                        "dict set failed");
-                goto out;
+    dict = dict_new();
+    ret = dict_set_str(dict, "volname", volname);
+    if (ret) {
+        gf_log(THIS->name, GF_LOG_ERROR, "dict set failed");
+        goto out;
+    }
+
+    if (!strcmp(volname, GLUSTER_SHARED_STORAGE)) {
+        question =
+            "Stopping the shared storage volume"
+            "(gluster_shared_storage), will affect features "
+            "like snapshot scheduler, geo-replication "
+            "and NFS-Ganesha. Do you still want to "
+            "continue?";
+    }
+
+    if (wordcount == 4) {
+        if (!strcmp("force", words[3])) {
+            flags |= GF_CLI_FLAG_OP_FORCE;
+        } else {
+            ret = -1;
+            cli_usage_out(word->pattern);
+            parse_error = 1;
+            goto out;
         }
+    }
 
-        answer = cli_cmd_get_confirmation (state, question);
+    ret = dict_set_int32(dict, "flags", flags);
+    if (ret) {
+        gf_log(THIS->name, GF_LOG_ERROR, "dict set failed");
+        goto out;
+    }
 
-        if (GF_ANSWER_NO == answer) {
-                ret = 0;
-                goto out;
-        }
+    answer = cli_cmd_get_confirmation(state, question);
 
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_STOP_VOLUME];
+    if (GF_ANSWER_NO == answer) {
+        ret = 0;
+        goto out;
+    }
 
-        CLI_LOCAL_INIT (local, words, frame, dict);
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_STOP_VOLUME];
 
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, dict);
-        }
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
 
-out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_out ("Volume stop on '%s' failed", volname);
-        }
+    CLI_LOCAL_INIT(local, words, frame, dict);
 
-        CLI_STACK_DESTROY (frame);
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, dict);
+    }
 
-        return ret;
+out:
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Volume stop on '%s' failed", volname);
+    }
+
+    CLI_STACK_DESTROY(frame);
+    if (dict)
+        dict_unref(dict);
+
+    if (ret == 0 && GF_ANSWER_YES == answer) {
+        gf_event(EVENT_VOLUME_STOP, "name=%s;force=%d", (char *)words[2],
+                 (flags & GF_CLI_FLAG_OP_FORCE));
+    }
+
+    return ret;
 }
 
-
 int
-cli_cmd_volume_rename_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                           const char **words, int wordcount)
+cli_cmd_volume_rename_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                          const char **words, int wordcount)
 {
-        int                     ret = -1;
-        rpc_clnt_procedure_t    *proc = NULL;
-        call_frame_t            *frame = NULL;
-        dict_t                  *dict = NULL;
-        int                     sent = 0;
-        int                     parse_error = 0;
-
-
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
-
-        dict = dict_new ();
-        if (!dict)
-                goto out;
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *dict = NULL;
+    int sent = 0;
+    int parse_error = 0;
+
+    if (wordcount != 4) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
 
-        if (wordcount != 4) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                goto out;
-        }
+    dict = dict_new();
+    if (!dict)
+        goto out;
 
-        ret = dict_set_str (dict, "old-volname", (char *)words[2]);
+    ret = dict_set_str(dict, "old-volname", (char *)words[2]);
 
-        if (ret)
-                goto out;
+    if (ret)
+        goto out;
 
-        ret = dict_set_str (dict, "new-volname", (char *)words[3]);
+    ret = dict_set_str(dict, "new-volname", (char *)words[3]);
 
-        if (ret)
-                goto out;
+    if (ret)
+        goto out;
 
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_RENAME_VOLUME];
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_RENAME_VOLUME];
 
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, dict);
+    if (proc->fn) {
+        frame = create_frame(THIS, THIS->ctx->pool);
+        if (!frame) {
+            ret = -1;
+            goto out;
         }
+        ret = proc->fn(frame, THIS, dict);
+    }
 
 out:
-        if (dict)
-                dict_destroy (dict);
+    if (dict)
+        dict_unref(dict);
 
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_out ("Volume rename on '%s' failed", (char *)words[2]);
-        }
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Volume rename on '%s' failed", (char *)words[2]);
+    }
 
-        CLI_STACK_DESTROY (frame);
+    CLI_STACK_DESTROY(frame);
 
-        return ret;
+    return ret;
 }
 
 int
-cli_cmd_volume_defrag_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                           const char **words, int wordcount)
+cli_cmd_volume_defrag_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                          const char **words, int wordcount)
 {
-        int                   ret     = -1;
-        rpc_clnt_procedure_t *proc    = NULL;
-        call_frame_t         *frame   = NULL;
-        dict_t               *dict = NULL;
-        int                     sent = 0;
-        int                     parse_error = 0;
-        cli_local_t          *local = NULL;
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *dict = NULL;
+    int sent = 0;
+    int parse_error = 0;
+    cli_local_t *local = NULL;
+#if (USE_EVENTS)
+    eventtypes_t event = EVENT_LAST;
+#endif
+
 #ifdef GF_SOLARIS_HOST_OS
-        cli_out ("Command not supported on Solaris");
-        goto out;
+    cli_out("Command not supported on Solaris");
+    goto out;
 #endif
 
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
+    ret = cli_cmd_volume_defrag_parse(words, wordcount, &dict);
 
-        ret = cli_cmd_volume_defrag_parse (words, wordcount, &dict);
+    if (ret) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+    }
 
-        if (ret) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-        }
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_DEFRAG_VOLUME];
 
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_DEFRAG_VOLUME];
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
 
-        CLI_LOCAL_INIT (local, words, frame, dict);
+    CLI_LOCAL_INIT(local, words, frame, dict);
 
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, dict);
-        }
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, dict);
+    }
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_out ("Volume rebalance failed");
-        }
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Volume rebalance failed");
+    } else {
+#if (USE_EVENTS)
+        if (!(strcmp(words[wordcount - 1], "start")) ||
+            !(strcmp(words[wordcount - 1], "force"))) {
+            event = EVENT_VOLUME_REBALANCE_START;
+        } else if (!strcmp(words[wordcount - 1], "stop")) {
+            event = EVENT_VOLUME_REBALANCE_STOP;
+        }
+
+        if (event != EVENT_LAST)
+            gf_event(event, "volume=%s", (char *)words[2]);
+#endif
+    }
 
-        CLI_STACK_DESTROY (frame);
+    CLI_STACK_DESTROY(frame);
 
-        return ret;
+    return ret;
 }
 
 int
-cli_cmd_volume_reset_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                          const char **words, int wordcount)
+cli_cmd_volume_reset_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                         const char **words, int wordcount)
 {
-        int                     sent = 0;
-        int                     parse_error = 0;
-        int                     ret = -1;
-        rpc_clnt_procedure_t    *proc = NULL;
-        call_frame_t            *frame = NULL;
-        dict_t                  *options = NULL;
-        cli_local_t             *local = NULL;
+    int sent = 0;
+    int parse_error = 0;
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *options = NULL;
+    cli_local_t *local = NULL;
+#if (USE_EVENTS)
+    int ret1 = -1;
+    char *tmp_opt = NULL;
+#endif
 
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_RESET_VOLUME];
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_RESET_VOLUME];
 
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
+    ret = cli_cmd_volume_reset_parse(words, wordcount, &options);
+    if (ret) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
 
-        ret = cli_cmd_volume_reset_parse (words, wordcount, &options);
-        if (ret) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                goto out;
-        }
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
 
-        CLI_LOCAL_INIT (local, words, frame, options);
+    CLI_LOCAL_INIT(local, words, frame, options);
 
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, options);
-        }
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, options);
+    }
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_out ("Volume reset failed");
-        }
-
-        CLI_STACK_DESTROY (frame);
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Volume reset failed");
+    }
+
+#if (USE_EVENTS)
+    if (ret == 0) {
+        ret1 = dict_get_str(options, "key", &tmp_opt);
+        if (ret1)
+            tmp_opt = "";
+
+        gf_event(EVENT_VOLUME_RESET, "name=%s;option=%s", (char *)words[2],
+                 tmp_opt);
+    }
+#endif
 
-        return ret;
+    CLI_STACK_DESTROY(frame);
 
+    return ret;
 }
 
 int
-cli_cmd_volume_profile_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                          const char **words, int wordcount)
+cli_cmd_volume_profile_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                           const char **words, int wordcount)
 {
-        int                     sent = 0;
-        int                     parse_error = 0;
+    int sent = 0;
+    int parse_error = 0;
 
-        int                     ret      = -1;
-        rpc_clnt_procedure_t    *proc    = NULL;
-        call_frame_t            *frame   = NULL;
-        dict_t                  *options = NULL;
-        cli_local_t             *local = NULL;
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *options = NULL;
+    cli_local_t *local = NULL;
 
-        ret = cli_cmd_volume_profile_parse (words, wordcount, &options);
+    ret = cli_cmd_volume_profile_parse(words, wordcount, &options);
 
-        if (ret) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                goto out;
-        }
+    if (ret) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
 
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_PROFILE_VOLUME];
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_PROFILE_VOLUME];
 
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        gf_log(THIS->name, GF_LOG_ERROR, "failed to create frame");
+        ret = -1;
+        goto out;
+    }
 
-        CLI_LOCAL_INIT (local, words, frame, options);
+    CLI_LOCAL_INIT(local, words, frame, options);
 
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, options);
-        }
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, options);
+    }
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_out ("Volume profile failed");
-        }
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Volume profile failed");
+    }
 
-        CLI_STACK_DESTROY (frame);
-
-        return ret;
+    CLI_STACK_DESTROY(frame);
 
+    return ret;
 }
 
 int
-cli_cmd_volume_set_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                        const char **words, int wordcount)
+cli_cmd_volume_set_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                       const char **words, int wordcount)
 {
-        int                     sent = 0;
-        int                     parse_error = 0;
-
-	int                     ret = -1;
-        rpc_clnt_procedure_t    *proc = NULL;
-        call_frame_t            *frame = NULL;
-        dict_t                  *options = NULL;
-        cli_local_t             *local = NULL;
-        char                    *op_errstr = NULL;
+    int sent = 0;
+    int parse_error = 0;
+
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *options = NULL;
+    cli_local_t *local = NULL;
+    char *op_errstr = NULL;
+
+#if (USE_EVENTS)
+    int ret1 = -1;
+    int i = 1;
+    char dict_key[50] = {
+        0,
+    };
+    char *tmp_opt = NULL;
+    char *opts_str = NULL;
+    int num_options = 0;
+#endif
 
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_SET_VOLUME];
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_SET_VOLUME];
 
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
+    ret = cli_cmd_volume_set_parse(state, words, wordcount, &options,
+                                   &op_errstr);
+    if (ret) {
+        if (op_errstr) {
+            cli_err("%s", op_errstr);
+            GF_FREE(op_errstr);
+        } else
+            cli_usage_out(word->pattern);
 
-        ret = cli_cmd_volume_set_parse (words, wordcount, &options, &op_errstr);
-        if (ret) {
-                if (op_errstr) {
-                    cli_err ("%s", op_errstr);
-                    GF_FREE (op_errstr);
-                } else
-                    cli_usage_out (word->pattern);
+        parse_error = 1;
+        goto out;
+    }
 
-                parse_error = 1;
-                goto out;
-        }
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
 
-        CLI_LOCAL_INIT (local, words, frame, options);
+    CLI_LOCAL_INIT(local, words, frame, options);
 
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, options);
-        }
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, options);
+    }
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_out ("Volume set failed");
-        }
-
-        CLI_STACK_DESTROY (frame);
-
-        return ret;
-
-}
-
-int
-cli_cmd_volume_add_brick_cbk (struct cli_state *state,
-                              struct cli_cmd_word *word, const char **words,
-                              int wordcount)
-{
-        int                     ret = -1;
-        rpc_clnt_procedure_t    *proc = NULL;
-        call_frame_t            *frame = NULL;
-        dict_t                  *options = NULL;
-        int                     sent = 0;
-        int                     parse_error = 0;
-        gf_answer_t             answer = GF_ANSWER_NO;
-        cli_local_t             *local = NULL;
-
-        const char *question = "Changing the 'stripe count' of the volume is "
-                "not a supported feature. In some cases it may result in data "
-                "loss on the volume. Also there may be issues with regular "
-                "filesystem operations on the volume after the change. Do you "
-                "really want to continue with 'stripe' count option ? ";
-
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
-
-        ret = cli_cmd_volume_add_brick_parse (words, wordcount, &options);
-        if (ret) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                goto out;
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Volume set failed");
+    }
+
+#if (USE_EVENTS)
+    if (ret == 0 && strcmp(words[2], "help") != 0) {
+        ret1 = dict_get_int32(options, "count", &num_options);
+        if (ret1) {
+            num_options = 0;
+            goto end;
+        } else {
+            num_options = num_options / 2;
         }
 
-        /* TODO: there are challenges in supporting changing of
-           stripe-count, until it is properly supported give warning to user */
-        if (dict_get (options, "stripe-count")) {
-                answer = cli_cmd_get_confirmation (state, question);
-
-                if (GF_ANSWER_NO == answer) {
-                        ret = 0;
-                        goto out;
-                }
+        char *free_list_key[num_options];
+        char *free_list_val[num_options];
+        for (i = 0; i < num_options; i++) {
+            free_list_key[i] = NULL;
+            free_list_val[i] = NULL;
         }
+        /* Initialize opts_str */
+        opts_str = "";
 
-        if (state->mode & GLUSTER_MODE_WIGNORE) {
-                ret = dict_set_int32 (options, "force", _gf_true);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to set force "
-                                "option");
-                        goto out;
-                }
-        }
+        /* Prepare String in format options=KEY1,VALUE1,KEY2,VALUE2 */
+        for (i = 1; i <= num_options; i++) {
+            sprintf(dict_key, "key%d", i);
+            ret1 = dict_get_str(options, dict_key, &tmp_opt);
+            if (ret1)
+                tmp_opt = "";
 
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_ADD_BRICK];
+            gf_asprintf(&opts_str, "%s,%s", opts_str, tmp_opt);
+            free_list_key[i - 1] = opts_str;
 
-        CLI_LOCAL_INIT (local, words, frame, options);
+            sprintf(dict_key, "value%d", i);
+            ret1 = dict_get_str(options, dict_key, &tmp_opt);
+            if (ret1)
+                tmp_opt = "";
 
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, options);
+            gf_asprintf(&opts_str, "%s,%s", opts_str, tmp_opt);
+            free_list_val[i - 1] = opts_str;
         }
 
-out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_out ("Volume add-brick failed");
+        gf_event(EVENT_VOLUME_SET, "name=%s;options=%s", (char *)words[2],
+                 opts_str);
+
+        /* Allocated by gf_strdup and gf_asprintf */
+        for (i = 0; i < num_options; i++) {
+            GF_FREE(free_list_key[i]);
+            GF_FREE(free_list_val[i]);
         }
+    }
+#endif
 
-        CLI_STACK_DESTROY (frame);
+end:
+    CLI_STACK_DESTROY(frame);
 
-        return ret;
+    return ret;
 }
 
 static int
-gf_cli_create_auxiliary_mount (char *volname)
+cli_event_remove_brick_str(dict_t *options, char **event_str,
+                           eventtypes_t *event)
 {
-        int      ret                     = -1;
-        char     mountdir[PATH_MAX]      = {0,};
-        char     pidfile_path[PATH_MAX]  = {0,};
-        char     logfile[PATH_MAX]       = {0,};
-        char     qpid [16]               = {0,};
-
-        GLUSTERFS_GET_AUX_MOUNT_PIDFILE (pidfile_path, volname);
-
-        if (gf_is_service_running (pidfile_path, NULL)) {
-                gf_log ("cli", GF_LOG_DEBUG, "Aux mount of volume %s is running"
-                        " already", volname);
-                ret = 0;
-                goto out;
-        }
+    int ret = -1;
+    char *bricklist = NULL;
+    char *brick = NULL;
+    char *volname = NULL;
+    char key[256] = {
+        0,
+    };
+    const char *eventstrformat = "volume=%s;bricks=%s";
+    int32_t command = 0;
+    int32_t i = 1;
+    int32_t count = 0;
+    int32_t eventstrlen = 1;
+    int bricklen = 0;
+    char *tmp_ptr = NULL;
+
+    if (!options || !event_str || !event)
+        goto out;
 
-        GLUSTERD_GET_QUOTA_AUX_MOUNT_PATH (mountdir, volname, "/");
-        ret = mkdir (mountdir, 0777);
-        if (ret && errno != EEXIST) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to create auxiliary mount "
-                        "directory %s. Reason : %s", mountdir,
-                        strerror (errno));
-                goto out;
-        }
+    ret = dict_get_str(options, "volname", &volname);
+    if (ret || !volname) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to fetch volname");
+        ret = -1;
+        goto out;
+    }
+    /* Get the list of bricks for the event */
+    ret = dict_get_int32(options, "command", &command);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to fetch command");
+        ret = -1;
+        goto out;
+    }
+
+    switch (command) {
+        case GF_OP_CMD_START:
+            *event = EVENT_VOLUME_REMOVE_BRICK_START;
+            break;
+        case GF_OP_CMD_COMMIT:
+            *event = EVENT_VOLUME_REMOVE_BRICK_COMMIT;
+            break;
+        case GF_OP_CMD_COMMIT_FORCE:
+            *event = EVENT_VOLUME_REMOVE_BRICK_FORCE;
+            break;
+        case GF_OP_CMD_STOP:
+            *event = EVENT_VOLUME_REMOVE_BRICK_STOP;
+            break;
+        default:
+            *event = EVENT_LAST;
+            break;
+    }
 
-        snprintf (logfile, PATH_MAX-1, "%s/quota-mount-%s.log",
-                  DEFAULT_LOG_FILE_DIRECTORY, volname);
-        snprintf(qpid, 15, "%d", GF_CLIENT_PID_QUOTA_MOUNT);
+    ret = -1;
 
-        ret = runcmd (SBIN_DIR"/glusterfs",
-                      "-s", "localhost",
-                      "--volfile-id", volname,
-                      "-l", logfile,
-                      "-p", pidfile_path,
-                      "--client-pid", qpid,
-                      mountdir,
-                      NULL);
+    if (*event == EVENT_LAST) {
+        goto out;
+    }
 
+    /* I could just get this from words[] but this is cleaner in case the
+     * format changes  */
+    while (i) {
+        snprintf(key, sizeof(key), "brick%d", i);
+        ret = dict_get_str(options, key, &brick);
         if (ret) {
-                gf_log ("cli", GF_LOG_WARNING, "failed to mount glusterfs "
-                        "client. Please check the log file %s for more details",
-                        logfile);
-                ret = -1;
-                goto out;
+            break;
         }
+        eventstrlen += strlen(brick) + 1;
+        i++;
+    }
 
-        ret = 0;
+    count = --i;
 
-out:
-        return ret;
-}
+    eventstrlen += 1;
 
-static int
-cli_stage_quota_op (char *volname, int op_code)
-{
-        int ret = -1;
-
-        switch (op_code) {
-                case GF_QUOTA_OPTION_TYPE_ENABLE:
-                case GF_QUOTA_OPTION_TYPE_LIMIT_USAGE:
-                case GF_QUOTA_OPTION_TYPE_REMOVE:
-                case GF_QUOTA_OPTION_TYPE_LIST:
-                        ret = gf_cli_create_auxiliary_mount (volname);
-                        if (ret) {
-                                cli_err ("quota: Could not start quota "
-                                         "auxiliary mount");
-                                goto out;
-                        }
-                        ret = 0;
-                        break;
-
-                default:
-                        ret = 0;
-                        break;
-        }
+    bricklist = GF_CALLOC(eventstrlen, sizeof(char), gf_common_mt_char);
+    if (!bricklist) {
+        gf_log(THIS->name, GF_LOG_ERROR,
+               "memory allocation failed for"
+               "bricklist");
+        ret = -1;
+        goto out;
+    }
 
-out:
-        return ret;
-}
+    tmp_ptr = bricklist;
 
-static void
-print_quota_list_header (void)
-{
-        //Header
-        cli_out ("                  Path                   Hard-limit "
-                 "Soft-limit   Used  Available  Soft-limit exceeded?"
-                 " Hard-limit exceeded?");
-        cli_out ("-----------------------------------------------------"
-                 "-----------------------------------------------------"
-                 "-----------------");
+    i = 1;
+    while (i <= count) {
+        snprintf(key, sizeof(key), "brick%d", i);
+        ret = dict_get_str(options, key, &brick);
+        if (ret) {
+            break;
+        }
+        snprintf(tmp_ptr, eventstrlen, "%s ", brick);
+        bricklen = strlen(brick);
+        eventstrlen -= (bricklen + 1);
+        tmp_ptr += (bricklen + 1);
+        i++;
+    }
+
+    if (!ret) {
+        gf_asprintf(event_str, eventstrformat, volname, bricklist);
+    } else {
+        gf_asprintf(event_str, eventstrformat, volname, "<unavailable>");
+    }
+
+    ret = 0;
+out:
+    GF_FREE(bricklist);
+    return ret;
 }
 
 int
-cli_get_soft_limit (dict_t *options, const char **words, dict_t *xdata)
+cli_cmd_volume_add_brick_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                             const char **words, int wordcount)
 {
-        call_frame_t            *frame          = NULL;
-        cli_local_t             *local          = NULL;
-        rpc_clnt_procedure_t    *proc           = NULL;
-        char                    *default_sl     = NULL;
-        char                    *default_sl_dup = NULL;
-        int                      ret  = -1;
-
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame) {
-                ret = -1;
-                goto out;
-        }
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *options = NULL;
+    int sent = 0;
+    int parse_error = 0;
+    gf_answer_t answer = GF_ANSWER_NO;
+    cli_local_t *local = NULL;
+
+#if (USE_EVENTS)
+    char *event_str = NULL;
+    char *bricks = NULL;
+    const char *eventstrformat = "volume=%s;bricks=%s";
+#endif
 
-        //We need a ref on @options to prevent CLI_STACK_DESTROY
-        //from destroying it prematurely.
-        dict_ref (options);
-        CLI_LOCAL_INIT (local, words, frame, options);
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_QUOTA];
-        ret = proc->fn (frame, THIS, options);
+    const char *question =
+        "Changing the 'stripe count' of the volume is "
+        "not a supported feature. In some cases it may result in data "
+        "loss on the volume. Also there may be issues with regular "
+        "filesystem operations on the volume after the change. Do you "
+        "really want to continue with 'stripe' count option ? ";
+
+    ret = cli_cmd_volume_add_brick_parse(state, words, wordcount, &options, 0);
+    if (ret) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
 
-        ret = dict_get_str (options, "default-soft-limit", &default_sl);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get default soft limit");
-                goto out;
-        }
+    /* TODO: there are challenges in supporting changing of
+       stripe-count, until it is properly supported give warning to user */
+    if (dict_get(options, "stripe-count")) {
+        answer = cli_cmd_get_confirmation(state, question);
 
-        default_sl_dup = gf_strdup (default_sl);
-        if (!default_sl_dup) {
-                ret = -1;
-                goto out;
+        if (GF_ANSWER_NO == answer) {
+            ret = 0;
+            goto out;
         }
+    }
+
+#if (USE_EVENTS)
+    /* Get the list of bricks for the event */
+
+    ret = dict_get_str(options, "bricks", &bricks);
 
-        ret = dict_set_dynstr (xdata, "default-soft-limit", default_sl_dup);
+    if (!ret) {
+        gf_asprintf(&event_str, eventstrformat, (char *)words[2],
+                    &bricks[1] /*Skip leading space*/);
+    } else {
+        gf_asprintf(&event_str, eventstrformat, (char *)words[2],
+                    "<unavailable>");
+    }
+#endif
+
+    if (state->mode & GLUSTER_MODE_WIGNORE) {
+        ret = dict_set_int32(options, "force", _gf_true);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to set default soft limit");
-                GF_FREE (default_sl_dup);
-                goto out;
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to set force "
+                   "option");
+            goto out;
         }
+    }
+
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_ADD_BRICK];
+
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
+
+    CLI_LOCAL_INIT(local, words, frame, options);
+
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, options);
+    }
 
 out:
-        CLI_STACK_DESTROY (frame);
-        return ret;
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Volume add-brick failed");
+    } else {
+#if (USE_EVENTS)
+        gf_event(EVENT_VOLUME_ADD_BRICK, "%s", event_str);
+#endif
+    }
+#if (USE_EVENTS)
+    GF_FREE(event_str);
+#endif
+
+    CLI_STACK_DESTROY(frame);
+    return ret;
 }
 
-#define QUOTA_CONF_HEADER                                                \
-        "GlusterFS Quota conf | version: v%d.%d\n"
 int
-cli_cmd_quota_conf_skip_header (int fd)
+cli_get_soft_limit(dict_t *options, const char **words, dict_t *xdata)
 {
-        char buf[PATH_MAX] = {0,};
-
-        snprintf (buf, sizeof(buf)-1, QUOTA_CONF_HEADER, 1, 1);
-        return gf_skip_header_section (fd, strlen (buf));
-}
+    call_frame_t *frame = NULL;
+    cli_local_t *local = NULL;
+    rpc_clnt_procedure_t *proc = NULL;
+    char *default_sl = NULL;
+    char *default_sl_dup = NULL;
+    int ret = -1;
+
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
+
+    // We need a ref on @options to prevent CLI_STACK_DESTROY
+    // from destroying it prematurely.
+    dict_ref(options);
+    CLI_LOCAL_INIT(local, words, frame, options);
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_QUOTA];
+    ret = proc->fn(frame, THIS, options);
+
+    ret = dict_get_str(options, "default-soft-limit", &default_sl);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get default soft limit");
+        goto out;
+    }
 
-/* Checks if at least one limit has been set on the volume
- *
- * Returns true if at least one limit is set. Returns false otherwise.
- */
-gf_boolean_t
-_limits_set_on_volume (char *volname) {
-        gf_boolean_t    limits_set = _gf_false;
-        int             ret = -1;
-        char            quota_conf_file[PATH_MAX] = {0,};
-        int             fd = -1;
-        char            buf[16] = {0,};
-
-        /* TODO: fix hardcoding; Need to perform an RPC call to glusterd
-         * to fetch working directory
-         */
-        sprintf (quota_conf_file, "%s/vols/%s/quota.conf",
-                 GLUSTERD_DEFAULT_WORKDIR,
-                 volname);
-        fd = open (quota_conf_file, O_RDONLY);
-        if (fd == -1)
-                goto out;
+    default_sl_dup = gf_strdup(default_sl);
+    if (!default_sl_dup) {
+        ret = -1;
+        goto out;
+    }
 
-        ret = cli_cmd_quota_conf_skip_header (fd);
-        if (ret)
-                goto out;
+    ret = dict_set_dynstr(xdata, "default-soft-limit", default_sl_dup);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to set default soft limit");
+        GF_FREE(default_sl_dup);
+        goto out;
+    }
 
-        /* Try to read atleast one gfid */
-        ret = read (fd, (void *)buf, 16);
-        if (ret == 16)
-                limits_set = _gf_true;
 out:
-        if (fd != -1)
-                close (fd);
-        return limits_set;
+    CLI_STACK_DESTROY(frame);
+    return ret;
 }
 
-/* Checks if the mount is connected to the bricks
+/* Checks if at least one limit has been set on the volume
  *
- * Returns true if connected and false if not
+ * Returns true if at least one limit is set. Returns false otherwise.
  */
 gf_boolean_t
-_quota_aux_mount_online (char *volname)
+_limits_set_on_volume(char *volname, int type)
 {
-        int         ret = 0;
-        char        mount_path[PATH_MAX + 1] = {0,};
-        struct stat buf = {0,};
+    gf_boolean_t limits_set = _gf_false;
+    int ret = -1;
+    char quota_conf_file[PATH_MAX] = {
+        0,
+    };
+    int fd = -1;
+    char buf[16] = {
+        0,
+    };
+    float version = 0.0f;
+    char gfid_type_stored = 0;
+    char gfid_type = 0;
+
+    /* TODO: fix hardcoding; Need to perform an RPC call to glusterd
+     * to fetch working directory
+     */
+    snprintf(quota_conf_file, sizeof quota_conf_file, "%s/vols/%s/quota.conf",
+             GLUSTERD_DEFAULT_WORKDIR, volname);
+    fd = open(quota_conf_file, O_RDONLY);
+    if (fd == -1)
+        goto out;
 
-        GF_ASSERT (volname);
+    ret = quota_conf_read_version(fd, &version);
+    if (ret)
+        goto out;
 
-        /* Try to create the aux mount before checking if bricks are online */
-        ret = gf_cli_create_auxiliary_mount (volname);
-        if (ret) {
-                cli_err ("quota: Could not start quota auxiliary mount");
-                return _gf_false;
-        }
+    if (type == GF_QUOTA_OPTION_TYPE_LIST)
+        gfid_type = GF_QUOTA_CONF_TYPE_USAGE;
+    else
+        gfid_type = GF_QUOTA_CONF_TYPE_OBJECTS;
 
-        GLUSTERD_GET_QUOTA_AUX_MOUNT_PATH (mount_path, volname, "/");
+    /* Try to read at least one gfid  of type 'gfid_type' */
+    while (1) {
+        ret = quota_conf_read_gfid(fd, buf, &gfid_type_stored, version);
+        if (ret <= 0)
+            break;
 
-        ret = sys_stat (mount_path, &buf);
-        if (ret) {
-                if (ENOTCONN == errno) {
-                        cli_err ("quota: Cannot connect to bricks. Check if "
-                                 "bricks are online.");
-                } else {
-                        cli_err ("quota: Error on quota auxiliary mount (%s).",
-                                 strerror (errno));
-                }
-                return _gf_false;
+        if (gfid_type_stored == gfid_type) {
+            limits_set = _gf_true;
+            break;
         }
-        return _gf_true;
+    }
+out:
+    if (fd != -1)
+        sys_close(fd);
+
+    return limits_set;
 }
 
 int
-cli_cmd_quota_handle_list_all (const char **words, dict_t *options)
+cli_cmd_quota_handle_list_all(const char **words, dict_t *options)
 {
-        int                      all_failed = 1;
-        int                      count      = 0;
-        int                      ret        = -1;
-        rpc_clnt_procedure_t    *proc       = NULL;
-        cli_local_t             *local      = NULL;
-        call_frame_t            *frame      = NULL;
-        dict_t                  *xdata      = NULL;
-        char                    *gfid_str   = NULL;
-        char                    *volname    = NULL;
-        char                    *volname_dup = NULL;
-        unsigned char            buf[16]   = {0};
-        int                      fd        = -1;
-        char                     quota_conf_file[PATH_MAX] = {0};
-
-        xdata = dict_new ();
-        if (!xdata) {
-                ret = -1;
-                goto out;
-        }
+    int all_failed = 1;
+    int count = 0;
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    cli_local_t *local = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *xdata = NULL;
+    char gfid_str[UUID_CANONICAL_FORM_LEN + 1];
+    char *volname = NULL;
+    char *volname_dup = NULL;
+    unsigned char buf[16] = {0};
+    int fd = -1;
+    char quota_conf_file[PATH_MAX] = {0};
+    gf_boolean_t xml_err_flag = _gf_false;
+    char err_str[NAME_MAX] = {
+        0,
+    };
+    int32_t type = 0;
+    char gfid_type = 0;
+    float version = 0.0f;
+    int32_t max_count = 0;
+
+    xdata = dict_new();
+    if (!xdata) {
+        ret = -1;
+        goto out;
+    }
 
-        ret = dict_get_str (options, "volname", &volname);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get volume name");
-                goto out;
-        }
+    ret = dict_get_str(options, "volname", &volname);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get volume name");
+        goto out;
+    }
 
-        ret = cli_get_soft_limit (options, words, xdata);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to fetch default "
-                        "soft-limit");
-                goto out;
-        }
+    ret = dict_get_int32(options, "type", &type);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get quota option type");
+        goto out;
+    }
 
-        /* Check if at least one limit is set on volume. No need to check for
-         * quota enabled as cli_get_soft_limit() handles that
-         */
-        if (!_limits_set_on_volume (volname)) {
-                cli_out ("quota: No quota configured on volume %s", volname);
-                ret = 0;
-                goto out;
-        }
+    ret = dict_set_int32(xdata, "type", type);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to set type in xdata");
+        goto out;
+    }
 
-        /* Check if the mount is online before doing any listing */
-        if (!_quota_aux_mount_online (volname)) {
-                ret = -1;
-                goto out;
+    ret = cli_get_soft_limit(options, words, xdata);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Failed to fetch default "
+               "soft-limit");
+        goto out;
+    }
+
+    /* Check if at least one limit is set on volume. No need to check for
+     * quota enabled as cli_get_soft_limit() handles that
+     */
+    if (!_limits_set_on_volume(volname, type)) {
+        snprintf(err_str, sizeof(err_str),
+                 "No%s quota configured on"
+                 " volume %s",
+                 (type == GF_QUOTA_OPTION_TYPE_LIST) ? "" : " inode", volname);
+        if (global_state->mode & GLUSTER_MODE_XML) {
+            xml_err_flag = _gf_true;
+        } else {
+            cli_out("quota: %s", err_str);
         }
+        ret = 0;
+        goto out;
+    }
 
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame) {
-                ret = -1;
-                goto out;
-        }
+    volname_dup = gf_strdup(volname);
+    if (!volname_dup) {
+        ret = -1;
+        goto out;
+    }
 
-        volname_dup = gf_strdup (volname);
-        if (!volname_dup) {
-                ret = -1;
-                goto out;
-        }
+    ret = dict_set_dynstr(xdata, "volume-uuid", volname_dup);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to set volume-uuid");
+        GF_FREE(volname_dup);
+        goto out;
+    }
+
+    // TODO: fix hardcoding; Need to perform an RPC call to glusterd
+    // to fetch working directory
+    snprintf(quota_conf_file, sizeof quota_conf_file, "%s/vols/%s/quota.conf",
+             GLUSTERD_DEFAULT_WORKDIR, volname);
+    fd = open(quota_conf_file, O_RDONLY);
+    if (fd == -1) {
+        // This may because no limits were yet set on the volume
+        gf_log("cli", GF_LOG_TRACE,
+               "Unable to open "
+               "quota.conf");
+        ret = 0;
+        goto out;
+    }
 
-        ret = dict_set_dynstr (xdata, "volume-uuid", volname_dup);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to set volume-uuid");
-                GF_FREE (volname_dup);
-                goto out;
-        }
+    ret = quota_conf_read_version(fd, &version);
+    if (ret)
+        goto out;
 
-        //TODO: fix hardcoding; Need to perform an RPC call to glusterd
-        //to fetch working directory
-        sprintf (quota_conf_file, "%s/vols/%s/quota.conf",
-                 GLUSTERD_DEFAULT_WORKDIR,
-                 volname);
-        fd = open (quota_conf_file, O_RDONLY);
-        if (fd == -1) {
-                //This may because no limits were yet set on the volume
-                gf_log ("cli", GF_LOG_TRACE, "Unable to open "
-                        "quota.conf");
-                ret = 0;
-                goto out;
-         }
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
+
+    CLI_LOCAL_INIT(local, words, frame, xdata);
+    proc = &cli_quotad_clnt.proctable[GF_AGGREGATOR_GETLIMIT];
+
+    for (count = 0;; count++) {
+        ret = quota_conf_read_gfid(fd, buf, &gfid_type, version);
+        if (ret == 0) {
+            break;
+        } else if (ret < 0) {
+            gf_log(THIS->name, GF_LOG_CRITICAL,
+                   "Quota "
+                   "configuration store may be corrupt.");
+            goto out;
+        }
+
+        if ((type == GF_QUOTA_OPTION_TYPE_LIST &&
+             gfid_type == GF_QUOTA_CONF_TYPE_OBJECTS) ||
+            (type == GF_QUOTA_OPTION_TYPE_LIST_OBJECTS &&
+             gfid_type == GF_QUOTA_CONF_TYPE_USAGE))
+            continue;
+
+        max_count++;
+    }
+    ret = dict_set_int32(xdata, "max_count", max_count);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to set max_count");
+        goto out;
+    }
+
+    ret = sys_lseek(fd, 0L, SEEK_SET);
+    if (ret < 0) {
+        gf_log(THIS->name, GF_LOG_ERROR,
+               "failed to move offset to "
+               "the beginning: %s",
+               strerror(errno));
+        goto out;
+    }
+    ret = quota_conf_read_version(fd, &version);
+    if (ret)
+        goto out;
 
-        ret = cli_cmd_quota_conf_skip_header (fd);
+    for (count = 0;; count++) {
+        ret = quota_conf_read_gfid(fd, buf, &gfid_type, version);
+        if (ret == 0) {
+            break;
+        } else if (ret < 0) {
+            gf_log(THIS->name, GF_LOG_CRITICAL,
+                   "Quota "
+                   "configuration store may be corrupt.");
+            goto out;
+        }
+
+        if ((type == GF_QUOTA_OPTION_TYPE_LIST &&
+             gfid_type == GF_QUOTA_CONF_TYPE_OBJECTS) ||
+            (type == GF_QUOTA_OPTION_TYPE_LIST_OBJECTS &&
+             gfid_type == GF_QUOTA_CONF_TYPE_USAGE))
+            continue;
+
+        uuid_utoa_r(buf, gfid_str);
+        ret = dict_set_str(xdata, "gfid", gfid_str);
         if (ret) {
-                goto out;
+            gf_log("cli", GF_LOG_ERROR, "Failed to set gfid");
+            goto out;
         }
-        CLI_LOCAL_INIT (local, words, frame, xdata);
-        proc = &cli_quotad_clnt.proctable[GF_AGGREGATOR_GETLIMIT];
 
-        print_quota_list_header ();
-        gfid_str = GF_CALLOC (1, gf_common_mt_char, 64);
-        if (!gfid_str) {
-                ret = -1;
-                goto out;
+        ret = proc->fn(frame, THIS, xdata);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to get quota "
+                   "limits for %s",
+                   uuid_utoa((unsigned char *)buf));
         }
-        for (count = 0;; count++) {
-                ret = read (fd, (void*) buf, 16);
-                if (ret <= 0) {
-                        //Finished reading all entries in the conf file
-                        break;
-                }
-                if (ret < 16) {
-                        //This should never happen. We must have a multiple of
-                        //entry_sz bytes in our configuration file.
-                        gf_log (THIS->name, GF_LOG_CRITICAL, "Quota "
-                                "configuration store may be corrupt.");
-                        goto out;
-                }
-                uuid_utoa_r (buf, gfid_str);
-                ret = dict_set_str (xdata, "gfid", gfid_str);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to set gfid");
-                        goto out;
-                }
 
-                ret = proc->fn (frame, THIS, xdata);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to get quota "
-                                "limits for %s", uuid_utoa ((unsigned char*)buf));
-                }
+        dict_del(xdata, "gfid");
+        all_failed = all_failed && ret;
+    }
 
-                dict_del (xdata, "gfid");
-                all_failed = all_failed && ret;
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_vol_quota_limit_list_end(local);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Error in printing "
+                   "xml output");
+            goto out;
         }
+    }
 
-        if (count > 0) {
-                ret = all_failed? -1: 0;
-        } else {
-                ret = 0;
-        }
-out:
-        if (fd != -1) {
-                close (fd);
-        }
+    if (count > 0) {
+        ret = all_failed ? -1 : 0;
+    } else {
+        ret = 0;
+    }
 
-        GF_FREE (gfid_str);
+out:
+    if (xml_err_flag) {
+        ret = cli_xml_output_str("volQuota", NULL, -1, 0, err_str);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not fetch and display quota"
-                        " limits");
-        }
-        CLI_STACK_DESTROY (frame);
-        return ret;
+            gf_log("cli", GF_LOG_ERROR,
+                   "Error outputting in "
+                   "xml format");
+        }
+    }
+    if (xdata)
+        dict_unref(xdata);
+
+    if (fd != -1) {
+        sys_close(fd);
+    }
+
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Could not fetch and display quota"
+               " limits");
+    }
+    CLI_STACK_DESTROY(frame);
+    return ret;
 }
 
 int
-cli_cmd_quota_cbk (struct cli_state *state, struct cli_cmd_word *word,
+cli_cmd_bitrot_cbk(struct cli_state *state, struct cli_cmd_word *word,
                    const char **words, int wordcount)
 {
+    int ret = -1;
+    int parse_err = 0;
+    call_frame_t *frame = NULL;
+    dict_t *options = NULL;
+    cli_local_t *local = NULL;
+    rpc_clnt_procedure_t *proc = NULL;
+    int sent = 0;
+#if (USE_EVENTS)
+    int cmd_type = -1;
+    int ret1 = -1;
+    int event_type = -1;
+    char *tmp = NULL;
+    char *events_str = NULL;
+    char *volname = NULL;
+#endif
 
-        int                      ret       = 0;
-        int                      parse_err = 0;
-        int32_t                  type      = 0;
-        rpc_clnt_procedure_t    *proc      = NULL;
-        call_frame_t            *frame     = NULL;
-        dict_t                  *options   = NULL;
-        gf_answer_t              answer    = GF_ANSWER_NO;
-        cli_local_t             *local     = NULL;
-        int                      sent      = 0;
-        char                    *volname   = NULL;
-        const char *question = "Disabling quota will delete all the quota "
-                               "configuration. Do you want to continue?";
-
-        //parse **words into options dictionary
-        ret = cli_cmd_quota_parse (words, wordcount, &options);
-        if (ret < 0) {
-                cli_usage_out (word->pattern);
-                parse_err = 1;
-                goto out;
-        }
+    ret = cli_cmd_bitrot_parse(words, wordcount, &options);
+    if (ret < 0) {
+        cli_usage_out(word->pattern);
+        parse_err = 1;
+        goto out;
+    }
 
-        ret = dict_get_int32 (options, "type", &type);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get opcode");
-                goto out;
+    if (ret == 1) {
+        /* this is 'volume bitrot help' */
+        cli_cmd_bitrot_help_cbk(state, word, words, wordcount);
+        ret = 0;
+        goto out2;
+    }
+
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
+
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_BITROT];
+
+    CLI_LOCAL_INIT(local, words, frame, options);
+
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, options);
+    }
+
+out:
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_err == 0))
+            cli_err(
+                "Bit rot command failed. Please check the cli "
+                "logs for more details");
+    }
+
+#if (USE_EVENTS)
+    if (ret == 0) {
+        ret1 = dict_get_int32(options, "type", &cmd_type);
+        if (ret1)
+            cmd_type = -1;
+        else {
+            ret1 = dict_get_str(options, "volname", &volname);
+            if (ret1)
+                volname = "";
         }
 
-        //handle quota-disable and quota-list-all different from others
-        switch (type) {
-        case GF_QUOTA_OPTION_TYPE_DISABLE:
-                answer = cli_cmd_get_confirmation (state, question);
-                if (answer == GF_ANSWER_NO)
-                        goto out;
+        switch (cmd_type) {
+            case GF_BITROT_OPTION_TYPE_ENABLE:
+                event_type = EVENT_BITROT_ENABLE;
                 break;
-        case GF_QUOTA_OPTION_TYPE_LIST:
-                if (wordcount != 4)
-                        break;
-                ret = cli_cmd_quota_handle_list_all (words, options);
-                goto out;
-        default:
+            case GF_BITROT_OPTION_TYPE_DISABLE:
+                event_type = EVENT_BITROT_DISABLE;
+                break;
+            case GF_BITROT_CMD_SCRUB_ONDEMAND:
+                event_type = EVENT_BITROT_SCRUB_ONDEMAND;
+                break;
+            case GF_BITROT_OPTION_TYPE_SCRUB_THROTTLE:
+                event_type = EVENT_BITROT_SCRUB_THROTTLE;
+                ret1 = dict_get_str(options, "scrub-throttle-value", &tmp);
+                if (ret1)
+                    tmp = "";
+                gf_asprintf(&events_str, "name=%s;value=%s", volname, tmp);
+                break;
+            case GF_BITROT_OPTION_TYPE_SCRUB_FREQ:
+                event_type = EVENT_BITROT_SCRUB_FREQ;
+                ret1 = dict_get_str(options, "scrub-frequency-value", &tmp);
+                if (ret1)
+                    tmp = "";
+                gf_asprintf(&events_str, "name=%s;value=%s", volname, tmp);
+                break;
+            case GF_BITROT_OPTION_TYPE_SCRUB:
+                event_type = EVENT_BITROT_SCRUB_OPTION;
+                ret1 = dict_get_str(options, "scrub-value", &tmp);
+                if (ret1)
+                    tmp = "";
+                gf_asprintf(&events_str, "name=%s;value=%s", volname, tmp);
+                break;
+            default:
                 break;
         }
 
-        ret = dict_get_str (options, "volname", &volname);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get volume name");
-                goto out;
-        }
+        if (event_type > -1)
+            gf_event(event_type, "%s", events_str);
 
-        //create auxiliary mount need for quota commands that operate on path
-        ret = cli_stage_quota_op (volname, type);
-        if (ret)
-                goto out;
+        if (events_str)
+            GF_FREE(events_str);
+    }
+#endif
 
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame) {
-                ret = -1;
-                goto out;
+    CLI_STACK_DESTROY(frame);
+out2:
+    return ret;
+}
+
+int
+cli_cmd_quota_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                  const char **words, int wordcount)
+{
+    int ret = 0;
+    int parse_err = 0;
+    int32_t type = 0;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *options = NULL;
+    gf_answer_t answer = GF_ANSWER_NO;
+    cli_local_t *local = NULL;
+    int sent = 0;
+    char *volname = NULL;
+    const char *question =
+        "Disabling quota will delete all the quota "
+        "configuration. Do you want to continue?";
+
+    // parse **words into options dictionary
+    if (strcmp(words[1], "inode-quota") == 0) {
+        ret = cli_cmd_inode_quota_parse(words, wordcount, &options);
+        if (ret < 0) {
+            cli_usage_out(word->pattern);
+            parse_err = 1;
+            goto out;
         }
+    } else {
+        ret = cli_cmd_quota_parse(words, wordcount, &options);
 
-        CLI_LOCAL_INIT (local, words, frame, options);
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_QUOTA];
-        if (proc == NULL) {
-                ret = -1;
-                goto out;
+        if (ret == 1) {
+            cli_cmd_quota_help_cbk(state, word, words, wordcount);
+            ret = 0;
+            goto out;
+        }
+        if (ret < 0) {
+            cli_usage_out(word->pattern);
+            parse_err = 1;
+            goto out;
         }
+    }
 
-        if (proc->fn)
-                ret = proc->fn (frame, THIS, options);
+    ret = dict_get_int32(options, "type", &type);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get opcode");
+        goto out;
+    }
+
+    // handle quota-disable and quota-list-all different from others
+    switch (type) {
+        case GF_QUOTA_OPTION_TYPE_DISABLE:
+            answer = cli_cmd_get_confirmation(state, question);
+            if (answer == GF_ANSWER_NO)
+                goto out;
+            break;
+        case GF_QUOTA_OPTION_TYPE_LIST:
+        case GF_QUOTA_OPTION_TYPE_LIST_OBJECTS:
+            if (wordcount != 4)
+                break;
+            ret = cli_cmd_quota_handle_list_all(words, options);
+            goto out;
+        default:
+            break;
+    }
+
+    ret = dict_get_str(options, "volname", &volname);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get volume name");
+        goto out;
+    }
+
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
+
+    CLI_LOCAL_INIT(local, words, frame, options);
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_QUOTA];
+
+    if (proc->fn)
+        ret = proc->fn(frame, THIS, options);
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if (sent == 0 && parse_err == 0)
-                        cli_out ("Quota command failed. Please check the cli "
-                                 "logs for more details");
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if (sent == 0 && parse_err == 0)
+            cli_out(
+                "Quota command failed. Please check the cli "
+                "logs for more details");
+    }
+    if (options)
+        dict_unref(options);
+
+    /* Events for Quota */
+    if (ret == 0) {
+        switch (type) {
+            case GF_QUOTA_OPTION_TYPE_ENABLE:
+                gf_event(EVENT_QUOTA_ENABLE, "volume=%s", volname);
+                break;
+            case GF_QUOTA_OPTION_TYPE_DISABLE:
+                gf_event(EVENT_QUOTA_DISABLE, "volume=%s", volname);
+                break;
+            case GF_QUOTA_OPTION_TYPE_LIMIT_USAGE:
+                gf_event(EVENT_QUOTA_SET_USAGE_LIMIT,
+                         "volume=%s;"
+                         "path=%s;limit=%s",
+                         volname, words[4], words[5]);
+                break;
+            case GF_QUOTA_OPTION_TYPE_LIMIT_OBJECTS:
+                gf_event(EVENT_QUOTA_SET_OBJECTS_LIMIT,
+                         "volume=%s;"
+                         "path=%s;limit=%s",
+                         volname, words[4], words[5]);
+                break;
+            case GF_QUOTA_OPTION_TYPE_REMOVE:
+                gf_event(EVENT_QUOTA_REMOVE_USAGE_LIMIT,
+                         "volume=%s;"
+                         "path=%s",
+                         volname, words[4]);
+                break;
+            case GF_QUOTA_OPTION_TYPE_REMOVE_OBJECTS:
+                gf_event(EVENT_QUOTA_REMOVE_OBJECTS_LIMIT,
+                         "volume=%s;"
+                         "path=%s",
+                         volname, words[4]);
+                break;
+            case GF_QUOTA_OPTION_TYPE_ALERT_TIME:
+                gf_event(EVENT_QUOTA_ALERT_TIME, "volume=%s;time=%s", volname,
+                         words[4]);
+                break;
+            case GF_QUOTA_OPTION_TYPE_SOFT_TIMEOUT:
+                gf_event(EVENT_QUOTA_SOFT_TIMEOUT,
+                         "volume=%s;"
+                         "soft-timeout=%s",
+                         volname, words[4]);
+                break;
+            case GF_QUOTA_OPTION_TYPE_HARD_TIMEOUT:
+                gf_event(EVENT_QUOTA_HARD_TIMEOUT,
+                         "volume=%s;"
+                         "hard-timeout=%s",
+                         volname, words[4]);
+                break;
+            case GF_QUOTA_OPTION_TYPE_DEFAULT_SOFT_LIMIT:
+                gf_event(EVENT_QUOTA_DEFAULT_SOFT_LIMIT,
+                         "volume=%s;"
+                         "default-soft-limit=%s",
+                         volname, words[4]);
+                break;
         }
+    }
 
-        CLI_STACK_DESTROY (frame);
-        return ret;
+    CLI_STACK_DESTROY(frame);
+    return ret;
 }
 
 int
-cli_cmd_volume_remove_brick_cbk (struct cli_state *state,
-                                 struct cli_cmd_word *word, const char **words,
-                                 int wordcount)
+cli_cmd_volume_remove_brick_cbk(struct cli_state *state,
+                                struct cli_cmd_word *word, const char **words,
+                                int wordcount)
 {
-        int                     ret = -1;
-        rpc_clnt_procedure_t    *proc = NULL;
-        call_frame_t            *frame = NULL;
-        dict_t                  *options = NULL;
-        gf_answer_t             answer = GF_ANSWER_NO;
-        int                     sent = 0;
-        int                     parse_error = 0;
-        int                     need_question = 0;
-        cli_local_t             *local = NULL;
-
-        const char *question = "Removing brick(s) can result in data loss. "
-                               "Do you want to Continue?";
-
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *options = NULL;
+    gf_answer_t answer = GF_ANSWER_NO;
+    int brick_count = 0;
+    int sent = 0;
+    int parse_error = 0;
+    int need_question = 0;
+    cli_local_t *local = NULL;
+    char *volname = NULL;
+#if (USE_EVENTS)
+    eventtypes_t event = EVENT_LAST;
+    char *event_str = NULL;
+    int event_ret = -1;
+#endif
+    int32_t command = GF_OP_CMD_NONE;
+    char *question = NULL;
+
+    ret = cli_cmd_volume_remove_brick_parse(state, words, wordcount, &options,
+                                            &need_question, &brick_count,
+                                            &command);
+    if (ret) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
+
+    if (command == GF_OP_CMD_COMMIT_FORCE) {
+        question =
+            "Remove-brick force will not migrate files from the "
+            "removed bricks, so they will no longer be available"
+            " on the volume.\nDo you want to continue?";
+    } else if (command == GF_OP_CMD_START) {
+        question =
+            "It is recommended that remove-brick be run with"
+            " cluster.force-migration option disabled to prevent"
+            " possible data corruption. Doing so will ensure that"
+            " files that receive writes during migration will not"
+            " be migrated and will need to be manually copied"
+            " after the remove-brick commit operation. Please"
+            " check the value of the option and update accordingly."
+            " \nDo you want to continue with your current"
+            " cluster.force-migration settings?";
+    }
+
+    if (!brick_count) {
+        cli_err("No bricks specified");
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        ret = -1;
+        goto out;
+    }
+    ret = dict_get_str(options, "volname", &volname);
+    if (ret || !volname) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to fetch volname");
+        ret = -1;
+        goto out;
+    }
 
-        ret = cli_cmd_volume_remove_brick_parse (words, wordcount, &options,
-                                                 &need_question);
-        if (ret) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                goto out;
-        }
+#if (USE_EVENTS)
+    event_ret = cli_event_remove_brick_str(options, &event_str, &event);
+#endif
 
-        if (!(state->mode & GLUSTER_MODE_SCRIPT) && need_question) {
-                /* we need to ask question only in case of 'commit or force' */
-                answer = cli_cmd_get_confirmation (state, question);
-                if (GF_ANSWER_NO == answer) {
-                        ret = 0;
-                        goto out;
-                }
+    if (!strcmp(volname, GLUSTER_SHARED_STORAGE)) {
+        question =
+            "Removing brick from the shared storage volume"
+            "(gluster_shared_storage), will affect features "
+            "like snapshot scheduler, geo-replication "
+            "and NFS-Ganesha. Do you still want to "
+            "continue?";
+        need_question = _gf_true;
+    }
+
+    if (!(state->mode & GLUSTER_MODE_SCRIPT) && need_question) {
+        answer = cli_cmd_get_confirmation(state, question);
+        if (GF_ANSWER_NO == answer) {
+            ret = 0;
+            goto out;
         }
+    }
 
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_REMOVE_BRICK];
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_REMOVE_BRICK];
 
-        CLI_LOCAL_INIT (local, words, frame, options);
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
 
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, options);
-        }
+    CLI_LOCAL_INIT(local, words, frame, options);
+
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, options);
+    }
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_out ("Volume remove-brick failed");
-        }
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Volume remove-brick failed");
+    }
+#if (USE_EVENTS)
+    if (!ret && !event_ret)
+        gf_event(event, "%s", event_str);
+    if (event_str)
+        GF_FREE(event_str);
 
-        CLI_STACK_DESTROY (frame);
+#endif
 
-        return ret;
+    CLI_STACK_DESTROY(frame);
+    if (options)
+        dict_unref(options);
 
+    return ret;
 }
 
 int
-cli_cmd_volume_replace_brick_cbk (struct cli_state *state,
-                                  struct cli_cmd_word *word,
-                                  const char **words,
-                                  int wordcount)
+cli_cmd_volume_reset_brick_cbk(struct cli_state *state,
+                               struct cli_cmd_word *word, const char **words,
+                               int wordcount)
 {
-        int                     ret = -1;
-        rpc_clnt_procedure_t    *proc = NULL;
-        call_frame_t            *frame = NULL;
-        dict_t                  *options = NULL;
-        int                     sent = 0;
-        int                     parse_error = 0;
-        cli_local_t             *local = NULL;
-        int                     replace_op = 0;
-        char                    *q = "All replace-brick commands except "
-                                     "commit force are deprecated. "
-                                     "Do you want to continue?";
-        gf_answer_t             answer = GF_ANSWER_NO;
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *options = NULL;
+    int sent = 0;
+    int parse_error = 0;
+    cli_local_t *local = NULL;
 
 #ifdef GF_SOLARIS_HOST_OS
-        cli_out ("Command not supported on Solaris");
-        goto out;
+    cli_out("Command not supported on Solaris");
+    goto out;
 #endif
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_REPLACE_BRICK];
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_RESET_BRICK];
 
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
+    ret = cli_cmd_volume_reset_brick_parse(words, wordcount, &options);
 
-        ret = cli_cmd_volume_replace_brick_parse (words, wordcount, &options);
+    if (ret) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
 
+    if (state->mode & GLUSTER_MODE_WIGNORE_PARTITION) {
+        ret = dict_set_int32(options, "ignore-partition", _gf_true);
         if (ret) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                goto out;
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to set ignore-"
+                   "partition option");
+            goto out;
         }
+    }
 
-        ret = dict_get_int32 (options, "operation", &replace_op);
-        if (replace_op != GF_REPLACE_OP_COMMIT_FORCE) {
-                answer = cli_cmd_get_confirmation (state, q);
-                if (GF_ANSWER_NO == answer) {
-                        ret = 0;
-                        goto out;
-                }
-        }
-
-        if (state->mode & GLUSTER_MODE_WIGNORE) {
-                ret = dict_set_int32 (options, "force", _gf_true);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to set force"
-                                "option");
-                        goto out;
-                }
-        }
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
 
-        CLI_LOCAL_INIT (local, words, frame, options);
+    CLI_LOCAL_INIT(local, words, frame, options);
 
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, options);
-        }
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, options);
+    }
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_out ("Volume replace-brick failed");
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Volume reset-brick failed");
+    } else {
+        if (wordcount > 5) {
+            gf_event(EVENT_BRICK_RESET_COMMIT,
+                     "Volume=%s;source-brick=%s;"
+                     "destination-brick=%s",
+                     (char *)words[2], (char *)words[3], (char *)words[4]);
+        } else {
+            gf_event(EVENT_BRICK_RESET_START, "Volume=%s;source-brick=%s",
+                     (char *)words[2], (char *)words[3]);
         }
+    }
+    CLI_STACK_DESTROY(frame);
 
-        CLI_STACK_DESTROY (frame);
-
-        return ret;
+    return ret;
 }
 
-
 int
-cli_cmd_volume_set_transport_cbk (struct cli_state *state,
-                                  struct cli_cmd_word *word,
-                                  const char **words, int wordcount)
+cli_cmd_volume_replace_brick_cbk(struct cli_state *state,
+                                 struct cli_cmd_word *word, const char **words,
+                                 int wordcount)
 {
-        cli_cmd_broadcast_response (0);
-        return 0;
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *options = NULL;
+    int sent = 0;
+    int parse_error = 0;
+    cli_local_t *local = NULL;
+
+#ifdef GF_SOLARIS_HOST_OS
+    cli_out("Command not supported on Solaris");
+    goto out;
+#endif
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_REPLACE_BRICK];
+
+    ret = cli_cmd_volume_replace_brick_parse(words, wordcount, &options);
+
+    if (ret) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
+
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
+
+    CLI_LOCAL_INIT(local, words, frame, options);
+
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, options);
+    }
+
+out:
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Volume replace-brick failed");
+    } else {
+        gf_event(EVENT_BRICK_REPLACE,
+                 "Volume=%s;source-brick=%s;destination-brick=%s",
+                 (char *)words[2], (char *)words[3], (char *)words[4]);
+    }
+    CLI_STACK_DESTROY(frame);
+
+    return ret;
 }
 
 int
-cli_cmd_volume_top_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                          const char **words, int wordcount)
+cli_cmd_volume_set_transport_cbk(struct cli_state *state,
+                                 struct cli_cmd_word *word, const char **words,
+                                 int wordcount)
 {
+    cli_cmd_broadcast_response(0);
+    return 0;
+}
 
-        int                     ret      = -1;
-        rpc_clnt_procedure_t    *proc    = NULL;
-        call_frame_t            *frame   = NULL;
-        dict_t                  *options = NULL;
-        int                     sent     = 0;
-        int                     parse_error = 0;
-        cli_local_t             *local = NULL;
-
-        ret = cli_cmd_volume_top_parse (words, wordcount, &options);
-
-        if (ret) {
-                parse_error = 1;
-                cli_usage_out (word->pattern);
-                goto out;
-        }
+int
+cli_cmd_volume_top_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                       const char **words, int wordcount)
+{
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *options = NULL;
+    int sent = 0;
+    int parse_error = 0;
+    cli_local_t *local = NULL;
+
+    ret = cli_cmd_volume_top_parse(words, wordcount, &options);
+
+    if (ret) {
+        parse_error = 1;
+        cli_usage_out(word->pattern);
+        goto out;
+    }
 
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_TOP_VOLUME];
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_TOP_VOLUME];
 
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        gf_log(THIS->name, GF_LOG_ERROR, "failed to create frame");
+        ret = -1;
+        goto out;
+    }
 
-        CLI_LOCAL_INIT (local, words, frame, options);
+    CLI_LOCAL_INIT(local, words, frame, options);
 
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, options);
-        }
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, options);
+    }
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_out ("Volume top failed");
-        }
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Volume top failed");
+    }
 
-        CLI_STACK_DESTROY (frame);
-
-        return ret;
+    CLI_STACK_DESTROY(frame);
 
+    return ret;
 }
 
-
 int
-cli_cmd_log_rotate_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                        const char **words, int wordcount)
+cli_cmd_log_rotate_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                       const char **words, int wordcount)
 {
-        int                     ret = -1;
-        rpc_clnt_procedure_t    *proc = NULL;
-        call_frame_t            *frame = NULL;
-        dict_t                  *options = NULL;
-        int                     sent = 0;
-        int                     parse_error = 0;
-        cli_local_t             *local = NULL;
-
-        if (!((wordcount == 4) || (wordcount == 5))) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                goto out;
-        }
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *options = NULL;
+    int sent = 0;
+    int parse_error = 0;
+    cli_local_t *local = NULL;
+
+    if (!((wordcount == 4) || (wordcount == 5))) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
 
-        if (!((strcmp ("rotate", words[2]) == 0) ||
-              (strcmp ("rotate", words[3]) == 0))) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                goto out;
-        }
+    if (!(strcmp("rotate", words[3]) == 0)) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
 
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_LOG_ROTATE];
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_LOG_ROTATE];
 
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
+    ret = cli_cmd_log_rotate_parse(words, wordcount, &options);
+    if (ret)
+        goto out;
 
-        ret = cli_cmd_log_rotate_parse (words, wordcount, &options);
-        if (ret)
-                goto out;
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        gf_log(THIS->name, GF_LOG_ERROR, "failed to create frame");
+        ret = -1;
+        goto out;
+    }
 
-        CLI_LOCAL_INIT (local, words, frame, options);
+    CLI_LOCAL_INIT(local, words, frame, options);
 
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, options);
-        }
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, options);
+    }
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_out ("Volume log rotate failed");
-        }
-        CLI_STACK_DESTROY (frame);
-
-        return ret;
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Volume log rotate failed");
+    }
+    CLI_STACK_DESTROY(frame);
+
+    return ret;
 }
 
 #if (SYNCDAEMON_COMPILE)
 static int
-cli_check_gsync_present ()
+cli_check_gsync_present()
 {
-        char                buff[PATH_MAX] = {0, };
-        runner_t            runner = {0,};
-        char                *ptr = NULL;
-        int                 ret = 0;
-
-        ret = setenv ("_GLUSTERD_CALLED_", "1", 1);
-        if (-1 == ret) {
-                gf_log ("", GF_LOG_WARNING, "setenv syscall failed, hence could"
-                        "not assert if geo-replication is installed");
-                goto out;
-        }
-
-        runinit (&runner);
-        runner_add_args (&runner, GSYNCD_PREFIX"/gsyncd", "--version", NULL);
-        runner_redir (&runner, STDOUT_FILENO, RUN_PIPE);
-        ret = runner_start (&runner);
-        if (ret == -1) {
-                gf_log ("", GF_LOG_INFO, "geo-replication not installed");
-                goto out;
-        }
+    char buff[PATH_MAX] = {
+        0,
+    };
+    runner_t runner = {
+        0,
+    };
+    char *ptr = NULL;
+    int ret = 0;
+
+    ret = setenv("_GLUSTERD_CALLED_", "1", 1);
+    if (-1 == ret) {
+        gf_log("", GF_LOG_WARNING,
+               "setenv syscall failed, hence could"
+               "not assert if geo-replication is installed");
+        goto out;
+    }
+
+    runinit(&runner);
+    runner_add_args(&runner, GSYNCD_PREFIX "/gsyncd", "--version", NULL);
+    runner_redir(&runner, STDOUT_FILENO, RUN_PIPE);
+    ret = runner_start(&runner);
+    if (ret == -1) {
+        gf_log("", GF_LOG_INFO, "geo-replication not installed");
+        goto out;
+    }
 
-        ptr = fgets(buff, sizeof(buff), runner_chio (&runner, STDOUT_FILENO));
-        if (ptr) {
-                if (!strstr (buff, "gsyncd")) {
-                        ret  = -1;
-                        goto out;
-                }
-        } else {
-                ret = -1;
-                goto out;
+    ptr = fgets(buff, sizeof(buff), runner_chio(&runner, STDOUT_FILENO));
+    if (ptr) {
+        if (!strstr(buff, "gsyncd")) {
+            ret = -1;
+            goto out;
         }
+    } else {
+        ret = -1;
+        goto out;
+    }
 
-        ret = runner_end (&runner);
+    ret = runner_end(&runner);
 
-        if (ret)
-                gf_log ("", GF_LOG_ERROR, "geo-replication not installed");
+    if (ret)
+        gf_log("", GF_LOG_ERROR, "geo-replication not installed");
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret ? -1 : 0;
-
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret ? -1 : 0;
 }
 
 void
-cli_cmd_check_gsync_exists_cbk (struct cli_cmd *this)
+cli_cmd_check_gsync_exists_cbk(struct cli_cmd *this)
 {
+    int ret = 0;
 
-        int             ret = 0;
-
-        ret = cli_check_gsync_present ();
-        if (ret)
-                this->disable = _gf_true;
-
+    ret = cli_check_gsync_present();
+    if (ret)
+        this->disable = _gf_true;
 }
 #endif
 
 int
-cli_cmd_volume_gsync_set_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                              const char **words, int wordcount)
+cli_cmd_volume_gsync_set_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                             const char **words, int wordcount)
 {
-        int                      ret     = 0;
-        int                      parse_err = 0;
-        dict_t                  *options = NULL;
-        rpc_clnt_procedure_t    *proc    = NULL;
-        call_frame_t            *frame   = NULL;
-        cli_local_t             *local   = NULL;
-
-        proc = &cli_rpc_prog->proctable [GLUSTER_CLI_GSYNC_SET];
-        if (proc == NULL) {
-                ret = -1;
-                goto out;
-        }
+    int ret = 0;
+    int parse_err = 0;
+    dict_t *options = NULL;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    cli_local_t *local = NULL;
+    char *errstr = NULL;
+#if (USE_EVENTS)
+    int ret1 = -1;
+    int cmd_type = -1;
+    int tmpi = 0;
+    char *tmp = NULL;
+    char *events_str = NULL;
+    int event_type = -1;
+#endif
 
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (frame == NULL) {
-                ret = -1;
-                goto out;
-        }
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_GSYNC_SET];
 
-        ret = cli_cmd_gsync_set_parse (words, wordcount, &options);
-        if (ret) {
-                cli_usage_out (word->pattern);
-                parse_err = 1;
-                goto out;
+    ret = cli_cmd_gsync_set_parse(state, words, wordcount, &options, &errstr);
+    if (ret) {
+        if (errstr) {
+            cli_err("%s", errstr);
+            GF_FREE(errstr);
+        } else {
+            cli_usage_out(word->pattern);
         }
+        parse_err = 1;
+        goto out;
+    }
+
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (frame == NULL) {
+        ret = -1;
+        goto out;
+    }
 
-        CLI_LOCAL_INIT (local, words, frame, options);
+    CLI_LOCAL_INIT(local, words, frame, options);
 
-        if (proc->fn)
-                ret = proc->fn (frame, THIS, options);
+    if (proc->fn)
+        ret = proc->fn(frame, THIS, options);
 
 out:
-        if (ret && parse_err == 0)
-                cli_out (GEOREP" command failed");
+    if (ret && parse_err == 0)
+        cli_out(GEOREP " command failed");
+
+#if (USE_EVENTS)
+    if (ret == 0) {
+        events_str = gf_strdup("");
+
+        /* Type of Geo-rep Action - Create, Start etc */
+        ret1 = dict_get_int32(options, "type", &cmd_type);
+        if (ret1)
+            cmd_type = -1;
+
+        /* Only capture Events for modification commands */
+        switch (cmd_type) {
+            case GF_GSYNC_OPTION_TYPE_CREATE:
+                event_type = EVENT_GEOREP_CREATE;
+                break;
+            case GF_GSYNC_OPTION_TYPE_START:
+                event_type = EVENT_GEOREP_START;
+                break;
+            case GF_GSYNC_OPTION_TYPE_STOP:
+                event_type = EVENT_GEOREP_STOP;
+                break;
+            case GF_GSYNC_OPTION_TYPE_PAUSE:
+                event_type = EVENT_GEOREP_PAUSE;
+                break;
+            case GF_GSYNC_OPTION_TYPE_RESUME:
+                event_type = EVENT_GEOREP_RESUME;
+                break;
+            case GF_GSYNC_OPTION_TYPE_DELETE:
+                event_type = EVENT_GEOREP_DELETE;
+                break;
+            case GF_GSYNC_OPTION_TYPE_CONFIG:
+                ret1 = dict_get_str(options, "subop", &tmp);
+                if (ret1)
+                    tmp = "";
+
+                /* For Config Set additionally capture key and value */
+                /* For Config Reset capture key */
+                if (strcmp(tmp, "set") == 0) {
+                    event_type = EVENT_GEOREP_CONFIG_SET;
+
+                    ret1 = dict_get_str(options, "op_name", &tmp);
+                    if (ret1)
+                        tmp = "";
+
+                    gf_asprintf_append(&events_str, "%soption=%s;", events_str,
+                                       tmp);
+
+                    ret1 = dict_get_str(options, "op_value", &tmp);
+                    if (ret1)
+                        tmp = "";
+
+                    gf_asprintf_append(&events_str, "%svalue=%s;", events_str,
+                                       tmp);
+                } else if (strcmp(tmp, "del") == 0) {
+                    event_type = EVENT_GEOREP_CONFIG_RESET;
+
+                    ret1 = dict_get_str(options, "op_name", &tmp);
+                    if (ret1)
+                        tmp = "";
+
+                    gf_asprintf_append(&events_str, "%soption=%s;", events_str,
+                                       tmp);
+                }
+                break;
+            default:
+                break;
+        }
+
+        if (event_type > -1) {
+            /* Capture all optional arguments used */
+            ret1 = dict_get_int32(options, "force", &tmpi);
+            if (ret1 == 0) {
+                gf_asprintf_append(&events_str, "%sforce=%d;", events_str,
+                                   tmpi);
+            }
+            ret1 = dict_get_int32(options, "push_pem", &tmpi);
+            if (ret1 == 0) {
+                gf_asprintf_append(&events_str, "%spush_pem=%d;", events_str,
+                                   tmpi);
+            }
+            ret1 = dict_get_int32(options, "no_verify", &tmpi);
+            if (ret1 == 0) {
+                gf_asprintf_append(&events_str, "%sno_verify=%d;", events_str,
+                                   tmpi);
+            }
+
+            ret1 = dict_get_int32(options, "ssh_port", &tmpi);
+            if (ret1 == 0) {
+                gf_asprintf_append(&events_str, "%sssh_port=%d;", events_str,
+                                   tmpi);
+            }
+
+            ret1 = dict_get_int32(options, "reset-sync-time", &tmpi);
+            if (ret1 == 0) {
+                gf_asprintf_append(&events_str, "%sreset_sync_time=%d;",
+                                   events_str, tmpi);
+            }
+            /* Capture Master and Slave Info */
+            ret1 = dict_get_str(options, "master", &tmp);
+            if (ret1)
+                tmp = "";
+            gf_asprintf_append(&events_str, "%smaster=%s;", events_str, tmp);
+
+            ret1 = dict_get_str(options, "slave", &tmp);
+            if (ret1)
+                tmp = "";
+            gf_asprintf_append(&events_str, "%sslave=%s", events_str, tmp);
+
+            gf_event(event_type, "%s", events_str);
+        }
+
+        /* Allocated by gf_strdup and gf_asprintf */
+        if (events_str)
+            GF_FREE(events_str);
+    }
+#endif
 
-        CLI_STACK_DESTROY (frame);
+    CLI_STACK_DESTROY(frame);
 
-        return ret;
+    return ret;
 }
 
 int
-cli_cmd_volume_status_cbk (struct cli_state *state,
-                           struct cli_cmd_word *word,
-                           const char **words, int wordcount)
+cli_cmd_volume_status_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                          const char **words, int wordcount)
 {
-        int                   ret         = -1;
-        rpc_clnt_procedure_t *proc        = NULL;
-        call_frame_t         *frame       = NULL;
-        dict_t               *dict        = NULL;
-        uint32_t              cmd         = 0;
-        cli_local_t          *local       = NULL;
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *dict = NULL;
+    uint32_t cmd = 0;
+    cli_local_t *local = NULL;
 
-        ret = cli_cmd_volume_status_parse (words, wordcount, &dict);
+    ret = cli_cmd_volume_status_parse(words, wordcount, &dict);
 
-        if (ret) {
-                cli_usage_out (word->pattern);
-                goto out;
-        }
+    if (ret) {
+        cli_usage_out(word->pattern);
+        goto out;
+    }
 
-        ret = dict_get_uint32 (dict, "cmd", &cmd);
-        if (ret)
-                goto out;
+    ret = dict_get_uint32(dict, "cmd", &cmd);
+    if (ret)
+        goto out;
 
-        if (!(cmd & GF_CLI_STATUS_ALL)) {
-                /* for one volume or brick */
-                proc = &cli_rpc_prog->proctable[GLUSTER_CLI_STATUS_VOLUME];
-        } else {
-                /* volume status all or all detail */
-                proc = &cli_rpc_prog->proctable[GLUSTER_CLI_STATUS_ALL];
-        }
+    if (!(cmd & GF_CLI_STATUS_ALL)) {
+        /* for one volume or brick */
+        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_STATUS_VOLUME];
+    } else {
+        /* volume status all or all detail */
+        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_STATUS_ALL];
+    }
 
-        if (!proc->fn)
-                goto out;
+    if (!proc->fn) {
+        ret = -1;
+        goto out;
+    }
 
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        gf_log(THIS->name, GF_LOG_ERROR, "failed to create frame");
+        ret = -1;
+        goto out;
+    }
 
-        CLI_LOCAL_INIT (local, words, frame, dict);
+    CLI_LOCAL_INIT(local, words, frame, dict);
 
-        ret = proc->fn (frame, THIS, dict);
+    ret = proc->fn(frame, THIS, dict);
 
 out:
-        CLI_STACK_DESTROY (frame);
+    CLI_STACK_DESTROY(frame);
 
-        return ret;
+    return ret;
 }
 
-
 int
-cli_get_detail_status (dict_t *dict, int i, cli_volume_status_t *status)
+cli_get_detail_status(dict_t *dict, int i, cli_volume_status_t *status)
 {
-        uint64_t                   free            = 0;
-        uint64_t                   total           = 0;
-        char                       key[1024]       = {0};
-        int                        ret             = 0;
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.free", i);
-        ret = dict_get_uint64 (dict, key, &free);
-
-        status->free = gf_uint64_2human_readable (free);
-        if (!status->free)
-                goto out;
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.total", i);
-        ret = dict_get_uint64 (dict, key, &total);
+    uint64_t free = 0;
+    uint64_t total = 0;
+    char key[1024] = {0};
+    int ret = 0;
 
-        status->total = gf_uint64_2human_readable (total);
-        if (!status->total)
-                goto out;
+    snprintf(key, sizeof(key), "brick%d.free", i);
+    ret = dict_get_uint64(dict, key, &free);
 
-#ifdef GF_LINUX_HOST_OS
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.device", i);
-        ret = dict_get_str (dict, key, &(status->device));
-        if (ret)
-                status->device = NULL;
-#endif
+    status->free = gf_uint64_2human_readable(free);
+    if (!status->free)
+        goto out;
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.block_size", i);
-        ret = dict_get_uint64 (dict, key, &(status->block_size));
-        if (ret) {
-                ret = 0;
-                status->block_size = 0;
-        }
+    snprintf(key, sizeof(key), "brick%d.total", i);
+    ret = dict_get_uint64(dict, key, &total);
 
-#ifdef GF_LINUX_HOST_OS
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.mnt_options", i);
-        ret = dict_get_str (dict, key, &(status->mount_options));
-        if (ret)
-                status->mount_options = NULL;
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.fs_name", i);
-        ret = dict_get_str (dict, key, &(status->fs_name));
-        if (ret) {
-                ret = 0;
-                status->fs_name = NULL;
-        }
+    status->total = gf_uint64_2human_readable(total);
+    if (!status->total)
+        goto out;
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.inode_size", i);
-        ret = dict_get_str (dict, key, &(status->inode_size));
-        if (ret)
-                status->inode_size = NULL;
-#endif /* GF_LINUX_HOST_OS */
+    snprintf(key, sizeof(key), "brick%d.device", i);
+    ret = dict_get_str(dict, key, &(status->device));
+    if (ret)
+        status->device = NULL;
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.total_inodes", i);
-        ret = dict_get_uint64 (dict, key,
-                        &(status->total_inodes));
-        if (ret)
-                status->total_inodes = 0;
+    snprintf(key, sizeof(key), "brick%d.block_size", i);
+    ret = dict_get_uint64(dict, key, &(status->block_size));
+    if (ret) {
+        ret = 0;
+        status->block_size = 0;
+    }
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.free_inodes", i);
-        ret = dict_get_uint64 (dict, key, &(status->free_inodes));
-        if (ret) {
-                ret = 0;
-                status->free_inodes = 0;
-        }
+    snprintf(key, sizeof(key), "brick%d.mnt_options", i);
+    ret = dict_get_str(dict, key, &(status->mount_options));
+    if (ret)
+        status->mount_options = NULL;
 
+    snprintf(key, sizeof(key), "brick%d.fs_name", i);
+    ret = dict_get_str(dict, key, &(status->fs_name));
+    if (ret) {
+        ret = 0;
+        status->fs_name = NULL;
+    }
+
+    snprintf(key, sizeof(key), "brick%d.inode_size", i);
+    ret = dict_get_str(dict, key, &(status->inode_size));
+    if (ret)
+        status->inode_size = NULL;
+
+    snprintf(key, sizeof(key), "brick%d.total_inodes", i);
+    ret = dict_get_uint64(dict, key, &(status->total_inodes));
+    if (ret)
+        status->total_inodes = 0;
+
+    snprintf(key, sizeof(key), "brick%d.free_inodes", i);
+    ret = dict_get_uint64(dict, key, &(status->free_inodes));
+    if (ret) {
+        ret = 0;
+        status->free_inodes = 0;
+    }
 
- out:
-        return ret;
+out:
+    return ret;
 }
 
 void
-cli_print_detailed_status (cli_volume_status_t *status)
+cli_print_detailed_status(cli_volume_status_t *status)
 {
-        cli_out ("%-20s : %-20s", "Brick", status->brick);
-        if (status->online)
-                cli_out ("%-20s : %-20d", "Port", status->port);
-        else
-                cli_out ("%-20s : %-20s", "Port", "N/A");
-        cli_out ("%-20s : %-20c", "Online", (status->online) ? 'Y' : 'N');
-        cli_out ("%-20s : %-20s", "Pid", status->pid_str);
-
-#ifdef GF_LINUX_HOST_OS
-        if (status->fs_name)
-                cli_out ("%-20s : %-20s", "File System", status->fs_name);
-        else
-                cli_out ("%-20s : %-20s", "File System", "N/A");
-
-        if (status->device)
-                cli_out ("%-20s : %-20s", "Device", status->device);
-        else
-                cli_out ("%-20s : %-20s", "Device", "N/A");
-
-        if (status->mount_options) {
-                cli_out ("%-20s : %-20s", "Mount Options",
-                         status->mount_options);
-        } else {
-                cli_out ("%-20s : %-20s", "Mount Options", "N/A");
-        }
-
-        if (status->inode_size) {
-                cli_out ("%-20s : %-20s", "Inode Size",
-                         status->inode_size);
-        } else {
-                cli_out ("%-20s : %-20s", "Inode Size", "N/A");
-        }
-#endif
-        if (status->free)
-                cli_out ("%-20s : %-20s", "Disk Space Free", status->free);
-        else
-                cli_out ("%-20s : %-20s", "Disk Space Free", "N/A");
-
-        if (status->total)
-                cli_out ("%-20s : %-20s", "Total Disk Space", status->total);
-        else
-                cli_out ("%-20s : %-20s", "Total Disk Space", "N/A");
-
-
-        if (status->total_inodes) {
-                cli_out ("%-20s : %-20"GF_PRI_INODE, "Inode Count",
-                         status->total_inodes);
-        } else {
-                cli_out ("%-20s : %-20s", "Inode Count", "N/A");
-        }
+    cli_out("%-20s : %-20s", "Brick", status->brick);
+
+    if (status->online) {
+        cli_out("%-20s : %-20d", "TCP Port", status->port);
+        cli_out("%-20s : %-20d", "RDMA Port", status->rdma_port);
+    } else {
+        cli_out("%-20s : %-20s", "TCP Port", "N/A");
+        cli_out("%-20s : %-20s", "RDMA Port", "N/A");
+    }
+
+    cli_out("%-20s : %-20c", "Online", (status->online) ? 'Y' : 'N');
+    cli_out("%-20s : %-20s", "Pid", status->pid_str);
+
+    if (status->fs_name)
+        cli_out("%-20s : %-20s", "File System", status->fs_name);
+    else
+        cli_out("%-20s : %-20s", "File System", "N/A");
+
+    if (status->device)
+        cli_out("%-20s : %-20s", "Device", status->device);
+    else
+        cli_out("%-20s : %-20s", "Device", "N/A");
+
+    if (status->mount_options) {
+        cli_out("%-20s : %-20s", "Mount Options", status->mount_options);
+    } else {
+        cli_out("%-20s : %-20s", "Mount Options", "N/A");
+    }
+
+    if (status->inode_size) {
+        cli_out("%-20s : %-20s", "Inode Size", status->inode_size);
+    } else {
+        cli_out("%-20s : %-20s", "Inode Size", "N/A");
+    }
+    if (status->free)
+        cli_out("%-20s : %-20s", "Disk Space Free", status->free);
+    else
+        cli_out("%-20s : %-20s", "Disk Space Free", "N/A");
+
+    if (status->total)
+        cli_out("%-20s : %-20s", "Total Disk Space", status->total);
+    else
+        cli_out("%-20s : %-20s", "Total Disk Space", "N/A");
+
+    if (status->total_inodes) {
+        cli_out("%-20s : %-20" GF_PRI_INODE, "Inode Count",
+                status->total_inodes);
+    } else {
+        cli_out("%-20s : %-20s", "Inode Count", "N/A");
+    }
+
+    if (status->free_inodes) {
+        cli_out("%-20s : %-20" GF_PRI_INODE, "Free Inodes",
+                status->free_inodes);
+    } else {
+        cli_out("%-20s : %-20s", "Free Inodes", "N/A");
+    }
+}
 
-        if (status->free_inodes) {
-                cli_out ("%-20s : %-20"GF_PRI_INODE, "Free Inodes",
-                         status->free_inodes);
+int
+cli_print_brick_status(cli_volume_status_t *status)
+{
+    int fieldlen = CLI_VOL_STATUS_BRICK_LEN;
+    int bricklen = 0;
+    char *p = NULL;
+    int num_spaces = 0;
+
+    p = status->brick;
+    bricklen = strlen(p);
+    while (bricklen > 0) {
+        if (bricklen > fieldlen) {
+            cli_out("%.*s", fieldlen, p);
+            p += fieldlen;
+            bricklen -= fieldlen;
         } else {
-                cli_out ("%-20s : %-20s", "Free Inodes", "N/A");
-        }
+            num_spaces = (fieldlen - bricklen) + 1;
+            printf("%s", p);
+            while (num_spaces-- != 0)
+                printf(" ");
+            if (status->port || status->rdma_port) {
+                if (status->online)
+                    cli_out("%-10d%-11d%-8c%-5s", status->port,
+                            status->rdma_port, status->online ? 'Y' : 'N',
+                            status->pid_str);
+                else
+                    cli_out("%-10s%-11s%-8c%-5s", "N/A", "N/A",
+                            status->online ? 'Y' : 'N', status->pid_str);
+            } else
+                cli_out("%-10s%-11s%-8c%-5s", "N/A", "N/A",
+                        status->online ? 'Y' : 'N', status->pid_str);
+            bricklen = 0;
+        }
+    }
+
+    return 0;
 }
 
+#define NEEDS_GLFS_HEAL(op)                                                    \
+    ((op == GF_SHD_OP_SBRAIN_HEAL_FROM_BIGGER_FILE) ||                         \
+     (op == GF_SHD_OP_SBRAIN_HEAL_FROM_LATEST_MTIME) ||                        \
+     (op == GF_SHD_OP_SBRAIN_HEAL_FROM_BRICK) ||                               \
+     (op == GF_SHD_OP_INDEX_SUMMARY) || (op == GF_SHD_OP_SPLIT_BRAIN_FILES) || \
+     (op == GF_SHD_OP_GRANULAR_ENTRY_HEAL_ENABLE) ||                           \
+     (op == GF_SHD_OP_HEAL_SUMMARY))
+
 int
-cli_print_brick_status (cli_volume_status_t *status)
+cli_launch_glfs_heal(int heal_op, dict_t *options)
 {
-        int  fieldlen = CLI_VOL_STATUS_BRICK_LEN;
-        int  bricklen = 0;
-        char *p = NULL;
-        int  num_tabs = 0;
-
-        p = status->brick;
-        bricklen = strlen (p);
-        while (bricklen > 0) {
-                if (bricklen > fieldlen) {
-                        cli_out ("%.*s", fieldlen, p);
-                        p += fieldlen;
-                        bricklen -= fieldlen;
-                } else {
-                        num_tabs = (fieldlen - bricklen) / CLI_TAB_LENGTH + 1;
-                        printf ("%s", p);
-                        while (num_tabs-- != 0)
-                                printf ("\t");
-                        if (status->port) {
-                                if (status->online)
-                                        cli_out ("%d\t%c\t%s",
-                                                 status->port,
-                                                 status->online?'Y':'N',
-                                                 status->pid_str);
-                                else
-                                        cli_out ("%s\t%c\t%s",
-                                                 "N/A",
-                                                 status->online?'Y':'N',
-                                                 status->pid_str);
-                        }
-                        else
-                                cli_out ("%s\t%c\t%s",
-                                         "N/A", status->online?'Y':'N',
-                                         status->pid_str);
-                        bricklen = 0;
-                }
-        }
+    char buff[PATH_MAX] = {0};
+    runner_t runner = {0};
+    char *filename = NULL;
+    char *hostname = NULL;
+    char *path = NULL;
+    char *volname = NULL;
+    char *out = NULL;
+    int ret = 0;
+
+    runinit(&runner);
+    ret = dict_get_str(options, "volname", &volname);
+    runner_add_args(&runner, GLFSHEAL_PREFIX "/glfsheal", volname, NULL);
+    runner_redir(&runner, STDOUT_FILENO, RUN_PIPE);
+
+    switch (heal_op) {
+        case GF_SHD_OP_INDEX_SUMMARY:
+            if (global_state->mode & GLUSTER_MODE_XML) {
+                runner_add_args(&runner, "--xml", NULL);
+            }
+            break;
+        case GF_SHD_OP_SBRAIN_HEAL_FROM_BIGGER_FILE:
+            ret = dict_get_str(options, "file", &filename);
+            runner_add_args(&runner, "bigger-file", filename, NULL);
+            break;
+        case GF_SHD_OP_SBRAIN_HEAL_FROM_LATEST_MTIME:
+            ret = dict_get_str(options, "file", &filename);
+            runner_add_args(&runner, "latest-mtime", filename, NULL);
+            break;
+        case GF_SHD_OP_SBRAIN_HEAL_FROM_BRICK:
+            ret = dict_get_str(options, "heal-source-hostname", &hostname);
+            ret = dict_get_str(options, "heal-source-brickpath", &path);
+            runner_add_args(&runner, "source-brick", NULL);
+            runner_argprintf(&runner, "%s:%s", hostname, path);
+            if (dict_get_str(options, "file", &filename) == 0)
+                runner_argprintf(&runner, "%s", filename);
+            break;
+        case GF_SHD_OP_SPLIT_BRAIN_FILES:
+            runner_add_args(&runner, "split-brain-info", NULL);
+            if (global_state->mode & GLUSTER_MODE_XML) {
+                runner_add_args(&runner, "--xml", NULL);
+            }
+            break;
+        case GF_SHD_OP_GRANULAR_ENTRY_HEAL_ENABLE:
+        case GF_SHD_OP_GRANULAR_ENTRY_HEAL_DISABLE:
+            runner_add_args(&runner, "granular-entry-heal-op", NULL);
+            break;
+        case GF_SHD_OP_HEAL_SUMMARY:
+            runner_add_args(&runner, "info-summary", NULL);
+            if (global_state->mode & GLUSTER_MODE_XML) {
+                runner_add_args(&runner, "--xml", NULL);
+            }
+            break;
+        default:
+            ret = -1;
+            goto out;
+    }
+    if (global_state->mode & GLUSTER_MODE_GLFSHEAL_NOLOG)
+        runner_add_args(&runner, "--nolog", NULL);
+    ret = runner_start(&runner);
+    if (ret == -1)
+        goto out;
+    while ((
+        out = fgets(buff, sizeof(buff), runner_chio(&runner, STDOUT_FILENO)))) {
+        printf("%s", out);
+    }
+    ret = runner_end(&runner);
 
-        return 0;
+out:
+    return ret;
 }
 
 int
-cli_cmd_volume_heal_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                          const char **words, int wordcount)
+cli_cmd_volume_heal_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                        const char **words, int wordcount)
 {
-        int                     ret = -1;
-        rpc_clnt_procedure_t    *proc = NULL;
-        call_frame_t            *frame = NULL;
-        int                     sent = 0;
-        int                     parse_error = 0;
-        dict_t                  *options = NULL;
-        xlator_t                *this = NULL;
-        cli_local_t             *local = NULL;
-        int                     heal_op = 0;
-        runner_t                runner = {0};
-        char                    buff[PATH_MAX] = {0};
-        char                    *out = NULL;
-
-        this = THIS;
-        frame = create_frame (this, this->ctx->pool);
-        if (!frame)
-                goto out;
-
-        if (wordcount < 3) {
-               cli_usage_out (word->pattern);
-               parse_error = 1;
-               goto out;
-        }
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    int sent = 0;
+    int parse_error = 0;
+    dict_t *options = NULL;
+    xlator_t *this = NULL;
+    cli_local_t *local = NULL;
+    int heal_op = 0;
+
+    this = THIS;
+
+    if (wordcount < 3) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
 
-        ret = cli_cmd_volume_heal_options_parse (words, wordcount, &options);
-        if (ret) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                goto out;
-        }
-        ret = dict_get_int32 (options, "heal-op", &heal_op);
+    ret = cli_cmd_volume_heal_options_parse(words, wordcount, &options);
+    if (ret) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
+    ret = dict_get_int32(options, "heal-op", &heal_op);
+    if (ret < 0)
+        goto out;
+    if (NEEDS_GLFS_HEAL(heal_op)) {
+        ret = cli_launch_glfs_heal(heal_op, options);
         if (ret < 0)
-                goto out;
+            goto out;
+        if (heal_op != GF_SHD_OP_GRANULAR_ENTRY_HEAL_ENABLE)
+            goto out;
+    }
 
-        if (heal_op == GF_AFR_OP_INDEX_SUMMARY) {
-                runinit (&runner);
-                runner_add_args (&runner, SBIN_DIR"/glfsheal", words[2], NULL);
-                runner_redir (&runner, STDOUT_FILENO, RUN_PIPE);
-                ret = runner_start (&runner);
-                if (ret == -1)
-                        goto out;
-                while ((out = fgets(buff, sizeof(buff),
-                                   runner_chio (&runner, STDOUT_FILENO)))) {
-                        printf ("%s", out);
-                }
-
-                ret = runner_end (&runner);
-                ret = WEXITSTATUS (ret);
-        }
-        else {
-                proc = &cli_rpc_prog->proctable[GLUSTER_CLI_HEAL_VOLUME];
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_HEAL_VOLUME];
 
-                CLI_LOCAL_INIT (local, words, frame, options);
+    frame = create_frame(this, this->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
 
-                if (proc->fn) {
-                        ret = proc->fn (frame, THIS, options);
-                }
-        }
+    CLI_LOCAL_INIT(local, words, frame, options);
 
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, options);
+    }
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_out ("Volume heal failed");
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0) &&
+            !(global_state->mode & GLUSTER_MODE_XML)) {
+            cli_out("Volume heal failed.");
         }
+    }
+
+    if (options)
+        dict_unref(options);
 
-        CLI_STACK_DESTROY (frame);
+    CLI_STACK_DESTROY(frame);
 
-        return ret;
+    return ret;
 }
 
 int
-cli_cmd_volume_statedump_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                              const char **words, int wordcount)
+cli_cmd_volume_statedump_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                             const char **words, int wordcount)
 {
-        int                             ret = -1;
-        rpc_clnt_procedure_t            *proc = NULL;
-        call_frame_t                    *frame = NULL;
-        dict_t                          *options = NULL;
-        int                             sent = 0;
-        int                             parse_error = 0;
-        cli_local_t                     *local = NULL;
-
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *options = NULL;
+    int sent = 0;
+    int parse_error = 0;
+    cli_local_t *local = NULL;
+
+    if (wordcount < 3) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
 
-        if (wordcount < 3) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                goto out;
+    if (wordcount >= 3) {
+        ret = cli_cmd_volume_statedump_options_parse(words, wordcount,
+                                                     &options);
+        if (ret) {
+            parse_error = 1;
+            gf_log("cli", GF_LOG_ERROR,
+                   "Error parsing "
+                   "statedump options");
+            cli_out("Error parsing options");
+            cli_usage_out(word->pattern);
         }
+    }
 
-        if (wordcount >= 3) {
-               ret = cli_cmd_volume_statedump_options_parse (words, wordcount,
-                                                              &options);
-               if (ret) {
-                       parse_error = 1;
-                       gf_log ("cli", GF_LOG_ERROR, "Error parsing "
-                               "statedump options");
-                       cli_out ("Error parsing options");
-                       cli_usage_out (word->pattern);
-               }
-        }
+    ret = dict_set_str(options, "volname", (char *)words[2]);
+    if (ret)
+        goto out;
 
-        ret = dict_set_str (options, "volname", (char *)words[2]);
-        if (ret)
-                goto out;
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_STATEDUMP_VOLUME];
 
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_STATEDUMP_VOLUME];
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
 
-        CLI_LOCAL_INIT (local, words, frame, options);
+    CLI_LOCAL_INIT(local, words, frame, options);
 
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, options);
-        }
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, options);
+    }
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error = 0))
-                        cli_out ("Volume statedump failed");
-        }
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Volume statedump failed");
+    }
 
-        CLI_STACK_DESTROY (frame);
+    CLI_STACK_DESTROY(frame);
 
-        return ret;
+    return ret;
 }
 
 int
-cli_cmd_volume_list_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                         const char **words, int wordcount)
+cli_cmd_volume_list_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                        const char **words, int wordcount)
 {
-        int                     ret = -1;
-        call_frame_t            *frame = NULL;
-        rpc_clnt_procedure_t    *proc = NULL;
-        int                     sent = 0;
-
-        frame = create_frame (THIS, THIS->ctx->pool);
+    int ret = -1;
+    call_frame_t *frame = NULL;
+    rpc_clnt_procedure_t *proc = NULL;
+    int sent = 0;
+
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_LIST_VOLUME];
+    if (proc->fn) {
+        frame = create_frame(THIS, THIS->ctx->pool);
         if (!frame)
-                goto out;
-
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_LIST_VOLUME];
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, NULL);
-        }
+            goto out;
+        ret = proc->fn(frame, THIS, NULL);
+    }
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if (sent == 0)
-                        cli_out ("Volume list failed");
-        }
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if (sent == 0)
+            cli_out("Volume list failed");
+    }
 
-        CLI_STACK_DESTROY (frame);
+    CLI_STACK_DESTROY(frame);
 
-        return ret;
+    return ret;
 }
 
 int
-cli_cmd_volume_clearlocks_cbk (struct cli_state *state,
-                               struct cli_cmd_word *word,
-                               const char **words, int wordcount)
+cli_cmd_volume_clearlocks_cbk(struct cli_state *state,
+                              struct cli_cmd_word *word, const char **words,
+                              int wordcount)
 {
-        int                             ret = -1;
-        rpc_clnt_procedure_t            *proc = NULL;
-        call_frame_t                    *frame = NULL;
-        dict_t                          *options = NULL;
-        int                             sent = 0;
-        int                             parse_error = 0;
-        cli_local_t                     *local = NULL;
-
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
-
-        if (wordcount < 7 || wordcount > 8) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                goto out;
-        }
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *options = NULL;
+    int sent = 0;
+    int parse_error = 0;
+    cli_local_t *local = NULL;
+
+    if (wordcount < 7 || wordcount > 8) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
+
+    ret = cli_cmd_volume_clrlks_opts_parse(words, wordcount, &options);
+    if (ret) {
+        parse_error = 1;
+        gf_log("cli", GF_LOG_ERROR,
+               "Error parsing "
+               "clear-locks options");
+        cli_out("Error parsing options");
+        cli_usage_out(word->pattern);
+    }
+
+    ret = dict_set_str(options, "volname", (char *)words[2]);
+    if (ret)
+        goto out;
 
-       ret = cli_cmd_volume_clrlks_opts_parse (words, wordcount, &options);
-       if (ret) {
-               parse_error = 1;
-               gf_log ("cli", GF_LOG_ERROR, "Error parsing "
-                       "clear-locks options");
-               cli_out ("Error parsing options");
-               cli_usage_out (word->pattern);
-       }
+    ret = dict_set_str(options, "path", (char *)words[3]);
+    if (ret)
+        goto out;
 
-        ret = dict_set_str (options, "volname", (char *)words[2]);
-        if (ret)
-                goto out;
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_CLRLOCKS_VOLUME];
 
-        ret = dict_set_str (options, "path", (char *)words[3]);
-        if (ret)
-                goto out;
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
 
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_CLRLOCKS_VOLUME];
+    CLI_LOCAL_INIT(local, words, frame, options);
 
-        CLI_LOCAL_INIT (local, words, frame, options);
-
-        if (proc->fn) {
-                ret = proc->fn (frame, THIS, options);
-        }
+    if (proc->fn) {
+        ret = proc->fn(frame, THIS, options);
+    }
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error = 0))
-                        cli_out ("Volume clear-locks failed");
-        }
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_out("Volume clear-locks failed");
+    }
 
-        CLI_STACK_DESTROY (frame);
+    CLI_STACK_DESTROY(frame);
 
-        return ret;
+    return ret;
 }
 
 int
-cli_cmd_volume_barrier_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                            const char **words, int wordcount)
+cli_cmd_volume_barrier_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                           const char **words, int wordcount)
 {
-        int ret = -1;
-        rpc_clnt_procedure_t *proc = NULL;
-        call_frame_t *frame = NULL;
-        dict_t *options = NULL;
-        int sent = 0;
-        int parse_error = 0;
-        cli_local_t *local = NULL;
-
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *options = NULL;
+    int sent = 0;
+    int parse_error = 0;
+    cli_local_t *local = NULL;
+
+    if (wordcount != 4) {
+        cli_usage_out(word->pattern);
+        parse_error = 1;
+        goto out;
+    }
 
-        if (wordcount != 4) {
-                cli_usage_out (word->pattern);
-                parse_error = 1;
-                goto out;
-        }
+    options = dict_new();
+    if (!options) {
+        ret = -1;
+        goto out;
+    }
+    ret = dict_set_str(options, "volname", (char *)words[2]);
+    if (ret)
+        goto out;
 
-        options = dict_new();
-        if (!options) {
-                ret = -1;
-                goto out;
-        }
-        ret = dict_set_str(options, "volname", (char *)words[2]);
-        if (ret)
-                goto out;
+    ret = dict_set_str(options, "barrier", (char *)words[3]);
+    if (ret)
+        goto out;
 
-        ret = dict_set_str (options, "barrier", (char *)words[3]);
-        if (ret)
-                goto out;
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_BARRIER_VOLUME];
 
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_BARRIER_VOLUME];
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
 
-        CLI_LOCAL_INIT (local, words, frame, options);
+    CLI_LOCAL_INIT(local, words, frame, options);
 
-        if (proc->fn)
-                ret = proc->fn (frame, THIS, options);
+    if (proc->fn)
+        ret = proc->fn(frame, THIS, options);
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_error == 0))
-                        cli_err ("Volume barrier failed");
-        }
-        CLI_STACK_DESTROY (frame);
-        if (options)
-                dict_unref (options);
-
-        return ret;
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_error == 0))
+            cli_err("Volume barrier failed");
+    }
+    CLI_STACK_DESTROY(frame);
+
+    return ret;
 }
 
 int
-cli_cmd_volume_getopt_cbk (struct cli_state *state, struct cli_cmd_word *word,
-                        const char **words, int wordcount)
+cli_cmd_volume_getopt_cbk(struct cli_state *state, struct cli_cmd_word *word,
+                          const char **words, int wordcount)
 {
-        int                   ret       = -1;
-        rpc_clnt_procedure_t *proc      = NULL;
-        call_frame_t         *frame     = NULL;
-        dict_t               *options   = NULL;
-        int                   sent      = 0;
-        int                   parse_err = 0;
-        cli_local_t          *local     = NULL;
-
-        if (wordcount != 4) {
-                cli_usage_out (word->pattern);
-                parse_err = 1;
-                goto out;
-        }
+    int ret = -1;
+    rpc_clnt_procedure_t *proc = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *options = NULL;
+    int sent = 0;
+    int parse_err = 0;
+    cli_local_t *local = NULL;
+
+    if (wordcount != 4) {
+        cli_usage_out(word->pattern);
+        parse_err = 1;
+        goto out;
+    }
 
-        frame = create_frame (THIS, THIS->ctx->pool);
-        if (!frame)
-                goto out;
+    options = dict_new();
+    if (!options)
+        goto out;
 
-        options = dict_new ();
-        if (!options)
-                goto out;
+    ret = dict_set_str(options, "volname", (char *)words[2]);
+    if (ret)
+        goto out;
 
-        ret = dict_set_str (options, "volname", (char *)words[2]);
-        if (ret)
-                goto out;
+    ret = dict_set_str(options, "key", (char *)words[3]);
+    if (ret)
+        goto out;
 
-        ret = dict_set_str (options, "key", (char *)words[3]);
-        if (ret)
-                goto out;
+    proc = &cli_rpc_prog->proctable[GLUSTER_CLI_GET_VOL_OPT];
 
-        proc = &cli_rpc_prog->proctable[GLUSTER_CLI_GET_VOL_OPT];
+    frame = create_frame(THIS, THIS->ctx->pool);
+    if (!frame) {
+        ret = -1;
+        goto out;
+    }
 
-        CLI_LOCAL_INIT (local, words, frame, options);
+    CLI_LOCAL_INIT(local, words, frame, options);
 
-        if (proc->fn)
-                ret = proc->fn (frame, THIS, options);
+    if (proc->fn)
+        ret = proc->fn(frame, THIS, options);
 
 out:
-        if (ret) {
-                cli_cmd_sent_status_get (&sent);
-                if ((sent == 0) && (parse_err == 0))
-                        cli_err ("Volume get option failed");
-        }
-        CLI_STACK_DESTROY (frame);
-        if (options)
-                dict_unref (options);
-        return ret;
+    if (ret) {
+        cli_cmd_sent_status_get(&sent);
+        if ((sent == 0) && (parse_err == 0))
+            cli_err("Volume get option failed");
+    }
+    CLI_STACK_DESTROY(frame);
+    return ret;
 }
 
+/* This is a bit of a hack to display the help. The current bitrot cmd
+ * format does not work well when registering the cmds.
+ * Ideally the should have been of the form
+ * gluster volume bitrot <subcommand> <volumename> ...
+ */
+
+struct cli_cmd bitrot_cmds[] = {
+
+    {"volume bitrot help", cli_cmd_bitrot_help_cbk,
+     "display help for volume bitrot commands"},
+
+    {"volume bitrot <VOLNAME> {enable|disable}", NULL, /*cli_cmd_bitrot_cbk,*/
+     "Enable/disable bitrot for volume <VOLNAME>"},
+
+    {"volume bitrot <VOLNAME> signing-time <time-in-secs>",
+     NULL, /*cli_cmd_bitrot_cbk,*/
+     "Waiting time for an object after last fd is closed to start signing "
+     "process"},
+
+    {"volume bitrot <VOLNAME> signer-threads <count>",
+     NULL, /*cli_cmd_bitrot_cbk,*/
+     "Number of signing process threads. Usually set to number of available "
+     "cores"},
+
+    {"volume bitrot <VOLNAME> scrub-throttle {lazy|normal|aggressive}",
+     NULL, /*cli_cmd_bitrot_cbk,*/
+     "Set the speed of the scrubber for volume <VOLNAME>"},
+
+    {"volume bitrot <VOLNAME> scrub-frequency {hourly|daily|weekly|biweekly"
+     "|monthly}",
+     NULL, /*cli_cmd_bitrot_cbk,*/
+     "Set the frequency of the scrubber for volume <VOLNAME>"},
+
+    {"volume bitrot <VOLNAME> scrub {pause|resume|status|ondemand}",
+     NULL, /*cli_cmd_bitrot_cbk,*/
+     "Pause/resume the scrubber for <VOLNAME>. Status displays the status of "
+     "the scrubber. ondemand starts the scrubber immediately."},
+
+    {"volume bitrot <VOLNAME> {enable|disable}\n"
+     "volume bitrot <VOLNAME> signing-time <time-in-secs>\n"
+     "volume bitrot <VOLNAME> signer-threads <count>\n"
+     "volume bitrot <volname> scrub-throttle {lazy|normal|aggressive}\n"
+     "volume bitrot <volname> scrub-frequency {hourly|daily|weekly|biweekly"
+     "|monthly}\n"
+     "volume bitrot <volname> scrub {pause|resume|status|ondemand}",
+     cli_cmd_bitrot_cbk, NULL},
+
+    {NULL, NULL, NULL}};
+
+struct cli_cmd quota_cmds[] = {
+
+    /* Quota commands */
+    {"volume quota help", cli_cmd_quota_help_cbk,
+     "display help for volume quota commands"},
+
+    {"volume quota <VOLNAME> {enable|disable|list [<path> ...]| "
+     "list-objects [<path> ...] | remove <path>| remove-objects <path> | "
+     "default-soft-limit <percent>}",
+     cli_cmd_quota_cbk, "Enable/disable and configure quota for <VOLNAME>"},
+
+    {"volume quota <VOLNAME> {limit-usage <path> <size> [<percent>]}",
+     cli_cmd_quota_cbk, "Set maximum size for <path> for <VOLNAME>"},
+
+    {"volume quota <VOLNAME> {limit-objects <path> <number> [<percent>]}",
+     cli_cmd_quota_cbk,
+     "Set the maximum number of entries allowed in <path> for <VOLNAME>"},
+
+    {"volume quota <VOLNAME> {alert-time|soft-timeout|hard-timeout} {<time>}",
+     cli_cmd_quota_cbk, "Set quota timeout for <VOLNAME>"},
+
+    {"volume inode-quota <VOLNAME> enable", cli_cmd_quota_cbk,
+     "Enable/disable inode-quota for <VOLNAME>"},
+
+    {"volume quota <VOLNAME> {enable|disable|list [<path> ...]| "
+     "list-objects [<path> ...] | remove <path>| remove-objects <path> | "
+     "default-soft-limit <percent>}\n"
+     "volume quota <VOLNAME> {limit-usage <path> <size> [<percent>]}\n"
+     "volume quota <VOLNAME> {limit-objects <path> <number> [<percent>]}\n"
+     "volume quota <VOLNAME> {alert-time|soft-timeout|hard-timeout} {<time>}",
+     cli_cmd_quota_cbk, NULL},
+
+    {NULL, NULL, NULL}};
+
 struct cli_cmd volume_cmds[] = {
-        { "volume info [all|<VOLNAME>]",
-          cli_cmd_volume_info_cbk,
-          "list information of all volumes"},
-
-        { "volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT>] "
-          "[disperse [<COUNT>]] [redundancy <COUNT>] "
-          "[transport <tcp|rdma|tcp,rdma>] <NEW-BRICK>"
-#ifdef HAVE_BD_XLATOR
-          "?<vg_name>"
-#endif
-          "... [force]",
+    {"volume help", cli_cmd_volume_help_cbk,
+     "display help for volume commands"},
 
-          cli_cmd_volume_create_cbk,
-          "create a new volume of specified type with mentioned bricks"},
+    {"volume info [all|<VOLNAME>]", cli_cmd_volume_info_cbk,
+     "list information of all volumes"},
 
-        { "volume delete <VOLNAME>",
-          cli_cmd_volume_delete_cbk,
-          "delete volume specified by <VOLNAME>"},
+    {"volume create <NEW-VOLNAME> [stripe <COUNT>] "
+     "[[replica <COUNT> [arbiter <COUNT>]]|[replica 2 thin-arbiter 1]] "
+     "[disperse [<COUNT>]] [disperse-data <COUNT>] [redundancy <COUNT>] "
+     "[transport <tcp|rdma|tcp,rdma>] <NEW-BRICK> <TA-BRICK>"
+     "... [force]",
 
-        { "volume start <VOLNAME> [force]",
-          cli_cmd_volume_start_cbk,
-          "start volume specified by <VOLNAME>"},
+     cli_cmd_volume_create_cbk,
+     "create a new volume of specified type with mentioned bricks"},
 
-        { "volume stop <VOLNAME> [force]",
-          cli_cmd_volume_stop_cbk,
-          "stop volume specified by <VOLNAME>"},
+    {"volume delete <VOLNAME>", cli_cmd_volume_delete_cbk,
+     "delete volume specified by <VOLNAME>"},
 
-        /*{ "volume rename <VOLNAME> <NEW-VOLNAME>",
-          cli_cmd_volume_rename_cbk,
-          "rename volume <VOLNAME> to <NEW-VOLNAME>"},*/
+    {"volume start <VOLNAME> [force]", cli_cmd_volume_start_cbk,
+     "start volume specified by <VOLNAME>"},
 
-        { "volume add-brick <VOLNAME> [<stripe|replica> <COUNT>] <NEW-BRICK> ... [force]",
-          cli_cmd_volume_add_brick_cbk,
-          "add brick to volume <VOLNAME>"},
+    {"volume stop <VOLNAME> [force]", cli_cmd_volume_stop_cbk,
+     "stop volume specified by <VOLNAME>"},
 
-        { "volume remove-brick <VOLNAME> [replica <COUNT>] <BRICK> ..."
-          " <start|stop|status|commit|force>",
-          cli_cmd_volume_remove_brick_cbk,
-          "remove brick from volume <VOLNAME>"},
+    /*{ "volume rename <VOLNAME> <NEW-VOLNAME>",
+      cli_cmd_volume_rename_cbk,
+      "rename volume <VOLNAME> to <NEW-VOLNAME>"},*/
 
-        { "volume rebalance <VOLNAME> {{fix-layout start} | {start [force]|stop|status}}",
-          cli_cmd_volume_defrag_cbk,
-          "rebalance operations"},
+    {"volume add-brick <VOLNAME> [<stripe|replica> <COUNT> "
+     "[arbiter <COUNT>]] <NEW-BRICK> ... [force]",
+     cli_cmd_volume_add_brick_cbk, "add brick to volume <VOLNAME>"},
 
-        { "volume replace-brick <VOLNAME> <BRICK> <NEW-BRICK> {start [force]|pause|abort|status|commit [force]}",
-          cli_cmd_volume_replace_brick_cbk,
-          "replace-brick operations"},
+    {"volume remove-brick <VOLNAME> [replica <COUNT>] <BRICK> ..."
+     " <start|stop|status|commit|force>",
+     cli_cmd_volume_remove_brick_cbk, "remove brick from volume <VOLNAME>"},
 
-        /*{ "volume set-transport <VOLNAME> <TRANSPORT-TYPE> [<TRANSPORT-TYPE>] ...",
-          cli_cmd_volume_set_transport_cbk,
-          "set transport type for volume <VOLNAME>"},*/
+    {"volume rebalance <VOLNAME> {{fix-layout start} | {start "
+     "[force]|stop|status}}",
+     cli_cmd_volume_defrag_cbk, "rebalance operations"},
 
-        { "volume set <VOLNAME> <KEY> <VALUE>",
-          cli_cmd_volume_set_cbk,
-         "set options for volume <VOLNAME>"},
+    {"volume replace-brick <VOLNAME> <SOURCE-BRICK> <NEW-BRICK> "
+     "{commit force}",
+     cli_cmd_volume_replace_brick_cbk, "replace-brick operations"},
 
-        { "volume help",
-          cli_cmd_volume_help_cbk,
-          "display help for the volume command"},
+    /*{ "volume set-transport <VOLNAME> <TRANSPORT-TYPE> [<TRANSPORT-TYPE>]
+      ...", cli_cmd_volume_set_transport_cbk, "set transport type for volume
+      <VOLNAME>"},*/
 
-        { "volume log <VOLNAME> rotate [BRICK]",
-          cli_cmd_log_rotate_cbk,
-         "rotate the log file for corresponding volume/brick"},
+    {"volume set <VOLNAME> <KEY> <VALUE>", cli_cmd_volume_set_cbk,
+     "set options for volume <VOLNAME>"},
 
-        { "volume log rotate <VOLNAME> [BRICK]",
-          cli_cmd_log_rotate_cbk,
-         "rotate the log file for corresponding volume/brick"
-         " NOTE: This is an old syntax, will be deprecated from next release."},
+    {"volume set <VOLNAME> group <GROUP>", cli_cmd_volume_set_cbk,
+     "This option can be used for setting multiple pre-defined volume options "
+     "where group_name is a file under /var/lib/glusterd/groups containing one "
+     "key value pair per line"},
 
-        { "volume sync <HOSTNAME> [all|<VOLNAME>]",
-          cli_cmd_sync_volume_cbk,
-         "sync the volume information from a peer"},
+    {"volume log <VOLNAME> rotate [BRICK]", cli_cmd_log_rotate_cbk,
+     "rotate the log file for corresponding volume/brick"},
 
-         { "volume reset <VOLNAME> [option] [force]",
-         cli_cmd_volume_reset_cbk,
-         "reset all the reconfigured options"},
+    {"volume sync <HOSTNAME> [all|<VOLNAME>]", cli_cmd_sync_volume_cbk,
+     "sync the volume information from a peer"},
+
+    {"volume reset <VOLNAME> [option] [force]", cli_cmd_volume_reset_cbk,
+     "reset all the reconfigured options"},
 
 #if (SYNCDAEMON_COMPILE)
-        {"volume "GEOREP" [<VOLNAME>] [<SLAVE-URL>] {create [push-pem] [force]"
-         "|start [force]|stop [force]|pause [force]|resume [force]|config|status [detail]|delete} [options...]",
-         cli_cmd_volume_gsync_set_cbk,
-         "Geo-sync operations",
-         cli_cmd_check_gsync_exists_cbk},
+    {"volume " GEOREP " [<MASTER-VOLNAME>] [<SLAVE-IP>]::[<SLAVE-VOLNAME>] {"
+     "\\\n create [[ssh-port n] [[no-verify] \\\n | [push-pem]]] [force] \\\n"
+     " | start [force] \\\n | stop [force] \\\n | pause [force] \\\n | resume "
+     "[force] \\\n"
+     " | config [[[\\!]<option>] [<value>]] \\\n | status "
+     "[detail] \\\n | delete [reset-sync-time]} ",
+     cli_cmd_volume_gsync_set_cbk, "Geo-sync operations",
+     cli_cmd_check_gsync_exists_cbk},
 #endif
 
-         { "volume profile <VOLNAME> {start|info [peek|incremental [peek]|cumulative|clear]|stop} [nfs]",
-           cli_cmd_volume_profile_cbk,
-           "volume profile operations"},
-
-        { "volume quota <VOLNAME> {enable|disable|list [<path> ...]|remove <path>| default-soft-limit <percent>} |\n"
-          "volume quota <VOLNAME> {limit-usage <path> <size> [<percent>]} |\n"
-          "volume quota <VOLNAME> {alert-time|soft-timeout|hard-timeout} {<time>}",
-          cli_cmd_quota_cbk,
-          "quota translator specific operations"},
-
-         { "volume top <VOLNAME> {open|read|write|opendir|readdir|clear} [nfs|brick <brick>] [list-cnt <value>] |\n"
-           "volume top <VOLNAME> {read-perf|write-perf} [bs <size> count <count>] [brick <brick>] [list-cnt <value>]",
-           cli_cmd_volume_top_cbk,
-           "volume top operations"},
-
-        { "volume status [all | <VOLNAME> [nfs|shd|<BRICK>|quotad]]"
-          " [detail|clients|mem|inode|fd|callpool|tasks]",
-          cli_cmd_volume_status_cbk,
-          "display status of all or specified volume(s)/brick"},
-
-        { "volume heal <VOLNAME> [{full | statistics {heal-count {replica <hostname:brickname>}} |info {healed | heal-failed | split-brain}}]",
-          cli_cmd_volume_heal_cbk,
-          "self-heal commands on volume specified by <VOLNAME>"},
-
-        {"volume statedump <VOLNAME> [nfs|quotad] [all|mem|iobuf|callpool|priv|fd|"
-         "inode|history]...",
-         cli_cmd_volume_statedump_cbk,
-         "perform statedump on bricks"},
-
-        {"volume list",
-         cli_cmd_volume_list_cbk,
-         "list all volumes in cluster"},
-
-        {"volume clear-locks <VOLNAME> <path> kind {blocked|granted|all}"
-          "{inode [range]|entry [basename]|posix [range]}",
-          cli_cmd_volume_clearlocks_cbk,
-          "Clear locks held on path"
-        },
-        {"volume barrier <VOLNAME> {enable|disable}",
-         cli_cmd_volume_barrier_cbk,
-         "Barrier/unbarrier file operations on a volume"
-        },
-        {"volume get <VOLNAME> <key|all>",
-         cli_cmd_volume_getopt_cbk,
-         "Get the value of the all options or given option for volume <VOLNAME>"},
-        { NULL, NULL, NULL }
-};
+    {"volume profile <VOLNAME> {start|info [peek|incremental "
+     "[peek]|cumulative|clear]|stop} [nfs]",
+     cli_cmd_volume_profile_cbk, "volume profile operations"},
+
+    {"volume top <VOLNAME> {open|read|write|opendir|readdir|clear} [nfs|brick "
+     "<brick>] [list-cnt <value>] | "
+     "{read-perf|write-perf} [bs <size> count <count>] "
+     "[brick <brick>] [list-cnt <value>]",
+     cli_cmd_volume_top_cbk, "volume top operations"},
+
+    {"volume status [all | <VOLNAME> [nfs|shd|<BRICK>|quotad]]"
+     " [detail|clients|mem|inode|fd|callpool|tasks|client-list]",
+     cli_cmd_volume_status_cbk,
+     "display status of all or specified volume(s)/brick"},
+
+    {"volume heal <VOLNAME> [enable | disable | full |"
+     "statistics [heal-count [replica <HOSTNAME:BRICKNAME>]] |"
+     "info [summary | split-brain] |"
+     "split-brain {bigger-file <FILE> | latest-mtime <FILE> |"
+     "source-brick <HOSTNAME:BRICKNAME> [<FILE>]} |"
+     "granular-entry-heal {enable | disable}]",
+     cli_cmd_volume_heal_cbk,
+     "self-heal commands on volume specified by <VOLNAME>"},
+
+    {"volume statedump <VOLNAME> [[nfs|quotad] [all|mem|iobuf|callpool|"
+     "priv|fd|inode|history]... | [client <hostname:process-id>]]",
+     cli_cmd_volume_statedump_cbk, "perform statedump on bricks"},
+
+    {"volume list", cli_cmd_volume_list_cbk, "list all volumes in cluster"},
+
+    {"volume clear-locks <VOLNAME> <path> kind {blocked|granted|all}"
+     "{inode [range]|entry [basename]|posix [range]}",
+     cli_cmd_volume_clearlocks_cbk, "Clear locks held on path"},
+    {"volume barrier <VOLNAME> {enable|disable}", cli_cmd_volume_barrier_cbk,
+     "Barrier/unbarrier file operations on a volume"},
+    {"volume get <VOLNAME|all> <key|all>", cli_cmd_volume_getopt_cbk,
+     "Get the value of the all options or given option for volume <VOLNAME>"
+     " or all option. gluster volume get all all is to get all global "
+     "options"},
+
+    {"volume reset-brick <VOLNAME> <SOURCE-BRICK> {{start} |"
+     " {<NEW-BRICK> commit}}",
+     cli_cmd_volume_reset_brick_cbk, "reset-brick operations"},
+
+    {NULL, NULL, NULL}};
 
 int
-cli_cmd_volume_help_cbk (struct cli_state *state, struct cli_cmd_word *in_word,
-                      const char **words, int wordcount)
+cli_cmd_quota_help_cbk(struct cli_state *state, struct cli_cmd_word *in_word,
+                       const char **words, int wordcount)
 {
-        struct cli_cmd        *cmd = NULL;
+    struct cli_cmd *cmd = NULL;
+    struct cli_cmd *quota_cmd = NULL;
+    int count = 0;
+
+    cmd = GF_MALLOC(sizeof(quota_cmds), cli_mt_cli_cmd);
+    memcpy(cmd, quota_cmds, sizeof(quota_cmds));
+    count = (sizeof(quota_cmds) / sizeof(struct cli_cmd));
+    cli_cmd_sort(cmd, count);
+
+    cli_out("\ngluster quota commands");
+    cli_out("=======================\n");
 
-        for (cmd = volume_cmds; cmd->pattern; cmd++)
-                if (_gf_false == cmd->disable)
-                        cli_out ("%s - %s", cmd->pattern, cmd->desc);
+    for (quota_cmd = cmd; quota_cmd->pattern; quota_cmd++)
+        if ((_gf_false == quota_cmd->disable) && (quota_cmd->desc))
+            cli_out("%s - %s", quota_cmd->pattern, quota_cmd->desc);
 
-        return 0;
+    cli_out("\n");
+    GF_FREE(cmd);
+
+    return 0;
 }
 
 int
-cli_cmd_volume_register (struct cli_state *state)
+cli_cmd_bitrot_help_cbk(struct cli_state *state, struct cli_cmd_word *in_word,
+                        const char **words, int wordcount)
 {
-        int  ret = 0;
-        struct cli_cmd *cmd = NULL;
+    struct cli_cmd *cmd = NULL;
+    struct cli_cmd *bitrot_cmd = NULL;
+    int count = 0;
 
-        for (cmd = volume_cmds; cmd->pattern; cmd++) {
+    cmd = GF_MALLOC(sizeof(bitrot_cmds), cli_mt_cli_cmd);
+    memcpy(cmd, bitrot_cmds, sizeof(bitrot_cmds));
+    count = (sizeof(bitrot_cmds) / sizeof(struct cli_cmd));
+    cli_cmd_sort(cmd, count);
+
+    cli_out("\ngluster bitrot commands");
+    cli_out("========================\n");
+
+    for (bitrot_cmd = cmd; bitrot_cmd->pattern; bitrot_cmd++)
+        if ((_gf_false == bitrot_cmd->disable) && (bitrot_cmd->desc))
+            cli_out("%s - %s", bitrot_cmd->pattern, bitrot_cmd->desc);
+
+    cli_out("\n");
+    GF_FREE(cmd);
+
+    return 0;
+}
+
+int
+cli_cmd_volume_help_cbk(struct cli_state *state, struct cli_cmd_word *in_word,
+                        const char **words, int wordcount)
+{
+    struct cli_cmd *cmd = NULL;
+    struct cli_cmd *vol_cmd = NULL;
+    int count = 0;
+
+    cmd = GF_MALLOC(sizeof(volume_cmds), cli_mt_cli_cmd);
+    memcpy(cmd, volume_cmds, sizeof(volume_cmds));
+    count = (sizeof(volume_cmds) / sizeof(struct cli_cmd));
+    cli_cmd_sort(cmd, count);
+
+    cli_out("\ngluster volume commands");
+    cli_out("========================\n");
+
+    for (vol_cmd = cmd; vol_cmd->pattern; vol_cmd++)
+        if (_gf_false == vol_cmd->disable)
+            cli_out("%s - %s", vol_cmd->pattern, vol_cmd->desc);
+
+    cli_out("\n");
+    GF_FREE(cmd);
+    return 0;
+}
+
+int
+cli_cmd_volume_register(struct cli_state *state)
+{
+    int ret = 0;
+    struct cli_cmd *cmd = NULL;
+
+    for (cmd = volume_cmds; cmd->pattern; cmd++) {
+        ret = cli_cmd_register(&state->tree, cmd);
+        if (ret)
+            goto out;
+    }
+
+    for (cmd = bitrot_cmds; cmd->pattern; cmd++) {
+        ret = cli_cmd_register(&state->tree, cmd);
+        if (ret)
+            goto out;
+    }
+
+    for (cmd = quota_cmds; cmd->pattern; cmd++) {
+        ret = cli_cmd_register(&state->tree, cmd);
+        if (ret)
+            goto out;
+    }
 
-                ret = cli_cmd_register (&state->tree, cmd);
-                if (ret)
-                        goto out;
-        }
 out:
-        return ret;
+    return ret;
+}
+
+static int
+gf_asprintf_append(char **string_ptr, const char *format, ...)
+{
+    va_list arg;
+    int rv = 0;
+    char *tmp = *string_ptr;
+
+    va_start(arg, format);
+    rv = gf_vasprintf(string_ptr, format, arg);
+    va_end(arg);
+
+    if (tmp)
+        GF_FREE(tmp);
+
+    return rv;
 }
diff --git a/cli/src/cli-cmd.c b/cli/src/cli-cmd.c
index b415569be28..2d458b16a56 100644
--- a/cli/src/cli-cmd.c
+++ b/cli/src/cli-cmd.c
@@ -13,11 +13,6 @@
 #include <stdint.h>
 #include <pthread.h>
 
-#ifndef _CONFIG_H
-#define _CONFIG_H
-#include "config.h"
-#endif
-
 #include "cli.h"
 #include "cli-cmd.h"
 #include "cli-mem-types.h"
@@ -27,364 +22,390 @@
 
 static int cmd_done;
 static int cmd_sent;
-static pthread_cond_t      cond  = PTHREAD_COND_INITIALIZER;
-static pthread_mutex_t     cond_mutex = PTHREAD_MUTEX_INITIALIZER;
-static pthread_cond_t      conn  = PTHREAD_COND_INITIALIZER;
-static pthread_mutex_t     conn_mutex = PTHREAD_MUTEX_INITIALIZER;
-
-int    cli_op_ret = 0;
-int    connected = 0;
+static pthread_cond_t cond = PTHREAD_COND_INITIALIZER;
+static pthread_mutex_t cond_mutex = PTHREAD_MUTEX_INITIALIZER;
+static pthread_cond_t conn = PTHREAD_COND_INITIALIZER;
+static pthread_mutex_t conn_mutex = PTHREAD_MUTEX_INITIALIZER;
 
-int cli_cmd_log_help_cbk (struct cli_state *state, struct cli_cmd_word *in_word,
-                      const char **words, int wordcount);
+int cli_op_ret = 0;
+static gf_boolean_t connected = _gf_false;
 
 static unsigned
-cli_cmd_needs_connection (struct cli_cmd_word *word)
+cli_cmd_needs_connection(struct cli_cmd_word *word)
 {
-        if (!strcasecmp ("quit", word->word))
-                return 0;
+    if (!strcasecmp("quit", word->word))
+        return 0;
 
-        if (!strcasecmp ("help", word->word))
-                return 0;
+    if (!strcasecmp("help", word->word))
+        return 0;
 
-        if (!strcasecmp ("getwd", word->word))
-                return 1;
+    if (!strcasecmp("getwd", word->word))
+        return 1;
 
-        if (!strcasecmp ("exit", word->word))
-                return 0;
+    if (!strcasecmp("exit", word->word))
+        return 0;
 
-        return CLI_DEFAULT_CONN_TIMEOUT;
+    return cli_default_conn_timeout;
 }
 
 int
-cli_cmd_status_reset (void)
+cli_cmd_status_reset(void)
 {
-        int ret = 0;
-
-        ret = cli_cmd_lock ();
-        {
-                if (ret == 0) {
-                        cmd_sent = 0;
-                        cmd_done = 0;
-                }
-        }
-        ret = cli_cmd_unlock ();
-        return ret;
+    int ret = 0;
 
+    ret = cli_cmd_lock();
+    {
+        if (ret == 0) {
+            cmd_sent = 0;
+            cmd_done = 0;
+        }
+    }
+    ret = cli_cmd_unlock();
+    return ret;
 }
 
 int
-cli_cmd_sent_status_get (int *status)
+cli_cmd_sent_status_get(int *status)
 {
-        int ret = 0;
-        GF_ASSERT (status);
-
-        ret = cli_cmd_lock ();
-        {
-                if (ret == 0)
-                        *status = cmd_sent;
-        }
-        ret = cli_cmd_unlock ();
-        return ret;
+    int ret = 0;
+    GF_ASSERT(status);
+
+    ret = cli_cmd_lock();
+    {
+        if (ret == 0)
+            *status = cmd_sent;
+    }
+    ret = cli_cmd_unlock();
+    return ret;
 }
 
 int
-cli_cmd_process (struct cli_state *state, int argc, char **argv)
+cli_cmd_process(struct cli_state *state, int argc, char **argv)
 {
-        int                  ret = 0;
-        struct cli_cmd_word *word = NULL;
-        struct cli_cmd_word *next = NULL;
-        int                  i = 0;
+    int ret = 0;
+    struct cli_cmd_word *word = NULL;
+    struct cli_cmd_word *next = NULL;
+    int i = 0;
 
-        word = &state->tree.root;
+    word = &state->tree.root;
 
-        if (!argc)
-                return 0;
+    if (!argc)
+        return 0;
 
-        for (i = 0; i < argc; i++) {
-                next = cli_cmd_nextword (word, argv[i]);
+    for (i = 0; i < argc; i++) {
+        next = cli_cmd_nextword(word, argv[i]);
 
-                word = next;
-                if (!word)
-                        break;
+        word = next;
+        if (!word)
+            break;
 
-                if (word->cbkfn)
-                        break;
-        }
+        if (word->cbkfn)
+            break;
+    }
 
-        if (!word) {
-                cli_out ("unrecognized word: %s (position %d)",
-                         argv[i], i);
-                return -1;
-        }
+    if (!word) {
+        cli_out("unrecognized word: %s (position %d)\n", argv[i], i);
+        usage();
+        return -1;
+    }
 
-        if (!word->cbkfn) {
-                cli_out ("unrecognized command");
-                return -1;
-        }
+    if (!word->cbkfn) {
+        cli_out("unrecognized command\n");
+        usage();
+        return -1;
+    }
 
-	if ( strcmp (word->word,"help")==0 )
-		goto callback;
+    if (strcmp(word->word, "help") == 0)
+        goto callback;
 
-        state->await_connected = cli_cmd_needs_connection (word);
+    state->await_connected = cli_cmd_needs_connection(word);
 
-        ret = cli_cmd_await_connected (state->await_connected);
-        if (ret) {
-                cli_out ("Connection failed. Please check if gluster "
-                          "daemon is operational.");
-                gf_log ("", GF_LOG_INFO, "Exiting with: %d", ret);
-                exit (ret);
-        }
+    ret = cli_cmd_await_connected(state->await_connected);
+    if (ret) {
+        cli_out(
+            "Connection failed. Please check if gluster "
+            "daemon is operational.");
+        gf_log("", GF_LOG_INFO, "Exiting with: %d", ret);
+        exit(ret);
+    }
 
 callback:
-        ret = word->cbkfn (state, word, (const char **)argv, argc);
-        (void) cli_cmd_status_reset ();
-        return ret;
+    ret = word->cbkfn(state, word, (const char **)argv, argc);
+    (void)cli_cmd_status_reset();
+    return ret;
 }
 
 int
-cli_cmd_input_token_count (const char *text)
+cli_cmd_input_token_count(const char *text)
 {
-        int          count = 0;
-        const char  *trav = NULL;
-        int          is_spc = 1;
-
-        for (trav = text; *trav; trav++) {
-                if (*trav == ' ') {
-                        is_spc = 1;
-                } else {
-                        if (is_spc) {
-                                count++;
-                                is_spc = 0;
-                        }
-                }
+    int count = 0;
+    const char *trav = NULL;
+    int is_spc = 1;
+
+    for (trav = text; *trav; trav++) {
+        if (*trav == ' ') {
+            is_spc = 1;
+        } else {
+            if (is_spc) {
+                count++;
+                is_spc = 0;
+            }
         }
+    }
 
-        return count;
+    return count;
 }
 
-
 int
-cli_cmd_process_line (struct cli_state *state, const char *text)
+cli_cmd_process_line(struct cli_state *state, const char *text)
 {
-        int     count = 0;
-        char  **tokens = NULL;
-        char  **tokenp = NULL;
-        char   *token = NULL;
-        char   *copy = NULL;
-        char   *saveptr = NULL;
-        int     i = 0;
-        int     ret = -1;
-
-        count = cli_cmd_input_token_count (text);
-
-        tokens = calloc (count + 1, sizeof (*tokens));
-        if (!tokens)
-                return -1;
-
-        copy = strdup (text);
-        if (!copy)
-                goto out;
-
-        tokenp = tokens;
-
-        for (token = strtok_r (copy, " \t\r\n", &saveptr); token;
-             token = strtok_r (NULL, " \t\r\n", &saveptr)) {
-                *tokenp = strdup (token);
-
-                if (!*tokenp)
-                        goto out;
-                tokenp++;
-                i++;
-
-        }
-
-        ret = cli_cmd_process (state, count, tokens);
+    int count = 0;
+    char **tokens = NULL;
+    char **tokenp = NULL;
+    char *token = NULL;
+    char *copy = NULL;
+    char *saveptr = NULL;
+    int i = 0;
+    int ret = -1;
+
+    count = cli_cmd_input_token_count(text);
+
+    tokens = calloc(count + 1, sizeof(*tokens));
+    if (!tokens)
+        return -1;
+
+    copy = strdup(text);
+    if (!copy)
+        goto out;
+
+    tokenp = tokens;
+
+    for (token = strtok_r(copy, " \t\r\n", &saveptr); token;
+         token = strtok_r(NULL, " \t\r\n", &saveptr)) {
+        *tokenp = strdup(token);
+
+        if (!*tokenp)
+            goto out;
+        tokenp++;
+        i++;
+    }
+
+    ret = cli_cmd_process(state, count, tokens);
 out:
-        free (copy);
+    free(copy);
 
-        if (tokens)
-                cli_cmd_tokens_destroy (tokens);
+    if (tokens)
+        cli_cmd_tokens_destroy(tokens);
 
-        return ret;
+    return ret;
 }
 
-
 int
-cli_cmds_register (struct cli_state *state)
+cli_cmds_register(struct cli_state *state)
 {
-        int  ret = 0;
-
-        ret = cli_cmd_volume_register (state);
-        if (ret)
-                goto out;
-
-        ret = cli_cmd_probe_register (state);
-        if (ret)
-                goto out;
-
-        ret = cli_cmd_system_register (state);
-        if (ret)
-                goto out;
-
-        ret = cli_cmd_misc_register (state);
-        if (ret)
-                goto out;
-
-        ret = cli_cmd_snapshot_register (state);
-        if (ret)
-                goto out;
+    int ret = 0;
+
+    ret = cli_cmd_volume_register(state);
+    if (ret)
+        goto out;
+
+    ret = cli_cmd_probe_register(state);
+    if (ret)
+        goto out;
+
+    ret = cli_cmd_system_register(state);
+    if (ret)
+        goto out;
+
+    ret = cli_cmd_misc_register(state);
+    if (ret)
+        goto out;
+
+    ret = cli_cmd_snapshot_register(state);
+    if (ret)
+        goto out;
+    ret = cli_cmd_global_register(state);
+    if (ret)
+        goto out;
 out:
-        return ret;
-}
-
-int
-cli_cmd_cond_init ()
-{
-
-       pthread_mutex_init (&cond_mutex, NULL);
-       pthread_cond_init (&cond, NULL);
-
-       pthread_mutex_init (&conn_mutex, NULL);
-       pthread_cond_init (&conn, NULL);
-
-       return 0;
+    return ret;
 }
 
 int
-cli_cmd_lock ()
+cli_cmd_lock()
 {
-       pthread_mutex_lock (&cond_mutex);
-       return 0;
+    pthread_mutex_lock(&cond_mutex);
+    return 0;
 }
 
 int
-cli_cmd_unlock ()
+cli_cmd_unlock()
 {
-        pthread_mutex_unlock (&cond_mutex);
-        return 0;
+    pthread_mutex_unlock(&cond_mutex);
+    return 0;
 }
 
 static void
-seconds_from_now (unsigned secs, struct timespec *ts)
+seconds_from_now(unsigned secs, struct timespec *ts)
 {
-        struct timeval tv = {0,};
+    struct timeval tv = {
+        0,
+    };
 
-        gettimeofday (&tv, NULL);
+    gettimeofday(&tv, NULL);
 
-        ts->tv_sec = tv.tv_sec + secs;
-        ts->tv_nsec = tv.tv_usec * 1000;
+    ts->tv_sec = tv.tv_sec + secs;
+    ts->tv_nsec = tv.tv_usec * 1000;
 }
 
 int
-cli_cmd_await_response (unsigned time)
+cli_cmd_await_response(unsigned time)
 {
-        struct  timespec        ts = {0,};
-        int                     ret = 0;
+    struct timespec ts = {
+        0,
+    };
+    int ret = 0;
 
-        cli_op_ret = -1;
+    cli_op_ret = -1;
 
-        seconds_from_now (time, &ts);
-        while (!cmd_done && !ret) {
-                ret = pthread_cond_timedwait (&cond, &cond_mutex,
-                                        &ts);
-        }
+    seconds_from_now(time, &ts);
+    while (!cmd_done && !ret) {
+        ret = pthread_cond_timedwait(&cond, &cond_mutex, &ts);
+    }
 
-        if (!cmd_done) {
-                if (ret == ETIMEDOUT)
-                        cli_out ("Error : Request timed out");
-                else
-                        cli_out ("Error : Command returned with error code:%d",
-                                 ret);
-        }
-        cmd_done = 0;
+    if (!cmd_done) {
+        if (ret == ETIMEDOUT)
+            cli_out("Error : Request timed out");
+        else
+            cli_out("Error : Command returned with error code:%d", ret);
+    }
+    cmd_done = 0;
 
-        return cli_op_ret;
+    return cli_op_ret;
 }
 
+/* This function must be called _only_ after all actions associated with
+ * command processing is complete. Otherwise, gluster process may exit before
+ * reporting results to stdout/stderr. */
 int
-cli_cmd_broadcast_response (int32_t status)
+cli_cmd_broadcast_response(int32_t status)
 {
-
-        pthread_mutex_lock (&cond_mutex);
-        {
-                if (!cmd_sent)
-                        goto out;
-                cmd_done = 1;
-                cli_op_ret = status;
-                pthread_cond_broadcast (&cond);
-        }
-
+    pthread_mutex_lock(&cond_mutex);
+    {
+        if (!cmd_sent)
+            goto out;
+        cmd_done = 1;
+        cli_op_ret = status;
+        pthread_cond_broadcast(&cond);
+    }
 
 out:
-        pthread_mutex_unlock (&cond_mutex);
-        return 0;
+    pthread_mutex_unlock(&cond_mutex);
+    return 0;
 }
 
 int32_t
-cli_cmd_await_connected (unsigned conn_timo)
+cli_cmd_await_connected(unsigned conn_timo)
 {
-        int32_t                 ret = 0;
-        struct  timespec        ts = {0,};
-
-        if (!conn_timo)
-                return 0;
-
-        pthread_mutex_lock (&conn_mutex);
-        {
-                seconds_from_now (conn_timo, &ts);
-                while (!connected && !ret) {
-                        ret = pthread_cond_timedwait (&conn, &conn_mutex,
-                                                      &ts);
-                }
-        }
-        pthread_mutex_unlock (&conn_mutex);
+    int32_t ret = 0;
+    struct timespec ts = {
+        0,
+    };
+
+    if (!conn_timo)
+        return 0;
 
+    pthread_mutex_lock(&conn_mutex);
+    {
+        seconds_from_now(conn_timo, &ts);
+        while (!connected && !ret) {
+            ret = pthread_cond_timedwait(&conn, &conn_mutex, &ts);
+        }
+    }
+    pthread_mutex_unlock(&conn_mutex);
 
-        return ret;
+    return ret;
 }
 
 int32_t
-cli_cmd_broadcast_connected ()
+cli_cmd_broadcast_connected(gf_boolean_t status)
 {
-        pthread_mutex_lock (&conn_mutex);
-        {
-                connected = 1;
-                pthread_cond_broadcast (&conn);
-        }
-
-        pthread_mutex_unlock (&conn_mutex);
-
-        return 0;
+    pthread_mutex_lock(&conn_mutex);
+    {
+        connected = status;
+        pthread_cond_broadcast(&conn);
+    }
+    pthread_mutex_unlock(&conn_mutex);
+
+    return 0;
 }
 
-int
-cli_cmd_submit (struct rpc_clnt* rpc, void *req, call_frame_t *frame,
-                rpc_clnt_prog_t *prog,
-                int procnum, struct iobref *iobref,
-                xlator_t *this, fop_cbk_fn_t cbkfn, xdrproc_t xdrproc)
+gf_boolean_t
+cli_cmd_connected(void)
 {
-        int             ret = -1;
-        unsigned        timeout = 0;
+    gf_boolean_t status;
 
-        if ((GLUSTER_CLI_PROFILE_VOLUME == procnum) ||
-            (GLUSTER_CLI_HEAL_VOLUME == procnum))
-                timeout = CLI_TEN_MINUTES_TIMEOUT;
-        else
-                timeout = CLI_DEFAULT_CMD_TIMEOUT;
+    pthread_mutex_lock(&conn_mutex);
+    {
+        status = connected;
+    }
+    pthread_mutex_unlock(&conn_mutex);
 
-        cli_cmd_lock ();
-        cmd_sent = 0;
-        ret = cli_submit_request (rpc, req, frame, prog,
-                                  procnum, NULL, this, cbkfn, xdrproc);
+    return status;
+}
 
-        if (!ret) {
-                cmd_sent = 1;
-                ret = cli_cmd_await_response (timeout);
-        }
+int
+cli_cmd_submit(struct rpc_clnt *rpc, void *req, call_frame_t *frame,
+               rpc_clnt_prog_t *prog, int procnum, struct iobref *iobref,
+               xlator_t *this, fop_cbk_fn_t cbkfn, xdrproc_t xdrproc)
+{
+    int ret = -1;
+    unsigned timeout = 0;
+
+    if ((GLUSTER_CLI_PROFILE_VOLUME == procnum) ||
+        (GLUSTER_CLI_HEAL_VOLUME == procnum) ||
+        (GLUSTER_CLI_GANESHA == procnum))
+        timeout = cli_ten_minutes_timeout;
+    else
+        timeout = cli_default_conn_timeout;
+
+    cli_cmd_lock();
+    cmd_sent = 0;
+    ret = cli_submit_request(rpc, req, frame, prog, procnum, NULL, this, cbkfn,
+                             xdrproc);
+
+    if (!ret) {
+        cmd_sent = 1;
+        ret = cli_cmd_await_response(timeout);
+    }
+
+    cli_cmd_unlock();
+
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
+}
 
-        cli_cmd_unlock ();
+int
+cli_cmd_pattern_cmp(void *a, void *b)
+{
+    struct cli_cmd *ia = NULL;
+    struct cli_cmd *ib = NULL;
+    int ret = 0;
+
+    ia = a;
+    ib = b;
+    if (strcmp(ia->pattern, ib->pattern) > 0)
+        ret = 1;
+    else if (strcmp(ia->pattern, ib->pattern) < 0)
+        ret = -1;
+    else
+        ret = 0;
+    return ret;
+}
 
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+void
+cli_cmd_sort(struct cli_cmd *cmd, int count)
+{
+    gf_array_insertionsort(cmd, 1, count - 2, sizeof(struct cli_cmd),
+                           cli_cmd_pattern_cmp);
 }
diff --git a/cli/src/cli-cmd.h b/cli/src/cli-cmd.h
index cf036928ddf..c1c068c7085 100644
--- a/cli/src/cli-cmd.h
+++ b/cli/src/cli-cmd.h
@@ -10,110 +10,120 @@
 #ifndef __CLI_CMD_H__
 #define __CLI_CMD_H__
 
-#ifndef _CONFIG_H
-#define _CONFIG_H
-#include "config.h"
-#endif
-
 #include <netdb.h>
 
 #include "cli.h"
-#include "list.h"
-
-#define CLI_LOCAL_INIT(local, words, frame, dictionary) \
-        do {                                                 \
-                local = cli_local_get ();                    \
-                                                             \
-                if (local) {                                 \
-                        local->words = words;                \
-                        if (dictionary)                      \
-                                local->dict = dictionary;    \
-                        if (frame)                           \
-                                frame->local = local;        \
-                }                                            \
-        } while (0)
-
-#define CLI_STACK_DESTROY(_frame)                                       \
-        do {                                                            \
-                if (_frame) {                                           \
-                        if (_frame->local) {                            \
-                                gf_log ("cli", GF_LOG_DEBUG, "frame->local " \
-                                        "is not NULL (%p)", _frame->local); \
-                                cli_local_wipe (_frame->local);         \
-                                _frame->local = NULL;                   \
-                        }                                               \
-                        STACK_DESTROY (_frame->root);                   \
-                }                                                       \
-        } while (0);
-
-typedef enum {
-        GF_ANSWER_YES = 1,
-        GF_ANSWER_NO  = 2
-} gf_answer_t;
+#include <glusterfs/list.h>
+
+#define GLUSTER_SHARED_STORAGE "gluster_shared_storage"
+
+#define CLI_LOCAL_INIT(local, words, frame, dictionary)                        \
+    do {                                                                       \
+        local = cli_local_get();                                               \
+                                                                               \
+        if (local) {                                                           \
+            local->words = words;                                              \
+            if (dictionary)                                                    \
+                local->dict = dictionary;                                      \
+            if (frame)                                                         \
+                frame->local = local;                                          \
+        }                                                                      \
+    } while (0)
+
+#define CLI_STACK_DESTROY(_frame)                                              \
+    do {                                                                       \
+        if (_frame) {                                                          \
+            if (_frame->local) {                                               \
+                gf_log("cli", GF_LOG_DEBUG,                                    \
+                       "frame->local "                                         \
+                       "is not NULL (%p)",                                     \
+                       _frame->local);                                         \
+                cli_local_wipe(_frame->local);                                 \
+                _frame->local = NULL;                                          \
+            }                                                                  \
+            STACK_DESTROY(_frame->root);                                       \
+        }                                                                      \
+    } while (0);
+
+typedef enum { GF_ANSWER_YES = 1, GF_ANSWER_NO = 2 } gf_answer_t;
 
 struct cli_cmd {
-        const char         *pattern;
-        cli_cmd_cbk_t      *cbk;
-        const char         *desc;
-        cli_cmd_reg_cbk_t  *reg_cbk; /* callback to check in runtime if the   *
-                                      * command should be enabled or disabled */
-        gf_boolean_t       disable;
+    const char *pattern;
+    cli_cmd_cbk_t *cbk;
+    const char *desc;
+    cli_cmd_reg_cbk_t *reg_cbk; /* callback to check in runtime if the   *
+                                 * command should be enabled or disabled */
+    gf_boolean_t disable;
 };
 
 struct cli_cmd_volume_get_ctx_ {
-        char            *volname;
-        int             flags;
+    char *volname;
+    int flags;
 };
 
 typedef struct cli_profile_info_ {
-        uint64_t fop_hits;
-        double min_latency;
-        double max_latency;
-        double avg_latency;
-        char   *fop_name;
-        double percentage_avg_latency;
+    uint64_t fop_hits;
+    double min_latency;
+    double max_latency;
+    double avg_latency;
+    char *fop_name;
+    double percentage_avg_latency;
 } cli_profile_info_t;
 
 typedef struct cli_cmd_volume_get_ctx_ cli_cmd_volume_get_ctx_t;
 
-int cli_cmd_volume_register (struct cli_state *state);
+int
+cli_cmd_volume_register(struct cli_state *state);
 
-int cli_cmd_probe_register (struct cli_state *state);
+int
+cli_cmd_probe_register(struct cli_state *state);
 
-int cli_cmd_system_register (struct cli_state *state);
+int
+cli_cmd_system_register(struct cli_state *state);
 
-int cli_cmd_snapshot_register (struct cli_state *state);
+int
+cli_cmd_snapshot_register(struct cli_state *state);
 
-int cli_cmd_misc_register (struct cli_state *state);
+int
+cli_cmd_global_register(struct cli_state *state);
 
-struct cli_cmd_word *cli_cmd_nextword (struct cli_cmd_word *word,
-                                       const char *text);
-void cli_cmd_tokens_destroy (char **tokens);
+int
+cli_cmd_misc_register(struct cli_state *state);
 
-int cli_cmd_await_response (unsigned time);
+struct cli_cmd_word *
+cli_cmd_nextword(struct cli_cmd_word *word, const char *text);
+void
+cli_cmd_tokens_destroy(char **tokens);
 
-int cli_cmd_broadcast_response (int32_t status);
+int
+cli_cmd_await_response(unsigned time);
 
-int cli_cmd_cond_init ();
+int
+cli_cmd_broadcast_response(int32_t status);
 
-int cli_cmd_lock ();
+int
+cli_cmd_lock();
 
-int cli_cmd_unlock ();
+int
+cli_cmd_unlock();
 
 int
-cli_cmd_submit (struct rpc_clnt *rpc, void *req, call_frame_t *frame,
-                rpc_clnt_prog_t *prog,
-                int procnum, struct iobref *iobref,
-                xlator_t *this, fop_cbk_fn_t cbkfn, xdrproc_t xdrproc);
+cli_cmd_submit(struct rpc_clnt *rpc, void *req, call_frame_t *frame,
+               rpc_clnt_prog_t *prog, int procnum, struct iobref *iobref,
+               xlator_t *this, fop_cbk_fn_t cbkfn, xdrproc_t xdrproc);
 
-gf_answer_t
-cli_cmd_get_confirmation (struct cli_state *state, const char *question);
-int cli_cmd_sent_status_get (int *status);
+int
+cli_cmd_pattern_cmp(void *a, void *b);
 
-gf_boolean_t
-_limits_set_on_volume (char *volname);
+void
+cli_cmd_sort(struct cli_cmd *cmd, int count);
+
+gf_answer_t
+cli_cmd_get_confirmation(struct cli_state *state, const char *question);
+int
+cli_cmd_sent_status_get(int *status);
 
 gf_boolean_t
-_quota_aux_mount_online (char *volname);
+_limits_set_on_volume(char *volname, int type);
 
 #endif /* __CLI_CMD_H__ */
diff --git a/cli/src/cli-mem-types.h b/cli/src/cli-mem-types.h
index 09fcb639bd0..b42b4dd86c2 100644
--- a/cli/src/cli-mem-types.h
+++ b/cli/src/cli-mem-types.h
@@ -10,20 +10,21 @@
 #ifndef __CLI_MEM_TYPES_H__
 #define __CLI_MEM_TYPES_H__
 
-#include "mem-types.h"
+#include <glusterfs/mem-types.h>
 
 #define CLI_MEM_TYPE_START (gf_common_mt_end + 1)
 
 enum cli_mem_types_ {
-        cli_mt_xlator_list_t = CLI_MEM_TYPE_START,
-        cli_mt_xlator_t,
-        cli_mt_xlator_cmdline_option_t,
-        cli_mt_char,
-        cli_mt_call_pool_t,
-        cli_mt_cli_local_t,
-        cli_mt_cli_get_vol_ctx_t,
-        cli_mt_append_str,
-        cli_mt_end
+    cli_mt_xlator_list_t = CLI_MEM_TYPE_START,
+    cli_mt_xlator_t,
+    cli_mt_xlator_cmdline_option_t,
+    cli_mt_char,
+    cli_mt_call_pool_t,
+    cli_mt_cli_local_t,
+    cli_mt_cli_get_vol_ctx_t,
+    cli_mt_append_str,
+    cli_mt_cli_cmd,
+    cli_mt_end
 
 };
 
diff --git a/cli/src/cli-quotad-client.c b/cli/src/cli-quotad-client.c
index f0efc8640bd..772b8f75bd9 100644
--- a/cli/src/cli-quotad-client.c
+++ b/cli/src/cli-quotad-client.c
@@ -7,148 +7,140 @@
    later), or the GNU General Public License, version 2 (GPLv2), in all
    cases as published by the Free Software Foundation.
 */
-#ifndef _CONFIG_H
-#define _CONFIG_H
-#include "config.h"
-#endif
-
 
 #include "cli-quotad-client.h"
 
-extern struct rpc_clnt global_quotad_rpc;
-extern struct rpc_clnt_program cli_quotad_clnt;
-
 int
-cli_quotad_submit_request (void *req, call_frame_t *frame,
-                           rpc_clnt_prog_t *prog,
-                           int procnum, struct iobref *iobref,
-                           xlator_t *this, fop_cbk_fn_t cbkfn,
-                           xdrproc_t xdrproc)
+cli_quotad_submit_request(void *req, call_frame_t *frame, rpc_clnt_prog_t *prog,
+                          int procnum, struct iobref *iobref, xlator_t *this,
+                          fop_cbk_fn_t cbkfn, xdrproc_t xdrproc)
 {
-        int           ret        = -1;
-        int           count      = 0;
-        struct iovec  iov        = {0, };
-        struct iobuf *iobuf      = NULL;
-        char          new_iobref = 0;
-        ssize_t       xdr_size   = 0;
-
-        GF_ASSERT (this);
-
-        if (req) {
-                xdr_size = xdr_sizeof (xdrproc, req);
-                iobuf = iobuf_get2 (this->ctx->iobuf_pool, xdr_size);
-                if (!iobuf) {
-                        goto out;
-                };
-
-                if (!iobref) {
-                        iobref = iobref_new ();
-                        if (!iobref) {
-                                goto out;
-                        }
-
-                        new_iobref = 1;
-                }
-
-                iobref_add (iobref, iobuf);
-
-                iov.iov_base = iobuf->ptr;
-                iov.iov_len  = iobuf_size (iobuf);
-
-                /* Create the xdr payload */
-                ret = xdr_serialize_generic (iov, req, xdrproc);
-                if (ret == -1) {
-                        goto out;
-                }
-                iov.iov_len = ret;
-                count = 1;
+    int ret = -1;
+    int count = 0;
+    struct iovec iov = {
+        0,
+    };
+    struct iobuf *iobuf = NULL;
+    char new_iobref = 0;
+    ssize_t xdr_size = 0;
+
+    GF_ASSERT(this);
+
+    if (req) {
+        xdr_size = xdr_sizeof(xdrproc, req);
+        iobuf = iobuf_get2(this->ctx->iobuf_pool, xdr_size);
+        if (!iobuf) {
+            goto out;
+        };
+
+        if (!iobref) {
+            iobref = iobref_new();
+            if (!iobref) {
+                goto out;
+            }
+
+            new_iobref = 1;
+        }
+
+        iobref_add(iobref, iobuf);
+
+        iov.iov_base = iobuf->ptr;
+        iov.iov_len = iobuf_size(iobuf);
+
+        /* Create the xdr payload */
+        ret = xdr_serialize_generic(iov, req, xdrproc);
+        if (ret == -1) {
+            goto out;
         }
+        iov.iov_len = ret;
+        count = 1;
+    }
 
-        /* Send the msg */
-        ret = rpc_clnt_submit (&global_quotad_rpc, prog, procnum, cbkfn,
-                               &iov, count,
-                               NULL, 0, iobref, frame, NULL, 0, NULL, 0, NULL);
-        ret = 0;
+    /* Send the msg */
+    ret = rpc_clnt_submit(global_quotad_rpc, prog, procnum, cbkfn, &iov, count,
+                          NULL, 0, iobref, frame, NULL, 0, NULL, 0, NULL);
+    ret = 0;
 
 out:
-        if (new_iobref)
-                iobref_unref (iobref);
-        if (iobuf)
-                iobuf_unref (iobuf);
+    if (new_iobref)
+        iobref_unref(iobref);
+    if (iobuf)
+        iobuf_unref(iobuf);
 
-        return ret;
+    return ret;
 }
 
 int
-cli_quotad_notify (struct rpc_clnt *rpc, void *mydata,
-                   rpc_clnt_event_t event, void *data)
+cli_quotad_notify(struct rpc_clnt *rpc, void *mydata, rpc_clnt_event_t event,
+                  void *data)
 {
-        xlator_t                *this = NULL;
-        int                     ret = 0;
+    xlator_t *this = NULL;
+    int ret = 0;
 
-        this = mydata;
+    this = mydata;
 
-        switch (event) {
-        case RPC_CLNT_CONNECT:
-        {
-                gf_log (this->name, GF_LOG_TRACE, "got RPC_CLNT_CONNECT");
-                break;
+    switch (event) {
+        case RPC_CLNT_CONNECT: {
+            gf_log(this->name, GF_LOG_TRACE, "got RPC_CLNT_CONNECT");
+            break;
         }
 
-        case RPC_CLNT_DISCONNECT:
-        {
-                gf_log (this->name, GF_LOG_TRACE, "got RPC_CLNT_DISCONNECT");
-                break;
+        case RPC_CLNT_DISCONNECT: {
+            gf_log(this->name, GF_LOG_TRACE, "got RPC_CLNT_DISCONNECT");
+            break;
         }
 
         default:
-                gf_log (this->name, GF_LOG_TRACE,
-                        "got some other RPC event %d", event);
-                ret = 0;
-                break;
-        }
+            gf_log(this->name, GF_LOG_TRACE, "got some other RPC event %d",
+                   event);
+            ret = 0;
+            break;
+    }
 
-        return ret;
+    return ret;
 }
 
 struct rpc_clnt *
-cli_quotad_clnt_init (xlator_t *this, dict_t *options)
+cli_quotad_clnt_init(xlator_t *this, dict_t *options)
 {
-        struct rpc_clnt *rpc  = NULL;
-        int              ret  = -1;
-
-
-        ret = dict_set_str (options, "transport.address-family", "unix");
-        if (ret)
-                goto out;
-
-        ret = dict_set_str (options, "transport-type", "socket");
-        if (ret)
-                goto out;
-
-        ret = dict_set_str (options, "transport.socket.connect-path",
-                            "/tmp/quotad.socket");
-        if (ret)
-                goto out;
-
-        rpc = rpc_clnt_new (options, this->ctx, this->name, 16);
-        if (!rpc)
-                goto out;
-
-        ret = rpc_clnt_register_notify (rpc, cli_quotad_notify, this);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "failed to register notify");
-                goto out;
-        }
-
-        rpc_clnt_start (rpc);
+    struct rpc_clnt *rpc = NULL;
+    int ret = -1;
+
+    ret = dict_set_nstrn(options, "transport.address-family",
+                         SLEN("transport.address-family"), "unix",
+                         SLEN("unix"));
+    if (ret)
+        goto out;
+
+    ret = dict_set_nstrn(options, "transport-type", SLEN("transport-type"),
+                         "socket", SLEN("socket"));
+    if (ret)
+        goto out;
+
+    ret = dict_set_nstrn(options, "transport.socket.connect-path",
+                         SLEN("transport.socket.connect-path"),
+                         "/var/run/gluster/quotad.socket",
+                         SLEN("/var/run/gluster/quotad.socket"));
+    if (ret)
+        goto out;
+
+    rpc = rpc_clnt_new(options, this, this->name, 16);
+    if (!rpc)
+        goto out;
+
+    ret = rpc_clnt_register_notify(rpc, cli_quotad_notify, this);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "failed to register notify");
+        goto out;
+    }
+
+    rpc_clnt_start(rpc);
 out:
-        if (ret) {
-                if (rpc)
-                        rpc_clnt_unref (rpc);
-                rpc = NULL;
-        }
+    if (ret) {
+        if (rpc)
+            rpc_clnt_unref(rpc);
+        rpc = NULL;
+    }
 
-        return rpc;
+    return rpc;
 }
-
diff --git a/cli/src/cli-quotad-client.h b/cli/src/cli-quotad-client.h
index aa0b42af38d..71a44e5916b 100644
--- a/cli/src/cli-quotad-client.h
+++ b/cli/src/cli-quotad-client.h
@@ -8,26 +8,22 @@
    cases as published by the Free Software Foundation.
 */
 #include "cli.h"
-#include "compat-errno.h"
-#include "compat.h"
+#include <glusterfs/compat-errno.h>
+#include <glusterfs/compat.h>
 #include "cli-cmd.h"
 #include "cli1-xdr.h"
 #include "xdr-generic.h"
 #include "protocol-common.h"
 #include "cli-mem-types.h"
 
-
 int
-cli_quotad_submit_request (void *req, call_frame_t *frame,
-                           rpc_clnt_prog_t *prog,
-                           int procnum, struct iobref *iobref,
-                           xlator_t *this, fop_cbk_fn_t cbkfn,
-                           xdrproc_t xdrproc);
+cli_quotad_submit_request(void *req, call_frame_t *frame, rpc_clnt_prog_t *prog,
+                          int procnum, struct iobref *iobref, xlator_t *this,
+                          fop_cbk_fn_t cbkfn, xdrproc_t xdrproc);
 
 struct rpc_clnt *
-cli_quotad_clnt_init (xlator_t *this, dict_t *options);
+cli_quotad_clnt_init(xlator_t *this, dict_t *options);
 
 int
-cli_quotad_notify (struct rpc_clnt *rpc, void *mydata,
-                   rpc_clnt_event_t event, void *data);
-
+cli_quotad_notify(struct rpc_clnt *rpc, void *mydata, rpc_clnt_event_t event,
+                  void *data);
diff --git a/cli/src/cli-rl.c b/cli/src/cli-rl.c
index ade1c8ebb37..7a38a0b882a 100644
--- a/cli/src/cli-rl.c
+++ b/cli/src/cli-rl.c
@@ -13,16 +13,11 @@
 #include <stdint.h>
 #include <pthread.h>
 
-#ifndef _CONFIG_H
-#define _CONFIG_H
-#include "config.h"
-#endif
-
 #include "cli.h"
 #include "cli-cmd.h"
 #include "cli-mem-types.h"
 
-#include "event.h"
+#include <glusterfs/gf-event.h>
 
 #include <fnmatch.h>
 
@@ -32,384 +27,376 @@
 #include <readline/readline.h>
 #include <readline/history.h>
 
-
 int
-cli_rl_out (struct cli_state *state, const char *fmt, va_list ap)
+cli_rl_out(struct cli_state *state, const char *fmt, va_list ap)
 {
-        int tmp_rl_point = rl_point;
-        int            n = rl_end;
-        int            ret = 0;
+    int tmp_rl_point = rl_point;
+    int n = rl_end;
+    int ret = 0;
 
-        if (rl_end >= 0 ) {
-                rl_kill_text (0, rl_end);
-                rl_redisplay ();
-        }
+    if (rl_end >= 0) {
+        rl_kill_text(0, rl_end);
+        rl_redisplay();
+    }
 
-        printf ("\r%*s\r", (int)strlen (state->prompt), "");
+    printf("\r%*s\r", (int)strlen(state->prompt), "");
 
-        ret = vprintf (fmt, ap);
+    ret = vprintf(fmt, ap);
 
-        printf ("\n");
-        fflush(stdout);
+    printf("\n");
+    fflush(stdout);
 
-        if (n) {
-                rl_do_undo ();
-                rl_point = tmp_rl_point;
-                rl_reset_line_state ();
-        }
+    if (n) {
+        rl_do_undo();
+        rl_point = tmp_rl_point;
+        rl_reset_line_state();
+    }
 
-        return ret;
+    return ret;
 }
 
 int
-cli_rl_err (struct cli_state *state, const char *fmt, va_list ap)
+cli_rl_err(struct cli_state *state, const char *fmt, va_list ap)
 {
-        int tmp_rl_point = rl_point;
-        int            n = rl_end;
-        int            ret = 0;
+    int tmp_rl_point = rl_point;
+    int n = rl_end;
+    int ret = 0;
 
-        if (rl_end >= 0 ) {
-                rl_kill_text (0, rl_end);
-                rl_redisplay ();
-        }
+    if (rl_end >= 0) {
+        rl_kill_text(0, rl_end);
+        rl_redisplay();
+    }
 
-        fprintf (stderr, "\r%*s\r", (int)strlen (state->prompt), "");
+    fprintf(stderr, "\r%*s\r", (int)strlen(state->prompt), "");
 
-        ret = vfprintf (stderr, fmt, ap);
+    ret = vfprintf(stderr, fmt, ap);
 
-        fprintf (stderr, "\n");
-        fflush(stderr);
+    fprintf(stderr, "\n");
+    fflush(stderr);
 
-        if (n) {
-                rl_do_undo ();
-                rl_point = tmp_rl_point;
-                rl_reset_line_state ();
-        }
+    if (n) {
+        rl_do_undo();
+        rl_point = tmp_rl_point;
+        rl_reset_line_state();
+    }
 
-        return ret;
+    return ret;
 }
 
-
 void
-cli_rl_process_line (char *line)
+cli_rl_process_line(char *line)
 {
-        struct cli_state *state = NULL;
-        int               ret = 0;
+    struct cli_state *state = NULL;
+    int ret = 0;
 
-        state = global_state;
+    state = global_state;
 
-        state->rl_processing = 1;
-        {
-                ret = cli_cmd_process_line (state, line);
-                if (ret)
-                        gf_log (THIS->name, GF_LOG_WARNING,
-                                "failed to process line");
-
-                add_history (line);
-        }
-        state->rl_processing = 0;
+    state->rl_processing = 1;
+    {
+        ret = cli_cmd_process_line(state, line);
+        if (ret)
+            gf_log(THIS->name, GF_LOG_WARNING, "failed to process line");
 
+        add_history(line);
+    }
+    state->rl_processing = 0;
 }
 
-
-int
-cli_rl_stdin (int fd, int idx, void *data,
-              int poll_out, int poll_in, int poll_err)
+void
+cli_rl_stdin(int fd, int idx, int gen, void *data, int poll_out, int poll_in,
+             int poll_err, char event_thread_died)
 {
-        rl_callback_read_char ();
+    struct cli_state *state = NULL;
 
-        return 0;
-}
+    state = data;
 
+    rl_callback_read_char();
+
+    gf_event_handled(state->ctx->event_pool, fd, idx, gen);
+
+    return;
+}
 
 char *
-cli_rl_autocomplete_entry (const char *text, int times)
+cli_rl_autocomplete_entry(const char *text, int times)
 {
-        struct cli_state  *state = NULL;
-        char              *retp = NULL;
+    struct cli_state *state = NULL;
+    char *retp = NULL;
 
-        state = global_state;
+    state = global_state;
 
-        if (!state->matchesp)
-                return NULL;
+    if (!state->matchesp)
+        return NULL;
 
-        retp = *state->matchesp;
+    retp = *state->matchesp;
 
-        state->matchesp++;
+    state->matchesp++;
 
-        return retp ? strdup (retp) : NULL;
+    return retp ? strdup(retp) : NULL;
 }
 
-
 int
-cli_rl_token_count (const char *text)
+cli_rl_token_count(const char *text)
 {
-        int          count = 0;
-        const char  *trav = NULL;
-        int          is_spc = 1;
-
-        for (trav = text; *trav; trav++) {
-                if (*trav == ' ') {
-                        is_spc = 1;
-                } else {
-                        if (is_spc) {
-                                count++;
-                                is_spc = 0;
-                        }
-                }
+    int count = 0;
+    const char *trav = NULL;
+    int is_spc = 1;
+
+    for (trav = text; *trav; trav++) {
+        if (*trav == ' ') {
+            is_spc = 1;
+        } else {
+            if (is_spc) {
+                count++;
+                is_spc = 0;
+            }
         }
+    }
 
-        if (is_spc)
-                /* what needs to be autocompleted is a full
-                   new word, and not extend the last word
-                */
-                count++;
+    if (is_spc)
+        /* what needs to be autocompleted is a full
+           new word, and not extend the last word
+        */
+        count++;
 
-        return count;
+    return count;
 }
 
-
 char **
-cli_rl_tokenize (const char *text)
+cli_rl_tokenize(const char *text)
 {
-        int     count = 0;
-        char  **tokens = NULL;
-        char  **tokenp = NULL;
-        char   *token = NULL;
-        char   *copy = NULL;
-        char   *saveptr = NULL;
-        int     i = 0;
-
-        count = cli_rl_token_count (text);
-
-        tokens = calloc (count + 1, sizeof (*tokens));
-        if (!tokens)
-                return NULL;
-
-        copy = strdup (text);
-        if (!copy)
-                goto out;
-
-        tokenp = tokens;
-
-        for (token = strtok_r (copy, " \t\r\n", &saveptr); token;
-             token = strtok_r (NULL, " \t\r\n", &saveptr)) {
-                *tokenp = strdup (token);
-
-                if (!*tokenp)
-                        goto out;
-                tokenp++;
-                i++;
-
-        }
+    int count = 0;
+    char **tokens = NULL;
+    char **tokenp = NULL;
+    char *token = NULL;
+    char *copy = NULL;
+    char *saveptr = NULL;
+    int i = 0;
+
+    count = cli_rl_token_count(text);
+
+    tokens = calloc(count + 1, sizeof(*tokens));
+    if (!tokens)
+        return NULL;
 
-        if (i < count) {
-                /* symbolize that what needs to be autocompleted is
-                   the full set of possible nextwords, and not extend
-                   the last word
-                */
-                *tokenp = strdup ("");
-                if (!*tokenp)
-                        goto out;
-                tokenp++;
-                i++;
-        }
+    copy = strdup(text);
+    if (!copy)
+        goto out;
+
+    tokenp = tokens;
+
+    for (token = strtok_r(copy, " \t\r\n", &saveptr); token;
+         token = strtok_r(NULL, " \t\r\n", &saveptr)) {
+        *tokenp = strdup(token);
+
+        if (!*tokenp)
+            goto out;
+        tokenp++;
+        i++;
+    }
+
+    if (i < count) {
+        /* symbolize that what needs to be autocompleted is
+           the full set of possible nextwords, and not extend
+           the last word
+        */
+        *tokenp = strdup("");
+        if (!*tokenp)
+            goto out;
+        tokenp++;
+        i++;
+    }
 
 out:
-        free (copy);
+    free(copy);
 
-        if (i < count) {
-                cli_cmd_tokens_destroy (tokens);
-                tokens = NULL;
-        }
+    if (i < count) {
+        cli_cmd_tokens_destroy(tokens);
+        tokens = NULL;
+    }
 
-        return tokens;
+    return tokens;
 }
 
-
 char **
-cli_rl_get_matches (struct cli_state *state, struct cli_cmd_word *word,
-                    const char *text)
+cli_rl_get_matches(struct cli_state *state, struct cli_cmd_word *word,
+                   const char *text)
 {
-        char                 **matches = NULL;
-        char                 **matchesp = NULL;
-        struct cli_cmd_word  **next = NULL;
-        int                    count = 0;
-        int                    len = 0;
+    char **matches = NULL;
+    char **matchesp = NULL;
+    struct cli_cmd_word **next = NULL;
+    int count = 0;
+    int len = 0;
 
-        len = strlen (text);
+    len = strlen(text);
 
-        if (!word->nextwords)
-                return NULL;
+    if (!word->nextwords)
+        return NULL;
 
-        for (next = word->nextwords; *next; next++)
-                count++;
+    for (next = word->nextwords; *next; next++)
+        count++;
 
-        matches = calloc (count + 1, sizeof (*matches));
-        matchesp = matches;
+    matches = calloc(count + 1, sizeof(*matches));
+    matchesp = matches;
 
-        for (next = word->nextwords; *next; next++) {
-                if ((*next)->match) {
-                        continue;
-                }
+    for (next = word->nextwords; *next; next++) {
+        if ((*next)->match) {
+            continue;
+        }
 
-                if (strncmp ((*next)->word, text, len) == 0) {
-                        *matchesp = strdup ((*next)->word);
-                        matchesp++;
-                }
+        if (strncmp((*next)->word, text, len) == 0) {
+            *matchesp = strdup((*next)->word);
+            matchesp++;
         }
+    }
 
-        return matches;
+    return matches;
 }
 
-
 int
-cli_rl_autocomplete_prepare (struct cli_state *state, const char *text)
+cli_rl_autocomplete_prepare(struct cli_state *state, const char *text)
 {
-        struct cli_cmd_word   *word = NULL;
-        struct cli_cmd_word   *next = NULL;
-        char                 **tokens = NULL;
-        char                 **tokenp = NULL;
-        char                  *token = NULL;
-        char                 **matches = NULL;
-
-        tokens = cli_rl_tokenize (text);
-        if (!tokens)
-                return 0;
-
-        word = &state->tree.root;
-
-        for (tokenp = tokens; (token = *tokenp); tokenp++) {
-                if (!*(tokenp+1)) {
-                        /* last word */
-                        break;
-                }
-
-                next = cli_cmd_nextword (word, token);
-                word = next;
-                if (!word)
-                        break;
+    struct cli_cmd_word *word = NULL;
+    struct cli_cmd_word *next = NULL;
+    char **tokens = NULL;
+    char **tokenp = NULL;
+    char *token = NULL;
+    char **matches = NULL;
+
+    tokens = cli_rl_tokenize(text);
+    if (!tokens)
+        return 0;
+
+    word = &state->tree.root;
+
+    for (tokenp = tokens; (token = *tokenp); tokenp++) {
+        if (!*(tokenp + 1)) {
+            /* last word */
+            break;
         }
 
+        next = cli_cmd_nextword(word, token);
+        word = next;
         if (!word)
-                goto out;
+            break;
+    }
+
+    if (!word || !token)
+        goto out;
 
-        matches = cli_rl_get_matches (state, word, token);
+    matches = cli_rl_get_matches(state, word, token);
 
-        state->matches = matches;
-        state->matchesp = matches;
+    state->matches = matches;
+    state->matchesp = matches;
 
 out:
-        cli_cmd_tokens_destroy (tokens);
-        return 0;
+    cli_cmd_tokens_destroy(tokens);
+    return 0;
 }
 
-
 int
-cli_rl_autocomplete_cleanup (struct cli_state *state)
+cli_rl_autocomplete_cleanup(struct cli_state *state)
 {
-        if (state->matches)
-                cli_cmd_tokens_destroy (state->matches);
+    if (state->matches)
+        cli_cmd_tokens_destroy(state->matches);
 
-        state->matches = NULL;
-        state->matchesp = NULL;
+    state->matches = NULL;
+    state->matchesp = NULL;
 
-        return 0;
+    return 0;
 }
 
-
 char **
-cli_rl_autocomplete (const char *text, int start, int end)
+cli_rl_autocomplete(const char *text, int start, int end)
 {
-        struct cli_state  *state = NULL;
-        char             **matches = NULL;
-        char               save = 0;
+    struct cli_state *state = NULL;
+    char **matches = NULL;
+    char save = 0;
 
-        state = global_state;
+    state = global_state;
 
-        /* hack to make the autocompletion code neater */
-        /* fake it as though the cursor is at the end of line */
+    /* hack to make the autocompletion code neater */
+    /* fake it as though the cursor is at the end of line */
 
-        save = rl_line_buffer[rl_point];
-        rl_line_buffer[rl_point] = 0;
+    save = rl_line_buffer[rl_point];
+    rl_line_buffer[rl_point] = 0;
 
-        cli_rl_autocomplete_prepare (state, rl_line_buffer);
+    cli_rl_autocomplete_prepare(state, rl_line_buffer);
 
-        matches = rl_completion_matches (text, cli_rl_autocomplete_entry);
+    matches = rl_completion_matches(text, cli_rl_autocomplete_entry);
 
-        cli_rl_autocomplete_cleanup (state);
+    cli_rl_autocomplete_cleanup(state);
 
-        rl_line_buffer[rl_point] = save;
+    rl_line_buffer[rl_point] = save;
 
-        return matches;
+    return matches;
 }
 
-
 static char *
-complete_none (const char *txt, int times)
+complete_none(const char *txt, int times)
 {
-        return NULL;
+    return NULL;
 }
 
-
 void *
-cli_rl_input (void *_data)
+cli_rl_input(void *_data)
 {
-        struct cli_state *state = NULL;
-        char             *line = NULL;
+    struct cli_state *state = NULL;
+    char *line = NULL;
 
-        state = _data;
+    state = _data;
 
-        for (;;) {
-                line = readline (state->prompt);
-                if (!line)
-                        exit(0);  //break;
+    fprintf(stderr,
+            "Welcome to gluster prompt, type 'help' to see the available "
+            "commands.\n");
+    for (;;) {
+        line = readline(state->prompt);
+        if (!line)
+            exit(0);  // break;
 
-                if (*line)
-                        cli_rl_process_line (line);
+        if (*line)
+            cli_rl_process_line(line);
 
-                free (line);
-        }
+        free(line);
+    }
 
-        return NULL;
+    return NULL;
 }
 
-
 int
-cli_rl_enable (struct cli_state *state)
+cli_rl_enable(struct cli_state *state)
 {
-        int ret = 0;
-
-        rl_pre_input_hook = NULL;
-        rl_attempted_completion_function = cli_rl_autocomplete;
-        rl_completion_entry_function = complete_none;
-
-        if (!state->rl_async) {
-                ret = pthread_create (&state->input, NULL,
-                                      cli_rl_input, state);
-                if (ret == 0)
-                        state->rl_enabled = 1;
-                goto out;
-        }
+    int ret = 0;
 
-        ret = event_register (state->ctx->event_pool, 0, cli_rl_stdin, state,
-                              1, 0);
-        if (ret == -1)
-                goto out;
+    rl_pre_input_hook = NULL;
+    rl_attempted_completion_function = cli_rl_autocomplete;
+    rl_completion_entry_function = complete_none;
 
-        state->rl_enabled = 1;
-        rl_callback_handler_install (state->prompt, cli_rl_process_line);
+    if (!state->rl_async) {
+        ret = pthread_create(&state->input, NULL, cli_rl_input, state);
+        if (ret == 0)
+            state->rl_enabled = 1;
+        goto out;
+    }
+
+    ret = gf_event_register(state->ctx->event_pool, 0, cli_rl_stdin, state, 1,
+                            0, 0);
+    if (ret == -1)
+        goto out;
+
+    state->rl_enabled = 1;
+    rl_callback_handler_install(state->prompt, cli_rl_process_line);
 
 out:
-        return state->rl_enabled;
+    return state->rl_enabled;
 }
 
 #else /* HAVE_READLINE */
 
 int
-cli_rl_enable (struct cli_state *state)
+cli_rl_enable(struct cli_state *state)
 {
-        return 0;
+    return 0;
 }
 
 #endif /* HAVE_READLINE */
diff --git a/cli/src/cli-rpc-ops.c b/cli/src/cli-rpc-ops.c
index f86fa63f68d..9b6b0c7fa50 100644
--- a/cli/src/cli-rpc-ops.c
+++ b/cli/src/cli-rpc-ops.c
@@ -8,68 +8,55 @@
    cases as published by the Free Software Foundation.
 */
 
-#ifndef _CONFIG_H
-#define _CONFIG_H
-#include "config.h"
-#endif
-
 /* Widths of various columns in top read/write-perf output
  * Total width of top read/write-perf should be 80 chars
  * including one space between column
  */
 #define VOL_TOP_PERF_FILENAME_DEF_WIDTH 47
 #define VOL_TOP_PERF_FILENAME_ALT_WIDTH 44
-#define VOL_TOP_PERF_SPEED_WIDTH        4
-#define VOL_TOP_PERF_TIME_WIDTH         26
+#define VOL_TOP_PERF_SPEED_WIDTH 4
+#define VOL_TOP_PERF_TIME_WIDTH 26
 
 #define INDENT_MAIN_HEAD "%-25s %s "
 
+#define RETURNING "Returning %d"
+#define XML_ERROR "Error outputting to xml"
+#define XDR_DECODE_FAIL "Failed to decode xdr response"
+#define DICT_SERIALIZE_FAIL "Failed to serialize to data to dictionary"
+#define DICT_UNSERIALIZE_FAIL "Failed to unserialize the dictionary"
+
+/* Do not show estimates if greater than this number */
+#define REBAL_ESTIMATE_SEC_UPPER_LIMIT (60 * 24 * 3600)
+#define REBAL_ESTIMATE_START_TIME 600
+
 #include "cli.h"
-#include "compat-errno.h"
+#include <glusterfs/compat-errno.h>
 #include "cli-cmd.h"
 #include <sys/uio.h>
 #include <stdlib.h>
 #include <sys/mount.h>
-#include "cli1-xdr.h"
-#include "xdr-generic.h"
-#include "protocol-common.h"
+#include <glusterfs/compat.h>
 #include "cli-mem-types.h"
-#include "compat.h"
-
-#include "syscall.h"
+#include <glusterfs/syscall.h>
 #include "glusterfs3.h"
 #include "portmap-xdr.h"
-#include "byte-order.h"
+#include <glusterfs/byte-order.h>
 
-#include "cli-quotad-client.h"
-#include "run.h"
+#include <glusterfs/run.h>
+#include <glusterfs/events.h>
 
-enum gf_task_types {
-        GF_TASK_TYPE_REBALANCE,
-        GF_TASK_TYPE_REMOVE_BRICK
-};
+enum gf_task_types { GF_TASK_TYPE_REBALANCE, GF_TASK_TYPE_REMOVE_BRICK };
 
-extern struct rpc_clnt *global_quotad_rpc;
-extern rpc_clnt_prog_t cli_quotad_clnt;
-extern rpc_clnt_prog_t *cli_rpc_prog;
-extern int              cli_op_ret;
-extern int              connected;
-
-char *cli_vol_type_str[] = {"Distribute",
-                            "Stripe",
-                            "Replicate",
-                            "Striped-Replicate",
-                            "Disperse",
-                            "Distributed-Stripe",
-                            "Distributed-Replicate",
-                            "Distributed-Striped-Replicate",
-                            "Distributed-Disperse",
-                           };
-
-char *cli_vol_status_str[] = {"Created",
-                              "Started",
-                              "Stopped",
-                             };
+rpc_clnt_prog_t cli_quotad_clnt;
+
+static int32_t
+gf_cli_remove_brick(call_frame_t *frame, xlator_t *this, void *data);
+
+char *cli_vol_status_str[] = {
+    "Created",
+    "Started",
+    "Stopped",
+};
 
 char *cli_vol_task_status_str[] = {"not started",
                                    "in progress",
@@ -80,7893 +67,8582 @@ char *cli_vol_task_status_str[] = {"not started",
                                    "fix-layout stopped",
                                    "fix-layout completed",
                                    "fix-layout failed",
-                                   "unknown"
-};
+                                   "unknown"};
 
-int32_t
-gf_cli_snapshot (call_frame_t *frame, xlator_t *this, void *data);
+static int32_t
+gf_cli_snapshot(call_frame_t *frame, xlator_t *this, void *data);
 
-int32_t
-gf_cli_get_volume (call_frame_t *frame, xlator_t *this,
-                      void *data);
+static int32_t
+gf_cli_get_volume(call_frame_t *frame, xlator_t *this, void *data);
 
-int
-cli_to_glusterd (gf_cli_req *req, call_frame_t *frame, fop_cbk_fn_t cbkfn,
-                 xdrproc_t xdrproc, dict_t *dict, int procnum, xlator_t *this,
-                 rpc_clnt_prog_t *prog, struct iobref *iobref);
-
-rpc_clnt_prog_t cli_handshake_prog = {
-        .progname  = "cli handshake",
-        .prognum   = GLUSTER_HNDSK_PROGRAM,
-        .progver   = GLUSTER_HNDSK_VERSION,
+static int
+cli_to_glusterd(gf_cli_req *req, call_frame_t *frame, fop_cbk_fn_t cbkfn,
+                xdrproc_t xdrproc, dict_t *dict, int procnum, xlator_t *this,
+                rpc_clnt_prog_t *prog, struct iobref *iobref);
+
+static int
+add_cli_cmd_timeout_to_dict(dict_t *dict);
+
+static rpc_clnt_prog_t cli_handshake_prog = {
+    .progname = "cli handshake",
+    .prognum = GLUSTER_HNDSK_PROGRAM,
+    .progver = GLUSTER_HNDSK_VERSION,
 };
 
-rpc_clnt_prog_t cli_pmap_prog = {
-        .progname   = "cli portmap",
-        .prognum    = GLUSTER_PMAP_PROGRAM,
-        .progver    = GLUSTER_PMAP_VERSION,
+static rpc_clnt_prog_t cli_pmap_prog = {
+    .progname = "cli portmap",
+    .prognum = GLUSTER_PMAP_PROGRAM,
+    .progver = GLUSTER_PMAP_VERSION,
 };
 
-int
-gf_cli_probe_cbk (struct rpc_req *req, struct iovec *iov,
-                        int count, void *myframe)
+static void
+gf_free_xdr_cli_rsp(gf_cli_rsp rsp)
 {
-        gf_cli_rsp            rsp   = {0,};
-        int                   ret   = -1;
-        char                  msg[1024] = {0,};
-
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
+    if (rsp.dict.dict_val) {
+        free(rsp.dict.dict_val);
+    }
+    if (rsp.op_errstr) {
+        free(rsp.op_errstr);
+    }
+}
 
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                //rsp.op_ret   = -1;
-                //rsp.op_errno = EINVAL;
-                goto out;
-        }
+static void
+gf_free_xdr_getspec_rsp(gf_getspec_rsp rsp)
+{
+    if (rsp.spec) {
+        free(rsp.spec);
+    }
+    if (rsp.xdata.xdata_val) {
+        free(rsp.xdata.xdata_val);
+    }
+}
 
-        gf_log ("cli", GF_LOG_INFO, "Received resp to probe");
+static void
+gf_free_xdr_fsm_log_rsp(gf1_cli_fsm_log_rsp rsp)
+{
+    if (rsp.op_errstr) {
+        free(rsp.op_errstr);
+    }
+    if (rsp.fsm_log.fsm_log_val) {
+        free(rsp.fsm_log.fsm_log_val);
+    }
+}
 
-        if (rsp.op_errstr && (strlen (rsp.op_errstr) > 0)) {
-                snprintf (msg, sizeof (msg), "%s", rsp.op_errstr);
-                if (rsp.op_ret)
-                        gf_log ("cli", GF_LOG_ERROR, "%s", msg);
+static int
+gf_cli_probe_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                 void *myframe)
+{
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    char msg[1024] = "success";
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        // rsp.op_ret   = -1;
+        // rsp.op_errno = EINVAL;
+        goto out;
+    }
+
+    gf_log("cli", GF_LOG_INFO, "Received resp to probe");
+
+    if (rsp.op_errstr && rsp.op_errstr[0] != '\0') {
+        snprintf(msg, sizeof(msg), "%s", rsp.op_errstr);
+        if (rsp.op_ret) {
+            gf_log("cli", GF_LOG_ERROR, "%s", msg);
         }
+    }
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_str (NULL,
-                                          (rsp.op_ret)? NULL : msg,
-                                          rsp.op_ret, rsp.op_errno,
-                                          (rsp.op_ret)? msg : NULL);
-                if (ret)
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
-                goto out;
-        }
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_str(NULL, (rsp.op_ret) ? NULL : msg, rsp.op_ret,
+                                 rsp.op_errno, (rsp.op_ret) ? msg : NULL);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
 
-        if (!rsp.op_ret)
-                cli_out ("peer probe: success. %s", msg);
-        else
-                cli_err ("peer probe: failed: %s", msg);
+    if (!rsp.op_ret)
+        cli_out("peer probe: %s", msg);
+    else
+        cli_err("peer probe: failed: %s", msg);
 
-        ret = rsp.op_ret;
+    ret = rsp.op_ret;
 
 out:
-        cli_cmd_broadcast_response (ret);
-        return ret;
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
+    return ret;
 }
 
-int
-gf_cli_deprobe_cbk (struct rpc_req *req, struct iovec *iov,
-                       int count, void *myframe)
+static int
+gf_cli_deprobe_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                   void *myframe)
 {
-        gf_cli_rsp            rsp   = {0,};
-        int                   ret   = -1;
-        char              msg[1024] = {0,};
-
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
-
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                //rsp.op_ret   = -1;
-                //rsp.op_errno = EINVAL;
-                goto out;
-        }
-
-        gf_log ("cli", GF_LOG_INFO, "Received resp to deprobe");
-
-        if (rsp.op_ret) {
-                if (strlen (rsp.op_errstr) > 0) {
-                        snprintf (msg, sizeof (msg), "%s", rsp.op_errstr);
-                        gf_log ("cli", GF_LOG_ERROR, "%s", rsp.op_errstr);
-                }
-        } else {
-                snprintf (msg, sizeof (msg), "success");
-        }
-
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_str (NULL,
-                                          (rsp.op_ret)? NULL : msg,
-                                          rsp.op_ret, rsp.op_errno,
-                                          (rsp.op_ret)? msg : NULL);
-                if (ret)
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
-                goto out;
-        }
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    char msg[1024] = "success";
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        // rsp.op_ret   = -1;
+        // rsp.op_errno = EINVAL;
+        goto out;
+    }
+
+    gf_log("cli", GF_LOG_INFO, "Received resp to deprobe");
+
+    if (rsp.op_ret) {
+        if (rsp.op_errstr[0] != '\0') {
+            snprintf(msg, sizeof(msg), "%s", rsp.op_errstr);
+            gf_log("cli", GF_LOG_ERROR, "%s", rsp.op_errstr);
+        }
+    }
+
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_str(NULL, (rsp.op_ret) ? NULL : msg, rsp.op_ret,
+                                 rsp.op_errno, (rsp.op_ret) ? msg : NULL);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
 
-        if (!rsp.op_ret)
-                cli_out ("peer detach: %s", msg);
-        else
-                cli_err ("peer detach: failed: %s", msg);
+    if (!rsp.op_ret)
+        cli_out("peer detach: %s", msg);
+    else
+        cli_err("peer detach: failed: %s", msg);
 
-        ret = rsp.op_ret;
+    ret = rsp.op_ret;
 
 out:
-        cli_cmd_broadcast_response (ret);
-        return ret;
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
+    return ret;
 }
 
-int
-gf_cli_output_peer_hostnames (dict_t *dict, int count, char *prefix)
+static int
+gf_cli_output_peer_hostnames(dict_t *dict, int count, const char *prefix)
 {
-        int ret = -1;
-        char key[256] = {0,};
-        int i = 0;
-        char *hostname = NULL;
-
-        cli_out ("Other names:");
-        /* Starting from friend.hostname1, as friend.hostname0 will be the same
-         * as friend.hostname
-         */
-        for (i = 1; i < count; i++) {
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.hostname%d", prefix, i);
-                ret = dict_get_str (dict, key, &hostname);
-                if (ret)
-                        break;
-                cli_out ("%s", hostname);
-                hostname = NULL;
-        }
+    int ret = -1;
+    char key[512] = {
+        0,
+    };
+    int i = 0;
+    char *hostname = NULL;
+
+    cli_out("Other names:");
+    /* Starting from friend.hostname1, as friend.hostname0 will be the same
+     * as friend.hostname
+     */
+    for (i = 1; i < count; i++) {
+        ret = snprintf(key, sizeof(key), "%s.hostname%d", prefix, i);
+        ret = dict_get_strn(dict, key, ret, &hostname);
+        if (ret)
+            break;
+        cli_out("%s", hostname);
+        hostname = NULL;
+    }
 
-        return ret;
+    return ret;
 }
 
-int
-gf_cli_output_peer_status (dict_t *dict, int count)
-{
-        int                        ret   = -1;
-        char                       *uuid_buf = NULL;
-        char                       *hostname_buf = NULL;
-        int32_t                    i = 1;
-        char                       key[256] = {0,};
-        char                       *state = NULL;
-        int32_t                    connected = 0;
-        char                       *connected_str = NULL;
-        int                        hostname_count = 0;
-
-        cli_out ("Number of Peers: %d", count);
-        i = 1;
-        while ( i <= count) {
-                snprintf (key, 256, "friend%d.uuid", i);
-                ret = dict_get_str (dict, key, &uuid_buf);
-                if (ret)
-                        goto out;
-
-                snprintf (key, 256, "friend%d.hostname", i);
-                ret = dict_get_str (dict, key, &hostname_buf);
-                if (ret)
-                        goto out;
+static int
+gf_cli_output_peer_status(dict_t *dict, int count)
+{
+    int ret = -1;
+    char *uuid_buf = NULL;
+    char *hostname_buf = NULL;
+    int32_t i = 1;
+    char key[256] = {
+        0,
+    };
+    int keylen;
+    char *state = NULL;
+    int32_t connected = 0;
+    const char *connected_str = NULL;
+    int hostname_count = 0;
+
+    cli_out("Number of Peers: %d", count);
+    i = 1;
+    while (i <= count) {
+        keylen = snprintf(key, sizeof(key), "friend%d.uuid", i);
+        ret = dict_get_strn(dict, key, keylen, &uuid_buf);
+        if (ret)
+            goto out;
 
-                snprintf (key, 256, "friend%d.connected", i);
-                ret = dict_get_int32 (dict, key, &connected);
-                if (ret)
-                        goto out;
-                if (connected)
-                        connected_str = "Connected";
-                else
-                        connected_str = "Disconnected";
+        keylen = snprintf(key, sizeof(key), "friend%d.hostname", i);
+        ret = dict_get_strn(dict, key, keylen, &hostname_buf);
+        if (ret)
+            goto out;
 
+        keylen = snprintf(key, sizeof(key), "friend%d.connected", i);
+        ret = dict_get_int32n(dict, key, keylen, &connected);
+        if (ret)
+            goto out;
+        if (connected)
+            connected_str = "Connected";
+        else
+            connected_str = "Disconnected";
 
-                snprintf (key, 256, "friend%d.state", i);
-                ret = dict_get_str (dict, key, &state);
-                if (ret)
-                        goto out;
+        keylen = snprintf(key, sizeof(key), "friend%d.state", i);
+        ret = dict_get_strn(dict, key, keylen, &state);
+        if (ret)
+            goto out;
 
-                cli_out ("\nHostname: %s\nUuid: %s\nState: %s (%s)",
-                         hostname_buf, uuid_buf, state, connected_str);
+        cli_out("\nHostname: %s\nUuid: %s\nState: %s (%s)", hostname_buf,
+                uuid_buf, state, connected_str);
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "friend%d.hostname_count", i);
-                ret = dict_get_int32 (dict, key, &hostname_count);
-                /* Print other addresses only if there are more than 1.
-                 */
-                if ((ret == 0) && (hostname_count > 1)) {
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key), "friend%d", i);
-                        ret = gf_cli_output_peer_hostnames (dict,
-                                                            hostname_count,
-                                                            key);
-                }
-                i++;
+        keylen = snprintf(key, sizeof(key), "friend%d.hostname_count", i);
+        ret = dict_get_int32n(dict, key, keylen, &hostname_count);
+        /* Print other addresses only if there are more than 1.
+         */
+        if ((ret == 0) && (hostname_count > 1)) {
+            snprintf(key, sizeof(key), "friend%d", i);
+            ret = gf_cli_output_peer_hostnames(dict, hostname_count, key);
+            if (ret) {
+                gf_log("cli", GF_LOG_WARNING,
+                       "error outputting peer other names");
+                goto out;
+            }
         }
+        i++;
+    }
 
-        ret = 0;
+    ret = 0;
 out:
-        return ret;
+    return ret;
 }
 
-int
-gf_cli_output_pool_list (dict_t *dict, int count)
-{
-        int                         ret           = -1;
-        char                       *uuid_buf      = NULL;
-        char                       *hostname_buf  = NULL;
-        int32_t                     hostname_len  = 8; /*min len 8 chars*/
-        int32_t                     i             = 1;
-        char                        key[256]      = {0,};
-        int32_t                     connected     = 0;
-        char                       *connected_str = NULL;
-
-        if (count <= 0)
-                goto out;
-
-        while (i <= count) {
-                snprintf (key, 256, "friend%d.hostname", i);
-                ret = dict_get_str (dict, key, &hostname_buf);
-                if (ret)
-                        goto out;
+static int
+gf_cli_output_pool_list(dict_t *dict, int count)
+{
+    int ret = -1;
+    char *uuid_buf = NULL;
+    char *hostname_buf = NULL;
+    int32_t hostname_len = 8; /*min len 8 chars*/
+    int32_t i = 1;
+    char key[64] = {
+        0,
+    };
+    int keylen;
+    int32_t connected = 0;
+    const char *connected_str = NULL;
+
+    if (count <= 0)
+        goto out;
+
+    while (i <= count) {
+        keylen = snprintf(key, sizeof(key), "friend%d.hostname", i);
+        ret = dict_get_strn(dict, key, keylen, &hostname_buf);
+        if (ret)
+            goto out;
 
-                ret = strlen(hostname_buf);
-                if (ret > hostname_len)
-                        hostname_len = ret;
+        ret = strlen(hostname_buf);
+        if (ret > hostname_len)
+            hostname_len = ret;
 
-                i++;
-        }
+        i++;
+    }
 
-        cli_out ("UUID\t\t\t\t\t%-*s\tState", hostname_len, "Hostname");
+    cli_out("UUID\t\t\t\t\t%-*s\tState", hostname_len, "Hostname");
 
-        i = 1;
-        while ( i <= count) {
-                snprintf (key, 256, "friend%d.uuid", i);
-                ret = dict_get_str (dict, key, &uuid_buf);
-                if (ret)
-                        goto out;
+    i = 1;
+    while (i <= count) {
+        keylen = snprintf(key, sizeof(key), "friend%d.uuid", i);
+        ret = dict_get_strn(dict, key, keylen, &uuid_buf);
+        if (ret)
+            goto out;
 
-                snprintf (key, 256, "friend%d.hostname", i);
-                ret = dict_get_str (dict, key, &hostname_buf);
-                if (ret)
-                        goto out;
+        keylen = snprintf(key, sizeof(key), "friend%d.hostname", i);
+        ret = dict_get_strn(dict, key, keylen, &hostname_buf);
+        if (ret)
+            goto out;
 
-                snprintf (key, 256, "friend%d.connected", i);
-                ret = dict_get_int32 (dict, key, &connected);
-                if (ret)
-                        goto out;
-                if (connected)
-                        connected_str = "Connected";
-                else
-                        connected_str = "Disconnected";
+        keylen = snprintf(key, sizeof(key), "friend%d.connected", i);
+        ret = dict_get_int32n(dict, key, keylen, &connected);
+        if (ret)
+            goto out;
+        if (connected)
+            connected_str = "Connected";
+        else
+            connected_str = "Disconnected";
 
-                cli_out ("%s\t%-*s\t%s ", uuid_buf, hostname_len, hostname_buf,
-                         connected_str);
-                i++;
-        }
+        cli_out("%s\t%-*s\t%s ", uuid_buf, hostname_len, hostname_buf,
+                connected_str);
+        i++;
+    }
 
-        ret = 0;
+    ret = 0;
 out:
-        return ret;
+    return ret;
 }
 
 /* function pointer for gf_cli_output_{pool_list,peer_status} */
-typedef int (*cli_friend_output_fn) (dict_t*, int);
+typedef int (*cli_friend_output_fn)(dict_t *, int);
 
-int
-gf_cli_list_friends_cbk (struct rpc_req *req, struct iovec *iov,
-                             int count, void *myframe)
-{
-        gf1_cli_peer_list_rsp      rsp   = {0,};
-        int                        ret   = -1;
-        dict_t                    *dict = NULL;
-        char                       msg[1024] = {0,};
-        char                      *cmd = NULL;
-        cli_friend_output_fn       friend_output_fn;
-        call_frame_t              *frame = NULL;
-        unsigned long              flags = 0;
-
-        frame = myframe;
-        flags = (long)frame->local;
-
-        if (flags == GF_CLI_LIST_POOL_NODES) {
-                cmd = "pool list";
-                friend_output_fn = &gf_cli_output_pool_list;
-        } else {
-                cmd = "peer status";
-                friend_output_fn = &gf_cli_output_peer_status;
+static int
+gf_cli_list_friends_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                        void *myframe)
+{
+    gf1_cli_peer_list_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    dict_t *dict = NULL;
+    char msg[1024] = {
+        0,
+    };
+    const char *cmd = NULL;
+    cli_friend_output_fn friend_output_fn;
+    call_frame_t *frame = NULL;
+    unsigned long flags = 0;
+
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
+
+    GF_ASSERT(myframe);
+
+    frame = myframe;
+
+    flags = (long)frame->local;
+
+    if (flags == GF_CLI_LIST_POOL_NODES) {
+        cmd = "pool list";
+        friend_output_fn = &gf_cli_output_pool_list;
+    } else {
+        cmd = "peer status";
+        friend_output_fn = &gf_cli_output_peer_status;
+    }
+
+    /* 'free' the flags set by gf_cli_list_friends */
+    frame->local = NULL;
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf1_cli_peer_list_rsp);
+    if (ret < 0) {
+        gf_log(frame->this->name, GF_LOG_ERROR, XDR_DECODE_FAIL);
+        // rsp.op_ret   = -1;
+        // rsp.op_errno = EINVAL;
+        goto out;
+    }
+
+    gf_log("cli", GF_LOG_DEBUG, "Received resp to list: %d", rsp.op_ret);
+
+    if (!rsp.op_ret) {
+        if (!rsp.friends.friends_len) {
+            snprintf(msg, sizeof(msg), "%s: No peers present", cmd);
+            if (global_state->mode & GLUSTER_MODE_XML) {
+                ret = cli_xml_output_peer_status(dict, rsp.op_ret, rsp.op_errno,
+                                                 msg);
+                if (ret)
+                    gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+                goto out;
+            }
+            cli_err("%s", msg);
+            ret = 0;
+            goto out;
         }
 
-        /* 'free' the flags set by gf_cli_list_friends */
-        frame->local = NULL;
+        dict = dict_new();
 
-        if (-1 == req->rpc_status) {
-                goto out;
+        if (!dict) {
+            ret = -1;
+            goto out;
         }
 
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf1_cli_peer_list_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                //rsp.op_ret   = -1;
-                //rsp.op_errno = EINVAL;
-                goto out;
+        ret = dict_unserialize(rsp.friends.friends_val, rsp.friends.friends_len,
+                               &dict);
+
+        if (ret) {
+            gf_log("", GF_LOG_ERROR, DICT_UNSERIALIZE_FAIL);
+            goto out;
         }
 
-        gf_log ("cli", GF_LOG_DEBUG, "Received resp to list: %d",
-                rsp.op_ret);
+        if (global_state->mode & GLUSTER_MODE_XML) {
+            ret = cli_xml_output_peer_status(dict, rsp.op_ret, rsp.op_errno,
+                                             msg);
+            if (ret)
+                gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+            goto out;
+        }
 
-        ret = rsp.op_ret;
+        ret = dict_get_int32_sizen(dict, "count", &count);
+        if (ret) {
+            goto out;
+        }
 
-        if (!rsp.op_ret) {
-
-                if (!rsp.friends.friends_len) {
-                        snprintf (msg, sizeof (msg),
-                                  "%s: No peers present", cmd);
-                        if (global_state->mode & GLUSTER_MODE_XML) {
-                                ret = cli_xml_output_peer_status (dict,
-                                                                  rsp.op_ret,
-                                                                  rsp.op_errno,
-                                                                  msg);
-                                if (ret)
-                                        gf_log ("cli", GF_LOG_ERROR,
-                                                "Error outputting to xml");
-                                goto out;
-                        }
-                        cli_err ("%s", msg);
-                        ret = 0;
-                        goto out;
-                }
+        ret = friend_output_fn(dict, count);
+        if (ret) {
+            goto out;
+        }
+    } else {
+        if (global_state->mode & GLUSTER_MODE_XML) {
+            ret = cli_xml_output_peer_status(dict, rsp.op_ret, rsp.op_errno,
+                                             NULL);
+            if (ret)
+                gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        } else {
+            ret = -1;
+        }
+        goto out;
+    }
 
-                dict = dict_new ();
+    ret = 0;
 
-                if (!dict) {
-                        ret = -1;
-                        goto out;
-                }
+out:
+    if (ret)
+        cli_err("%s: failed", cmd);
 
-                ret = dict_unserialize (rsp.friends.friends_val,
-                                        rsp.friends.friends_len,
-                                        &dict);
+    cli_cmd_broadcast_response(ret);
 
-                if (ret) {
-                        gf_log ("", GF_LOG_ERROR,
-                                        "Unable to allocate memory");
-                        goto out;
-                }
+    if (dict)
+        dict_unref(dict);
 
-                if (global_state->mode & GLUSTER_MODE_XML) {
-                        ret = cli_xml_output_peer_status (dict, rsp.op_ret,
-                                                          rsp.op_errno, msg);
-                        if (ret)
-                                gf_log ("cli", GF_LOG_ERROR,
-                                        "Error outputting to xml");
-                        goto out;
-                }
+    if (rsp.friends.friends_val) {
+        free(rsp.friends.friends_val);
+    }
+    return ret;
+}
 
-                ret = dict_get_int32 (dict, "count", &count);
-                if (ret) {
-                        goto out;
-                }
+static int
+gf_cli_get_state_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                     void *myframe)
+{
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    dict_t *dict = NULL;
+    char *daemon_name = NULL;
+    char *ofilepath = NULL;
+
+    GF_VALIDATE_OR_GOTO("cli", myframe, out);
+
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    dict = dict_new();
+
+    if (!dict) {
+        ret = -1;
+        goto out;
+    }
+
+    ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &dict);
+    if (ret)
+        goto out;
+
+    if (rsp.op_ret) {
+        if (strcmp(rsp.op_errstr, ""))
+            cli_err("Failed to get daemon state: %s", rsp.op_errstr);
+        else
+            cli_err(
+                "Failed to get daemon state. Check glusterd"
+                " log file for more details");
+    } else {
+        ret = dict_get_str_sizen(dict, "daemon", &daemon_name);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, "Couldn't get daemon name");
 
-                ret = friend_output_fn (dict, count);
-                if (ret) {
-                        goto out;
-                }
-        } else {
-                if (global_state->mode & GLUSTER_MODE_XML) {
-                        ret = cli_xml_output_peer_status (dict, rsp.op_ret,
-                                                          rsp.op_errno, NULL);
-                        if (ret)
-                                gf_log ("cli", GF_LOG_ERROR,
-                                        "Error outputting to xml");
-                } else {
-                        ret = -1;
-                }
-                goto out;
-        }
+        ret = dict_get_str_sizen(dict, "ofilepath", &ofilepath);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, "Couldn't get filepath");
 
+        if (daemon_name && ofilepath)
+            cli_out("%s state dumped to %s", daemon_name, ofilepath);
+    }
 
-        ret = 0;
+    ret = rsp.op_ret;
 
 out:
-        cli_cmd_broadcast_response (ret);
-        if (ret)
-                cli_err ("%s: failed", cmd);
+    if (dict)
+        dict_unref(dict);
 
-        if (dict)
-                dict_destroy (dict);
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
 
-        return ret;
+    return ret;
 }
 
-void
-cli_out_options ( char *substr, char *optstr, char *valstr)
+static void
+cli_out_options(char *substr, char *optstr, char *valstr)
 {
-        char                    *ptr1 = NULL;
-        char                    *ptr2 = NULL;
+    char *ptr1 = NULL;
+    char *ptr2 = NULL;
+
+    ptr1 = substr;
+    ptr2 = optstr;
+
+    while (ptr1) {
+        /* Avoiding segmentation fault. */
+        if (!ptr2)
+            return;
+        if (*ptr1 != *ptr2)
+            break;
+        ptr1++;
+        ptr2++;
+    }
+
+    if (*ptr2 == '\0')
+        return;
+    cli_out("%s: %s", ptr2, valstr);
+}
 
-        ptr1 = substr;
-        ptr2 = optstr;
+static int
+_gf_cli_output_volinfo_opts(dict_t *d, char *k, data_t *v, void *tmp)
+{
+    int ret = 0;
+    char *key = NULL;
+    char *ptr = NULL;
+    data_t *value = NULL;
+
+    key = tmp;
+
+    ptr = strstr(k, "option.");
+    if (ptr) {
+        value = v;
+        if (!value) {
+            ret = -1;
+            goto out;
+        }
+        cli_out_options(key, k, v->data);
+    }
+out:
+    return ret;
+}
 
-        while (ptr1)
-        {
-                if (*ptr1 != *ptr2)
-                        break;
-                ptr1++;
-                ptr2++;
-                if (!ptr1)
-                        return;
-                if (!ptr2)
-                        return;
-        }
+static int
+print_brick_details(dict_t *dict, int volcount, int start_index, int end_index,
+                    int replica_count)
+{
+    char key[64] = {
+        0,
+    };
+    int keylen;
+    int index = start_index;
+    int isArbiter = 0;
+    int ret = -1;
+    char *brick = NULL;
+
+    while (index <= end_index) {
+        keylen = snprintf(key, sizeof(key), "volume%d.brick%d", volcount,
+                          index);
+        ret = dict_get_strn(dict, key, keylen, &brick);
+        if (ret)
+            goto out;
+        keylen = snprintf(key, sizeof(key), "volume%d.brick%d.isArbiter",
+                          volcount, index);
+        if (dict_getn(dict, key, keylen))
+            isArbiter = 1;
+        else
+            isArbiter = 0;
 
-        if (*ptr2 == '\0')
-                return;
-        cli_out ("%s: %s",ptr2 , valstr);
+        if (isArbiter)
+            cli_out("Brick%d: %s (arbiter)", index, brick);
+        else
+            cli_out("Brick%d: %s", index, brick);
+        index++;
+    }
+    ret = 0;
+out:
+    return ret;
 }
 
-static int
-_gf_cli_output_volinfo_opts (dict_t *d, char *k,
-                             data_t *v, void *tmp)
-{
-        int     ret   = 0;
-        char   *key   = NULL;
-        char   *ptr   = NULL;
-        data_t *value = NULL;
-
-        key = tmp;
-
-        ptr = strstr (k, "option.");
-        if (ptr) {
-                value = v;
-                if (!value) {
-                        ret = -1;
-                        goto out;
-                }
-                cli_out_options (key, k, v->data);
+static void
+gf_cli_print_number_of_bricks(int type, int brick_count, int dist_count,
+                              int stripe_count, int replica_count,
+                              int disperse_count, int redundancy_count,
+                              int arbiter_count)
+{
+    if (type == GF_CLUSTER_TYPE_NONE) {
+        cli_out("Number of Bricks: %d", brick_count);
+    } else if (type == GF_CLUSTER_TYPE_DISPERSE) {
+        cli_out("Number of Bricks: %d x (%d + %d) = %d",
+                (brick_count / dist_count), disperse_count - redundancy_count,
+                redundancy_count, brick_count);
+    } else {
+        /* For both replicate and stripe, dist_count is
+           good enough */
+        if (arbiter_count == 0) {
+            cli_out("Number of Bricks: %d x %d = %d",
+                    (brick_count / dist_count), dist_count, brick_count);
+        } else {
+            cli_out("Number of Bricks: %d x (%d + %d) = %d",
+                    (brick_count / dist_count), dist_count - arbiter_count,
+                    arbiter_count, brick_count);
         }
-out:
-        return ret;
+    }
 }
 
+static int
+gf_cli_get_volume_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                      void *myframe)
+{
+    int ret = -1;
+    int opt_count = 0;
+    int32_t i = 0;
+    int32_t j = 1;
+    int32_t status = 0;
+    int32_t type = 0;
+    int32_t brick_count = 0;
+    int32_t dist_count = 0;
+    int32_t stripe_count = 0;
+    int32_t replica_count = 0;
+    int32_t disperse_count = 0;
+    int32_t redundancy_count = 0;
+    int32_t arbiter_count = 0;
+    int32_t snap_count = 0;
+    int32_t thin_arbiter_count = 0;
+    int32_t vol_type = 0;
+    int32_t transport = 0;
+    char *volume_id_str = NULL;
+    char *volname = NULL;
+    char *ta_brick = NULL;
+    dict_t *dict = NULL;
+    cli_local_t *local = NULL;
+    char key[64] = {0};
+    int keylen;
+    char err_str[2048] = {0};
+    gf_cli_rsp rsp = {0};
+    char *caps __attribute__((unused)) = NULL;
+    int k __attribute__((unused)) = 0;
+    call_frame_t *frame = NULL;
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status)
+        goto out;
+
+    frame = myframe;
+
+    GF_ASSERT(frame->local);
+
+    local = frame->local;
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(frame->this->name, GF_LOG_ERROR, XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    gf_log("cli", GF_LOG_INFO, "Received resp to get vol: %d", rsp.op_ret);
+
+    if (!rsp.dict.dict_len) {
+        if (global_state->mode & GLUSTER_MODE_XML)
+            goto xml_output;
+        cli_err("No volumes present");
+        ret = 0;
+        goto out;
+    }
+
+    dict = dict_new();
 
-int
-gf_cli_get_volume_cbk (struct rpc_req *req, struct iovec *iov,
-                          int count, void *myframe)
-{
-        int                        ret                  = -1;
-        int                        opt_count            = 0;
-        int32_t                    i                    = 0;
-        int32_t                    j                    = 1;
-        int32_t                    status               = 0;
-        int32_t                    type                 = 0;
-        int32_t                    brick_count          = 0;
-        int32_t                    dist_count           = 0;
-        int32_t                    stripe_count         = 0;
-        int32_t                    replica_count        = 0;
-        int32_t                    disperse_count       = 0;
-        int32_t                    redundancy_count     = 0;
-        int32_t                    vol_type             = 0;
-        int32_t                    transport            = 0;
-        char                      *volume_id_str        = NULL;
-        char                      *brick                = NULL;
-        char                      *volname              = NULL;
-        dict_t                    *dict                 = NULL;
-        cli_local_t               *local                = NULL;
-        char                       key[1024]            = {0};
-        char                       err_str[2048]        = {0};
-        gf_cli_rsp                 rsp                  = {0};
-        char                      *caps                 = NULL;
-        int                        k __attribute__((unused)) = 0;
-
-        if (-1 == req->rpc_status)
-                goto out;
+    if (!dict) {
+        ret = -1;
+        goto out;
+    }
 
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
+    ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &dict);
 
-        gf_log ("cli", GF_LOG_INFO, "Received resp to get vol: %d",
-                rsp.op_ret);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, DICT_UNSERIALIZE_FAIL);
+        goto out;
+    }
 
-        if (rsp.op_ret) {
-                ret = -1;
-                goto out;
-        }
+    ret = dict_get_int32_sizen(dict, "count", &count);
+    if (ret)
+        goto out;
 
-        if (!rsp.dict.dict_len) {
-                if (global_state->mode & GLUSTER_MODE_XML)
-                        goto xml_output;
-                cli_err ("No volumes present");
+    if (!count) {
+        switch (local->get_vol.flags) {
+            case GF_CLI_GET_NEXT_VOLUME:
+                GF_FREE(local->get_vol.volname);
+                local->get_vol.volname = NULL;
                 ret = 0;
                 goto out;
-        }
-
-        dict = dict_new ();
 
-        if (!dict) {
+            case GF_CLI_GET_VOLUME:
+                snprintf(err_str, sizeof(err_str), "Volume %s does not exist",
+                         local->get_vol.volname);
                 ret = -1;
-                goto out;
+                if (!(global_state->mode & GLUSTER_MODE_XML))
+                    goto out;
         }
+    }
 
-        ret = dict_unserialize (rsp.dict.dict_val,
-                                rsp.dict.dict_len,
-                                &dict);
+    if (rsp.op_ret) {
+        if (global_state->mode & GLUSTER_MODE_XML)
+            goto xml_output;
+        ret = -1;
+        goto out;
+    }
 
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR,
-                        "Unable to allocate memory");
+xml_output:
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        /* For GET_NEXT_VOLUME output is already begun in
+         * and will also end in gf_cli_get_next_volume()
+         */
+        if (local->get_vol.flags == GF_CLI_GET_VOLUME) {
+            ret = cli_xml_output_vol_info_begin(local, rsp.op_ret, rsp.op_errno,
+                                                rsp.op_errstr);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR, XML_ERROR);
                 goto out;
+            }
         }
 
-        ret = dict_get_int32 (dict, "count", &count);
-        if (ret)
+        if (dict) {
+            ret = cli_xml_output_vol_info(local, dict);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR, XML_ERROR);
                 goto out;
+            }
+        }
 
-        local = ((call_frame_t *)myframe)->local;
-
-        if (!count) {
-                switch (local->get_vol.flags) {
-
-                case GF_CLI_GET_NEXT_VOLUME:
-                        GF_FREE (local->get_vol.volname);
-                        local->get_vol.volname = NULL;
-                        ret = 0;
-                        goto out;
-
-                case GF_CLI_GET_VOLUME:
-                        memset (err_str, 0, sizeof (err_str));
-                        snprintf (err_str, sizeof (err_str),
-                                  "Volume %s does not exist",
-                                  local->get_vol.volname);
-                        ret = -1;
-                        if (!(global_state->mode & GLUSTER_MODE_XML))
-                                goto out;
-                }
+        if (local->get_vol.flags == GF_CLI_GET_VOLUME) {
+            ret = cli_xml_output_vol_info_end(local);
+            if (ret)
+                gf_log("cli", GF_LOG_ERROR, XML_ERROR);
         }
+        goto out;
+    }
 
-xml_output:
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                /* For GET_NEXT_VOLUME output is already begun in
-                 * and will also end in gf_cli_get_next_volume()
-                 */
-                if (local->get_vol.flags == GF_CLI_GET_VOLUME) {
-                        ret = cli_xml_output_vol_info_begin
-                                (local, rsp.op_ret, rsp.op_errno,
-                                 rsp.op_errstr);
-                        if (ret) {
-                                gf_log ("cli", GF_LOG_ERROR,
-                                        "Error outputting to xml");
-                                goto out;
-                        }
-                }
+    while (i < count) {
+        cli_out(" ");
+        keylen = snprintf(key, sizeof(key), "volume%d.name", i);
+        ret = dict_get_strn(dict, key, keylen, &volname);
+        if (ret)
+            goto out;
 
-                if (dict) {
-                        ret = cli_xml_output_vol_info (local, dict);
-                        if (ret) {
-                                gf_log ("cli", GF_LOG_ERROR,
-                                        "Error outputting to xml");
-                                goto out;
-                        }
-                }
+        keylen = snprintf(key, sizeof(key), "volume%d.type", i);
+        ret = dict_get_int32n(dict, key, keylen, &type);
+        if (ret)
+            goto out;
 
-                if (local->get_vol.flags == GF_CLI_GET_VOLUME) {
-                        ret = cli_xml_output_vol_info_end (local);
-                        if (ret)
-                                gf_log ("cli", GF_LOG_ERROR,
-                                        "Error outputting to xml");
-                }
-                goto out;
-        }
+        keylen = snprintf(key, sizeof(key), "volume%d.status", i);
+        ret = dict_get_int32n(dict, key, keylen, &status);
+        if (ret)
+            goto out;
 
-        while ( i < count) {
-                cli_out (" ");
-                snprintf (key, 256, "volume%d.name", i);
-                ret = dict_get_str (dict, key, &volname);
-                if (ret)
-                        goto out;
+        keylen = snprintf(key, sizeof(key), "volume%d.brick_count", i);
+        ret = dict_get_int32n(dict, key, keylen, &brick_count);
+        if (ret)
+            goto out;
 
-                snprintf (key, 256, "volume%d.type", i);
-                ret = dict_get_int32 (dict, key, &type);
-                if (ret)
-                        goto out;
+        keylen = snprintf(key, sizeof(key), "volume%d.dist_count", i);
+        ret = dict_get_int32n(dict, key, keylen, &dist_count);
+        if (ret)
+            goto out;
 
-                snprintf (key, 256, "volume%d.status", i);
-                ret = dict_get_int32 (dict, key, &status);
-                if (ret)
-                        goto out;
+        keylen = snprintf(key, sizeof(key), "volume%d.stripe_count", i);
+        ret = dict_get_int32n(dict, key, keylen, &stripe_count);
+        if (ret)
+            goto out;
 
-                snprintf (key, 256, "volume%d.brick_count", i);
-                ret = dict_get_int32 (dict, key, &brick_count);
-                if (ret)
-                        goto out;
+        keylen = snprintf(key, sizeof(key), "volume%d.replica_count", i);
+        ret = dict_get_int32n(dict, key, keylen, &replica_count);
+        if (ret)
+            goto out;
 
-                snprintf (key, 256, "volume%d.dist_count", i);
-                ret = dict_get_int32 (dict, key, &dist_count);
-                if (ret)
-                        goto out;
+        keylen = snprintf(key, sizeof(key), "volume%d.disperse_count", i);
+        ret = dict_get_int32n(dict, key, keylen, &disperse_count);
+        if (ret)
+            goto out;
 
-                snprintf (key, 256, "volume%d.stripe_count", i);
-                ret = dict_get_int32 (dict, key, &stripe_count);
-                if (ret)
-                        goto out;
+        keylen = snprintf(key, sizeof(key), "volume%d.redundancy_count", i);
+        ret = dict_get_int32n(dict, key, keylen, &redundancy_count);
+        if (ret)
+            goto out;
 
-                snprintf (key, 256, "volume%d.replica_count", i);
-                ret = dict_get_int32 (dict, key, &replica_count);
-                if (ret)
-                        goto out;
+        keylen = snprintf(key, sizeof(key), "volume%d.arbiter_count", i);
+        ret = dict_get_int32n(dict, key, keylen, &arbiter_count);
+        if (ret)
+            goto out;
 
-                snprintf (key, 256, "volume%d.disperse_count", i);
-                ret = dict_get_int32 (dict, key, &disperse_count);
-                if (ret)
-                        goto out;
+        keylen = snprintf(key, sizeof(key), "volume%d.transport", i);
+        ret = dict_get_int32n(dict, key, keylen, &transport);
+        if (ret)
+            goto out;
 
-                snprintf (key, 256, "volume%d.redundancy_count", i);
-                ret = dict_get_int32 (dict, key, &redundancy_count);
-                if (ret)
-                        goto out;
+        keylen = snprintf(key, sizeof(key), "volume%d.volume_id", i);
+        ret = dict_get_strn(dict, key, keylen, &volume_id_str);
+        if (ret)
+            goto out;
 
-                snprintf (key, 256, "volume%d.transport", i);
-                ret = dict_get_int32 (dict, key, &transport);
-                if (ret)
-                        goto out;
+        keylen = snprintf(key, sizeof(key), "volume%d.snap_count", i);
+        ret = dict_get_int32n(dict, key, keylen, &snap_count);
+        if (ret)
+            goto out;
 
-                snprintf (key, 256, "volume%d.volume_id", i);
-                ret = dict_get_str (dict, key, &volume_id_str);
-                if (ret)
-                        goto out;
+        keylen = snprintf(key, sizeof(key), "volume%d.thin_arbiter_count", i);
+        ret = dict_get_int32n(dict, key, keylen, &thin_arbiter_count);
+        if (ret)
+            goto out;
 
-                vol_type = type;
+        // Distributed (stripe/replicate/stripe-replica) setups
+        vol_type = get_vol_type(type, dist_count, brick_count);
 
-                // Distributed (stripe/replicate/stripe-replica) setups
-                if ((type > 0) && ( dist_count < brick_count))
-                        vol_type = type + 4;
+        cli_out("Volume Name: %s", volname);
+        cli_out("Type: %s", vol_type_str[vol_type]);
+        cli_out("Volume ID: %s", volume_id_str);
+        cli_out("Status: %s", cli_vol_status_str[status]);
+        cli_out("Snapshot Count: %d", snap_count);
 
-                cli_out ("Volume Name: %s", volname);
-                cli_out ("Type: %s", cli_vol_type_str[vol_type]);
-                cli_out ("Volume ID: %s", volume_id_str);
-                cli_out ("Status: %s", cli_vol_status_str[status]);
+        gf_cli_print_number_of_bricks(
+            type, brick_count, dist_count, stripe_count, replica_count,
+            disperse_count, redundancy_count, arbiter_count);
 
-#ifdef HAVE_BD_XLATOR
-                k = 0;
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "volume%d.xlator%d", i, k);
-                ret = dict_get_str (dict, key, &caps);
-                if (ret)
-                        goto next;
-                do {
-                        j = 0;
-                        cli_out ("Xlator %d: %s", k + 1, caps);
-                        do {
-                                memset (key, 0, sizeof (key));
-                                snprintf (key, sizeof (key),
-                                          "volume%d.xlator%d.caps%d",
-                                          i, k, j++);
-                                ret = dict_get_str (dict, key, &caps);
-                                if (ret)
-                                        break;
-                                cli_out ("Capability %d: %s", j, caps);
-                        } while (1);
-
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key),
-                                  "volume%d.xlator%d", i, ++k);
-                        ret = dict_get_str (dict, key, &caps);
-                        if (ret)
-                                break;
-                } while (1);
-
-next:
-#else
-                caps = 0; /* Avoid compiler warnings when BD not enabled */
-#endif
+        cli_out("Transport-type: %s",
+                ((transport == 0) ? "tcp"
+                                  : (transport == 1) ? "rdma" : "tcp,rdma"));
+        j = 1;
 
-                if (type == GF_CLUSTER_TYPE_STRIPE_REPLICATE) {
-                        cli_out ("Number of Bricks: %d x %d x %d = %d",
-                                 (brick_count / dist_count),
-                                 stripe_count,
-                                 replica_count,
-                                 brick_count);
-                } else if (type == GF_CLUSTER_TYPE_NONE) {
-                        cli_out ("Number of Bricks: %d", brick_count);
-                } else if (type == GF_CLUSTER_TYPE_DISPERSE) {
-                        cli_out ("Number of Bricks: %d x (%d + %d) = %d",
-                                 (brick_count / dist_count),
-                                 disperse_count - redundancy_count,
-                                 redundancy_count, brick_count);
-                } else {
-                        /* For both replicate and stripe, dist_count is
-                           good enough */
-                        cli_out ("Number of Bricks: %d x %d = %d",
-                                 (brick_count / dist_count),
-                                 dist_count, brick_count);
-                }
+        GF_FREE(local->get_vol.volname);
+        local->get_vol.volname = gf_strdup(volname);
 
-                cli_out ("Transport-type: %s",
-                         ((transport == 0)?"tcp":
-                          (transport == 1)?"rdma":
-                          "tcp,rdma"));
-                j = 1;
-
-                GF_FREE (local->get_vol.volname);
-                local->get_vol.volname = gf_strdup (volname);
-
-                if (brick_count)
-                        cli_out ("Bricks:");
-
-                while (j <= brick_count) {
-                        snprintf (key, 1024, "volume%d.brick%d", i, j);
-                        ret = dict_get_str (dict, key, &brick);
-                        if (ret)
-                                goto out;
-
-                        cli_out ("Brick%d: %s", j, brick);
-#ifdef HAVE_BD_XLATOR
-                        snprintf (key, 256, "volume%d.vg%d", i, j);
-                        ret = dict_get_str (dict, key, &caps);
-                        if (!ret)
-                                cli_out ("Brick%d VG: %s", j, caps);
-#endif
-                        j++;
-                }
+        cli_out("Bricks:");
+        ret = print_brick_details(dict, i, j, brick_count, replica_count);
+        if (ret)
+            goto out;
 
-                snprintf (key, 256, "volume%d.opt_count",i);
-                ret = dict_get_int32 (dict, key, &opt_count);
-                if (ret)
-                        goto out;
+        if (thin_arbiter_count) {
+            snprintf(key, sizeof(key), "volume%d.thin_arbiter_brick", i);
+            ret = dict_get_str(dict, key, &ta_brick);
+            if (ret)
+                goto out;
+            cli_out("Thin-arbiter-path: %s", ta_brick);
+        }
 
-                if (!opt_count)
-                        goto out;
+        snprintf(key, sizeof(key), "volume%d.opt_count", i);
+        ret = dict_get_int32(dict, key, &opt_count);
+        if (ret)
+            goto out;
 
-                cli_out ("Options Reconfigured:");
+        if (!opt_count)
+            goto out;
 
-                snprintf (key, 256, "volume%d.option.",i);
+        cli_out("Options Reconfigured:");
 
-                ret = dict_foreach (dict, _gf_cli_output_volinfo_opts, key);
-                if (ret)
-                        goto out;
+        snprintf(key, sizeof(key), "volume%d.option.", i);
 
-                i++;
-        }
+        ret = dict_foreach(dict, _gf_cli_output_volinfo_opts, key);
+        if (ret)
+            goto out;
 
+        i++;
+    }
 
-        ret = 0;
+    ret = 0;
 out:
-        cli_cmd_broadcast_response (ret);
-        if (ret)
-                cli_err ("%s", err_str);
+    if (ret)
+        cli_err("%s", err_str);
 
-        if (dict)
-                dict_destroy (dict);
+    cli_cmd_broadcast_response(ret);
 
-        free (rsp.dict.dict_val);
+    if (dict)
+        dict_unref(dict);
 
-        free (rsp.op_errstr);
+    gf_free_xdr_cli_rsp(rsp);
 
-        gf_log ("cli", GF_LOG_INFO, "Returning: %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
+    return ret;
 }
 
-int
-gf_cli_create_volume_cbk (struct rpc_req *req, struct iovec *iov,
-                             int count, void *myframe)
+static int
+gf_cli_create_volume_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                         void *myframe)
 {
-        gf_cli_rsp              rsp   = {0,};
-        int                     ret   = -1;
-        cli_local_t             *local = NULL;
-        char                    *volname = NULL;
-        dict_t                  *dict = NULL;
-        dict_t                  *rsp_dict = NULL;
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    cli_local_t *local = NULL;
+    char *volname = NULL;
+    dict_t *rsp_dict = NULL;
+    call_frame_t *frame = NULL;
 
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
+    GF_ASSERT(myframe);
 
-        local = ((call_frame_t *) (myframe))->local;
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
 
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
+    frame = myframe;
 
-        gf_log ("cli", GF_LOG_INFO, "Received resp to create volume");
+    GF_ASSERT(frame->local);
 
-        dict = local->dict;
+    local = frame->local;
 
-        ret = dict_get_str (dict, "volname", &volname);
-        if (ret)
-                goto out;
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(frame->this->name, GF_LOG_ERROR, XDR_DECODE_FAIL);
+        goto out;
+    }
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                if (rsp.op_ret == 0) {
-                        rsp_dict = dict_new ();
-                        ret = dict_unserialize (rsp.dict.dict_val,
-                                                rsp.dict.dict_len,
-                                                &rsp_dict);
-                        if (ret) {
-                                gf_log ("cli", GF_LOG_ERROR,
-                                        "Failed rsp_dict unserialization");
-                                goto out;
-                        }
-                }
+    gf_log("cli", GF_LOG_INFO, "Received resp to create volume");
 
-                ret = cli_xml_output_vol_create (rsp_dict, rsp.op_ret,
-                                                 rsp.op_errno, rsp.op_errstr);
-                if (ret)
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        if (rsp.op_ret == 0) {
+            rsp_dict = dict_new();
+            ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len,
+                                   &rsp_dict);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR, DICT_UNSERIALIZE_FAIL);
                 goto out;
+            }
         }
 
-        if (rsp.op_ret && strcmp (rsp.op_errstr, ""))
-                cli_err ("volume create: %s: failed: %s", volname,
-                         rsp.op_errstr);
-        else if (rsp.op_ret)
-                cli_err ("volume create: %s: failed", volname);
-        else
-                cli_out ("volume create: %s: success: "
-                         "please start the volume to access data", volname);
-
-        ret = rsp.op_ret;
+        ret = cli_xml_output_vol_create(rsp_dict, rsp.op_ret, rsp.op_errno,
+                                        rsp.op_errstr);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
+
+    ret = dict_get_str_sizen(local->dict, "volname", &volname);
+    if (ret)
+        goto out;
+
+    if (rsp.op_ret && strcmp(rsp.op_errstr, ""))
+        cli_err("volume create: %s: failed: %s", volname, rsp.op_errstr);
+    else if (rsp.op_ret)
+        cli_err("volume create: %s: failed", volname);
+    else
+        cli_out(
+            "volume create: %s: success: "
+            "please start the volume to access data",
+            volname);
+
+    ret = rsp.op_ret;
 
 out:
-        cli_cmd_broadcast_response (ret);
-        free (rsp.dict.dict_val);
-        free (rsp.op_errstr);
-        return ret;
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
+
+    if (rsp_dict)
+        dict_unref(rsp_dict);
+    return ret;
 }
 
-int
-gf_cli_delete_volume_cbk (struct rpc_req *req, struct iovec *iov,
-                             int count, void *myframe)
-{
-        gf_cli_rsp              rsp   = {0,};
-        int                     ret   = -1;
-        cli_local_t             *local = NULL;
-        char                    *volname = NULL;
-        call_frame_t            *frame = NULL;
-        dict_t                  *dict = NULL;
-        dict_t                  *rsp_dict = NULL;
-
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
+static int
+gf_cli_delete_volume_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                         void *myframe)
+{
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    cli_local_t *local = NULL;
+    char *volname = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *rsp_dict = NULL;
 
-        frame = myframe;
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
 
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
+    GF_ASSERT(myframe);
+    frame = myframe;
 
-        local = frame->local;
+    GF_ASSERT(frame->local);
 
-        if (local)
-                dict = local->dict;
-        ret = dict_get_str (dict, "volname", &volname);
-        if (ret) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "dict get failed");
-                goto out;
-        }
+    local = frame->local;
 
-        gf_log ("cli", GF_LOG_INFO, "Received resp to delete volume");
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(frame->this->name, GF_LOG_ERROR, XDR_DECODE_FAIL);
+        goto out;
+    }
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                if (rsp.op_ret == 0) {
-                        rsp_dict = dict_new ();
-                        ret = dict_unserialize (rsp.dict.dict_val,
-                                                rsp.dict.dict_len,
-                                                &rsp_dict);
-                        if (ret) {
-                                gf_log ("cli", GF_LOG_ERROR,
-                                        "Failed rsp_dict unserialization");
-                                goto out;
-                        }
-                }
+    gf_log("cli", GF_LOG_INFO, "Received resp to delete volume");
 
-                ret = cli_xml_output_generic_volume ("volDelete", rsp_dict,
-                                                     rsp.op_ret, rsp.op_errno,
-                                                     rsp.op_errstr);
-                if (ret)
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        if (rsp.op_ret == 0) {
+            rsp_dict = dict_new();
+            ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len,
+                                   &rsp_dict);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR, DICT_UNSERIALIZE_FAIL);
                 goto out;
+            }
         }
 
-        if (rsp.op_ret && strcmp (rsp.op_errstr, ""))
-                cli_err ("volume delete: %s: failed: %s", volname,
-                         rsp.op_errstr);
-        else if (rsp.op_ret)
-                cli_err ("volume delete: %s: failed", volname);
-        else
-                cli_out ("volume delete: %s: success", volname);
+        ret = cli_xml_output_generic_volume("volDelete", rsp_dict, rsp.op_ret,
+                                            rsp.op_errno, rsp.op_errstr);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
 
-        ret = rsp.op_ret;
+    ret = dict_get_str_sizen(local->dict, "volname", &volname);
+    if (ret) {
+        gf_log(frame->this->name, GF_LOG_ERROR, "dict get failed");
+        goto out;
+    }
+
+    if (rsp.op_ret && strcmp(rsp.op_errstr, ""))
+        cli_err("volume delete: %s: failed: %s", volname, rsp.op_errstr);
+    else if (rsp.op_ret)
+        cli_err("volume delete: %s: failed", volname);
+    else
+        cli_out("volume delete: %s: success", volname);
+
+    ret = rsp.op_ret;
 
 out:
-        cli_cmd_broadcast_response (ret);
-        free (rsp.dict.dict_val);
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
 
-        gf_log ("", GF_LOG_INFO, "Returning with %d", ret);
-        return ret;
+    if (rsp_dict)
+        dict_unref(rsp_dict);
+    gf_log("", GF_LOG_DEBUG, RETURNING, ret);
+    return ret;
 }
 
-int
-gf_cli3_1_uuid_get_cbk (struct rpc_req *req, struct iovec *iov,
-                        int count, void *myframe)
+static int
+gf_cli3_1_uuid_get_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                       void *myframe)
 {
-        char                    *uuid_str = NULL;
-        gf_cli_rsp              rsp   = {0,};
-        int                     ret   = -1;
-        cli_local_t             *local = NULL;
-        call_frame_t            *frame = NULL;
-        dict_t                  *dict = NULL;
+    char *uuid_str = NULL;
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    cli_local_t *local = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *dict = NULL;
 
-        if (-1 == req->rpc_status)
-                goto out;
+    GF_ASSERT(myframe);
 
-        frame = myframe;
+    if (-1 == req->rpc_status)
+        goto out;
 
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
+    frame = myframe;
 
-        local = frame->local;
-        frame->local = NULL;
+    GF_ASSERT(frame->local);
 
-        gf_log ("cli", GF_LOG_INFO, "Received resp to uuid get");
+    local = frame->local;
 
-        dict = dict_new ();
-        if (!dict) {
-                ret = -1;
-                goto out;
-        }
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(frame->this->name, GF_LOG_ERROR, XDR_DECODE_FAIL);
+        goto out;
+    }
 
-        ret = dict_unserialize (rsp.dict.dict_val, rsp.dict.dict_len,
-                                &dict);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to unserialize "
-                        "response for uuid get");
-                goto out;
-        }
+    frame->local = NULL;
 
-        ret = dict_get_str (dict, "uuid", &uuid_str);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get uuid "
-                        "from dictionary");
-                goto out;
-        }
+    gf_log("cli", GF_LOG_INFO, "Received resp to uuid get");
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_dict ("uuidGenerate", dict, rsp.op_ret,
-                                           rsp.op_errno, rsp.op_errstr);
-                if (ret)
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
-                goto out;
-        }
+    dict = dict_new();
+    if (!dict) {
+        ret = -1;
+        goto out;
+    }
 
-        if (rsp.op_ret) {
-                if (strcmp (rsp.op_errstr, "") == 0)
-                        cli_err ("Get uuid was unsuccessful");
-                else
-                        cli_err ("%s", rsp.op_errstr);
+    ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &dict);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, DICT_UNSERIALIZE_FAIL);
+        goto out;
+    }
 
-        } else {
-                cli_out ("UUID: %s", uuid_str);
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_dict("uuidGenerate", dict, rsp.op_ret,
+                                  rsp.op_errno, rsp.op_errstr);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
 
+    if (rsp.op_ret) {
+        if (strcmp(rsp.op_errstr, "") == 0)
+            cli_err("Get uuid was unsuccessful");
+        else
+            cli_err("%s", rsp.op_errstr);
+
+    } else {
+        ret = dict_get_str_sizen(dict, "uuid", &uuid_str);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, "Failed to get uuid from dictionary");
+            goto out;
         }
-        ret = rsp.op_ret;
+        cli_out("UUID: %s", uuid_str);
+    }
+    ret = rsp.op_ret;
 
 out:
-        cli_cmd_broadcast_response (ret);
-        cli_local_wipe (local);
-        if (rsp.dict.dict_val)
-                free (rsp.dict.dict_val);
-        if (dict)
-                dict_unref (dict);
+    cli_cmd_broadcast_response(ret);
+    cli_local_wipe(local);
+    gf_free_xdr_cli_rsp(rsp);
 
-        gf_log ("", GF_LOG_DEBUG, "Returning with %d", ret);
-        return ret;
+    if (dict)
+        dict_unref(dict);
+
+    gf_log("", GF_LOG_DEBUG, RETURNING, ret);
+    return ret;
 }
 
-int
-gf_cli3_1_uuid_reset_cbk (struct rpc_req *req, struct iovec *iov,
-                             int count, void *myframe)
+static int
+gf_cli3_1_uuid_reset_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                         void *myframe)
 {
-        gf_cli_rsp              rsp   = {0,};
-        int                     ret   = -1;
-        cli_local_t             *local = NULL;
-        call_frame_t            *frame = NULL;
-        dict_t                  *dict = NULL;
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    cli_local_t *local = NULL;
+    call_frame_t *frame = NULL;
 
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
+    GF_ASSERT(myframe);
 
-        frame = myframe;
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
 
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
+    frame = myframe;
 
-        local = frame->local;
-        frame->local = NULL;
+    GF_ASSERT(frame->local);
 
-        gf_log ("cli", GF_LOG_INFO, "Received resp to uuid reset");
+    local = frame->local;
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_dict ("uuidReset", dict, rsp.op_ret,
-                                           rsp.op_errno, rsp.op_errstr);
-                if (ret)
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
-                goto out;
-        }
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(frame->this->name, GF_LOG_ERROR, XDR_DECODE_FAIL);
+        goto out;
+    }
 
-        if (rsp.op_ret && strcmp (rsp.op_errstr, ""))
-                cli_err ("%s", rsp.op_errstr);
-        else
-                cli_out ("resetting the peer uuid has been %s",
-                         (rsp.op_ret) ? "unsuccessful": "successful");
-        ret = rsp.op_ret;
+    frame->local = NULL;
+
+    gf_log("cli", GF_LOG_INFO, "Received resp to uuid reset");
+
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_dict("uuidReset", NULL, rsp.op_ret, rsp.op_errno,
+                                  rsp.op_errstr);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
+
+    if (rsp.op_ret && strcmp(rsp.op_errstr, ""))
+        cli_err("%s", rsp.op_errstr);
+    else
+        cli_out("resetting the peer uuid has been %s",
+                (rsp.op_ret) ? "unsuccessful" : "successful");
+    ret = rsp.op_ret;
 
 out:
-        cli_cmd_broadcast_response (ret);
-        cli_local_wipe (local);
-        if (rsp.dict.dict_val)
-                free (rsp.dict.dict_val);
-        if (dict)
-                dict_unref (dict);
+    cli_cmd_broadcast_response(ret);
+    cli_local_wipe(local);
+    gf_free_xdr_cli_rsp(rsp);
 
-        gf_log ("", GF_LOG_INFO, "Returning with %d", ret);
-        return ret;
+    gf_log("", GF_LOG_DEBUG, RETURNING, ret);
+    return ret;
 }
 
-int
-gf_cli_start_volume_cbk (struct rpc_req *req, struct iovec *iov,
-                             int count, void *myframe)
-{
-        gf_cli_rsp              rsp   = {0,};
-        int                     ret   = -1;
-        cli_local_t             *local = NULL;
-        char                    *volname = NULL;
-        call_frame_t            *frame = NULL;
-        dict_t                  *dict = NULL;
-        dict_t                  *rsp_dict = NULL;
-
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
+static int
+gf_cli_start_volume_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                        void *myframe)
+{
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    cli_local_t *local = NULL;
+    char *volname = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *rsp_dict = NULL;
 
-        frame = myframe;
+    GF_ASSERT(myframe);
 
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
 
-        if (frame)
-                local = frame->local;
+    frame = myframe;
 
-        if (local)
-                dict = local->dict;
+    GF_ASSERT(frame->local);
 
-        ret = dict_get_str (dict, "volname", &volname);
-        if (ret) {
-                gf_log (frame->this->name, GF_LOG_ERROR, "dict get failed");
-                goto out;
-        }
+    local = frame->local;
 
-        gf_log ("cli", GF_LOG_INFO, "Received resp to start volume");
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(frame->this->name, GF_LOG_ERROR, XDR_DECODE_FAIL);
+        goto out;
+    }
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                if (rsp.op_ret == 0) {
-                        rsp_dict = dict_new ();
-                        ret = dict_unserialize (rsp.dict.dict_val,
-                                                rsp.dict.dict_len,
-                                                &rsp_dict);
-                        if (ret) {
-                                gf_log ("cli", GF_LOG_ERROR,
-                                        "Failed rsp_dict unserialization");
-                                goto out;
-                        }
-                }
+    gf_log("cli", GF_LOG_INFO, "Received resp to start volume");
 
-                ret = cli_xml_output_generic_volume ("volStart", rsp_dict,
-                                                     rsp.op_ret, rsp.op_errno,
-                                                     rsp.op_errstr);
-                if (ret)
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        if (rsp.op_ret == 0) {
+            rsp_dict = dict_new();
+            ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len,
+                                   &rsp_dict);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR, DICT_UNSERIALIZE_FAIL);
                 goto out;
+            }
         }
 
-        if (rsp.op_ret && strcmp (rsp.op_errstr, ""))
-                cli_err ("volume start: %s: failed: %s", volname,
-                         rsp.op_errstr);
-        else if (rsp.op_ret)
-                cli_err ("volume start: %s: failed", volname);
-        else
-                cli_out ("volume start: %s: success", volname);
-
-        ret = rsp.op_ret;
+        ret = cli_xml_output_generic_volume("volStart", rsp_dict, rsp.op_ret,
+                                            rsp.op_errno, rsp.op_errstr);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
 
-out:
-        cli_cmd_broadcast_response (ret);
-        free (rsp.dict.dict_val);
-        free (rsp.op_errstr);
-        return ret;
-}
+    ret = dict_get_str_sizen(local->dict, "volname", &volname);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "dict get failed");
+        goto out;
+    }
 
-int
-gf_cli_stop_volume_cbk (struct rpc_req *req, struct iovec *iov,
-                             int count, void *myframe)
-{
-        gf_cli_rsp            rsp   = {0,};
-        int                   ret   = -1;
-        cli_local_t           *local = NULL;
-        char                  *volname = NULL;
-        call_frame_t          *frame = NULL;
-        dict_t                *dict = NULL;
-        dict_t                *rsp_dict = NULL;
-
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
+    if (rsp.op_ret && strcmp(rsp.op_errstr, ""))
+        cli_err("volume start: %s: failed: %s", volname, rsp.op_errstr);
+    else if (rsp.op_ret)
+        cli_err("volume start: %s: failed", volname);
+    else
+        cli_out("volume start: %s: success", volname);
 
-        frame = myframe;
+    ret = rsp.op_ret;
 
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
+out:
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
 
-        if (frame)
-                local = frame->local;
+    if (rsp_dict)
+        dict_unref(rsp_dict);
+    return ret;
+}
 
-        if (local) {
-                dict = local->dict;
-                ret = dict_get_str (dict, "volname", &volname);
-                if (ret) {
-                        gf_log (frame->this->name, GF_LOG_ERROR,
-                                "Unable to get volname from dict");
-                        goto out;
-                }
-        }
+static int
+gf_cli_stop_volume_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                       void *myframe)
+{
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    cli_local_t *local = NULL;
+    char *volname = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *rsp_dict = NULL;
 
-        gf_log ("cli", GF_LOG_INFO, "Received resp to stop volume");
+    GF_ASSERT(myframe);
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                if (rsp.op_ret == 0) {
-                        rsp_dict = dict_new ();
-                        ret = dict_unserialize (rsp.dict.dict_val,
-                                                rsp.dict.dict_len,
-                                                &rsp_dict);
-                        if (ret) {
-                                gf_log ("cli", GF_LOG_ERROR,
-                                        "Failed rsp_dict unserialization");
-                                goto out;
-                        }
-                }
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
 
-                ret = cli_xml_output_generic_volume ("volStop", rsp_dict,
-                                                     rsp.op_ret, rsp.op_errno,
-                                                     rsp.op_errstr);
-                if (ret)
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
-                goto out;
-        }
+    frame = myframe;
 
-        if (rsp.op_ret && strcmp (rsp.op_errstr, ""))
-                cli_err ("volume stop: %s: failed: %s", volname, rsp.op_errstr);
-        else if (rsp.op_ret)
-                cli_err ("volume stop: %s: failed", volname);
-        else
-                cli_out ("volume stop: %s: success", volname);
+    GF_ASSERT(frame->local);
 
-        ret = rsp.op_ret;
+    local = frame->local;
 
-out:
-        cli_cmd_broadcast_response (ret);
-        free (rsp.op_errstr);
-        free (rsp.dict.dict_val);
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(frame->this->name, GF_LOG_ERROR, XDR_DECODE_FAIL);
+        goto out;
+    }
 
-        return ret;
-}
+    gf_log("cli", GF_LOG_INFO, "Received resp to stop volume");
 
-int
-gf_cli_print_rebalance_status (dict_t *dict, enum gf_task_types task_type)
-{
-        int                ret          = -1;
-        int                count        = 0;
-        int                i            = 1;
-        char               key[256]     = {0,};
-        gf_defrag_status_t status_rcd   = GF_DEFRAG_STATUS_NOT_STARTED;
-        uint64_t           files        = 0;
-        uint64_t           size         = 0;
-        uint64_t           lookup       = 0;
-        char               *node_name   = NULL;
-        uint64_t           failures     = 0;
-        uint64_t           skipped      = 0;
-        double             elapsed      = 0;
-        char               *status_str  = NULL;
-        char               *size_str    = NULL;
-
-        ret = dict_get_int32 (dict, "count", &count);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "count not set");
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        if (rsp.op_ret == 0) {
+            rsp_dict = dict_new();
+            ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len,
+                                   &rsp_dict);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR, DICT_UNSERIALIZE_FAIL);
                 goto out;
+            }
         }
 
+        ret = cli_xml_output_generic_volume("volStop", rsp_dict, rsp.op_ret,
+                                            rsp.op_errno, rsp.op_errstr);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
+
+    ret = dict_get_str_sizen(local->dict, "volname", &volname);
+    if (ret) {
+        gf_log(frame->this->name, GF_LOG_ERROR,
+               "Unable to get volname from dict");
+        goto out;
+    }
+
+    if (rsp.op_ret && strcmp(rsp.op_errstr, ""))
+        cli_err("volume stop: %s: failed: %s", volname, rsp.op_errstr);
+    else if (rsp.op_ret)
+        cli_err("volume stop: %s: failed", volname);
+    else
+        cli_out("volume stop: %s: success", volname);
+
+    ret = rsp.op_ret;
 
-        cli_out ("%40s %16s %13s %13s %13s %13s %20s %18s", "Node",
-                 "Rebalanced-files", "size", "scanned", "failures", "skipped",
-                 "status", "run time in secs");
-        cli_out ("%40s %16s %13s %13s %13s %13s %20s %18s", "---------",
-                 "-----------", "-----------", "-----------", "-----------",
-                 "-----------", "------------", "--------------");
-        for (i = 1; i <= count; i++) {
-                /* Reset the variables to prevent carryover of values */
-                node_name = NULL;
-                files = 0;
-                size = 0;
-                lookup = 0;
-                skipped = 0;
-                status_str = NULL;
-                elapsed = 0;
-
-                /* Check if status is NOT_STARTED, and continue early */
-                memset (key, 0, 256);
-                snprintf (key, 256, "status-%d", i);
-                ret = dict_get_int32 (dict, key, (int32_t *)&status_rcd);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_TRACE, "failed to get status");
-                        goto out;
-                }
-                if (GF_DEFRAG_STATUS_NOT_STARTED == status_rcd)
-                        continue;
+out:
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
 
+    if (rsp_dict)
+        dict_unref(rsp_dict);
 
-                snprintf (key, 256, "node-name-%d", i);
-                ret = dict_get_str (dict, key, &node_name);
-                if (ret)
-                        gf_log ("cli", GF_LOG_TRACE, "failed to get node-name");
+    return ret;
+}
 
-                memset (key, 0, 256);
-                snprintf (key, 256, "files-%d", i);
-                ret = dict_get_uint64 (dict, key, &files);
-                if (ret)
-                        gf_log ("cli", GF_LOG_TRACE,
-                                "failed to get file count");
+static int
+gf_cli_print_rebalance_status(dict_t *dict, enum gf_task_types task_type)
+{
+    int ret = -1;
+    int count = 0;
+    int i = 1;
+    char key[64] = {
+        0,
+    };
+    int keylen;
+    gf_defrag_status_t status_rcd = GF_DEFRAG_STATUS_NOT_STARTED;
+    uint64_t files = 0;
+    uint64_t size = 0;
+    uint64_t lookup = 0;
+    char *node_name = NULL;
+    uint64_t failures = 0;
+    uint64_t skipped = 0;
+    double elapsed = 0;
+    char *status_str = NULL;
+    char *size_str = NULL;
+    int32_t hrs = 0;
+    uint32_t min = 0;
+    uint32_t sec = 0;
+    gf_boolean_t fix_layout = _gf_false;
+    uint64_t max_time = 0;
+    uint64_t max_elapsed = 0;
+    uint64_t time_left = 0;
+    gf_boolean_t show_estimates = _gf_false;
+
+    ret = dict_get_int32_sizen(dict, "count", &count);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "count not set");
+        goto out;
+    }
+
+    for (i = 1; i <= count; i++) {
+        keylen = snprintf(key, sizeof(key), "status-%d", i);
+        ret = dict_get_int32n(dict, key, keylen, (int32_t *)&status_rcd);
+        /* If information from a node is missing we should skip
+         * the node and try to fetch information of other nodes.
+         * If information is not found for all nodes, we should
+         * error out.
+         */
+        if (!ret)
+            break;
+        if (ret && i == count) {
+            gf_log("cli", GF_LOG_TRACE, "failed to get status");
+            goto out;
+        }
+    }
+
+    /* Fix layout will be sent to all nodes for the volume
+       so every status should be of type
+       GF_DEFRAG_STATUS_LAYOUT_FIX*
+    */
+
+    if ((task_type == GF_TASK_TYPE_REBALANCE) &&
+        (status_rcd >= GF_DEFRAG_STATUS_LAYOUT_FIX_STARTED)) {
+        fix_layout = _gf_true;
+    }
+
+    if (fix_layout) {
+        cli_out("%35s %41s %27s", "Node", "status", "run time in h:m:s");
+        cli_out("%35s %41s %27s", "---------", "-----------", "------------");
+    } else {
+        cli_out("%40s %16s %13s %13s %13s %13s %20s %18s", "Node",
+                "Rebalanced-files", "size", "scanned", "failures", "skipped",
+                "status",
+                "run time in"
+                " h:m:s");
+        cli_out("%40s %16s %13s %13s %13s %13s %20s %18s", "---------",
+                "-----------", "-----------", "-----------", "-----------",
+                "-----------", "------------", "--------------");
+    }
+
+    for (i = 1; i <= count; i++) {
+        /* Reset the variables to prevent carryover of values */
+        node_name = NULL;
+        files = 0;
+        size = 0;
+        lookup = 0;
+        skipped = 0;
+        status_str = NULL;
+        elapsed = 0;
+        time_left = 0;
+
+        /* Check if status is NOT_STARTED, and continue early */
+        keylen = snprintf(key, sizeof(key), "status-%d", i);
+
+        ret = dict_get_int32n(dict, key, keylen, (int32_t *)&status_rcd);
+        if (ret == -ENOENT) {
+            gf_log("cli", GF_LOG_TRACE, "count %d %d", count, i);
+            gf_log("cli", GF_LOG_TRACE, "failed to get status");
+            gf_log("cli", GF_LOG_ERROR, "node down and has failed to set dict");
+            continue;
+            /* skip this node if value not available*/
+        } else if (ret) {
+            gf_log("cli", GF_LOG_TRACE, "count %d %d", count, i);
+            gf_log("cli", GF_LOG_TRACE, "failed to get status");
+            continue;
+            /* skip this node if value not available*/
+        }
+
+        if (GF_DEFRAG_STATUS_NOT_STARTED == status_rcd)
+            continue;
+
+        if (GF_DEFRAG_STATUS_STARTED == status_rcd)
+            show_estimates = _gf_true;
+
+        keylen = snprintf(key, sizeof(key), "node-name-%d", i);
+        ret = dict_get_strn(dict, key, keylen, &node_name);
+        if (ret)
+            gf_log("cli", GF_LOG_TRACE, "failed to get node-name");
 
-                memset (key, 0, 256);
-                snprintf (key, 256, "size-%d", i);
-                ret = dict_get_uint64 (dict, key, &size);
-                if (ret)
-                        gf_log ("cli", GF_LOG_TRACE,
-                                "failed to get size of xfer");
+        snprintf(key, sizeof(key), "files-%d", i);
+        ret = dict_get_uint64(dict, key, &files);
+        if (ret)
+            gf_log("cli", GF_LOG_TRACE, "failed to get file count");
 
-                memset (key, 0, 256);
-                snprintf (key, 256, "lookups-%d", i);
-                ret = dict_get_uint64 (dict, key, &lookup);
-                if (ret)
-                        gf_log ("cli", GF_LOG_TRACE,
-                                "failed to get lookedup file count");
+        snprintf(key, sizeof(key), "size-%d", i);
+        ret = dict_get_uint64(dict, key, &size);
+        if (ret)
+            gf_log("cli", GF_LOG_TRACE, "failed to get size of xfer");
 
-                memset (key, 0, 256);
-                snprintf (key, 256, "failures-%d", i);
-                ret = dict_get_uint64 (dict, key, &failures);
-                if (ret)
-                        gf_log ("cli", GF_LOG_TRACE,
-                                "failed to get failures count");
+        snprintf(key, sizeof(key), "lookups-%d", i);
+        ret = dict_get_uint64(dict, key, &lookup);
+        if (ret)
+            gf_log("cli", GF_LOG_TRACE, "failed to get lookedup file count");
 
-                memset (key, 0, 256);
-                snprintf (key, 256, "skipped-%d", i);
-                ret = dict_get_uint64 (dict, key, &skipped);
-                if (ret)
-                        gf_log ("cli", GF_LOG_TRACE,
-                                "failed to get skipped count");
+        snprintf(key, sizeof(key), "failures-%d", i);
+        ret = dict_get_uint64(dict, key, &failures);
+        if (ret)
+            gf_log("cli", GF_LOG_TRACE, "failed to get failures count");
 
-                /* For remove-brick include skipped count into failure count*/
-                if (task_type != GF_TASK_TYPE_REBALANCE) {
-                        failures += skipped;
-                        skipped = 0;
-                }
+        snprintf(key, sizeof(key), "skipped-%d", i);
+        ret = dict_get_uint64(dict, key, &skipped);
+        if (ret)
+            gf_log("cli", GF_LOG_TRACE, "failed to get skipped count");
 
-                memset (key, 0, 256);
-                snprintf (key, 256, "run-time-%d", i);
-                ret = dict_get_double (dict, key, &elapsed);
-                if (ret)
-                        gf_log ("cli", GF_LOG_TRACE, "failed to get run-time");
-
-                /* Check for array bound */
-                if (status_rcd >= GF_DEFRAG_STATUS_MAX)
-                        status_rcd = GF_DEFRAG_STATUS_MAX;
-
-                status_str = cli_vol_task_status_str[status_rcd];
-                size_str = gf_uint64_2human_readable(size);
-                if (size_str) {
-                        cli_out ("%40s %16"PRIu64 " %13s" " %13"PRIu64 " %13"
-                                 PRIu64" %13"PRIu64 " %20s %18.2f", node_name,
-                                 files, size_str, lookup, failures, skipped,
-                                 status_str, elapsed);
-                } else {
-                        cli_out ("%40s %16"PRIu64 " %13"PRIu64 " %13"PRIu64
-                                 " %13"PRIu64" %13"PRIu64 " %20s %18.2f",
-                                 node_name, files, size, lookup, failures,
-                                 skipped, status_str, elapsed);
-                }
-                GF_FREE(size_str);
+        /* For remove-brick include skipped count into failure count*/
+        if (task_type != GF_TASK_TYPE_REBALANCE) {
+            failures += skipped;
+            skipped = 0;
         }
-out:
-        return ret;
-}
 
-int
-gf_cli_defrag_volume_cbk (struct rpc_req *req, struct iovec *iov,
-                             int count, void *myframe)
-{
-        gf_cli_rsp   rsp          = {0,};
-        cli_local_t  *local       = NULL;
-        char         *volname     = NULL;
-        call_frame_t *frame       = NULL;
-        int          cmd          = 0;
-        int          ret          = -1;
-        dict_t       *dict        = NULL;
-        dict_t       *local_dict  = NULL;
-        char         msg[1024]    = {0,};
-        char         *task_id_str = NULL;
-
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
+        snprintf(key, sizeof(key), "run-time-%d", i);
+        ret = dict_get_double(dict, key, &elapsed);
+        if (ret)
+            gf_log("cli", GF_LOG_TRACE, "failed to get run-time");
 
-        frame = myframe;
+        snprintf(key, sizeof(key), "time-left-%d", i);
+        ret = dict_get_uint64(dict, key, &time_left);
+        if (ret)
+            gf_log("cli", GF_LOG_TRACE, "failed to get time left");
 
-        ret = xdr_to_generic (*iov, &rsp,
-                              (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
+        if (elapsed > max_elapsed)
+            max_elapsed = elapsed;
 
-        if (frame)
-                local = frame->local;
+        if (time_left > max_time)
+            max_time = time_left;
 
-        if (local)
-                local_dict = local->dict;
+        /* Check for array bound */
+        if (status_rcd >= GF_DEFRAG_STATUS_MAX)
+            status_rcd = GF_DEFRAG_STATUS_MAX;
 
-        ret = dict_get_str (local_dict, "volname", &volname);
-        if (ret) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "Failed to get volname");
-                goto out;
-        }
+        status_str = cli_vol_task_status_str[status_rcd];
+        size_str = gf_uint64_2human_readable(size);
+        hrs = elapsed / 3600;
+        min = ((uint64_t)elapsed % 3600) / 60;
+        sec = ((uint64_t)elapsed % 3600) % 60;
 
-        ret = dict_get_int32 (local_dict, "rebalance-command", (int32_t*)&cmd);
-        if (ret) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "Failed to get command");
-                goto out;
+        if (fix_layout) {
+            cli_out("%35s %50s %8d:%d:%d", node_name, status_str, hrs, min,
+                    sec);
+        } else {
+            if (size_str) {
+                cli_out("%40s %16" PRIu64
+                        " %13s"
+                        " %13" PRIu64 " %13" PRIu64 " %13" PRIu64
+                        " %20s "
+                        "%8d:%02d:%02d",
+                        node_name, files, size_str, lookup, failures, skipped,
+                        status_str, hrs, min, sec);
+            } else {
+                cli_out("%40s %16" PRIu64 " %13" PRIu64 " %13" PRIu64
+                        " %13" PRIu64 " %13" PRIu64
+                        " %20s"
+                        " %8d:%02d:%02d",
+                        node_name, files, size, lookup, failures, skipped,
+                        status_str, hrs, min, sec);
+            }
+        }
+        GF_FREE(size_str);
+    }
+
+    /* Max time will be non-zero if rebalance is still running */
+    if (max_time) {
+        hrs = max_time / 3600;
+        min = (max_time % 3600) / 60;
+        sec = (max_time % 3600) % 60;
+
+        if (hrs < REBAL_ESTIMATE_SEC_UPPER_LIMIT) {
+            cli_out(
+                "Estimated time left for rebalance to "
+                "complete : %8d:%02d:%02d",
+                hrs, min, sec);
+        } else {
+            cli_out(
+                "Estimated time left for rebalance to "
+                "complete : > 2 months. Please try again "
+                "later.");
+        }
+    } else {
+        /* Rebalance will return 0 if it could not calculate the
+         * estimates or if it is complete.
+         */
+        if (!show_estimates) {
+            goto out;
+        }
+        if (max_elapsed <= REBAL_ESTIMATE_START_TIME) {
+            cli_out(
+                "The estimated time for rebalance to complete "
+                "will be unavailable for the first 10 "
+                "minutes.");
+        } else {
+            cli_out(
+                "Rebalance estimated time unavailable. Please "
+                "try again later.");
         }
+    }
+out:
+    return ret;
+}
 
-        if (rsp.dict.dict_len) {
-                /* Unserialize the dictionary */
-                dict  = dict_new ();
-
-                ret = dict_unserialize (rsp.dict.dict_val,
-                                        rsp.dict.dict_len,
-                                        &dict);
-                if (ret < 0) {
-                        gf_log ("glusterd", GF_LOG_ERROR,
-                                "failed to "
-                                "unserialize req-buffer to dictionary");
-                        goto out;
-                }
+static int
+gf_cli_defrag_volume_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                         void *myframe)
+{
+    gf_cli_rsp rsp = {
+        0,
+    };
+    cli_local_t *local = NULL;
+    char *volname = NULL;
+    call_frame_t *frame = NULL;
+    int cmd = 0;
+    int ret = -1;
+    dict_t *dict = NULL;
+    char msg[1024] = {
+        0,
+    };
+    char *task_id_str = NULL;
+
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
+
+    GF_ASSERT(myframe);
+
+    frame = myframe;
+
+    GF_ASSERT(frame->local);
+
+    local = frame->local;
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(frame->this->name, GF_LOG_ERROR, XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    ret = dict_get_str_sizen(local->dict, "volname", &volname);
+    if (ret) {
+        gf_log(frame->this->name, GF_LOG_ERROR, "Failed to get volname");
+        goto out;
+    }
+
+    ret = dict_get_int32_sizen(local->dict, "rebalance-command",
+                               (int32_t *)&cmd);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get command");
+        goto out;
+    }
+
+    if (rsp.dict.dict_len) {
+        /* Unserialize the dictionary */
+        dict = dict_new();
+
+        ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &dict);
+        if (ret < 0) {
+            gf_log("glusterd", GF_LOG_ERROR, DICT_UNSERIALIZE_FAIL);
+            goto out;
         }
+    }
 
-        if (!((cmd == GF_DEFRAG_CMD_STOP) || (cmd == GF_DEFRAG_CMD_STATUS)) &&
-             !(global_state->mode & GLUSTER_MODE_XML)) {
-                /* All other possibilites are about starting a rebalance */
-                ret = dict_get_str (dict, GF_REBALANCE_TID_KEY, &task_id_str);
-                if (rsp.op_ret && strcmp (rsp.op_errstr, "")) {
-                        snprintf (msg, sizeof (msg), "%s", rsp.op_errstr);
-                } else {
-                         if (!rsp.op_ret) {
-                                /* append errstr in the cli msg for successful
-                                 * case since unlock failures can be highlighted
-                                 * event though rebalance command was successful
-                                 */
-                                snprintf (msg, sizeof (msg),
-                                          "Rebalance on %s has been started "
-                                          "successfully. Use rebalance status "
-                                          "command to check status of the "
-                                          "rebalance process.\nID: %s\n%s",
-                                          volname, task_id_str, rsp.op_errstr);
-                         } else {
-                                snprintf (msg, sizeof (msg),
-                                          "Starting rebalance on volume %s has "
-                                          "been unsuccessful.", volname);
-                        }
-                }
-                goto done;
-        }
-
-        if (cmd == GF_DEFRAG_CMD_STOP) {
-                if (rsp.op_ret == -1) {
-                        if (strcmp (rsp.op_errstr, ""))
-                                snprintf (msg, sizeof (msg),
-                                          "%s", rsp.op_errstr);
-                        else
-                                snprintf (msg, sizeof (msg),
-                                          "rebalance volume %s stop failed",
-                                          volname);
-                        goto done;
-                } else {
-                        /* append errstr in the cli msg for successful case
-                         * since unlock failures can be highlighted event though
-                         * rebalance command was successful */
-                        snprintf (msg, sizeof (msg),
-                                  "rebalance process may be in the middle of a "
-                                  "file migration.\nThe process will be fully "
-                                  "stopped once the migration of the file is "
-                                  "complete.\nPlease check rebalance process "
-                                  "for completion before doing any further "
-                                  "brick related tasks on the volume.\n%s",
-                                  rsp.op_errstr);
-                }
+    if (!((cmd == GF_DEFRAG_CMD_STOP) || (cmd == GF_DEFRAG_CMD_STATUS)) &&
+        !(global_state->mode & GLUSTER_MODE_XML)) {
+        ret = dict_get_str_sizen(dict, GF_REBALANCE_TID_KEY, &task_id_str);
+        if (ret) {
+            gf_log("cli", GF_LOG_WARNING, "failed to get %s from dict",
+                   GF_REBALANCE_TID_KEY);
         }
-        if (cmd == GF_DEFRAG_CMD_STATUS) {
-                if (rsp.op_ret == -1) {
-                        if (strcmp (rsp.op_errstr, ""))
-                                snprintf (msg, sizeof (msg),
-                                          "%s", rsp.op_errstr);
-                        else
-                                snprintf (msg, sizeof (msg),
-                                          "Failed to get the status of "
-                                          "rebalance process");
-                        goto done;
-                } else {
-                        snprintf (msg, sizeof (msg), "%s", rsp.op_errstr);
-                }
+        if (rsp.op_ret && strcmp(rsp.op_errstr, "")) {
+            snprintf(msg, sizeof(msg), "%s", rsp.op_errstr);
+        } else {
+            if (!rsp.op_ret) {
+                /* append errstr in the cli msg for successful
+                 * case since unlock failures can be highlighted
+                 * event though rebalance command was successful
+                 */
+                snprintf(msg, sizeof(msg),
+                         "Rebalance on %s has been "
+                         "started successfully. Use "
+                         "rebalance status command to"
+                         " check status of the "
+                         "rebalance process.\nID: %s",
+                         volname, task_id_str);
+            } else {
+                snprintf(msg, sizeof(msg),
+                         "Starting rebalance on volume %s has "
+                         "been unsuccessful.",
+                         volname);
+            }
+        }
+        goto done;
+    }
+
+    if (cmd == GF_DEFRAG_CMD_STOP) {
+        if (rsp.op_ret == -1) {
+            if (strcmp(rsp.op_errstr, ""))
+                snprintf(msg, sizeof(msg), "%s", rsp.op_errstr);
+            else
+                snprintf(msg, sizeof(msg), "rebalance volume %s stop failed",
+                         volname);
+            goto done;
+        } else {
+            /* append errstr in the cli msg for successful case
+             * since unlock failures can be highlighted event though
+             * rebalance command was successful */
+            snprintf(msg, sizeof(msg),
+                     "rebalance process may be in the middle of a "
+                     "file migration.\nThe process will be fully "
+                     "stopped once the migration of the file is "
+                     "complete.\nPlease check rebalance process "
+                     "for completion before doing any further "
+                     "brick related tasks on the volume.\n%s",
+                     rsp.op_errstr);
+        }
+    }
+    if (cmd == GF_DEFRAG_CMD_STATUS) {
+        if (rsp.op_ret == -1) {
+            if (strcmp(rsp.op_errstr, ""))
+                snprintf(msg, sizeof(msg), "%s", rsp.op_errstr);
+            else
+                snprintf(msg, sizeof(msg),
+                         "Failed to get the status of rebalance process");
+            goto done;
+        } else {
+            snprintf(msg, sizeof(msg), "%s", rsp.op_errstr);
         }
+    }
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_vol_rebalance (cmd, dict, rsp.op_ret,
-                                                    rsp.op_errno,
-                                                    rsp.op_errstr);
-                goto out;
-        }
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_vol_rebalance(cmd, dict, rsp.op_ret, rsp.op_errno,
+                                           rsp.op_errstr);
+        goto out;
+    }
 
-        ret = gf_cli_print_rebalance_status (dict, GF_TASK_TYPE_REBALANCE);
-        if (ret)
-                gf_log ("cli", GF_LOG_ERROR,
-                        "Failed to print rebalance status");
+    ret = gf_cli_print_rebalance_status(dict, GF_TASK_TYPE_REBALANCE);
+    if (ret)
+        gf_log("cli", GF_LOG_ERROR, "Failed to print rebalance status");
 
 done:
-        if (global_state->mode & GLUSTER_MODE_XML)
-                cli_xml_output_str ("volRebalance", msg,
-                                    rsp.op_ret, rsp.op_errno,
-                                    rsp.op_errstr);
-        else {
-                if (rsp.op_ret)
-                        cli_err ("volume rebalance: %s: failed: %s", volname,
-                                 msg);
-                else
-                        cli_out ("volume rebalance: %s: success: %s", volname,
-                                 msg);
-        }
-        ret = rsp.op_ret;
+    if (global_state->mode & GLUSTER_MODE_XML)
+        cli_xml_output_str("volRebalance", msg, rsp.op_ret, rsp.op_errno,
+                           rsp.op_errstr);
+    else {
+        if (rsp.op_ret)
+            cli_err("volume rebalance: %s: failed%s%s", volname,
+                    strlen(msg) ? ": " : "", msg);
+        else
+            cli_out("volume rebalance: %s: success%s%s", volname,
+                    strlen(msg) ? ": " : "", msg);
+    }
+    ret = rsp.op_ret;
 
 out:
-        free (rsp.op_errstr); //malloced by xdr
-        free (rsp.dict.dict_val); //malloced by xdr
-        if (dict)
-                dict_unref (dict);
-        cli_cmd_broadcast_response (ret);
-        return ret;
+    gf_free_xdr_cli_rsp(rsp);
+    if (dict)
+        dict_unref(dict);
+    cli_cmd_broadcast_response(ret);
+    return ret;
 }
 
-int
-gf_cli_rename_volume_cbk (struct rpc_req *req, struct iovec *iov,
-                             int count, void *myframe)
+static int
+gf_cli_rename_volume_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                         void *myframe)
 {
-        gf_cli_rsp              rsp   = {0,};
-        int                     ret   = -1;
-        char                    msg[1024] = {0,};
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    char msg[1024] = {
+        0,
+    };
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    gf_log("cli", GF_LOG_INFO, "Received resp to probe");
+    snprintf(msg, sizeof(msg), "Rename volume %s",
+             (rsp.op_ret) ? "unsuccessful" : "successful");
+
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_str("volRename", msg, rsp.op_ret, rsp.op_errno,
+                                 rsp.op_errstr);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
 
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
-
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
+    if (rsp.op_ret)
+        cli_err("volume rename: failed");
+    else
+        cli_out("volume rename: success");
 
+    ret = rsp.op_ret;
 
-        gf_log ("cli", GF_LOG_INFO, "Received resp to probe");
-        snprintf (msg, sizeof (msg), "Rename volume %s",
-                  (rsp.op_ret) ? "unsuccessful": "successful");
+out:
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
+    return ret;
+}
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_str ("volRename", msg, rsp.op_ret,
-                                          rsp.op_errno, rsp.op_errstr);
-                if (ret)
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
-                goto out;
-        }
+static int
+gf_cli_reset_volume_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                        void *myframe)
+{
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    char msg[1024] = {
+        0,
+    };
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    gf_log("cli", GF_LOG_INFO, "Received resp to reset");
+
+    if (strcmp(rsp.op_errstr, ""))
+        snprintf(msg, sizeof(msg), "%s", rsp.op_errstr);
+    else
+        snprintf(msg, sizeof(msg), "reset volume %s",
+                 (rsp.op_ret) ? "unsuccessful" : "successful");
+
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_str("volReset", msg, rsp.op_ret, rsp.op_errno,
+                                 rsp.op_errstr);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
 
-        if (rsp.op_ret)
-                cli_err ("volume rename: failed");
-        else
-                cli_out ("volume rename: success");
+    if (rsp.op_ret)
+        cli_err("volume reset: failed: %s", msg);
+    else
+        cli_out("volume reset: success: %s", msg);
 
-        ret = rsp.op_ret;
+    ret = rsp.op_ret;
 
 out:
-        cli_cmd_broadcast_response (ret);
-        return ret;
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
+    return ret;
 }
 
-int
-gf_cli_reset_volume_cbk (struct rpc_req *req, struct iovec *iov,
-                             int count, void *myframe)
+static int
+gf_cli_ganesha_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                   void *myframe)
 {
-        gf_cli_rsp           rsp   = {0,};
-        int                  ret   = -1;
-        char                 msg[1024] = {0,};
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    dict_t *dict = NULL;
 
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
+    GF_ASSERT(myframe);
 
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
 
-        gf_log ("cli", GF_LOG_INFO, "Received resp to reset");
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
 
-        if (strcmp (rsp.op_errstr, ""))
-                snprintf (msg, sizeof (msg), "%s", rsp.op_errstr);
-        else
-                snprintf (msg, sizeof (msg), "reset volume %s",
-                          (rsp.op_ret) ? "unsuccessful": "successful");
+    gf_log("cli", GF_LOG_DEBUG, "Received resp to ganesha");
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_str ("volReset", msg, rsp.op_ret,
-                                          rsp.op_errno, rsp.op_errstr);
-                if (ret)
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
-                goto out;
-        }
+    dict = dict_new();
 
-        if (rsp.op_ret)
-                cli_err ("volume reset: failed: %s", msg);
+    if (!dict) {
+        ret = -1;
+        goto out;
+    }
+
+    ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &dict);
+    if (ret)
+        goto out;
+
+    if (rsp.op_ret) {
+        if (strcmp(rsp.op_errstr, ""))
+            cli_err("nfs-ganesha: failed: %s", rsp.op_errstr);
         else
-                cli_out ("volume reset: success: %s", msg);
+            cli_err("nfs-ganesha: failed");
+    }
 
-        ret = rsp.op_ret;
+    else {
+        cli_out("nfs-ganesha : success ");
+    }
+
+    ret = rsp.op_ret;
 
 out:
-        cli_cmd_broadcast_response (ret);
-        return ret;
+    if (dict)
+        dict_unref(dict);
+    cli_cmd_broadcast_response(ret);
+    return ret;
 }
 
-char *
-is_server_debug_xlator (void *myframe)
+static char *
+is_server_debug_xlator(void *myframe)
 {
-        call_frame_t         *frame        = NULL;
-        cli_local_t          *local        = NULL;
-        char                 **words       = NULL;
-        char                 *key          = NULL;
-        char                 *value        = NULL;
-        char                 *debug_xlator = NULL;
-
-        frame = myframe;
-        local = frame->local;
-        words = (char **)local->words;
-
-        while (*words != NULL) {
-                if (strstr (*words, "trace") == NULL &&
-                    strstr (*words, "error-gen") == NULL) {
-                        words++;
-                        continue;
-                }
-
-                key = *words;
+    call_frame_t *frame = NULL;
+    cli_local_t *local = NULL;
+    char **words = NULL;
+    char *key = NULL;
+    char *value = NULL;
+    char *debug_xlator = NULL;
+
+    frame = myframe;
+    local = frame->local;
+    words = (char **)local->words;
+
+    while (*words != NULL) {
+        if (strstr(*words, "trace") == NULL &&
+            strstr(*words, "error-gen") == NULL) {
+            words++;
+            continue;
+        }
+
+        key = *words;
+        words++;
+        value = *words;
+        if (value == NULL)
+            break;
+        if (strstr(value, "client")) {
+            words++;
+            continue;
+        } else {
+            if (!(strstr(value, "posix") || strstr(value, "acl") ||
+                  strstr(value, "locks") || strstr(value, "io-threads") ||
+                  strstr(value, "marker") || strstr(value, "index"))) {
                 words++;
-                value = *words;
-                if (value == NULL)
-                        break;
-                if (strstr (value, "client")) {
-                        words++;
-                        continue;
-                } else {
-                        if (!(strstr (value, "posix") || strstr (value, "acl")
-                              || strstr (value, "locks") ||
-                              strstr (value, "io-threads") ||
-                              strstr (value, "marker") ||
-                              strstr (value, "index"))) {
-                                words++;
-                                continue;
-                        } else {
-                                debug_xlator = gf_strdup (key);
-                                break;
-                        }
-                }
+                continue;
+            } else {
+                debug_xlator = gf_strdup(key);
+                break;
+            }
         }
+    }
 
-        return debug_xlator;
+    return debug_xlator;
 }
 
-int
-gf_cli_set_volume_cbk (struct rpc_req *req, struct iovec *iov,
-                             int count, void *myframe)
-{
-        gf_cli_rsp           rsp   = {0,};
-        int                  ret   = -1;
-        dict_t               *dict = NULL;
-        char                 *help_str = NULL;
-        char                 msg[1024] = {0,};
-        char                 *debug_xlator = NULL;
-        char                 tmp_str[512] = {0,};
-
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
-
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
-
-        gf_log ("cli", GF_LOG_INFO, "Received resp to set");
-
-        dict = dict_new ();
-
-        if (!dict) {
-                ret = -1;
-                goto out;
-        }
-
-        ret = dict_unserialize (rsp.dict.dict_val, rsp.dict.dict_len, &dict);
-
-        /* For brick processes graph change does not happen on the fly.
-         * The process has to be restarted. So this is a check from the
-         * volume set option such that if debug xlators such as trace/errorgen
-         * are provided in the set command, warn the user.
-         */
-        debug_xlator = is_server_debug_xlator (myframe);
-
-        if (dict_get_str (dict, "help-str", &help_str) && !msg[0])
-                snprintf (msg, sizeof (msg), "Set volume %s",
-                          (rsp.op_ret) ? "unsuccessful": "successful");
-        if (rsp.op_ret == 0 && debug_xlator) {
-                snprintf (tmp_str, sizeof (tmp_str), "\n%s translator has been "
-                          "added to the server volume file. Please restart the"
-                          " volume for enabling the translator", debug_xlator);
-        }
-
-        if ((global_state->mode & GLUSTER_MODE_XML) && (help_str == NULL)) {
-                ret = cli_xml_output_str ("volSet", msg, rsp.op_ret,
-                                          rsp.op_errno, rsp.op_errstr);
-                if (ret)
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
-                goto out;
-        }
+static int
+gf_cli_set_volume_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                      void *myframe)
+{
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    dict_t *dict = NULL;
+    char *help_str = NULL;
+    char msg[1024] = {
+        0,
+    };
+    char *debug_xlator = NULL;
+    char tmp_str[512] = {
+        0,
+    };
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    gf_log("cli", GF_LOG_INFO, "Received resp to set");
+
+    dict = dict_new();
+
+    if (!dict) {
+        ret = -1;
+        goto out;
+    }
+
+    ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &dict);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, DICT_UNSERIALIZE_FAIL);
+        goto out;
+    }
+
+    /* For brick processes graph change does not happen on the fly.
+     * The process has to be restarted. So this is a check from the
+     * volume set option such that if debug xlators such as trace/errorgen
+     * are provided in the set command, warn the user.
+     */
+    debug_xlator = is_server_debug_xlator(myframe);
+
+    if (dict_get_str_sizen(dict, "help-str", &help_str) && !msg[0])
+        snprintf(msg, sizeof(msg), "Set volume %s",
+                 (rsp.op_ret) ? "unsuccessful" : "successful");
+    if (rsp.op_ret == 0 && debug_xlator) {
+        snprintf(tmp_str, sizeof(tmp_str),
+                 "\n%s translator has been "
+                 "added to the server volume file. Please restart the"
+                 " volume for enabling the translator",
+                 debug_xlator);
+    }
+
+    if ((global_state->mode & GLUSTER_MODE_XML) && (help_str == NULL)) {
+        ret = cli_xml_output_str("volSet", msg, rsp.op_ret, rsp.op_errno,
+                                 rsp.op_errstr);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
 
-        if (rsp.op_ret) {
-                if (strcmp (rsp.op_errstr, ""))
-                        cli_err ("volume set: failed: %s", rsp.op_errstr);
-                else
-                        cli_err ("volume set: failed");
+    if (rsp.op_ret) {
+        if (strcmp(rsp.op_errstr, ""))
+            cli_err("volume set: failed: %s", rsp.op_errstr);
+        else
+            cli_err("volume set: failed");
+    } else {
+        if (help_str == NULL) {
+            if (debug_xlator == NULL)
+                cli_out("volume set: success");
+            else
+                cli_out("volume set: success%s", tmp_str);
         } else {
-                if (help_str == NULL) {
-                        if (debug_xlator == NULL)
-                                cli_out ("volume set: success");
-                        else
-                                cli_out ("volume set: success%s", tmp_str);
-                }else {
-                        cli_out ("%s", help_str);
-                }
+            cli_out("%s", help_str);
         }
+    }
 
-        ret = rsp.op_ret;
+    ret = rsp.op_ret;
 
 out:
-        if (dict)
-                dict_unref (dict);
-        GF_FREE (debug_xlator);
-        cli_cmd_broadcast_response (ret);
-        return ret;
+    if (dict)
+        dict_unref(dict);
+    GF_FREE(debug_xlator);
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
+    return ret;
 }
 
 int
-gf_cli_add_brick_cbk (struct rpc_req *req, struct iovec *iov,
-                             int count, void *myframe)
+gf_cli_add_brick_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                     void *myframe)
 {
-        gf_cli_rsp                  rsp   = {0,};
-        int                         ret   = -1;
-        char                        msg[1024] = {0,};
-
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    char msg[1024] = {
+        0,
+    };
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    gf_log("cli", GF_LOG_INFO, "Received resp to add brick");
+
+    if (rsp.op_ret && strcmp(rsp.op_errstr, ""))
+        snprintf(msg, sizeof(msg), "%s", rsp.op_errstr);
+    else
+        snprintf(msg, sizeof(msg), "Add Brick %s",
+                 (rsp.op_ret) ? "unsuccessful" : "successful");
+
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_str("volAddBrick", msg, rsp.op_ret, rsp.op_errno,
+                                 rsp.op_errstr);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
 
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
+    if (rsp.op_ret)
+        cli_err("volume add-brick: failed: %s", rsp.op_errstr);
+    else
+        cli_out("volume add-brick: success");
+    ret = rsp.op_ret;
 
+out:
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
+    return ret;
+}
 
-        gf_log ("cli", GF_LOG_INFO, "Received resp to add brick");
+static int
+gf_cli3_remove_brick_status_cbk(struct rpc_req *req, struct iovec *iov,
+                                int count, void *myframe)
+{
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    dict_t *dict = NULL;
+    char msg[1024] = {
+        0,
+    };
+    int32_t command = 0;
+    gf1_op_commands cmd = GF_OP_CMD_NONE;
+    cli_local_t *local = NULL;
+    call_frame_t *frame = NULL;
+    const char *cmd_str;
 
-        if (rsp.op_ret && strcmp (rsp.op_errstr, ""))
-                snprintf (msg, sizeof (msg), "%s", rsp.op_errstr);
-        else
-                snprintf (msg, sizeof (msg), "Add Brick %s",
-                          (rsp.op_ret) ? "unsuccessful": "successful");
+    GF_ASSERT(myframe);
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_str ("volAddBrick", msg, rsp.op_ret,
-                                          rsp.op_errno, rsp.op_errstr);
-                if (ret)
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
-                goto out;
-        }
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
 
-        if (rsp.op_ret)
-                cli_err ("volume add-brick: failed: %s", rsp.op_errstr);
-        else
-                cli_out ("volume add-brick: success");
-        ret = rsp.op_ret;
+    frame = myframe;
 
-out:
-        cli_cmd_broadcast_response (ret);
-        free (rsp.dict.dict_val);
-        free (rsp.op_errstr);
-        return ret;
-}
+    GF_ASSERT(frame->local);
 
-int
-gf_cli3_remove_brick_status_cbk (struct rpc_req *req, struct iovec *iov,
-                                 int count, void *myframe)
-{
-        gf_cli_rsp               rsp     = {0,};
-        int                      ret     = -1;
-        dict_t                  *dict    = NULL;
-        char                     msg[1024] = {0,};
-        int32_t                  command = 0;
-        gf1_op_commands          cmd = GF_OP_CMD_NONE;
-        cli_local_t             *local = NULL;
-        call_frame_t            *frame = NULL;
-        char                    *cmd_str = "unknown";
-
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
+    local = frame->local;
 
-        frame = myframe;
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(frame->this->name, GF_LOG_ERROR, XDR_DECODE_FAIL);
+        goto out;
+    }
 
-        ret = xdr_to_generic (*iov, &rsp,
-                              (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
+    ret = dict_get_int32_sizen(local->dict, "command", &command);
+    if (ret)
+        goto out;
 
-        if (frame)
-                local = frame->local;
-        ret = dict_get_int32 (local->dict, "command", &command);
-        if (ret)
-                goto out;
-        cmd = command;
+    cmd = command;
 
-        switch (cmd) {
+    switch (cmd) {
         case GF_OP_CMD_STOP:
-                cmd_str = "stop";
-                break;
+            cmd_str = "stop";
+            break;
         case GF_OP_CMD_STATUS:
-                cmd_str = "status";
-                break;
+            cmd_str = "status";
+            break;
         default:
-                break;
-        }
+            cmd_str = "unknown";
+            break;
+    }
+
+    ret = rsp.op_ret;
+    if (rsp.op_ret == -1) {
+        if (strcmp(rsp.op_errstr, ""))
+            snprintf(msg, sizeof(msg), "volume remove-brick %s: failed: %s",
+                     cmd_str, rsp.op_errstr);
+        else
+            snprintf(msg, sizeof(msg), "volume remove-brick %s: failed",
+                     cmd_str);
 
-        ret = rsp.op_ret;
-        if (rsp.op_ret == -1) {
-                if (strcmp (rsp.op_errstr, ""))
-                        snprintf (msg, sizeof (msg), "volume remove-brick %s: "
-                                  "failed: %s", cmd_str, rsp.op_errstr);
-                else
-                        snprintf (msg, sizeof (msg), "volume remove-brick %s: "
-                                  "failed", cmd_str);
+        if (global_state->mode & GLUSTER_MODE_XML)
+            goto xml_output;
 
-                if (global_state->mode & GLUSTER_MODE_XML)
-                        goto xml_output;
+        cli_err("%s", msg);
+        goto out;
+    }
 
-                cli_err ("%s", msg);
-                goto out;
-        }
-
-        if (rsp.dict.dict_len) {
-                /* Unserialize the dictionary */
-                dict  = dict_new ();
+    if (rsp.dict.dict_len) {
+        /* Unserialize the dictionary */
+        dict = dict_new();
 
-                ret = dict_unserialize (rsp.dict.dict_val,
-                                        rsp.dict.dict_len,
-                                        &dict);
-                if (ret < 0) {
-                        strncpy (msg, "failed to unserialize req-buffer to "
-                                 "dictionary", sizeof (msg));
+        ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &dict);
+        if (ret < 0) {
+            strncpy(msg, DICT_UNSERIALIZE_FAIL, sizeof(msg));
 
-                        if (global_state->mode & GLUSTER_MODE_XML) {
-                                rsp.op_ret = -1;
-                                goto xml_output;
-                        }
+            if (global_state->mode & GLUSTER_MODE_XML) {
+                rsp.op_ret = -1;
+                goto xml_output;
+            }
 
-                        gf_log ("cli", GF_LOG_ERROR, "%s", msg);
-                        goto out;
-                }
+            gf_log("cli", GF_LOG_ERROR, "%s", msg);
+            goto out;
         }
+    }
 
 xml_output:
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                if (strcmp (rsp.op_errstr, "")) {
-                        ret = cli_xml_output_vol_remove_brick (_gf_true, dict,
-                                                               rsp.op_ret,
-                                                               rsp.op_errno,
-                                                               rsp.op_errstr);
-                } else {
-                        ret = cli_xml_output_vol_remove_brick (_gf_true, dict,
-                                                               rsp.op_ret,
-                                                               rsp.op_errno,
-                                                               msg);
-                }
-                goto out;
-        }
-
-        ret = gf_cli_print_rebalance_status (dict, GF_TASK_TYPE_REMOVE_BRICK);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to print remove-brick "
-                        "rebalance status");
-                goto out;
-        }
-
-        if ((cmd == GF_OP_CMD_STOP) && (rsp.op_ret == 0)) {
-                cli_out ("'remove-brick' process may be in the middle of a "
-                         "file migration.\nThe process will be fully stopped "
-                         "once the migration of the file is complete.\nPlease "
-                         "check remove-brick process for completion before "
-                         "doing any further brick related tasks on the "
-                         "volume.");
-        }
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        if (strcmp(rsp.op_errstr, "")) {
+            ret = cli_xml_output_vol_remove_brick(_gf_true, dict, rsp.op_ret,
+                                                  rsp.op_errno, rsp.op_errstr,
+                                                  "volRemoveBrick");
+        } else {
+            ret = cli_xml_output_vol_remove_brick(_gf_true, dict, rsp.op_ret,
+                                                  rsp.op_errno, msg,
+                                                  "volRemoveBrick");
+        }
+        goto out;
+    }
+
+    ret = gf_cli_print_rebalance_status(dict, GF_TASK_TYPE_REMOVE_BRICK);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Failed to print remove-brick rebalance status");
+        goto out;
+    }
+
+    if ((cmd == GF_OP_CMD_STOP) && (rsp.op_ret == 0)) {
+        cli_out(
+            "'remove-brick' process may be in the middle of a "
+            "file migration.\nThe process will be fully stopped "
+            "once the migration of the file is complete.\nPlease "
+            "check remove-brick process for completion before "
+            "doing any further brick related tasks on the "
+            "volume.");
+    }
 
 out:
-        free (rsp.dict.dict_val); //malloced by xdr
-        if (dict)
-                dict_unref (dict);
-        cli_cmd_broadcast_response (ret);
-        return ret;
+    if (dict)
+        dict_unref(dict);
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
+    return ret;
 }
 
-
-int
-gf_cli_remove_brick_cbk (struct rpc_req *req, struct iovec *iov,
-                             int count, void *myframe)
-{
-        gf_cli_rsp                      rsp   = {0,};
-        int                             ret   = -1;
-        char                            msg[1024] = {0,};
-        gf1_op_commands                 cmd = GF_OP_CMD_NONE;
-        char                           *cmd_str = "unknown";
-        cli_local_t                    *local = NULL;
-        call_frame_t                   *frame = NULL;
-        char                           *task_id_str = NULL;
-        dict_t                         *rsp_dict = NULL;
-
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
-
-        frame = myframe;
-        local = frame->local;
-
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
-
-        ret = dict_get_int32 (local->dict, "command", (int32_t *)&cmd);
+static int
+gf_cli_remove_brick_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                        void *myframe)
+{
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    char msg[1024] = {
+        0,
+    };
+    gf1_op_commands cmd = GF_OP_CMD_NONE;
+    char *cmd_str = "unknown";
+    cli_local_t *local = NULL;
+    call_frame_t *frame = NULL;
+    char *task_id_str = NULL;
+    dict_t *rsp_dict = NULL;
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
+
+    frame = myframe;
+
+    GF_ASSERT(frame->local);
+
+    local = frame->local;
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(frame->this->name, GF_LOG_ERROR, XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    ret = dict_get_int32_sizen(local->dict, "command", (int32_t *)&cmd);
+    if (ret) {
+        gf_log("", GF_LOG_ERROR, "failed to get command");
+        goto out;
+    }
+
+    if (rsp.dict.dict_len) {
+        rsp_dict = dict_new();
+        if (!rsp_dict) {
+            ret = -1;
+            goto out;
+        }
+
+        ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &rsp_dict);
         if (ret) {
-                 gf_log ("", GF_LOG_ERROR, "failed to get command");
-                 goto out;
-        }
-
-        if (rsp.dict.dict_len) {
-                rsp_dict = dict_new ();
-                if (!rsp_dict) {
-                        ret = -1;
-                        goto out;
-                }
-
-                ret = dict_unserialize (rsp.dict.dict_val, rsp.dict.dict_len,
-                                        &rsp_dict);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Failed to unserialize rsp_dict");
-                        goto out;
-                }
+            gf_log("cli", GF_LOG_ERROR, DICT_UNSERIALIZE_FAIL);
+            goto out;
         }
+    }
 
-        switch (cmd) {
+    switch (cmd) {
+        case GF_OP_CMD_DETACH_START:
         case GF_OP_CMD_START:
-                cmd_str = "start";
-
-                ret = dict_get_str (rsp_dict, GF_REMOVE_BRICK_TID_KEY, &task_id_str);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "remove-brick-id is not present in dict");
-                }
-                break;
+            cmd_str = "start";
+
+            ret = dict_get_str_sizen(rsp_dict, GF_REMOVE_BRICK_TID_KEY,
+                                     &task_id_str);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "remove-brick-id is not present in dict");
+            }
+            break;
         case GF_OP_CMD_COMMIT:
-                cmd_str = "commit";
-                break;
+            cmd_str = "commit";
+            break;
         case GF_OP_CMD_COMMIT_FORCE:
-                cmd_str = "commit force";
-                break;
+            cmd_str = "commit force";
+            break;
         default:
-                cmd_str = "unknown";
-                break;
-        }
+            cmd_str = "unknown";
+            break;
+    }
+
+    gf_log("cli", GF_LOG_INFO, "Received resp to remove brick");
+
+    if (rsp.op_ret && strcmp(rsp.op_errstr, ""))
+        snprintf(msg, sizeof(msg), "%s", rsp.op_errstr);
+    else
+        snprintf(msg, sizeof(msg), "Remove Brick %s %s", cmd_str,
+                 (rsp.op_ret) ? "unsuccessful" : "successful");
+
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_vol_remove_brick(_gf_false, rsp_dict, rsp.op_ret,
+                                              rsp.op_errno, msg,
+                                              "volRemoveBrick");
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
+
+    if (rsp.op_ret) {
+        cli_err("volume remove-brick %s: failed: %s", cmd_str, msg);
+    } else {
+        cli_out("volume remove-brick %s: success", cmd_str);
+        if (GF_OP_CMD_START == cmd && task_id_str != NULL)
+            cli_out("ID: %s", task_id_str);
+        if (GF_OP_CMD_COMMIT == cmd)
+            cli_out(
+                "Check the removed bricks to ensure all files "
+                "are migrated.\nIf files with data are "
+                "found on the brick path, copy them via a "
+                "gluster mount point before re-purposing the "
+                "removed brick. ");
+    }
+
+    ret = rsp.op_ret;
 
-        gf_log ("cli", GF_LOG_INFO, "Received resp to remove brick");
+out:
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
 
-        if (rsp.op_ret && strcmp (rsp.op_errstr, ""))
-                snprintf (msg, sizeof (msg), "%s", rsp.op_errstr);
-        else
-                snprintf (msg, sizeof (msg), "Remove Brick %s %s", cmd_str,
-                          (rsp.op_ret) ? "unsuccessful": "successful");
+    if (rsp_dict)
+        dict_unref(rsp_dict);
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_vol_remove_brick (_gf_false, rsp_dict,
-                                                       rsp.op_ret, rsp.op_errno,
-                                                       msg);
-                if (ret)
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
-                goto out;
-        }
+    return ret;
+}
 
-        if (rsp.op_ret) {
-                cli_err ("volume remove-brick %s: failed: %s", cmd_str,
-                         msg);
-        } else {
-                cli_out ("volume remove-brick %s: success", cmd_str);
-                if (GF_OP_CMD_START == cmd && task_id_str != NULL)
-                        cli_out ("ID: %s", task_id_str);
-                if (GF_OP_CMD_COMMIT == cmd)
-                        cli_out ("Check the removed bricks to ensure all files "
-                                 "are migrated.\nIf files with data are "
-                                 "found on the brick path, copy them via a "
-                                 "gluster mount point before re-purposing the "
-                                 "removed brick. ");
-        }
+static int
+gf_cli_reset_brick_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                       void *myframe)
+{
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    cli_local_t *local = NULL;
+    call_frame_t *frame = NULL;
+    const char *rb_operation_str = NULL;
+    dict_t *rsp_dict = NULL;
+    char msg[1024] = {
+        0,
+    };
+    char *reset_op = NULL;
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
+
+    frame = myframe;
+
+    GF_ASSERT(frame->local);
+
+    local = frame->local;
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(frame->this->name, GF_LOG_ERROR, XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    ret = dict_get_str_sizen(local->dict, "operation", &reset_op);
+    if (ret) {
+        gf_log(frame->this->name, GF_LOG_ERROR, "dict_get on operation failed");
+        goto out;
+    }
+
+    if (strcmp(reset_op, "GF_RESET_OP_START") &&
+        strcmp(reset_op, "GF_RESET_OP_COMMIT") &&
+        strcmp(reset_op, "GF_RESET_OP_COMMIT_FORCE")) {
+        ret = -1;
+        goto out;
+    }
+
+    if (rsp.dict.dict_len) {
+        /* Unserialize the dictionary */
+        rsp_dict = dict_new();
+
+        ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &rsp_dict);
+        if (ret < 0) {
+            gf_log(frame->this->name, GF_LOG_ERROR, DICT_UNSERIALIZE_FAIL);
+            goto out;
+        }
+    }
+
+    if (rsp.op_ret && (strcmp(rsp.op_errstr, ""))) {
+        rb_operation_str = rsp.op_errstr;
+    } else {
+        if (!strcmp(reset_op, "GF_RESET_OP_START")) {
+            if (rsp.op_ret)
+                rb_operation_str = "reset-brick start operation failed";
+            else
+                rb_operation_str = "reset-brick start operation successful";
+        } else if (!strcmp(reset_op, "GF_RESET_OP_COMMIT")) {
+            if (rsp.op_ret)
+                rb_operation_str = "reset-brick commit operation failed";
+            else
+                rb_operation_str = "reset-brick commit operation successful";
+        } else if (!strcmp(reset_op, "GF_RESET_OP_COMMIT_FORCE")) {
+            if (rsp.op_ret)
+                rb_operation_str = "reset-brick commit force operation failed";
+            else
+                rb_operation_str =
+                    "reset-brick commit force operation successful";
+        }
+    }
+
+    gf_log("cli", GF_LOG_INFO, "Received resp to reset brick");
+    snprintf(msg, sizeof(msg), "%s",
+             rb_operation_str ? rb_operation_str : "Unknown operation");
+
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_vol_replace_brick(rsp_dict, rsp.op_ret,
+                                               rsp.op_errno, msg);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
 
-        ret = rsp.op_ret;
+    if (rsp.op_ret)
+        cli_err("volume reset-brick: failed: %s", msg);
+    else
+        cli_out("volume reset-brick: success: %s", msg);
+    ret = rsp.op_ret;
 
 out:
-        cli_cmd_broadcast_response (ret);
-        free (rsp.dict.dict_val);
-        free (rsp.op_errstr);
-
-        return ret;
-}
-
-
+    if (frame)
+        frame->local = NULL;
 
-int
-gf_cli_replace_brick_cbk (struct rpc_req *req, struct iovec *iov,
-                             int count, void *myframe)
-{
-        gf_cli_rsp                       rsp              = {0,};
-        int                              ret              = -1;
-        cli_local_t                     *local            = NULL;
-        call_frame_t                    *frame            = NULL;
-        dict_t                          *dict             = NULL;
-        char                            *src_brick        = NULL;
-        char                            *dst_brick        = NULL;
-        char                            *status_reply     = NULL;
-        gf1_cli_replace_op               replace_op       = 0;
-        char                            *rb_operation_str = NULL;
-        dict_t                          *rsp_dict         = NULL;
-        char                             msg[1024]        = {0,};
-        char                            *task_id_str      = NULL;
-
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
+    if (local)
+        cli_local_wipe(local);
 
-        frame = (call_frame_t *) myframe;
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
+    if (rsp_dict)
+        dict_unref(rsp_dict);
 
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    return ret;
+}
+static int
+gf_cli_replace_brick_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                         void *myframe)
+{
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    cli_local_t *local = NULL;
+    call_frame_t *frame = NULL;
+    const char *rb_operation_str = NULL;
+    dict_t *rsp_dict = NULL;
+    char msg[1024] = {
+        0,
+    };
+    char *replace_op = NULL;
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
+
+    frame = myframe;
+
+    GF_ASSERT(frame->local);
+
+    local = frame->local;
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(frame->this->name, GF_LOG_ERROR, XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    ret = dict_get_str_sizen(local->dict, "operation", &replace_op);
+    if (ret) {
+        gf_log(frame->this->name, GF_LOG_ERROR, "dict_get on operation failed");
+        goto out;
+    }
+
+    if (rsp.dict.dict_len) {
+        /* Unserialize the dictionary */
+        rsp_dict = dict_new();
+
+        ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &rsp_dict);
         if (ret < 0) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
-
-        local = frame->local;
-        GF_ASSERT (local);
-        dict = local->dict;
-
-        ret = dict_get_int32 (dict, "operation", (int32_t *)&replace_op);
-        if (ret) {
-                gf_log ("", GF_LOG_DEBUG,
-                        "dict_get on operation failed");
-                goto out;
+            gf_log(frame->this->name, GF_LOG_ERROR, DICT_UNSERIALIZE_FAIL);
+            goto out;
         }
+    }
 
-        if (rsp.dict.dict_len) {
-                /* Unserialize the dictionary */
-                rsp_dict  = dict_new ();
-
-                ret = dict_unserialize (rsp.dict.dict_val,
-                                rsp.dict.dict_len,
-                                &rsp_dict);
-                if (ret < 0) {
-                        gf_log ("glusterd", GF_LOG_ERROR,
-                                        "failed to "
-                                        "unserialize rsp buffer to dictionary");
-                        goto out;
-                }
-        }
-
-        switch (replace_op) {
-        case GF_REPLACE_OP_START:
-                if (rsp.op_ret) {
-                        rb_operation_str = gf_strdup ("replace-brick failed to"
-                                                      " start");
-                } else {
-                        ret = dict_get_str (rsp_dict, GF_REPLACE_BRICK_TID_KEY,
-                                            &task_id_str);
-                        if (ret) {
-                                gf_log ("cli", GF_LOG_ERROR, "Failed to get "
-                                        "\"replace-brick-id\" from dict");
-                                goto out;
-                        }
-                        ret = gf_asprintf (&rb_operation_str,
-                                           "replace-brick started successfully"
-                                           "\nID: %s", task_id_str);
-                        if (ret < 0)
-                                goto out;
-                }
-                break;
-
-        case GF_REPLACE_OP_STATUS:
-
-                if (rsp.op_ret || ret) {
-                        rb_operation_str = gf_strdup ("replace-brick status "
-                                                      "unknown");
-                } else {
-                        ret = dict_get_str (rsp_dict, "status-reply",
-                                            &status_reply);
-                        if (ret) {
-                                gf_log (frame->this->name, GF_LOG_ERROR, "failed to"
-                                        "get status");
-                                goto out;
-                        }
-
-                        rb_operation_str = gf_strdup (status_reply);
-                }
-
-                break;
-
-        case GF_REPLACE_OP_PAUSE:
-                if (rsp.op_ret)
-                        rb_operation_str = gf_strdup ("replace-brick pause "
-                                                      "failed");
-                else
-                        rb_operation_str = gf_strdup ("replace-brick paused "
-                                                      "successfully");
-                break;
-
-        case GF_REPLACE_OP_ABORT:
-                if (rsp.op_ret)
-                        rb_operation_str = gf_strdup ("replace-brick abort "
-                                                      "failed");
-                else
-                        rb_operation_str = gf_strdup ("replace-brick aborted "
-                                                      "successfully");
-                break;
-
-        case GF_REPLACE_OP_COMMIT:
-        case GF_REPLACE_OP_COMMIT_FORCE:
-                ret = dict_get_str (dict, "src-brick", &src_brick);
-                if (ret) {
-                        gf_log ("", GF_LOG_DEBUG,
-                                "dict_get on src-brick failed");
-                        goto out;
-                }
-
-                ret = dict_get_str (dict, "dst-brick", &dst_brick);
-                if (ret) {
-                        gf_log ("", GF_LOG_DEBUG,
-                                "dict_get on dst-brick failed");
-                        goto out;
-                }
-
+    if (!strcmp(replace_op, "GF_REPLACE_OP_COMMIT_FORCE")) {
+        if (rsp.op_ret || ret)
+            rb_operation_str = "replace-brick commit force operation failed";
+        else
+            rb_operation_str =
+                "replace-brick commit force operation successful";
+    } else {
+        gf_log(frame->this->name, GF_LOG_DEBUG, "Unknown operation");
+    }
+
+    if (rsp.op_ret && (strcmp(rsp.op_errstr, ""))) {
+        rb_operation_str = rsp.op_errstr;
+    }
+
+    gf_log("cli", GF_LOG_INFO, "Received resp to replace brick");
+    snprintf(msg, sizeof(msg), "%s",
+             rb_operation_str ? rb_operation_str : "Unknown operation");
+
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_vol_replace_brick(rsp_dict, rsp.op_ret,
+                                               rsp.op_errno, msg);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
 
-                if (rsp.op_ret || ret)
-                        rb_operation_str = gf_strdup ("replace-brick commit "
-                                                      "failed");
-                else
-                        rb_operation_str = gf_strdup ("replace-brick commit "
-                                                      "successful");
+    if (rsp.op_ret)
+        cli_err("volume replace-brick: failed: %s", msg);
+    else
+        cli_out("volume replace-brick: success: %s", msg);
+    ret = rsp.op_ret;
 
-                break;
+out:
+    if (frame)
+        frame->local = NULL;
 
-        default:
-                gf_log ("", GF_LOG_DEBUG,
-                        "Unknown operation");
-                break;
-        }
+    if (local)
+        cli_local_wipe(local);
 
-        if (rsp.op_ret && (strcmp (rsp.op_errstr, ""))) {
-                rb_operation_str = gf_strdup (rsp.op_errstr);
-        }
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
+    if (rsp_dict)
+        dict_unref(rsp_dict);
 
-        gf_log ("cli", GF_LOG_INFO, "Received resp to replace brick");
-        snprintf (msg, sizeof (msg), "%s",
-                  rb_operation_str ? rb_operation_str : "Unknown operation");
+    return ret;
+}
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_vol_replace_brick (replace_op, rsp_dict,
-                                                        rsp.op_ret,
-                                                        rsp.op_errno, msg);
-                if (ret)
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
-                goto out;
-        }
+static int
+gf_cli_log_rotate_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                      void *myframe)
+{
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    char msg[1024] = {
+        0,
+    };
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    gf_log("cli", GF_LOG_DEBUG, "Received resp to log rotate");
+
+    if (rsp.op_ret && strcmp(rsp.op_errstr, ""))
+        snprintf(msg, sizeof(msg), "%s", rsp.op_errstr);
+    else
+        snprintf(msg, sizeof(msg), "log rotate %s",
+                 (rsp.op_ret) ? "unsuccessful" : "successful");
+
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_str("volLogRotate", msg, rsp.op_ret, rsp.op_errno,
+                                 rsp.op_errstr);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
 
-        if (rsp.op_ret)
-                cli_err ("volume replace-brick: failed: %s", msg);
-        else
-                cli_out ("volume replace-brick: success: %s", msg);
-        ret = rsp.op_ret;
+    if (rsp.op_ret)
+        cli_err("volume log-rotate: failed: %s", msg);
+    else
+        cli_out("volume log-rotate: success");
+    ret = rsp.op_ret;
 
 out:
-        if (frame)
-                frame->local = NULL;
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
 
-        if (local) {
-                dict_unref (local->dict);
-                cli_local_wipe (local);
-        }
+    return ret;
+}
 
-        if (rb_operation_str)
-                GF_FREE (rb_operation_str);
+static int
+gf_cli_sync_volume_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                       void *myframe)
+{
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    char msg[1024] = {
+        0,
+    };
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    gf_log("cli", GF_LOG_DEBUG, "Received resp to sync");
+
+    if (rsp.op_ret && strcmp(rsp.op_errstr, ""))
+        snprintf(msg, sizeof(msg), "volume sync: failed: %s", rsp.op_errstr);
+    else
+        snprintf(msg, sizeof(msg), "volume sync: %s",
+                 (rsp.op_ret) ? "failed" : "success");
+
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_str("volSync", msg, rsp.op_ret, rsp.op_errno,
+                                 rsp.op_errstr);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
 
-        cli_cmd_broadcast_response (ret);
-        free (rsp.dict.dict_val);
-        if (rsp_dict)
-                dict_unref (rsp_dict);
+    if (rsp.op_ret)
+        cli_err("%s", msg);
+    else
+        cli_out("%s", msg);
+    ret = rsp.op_ret;
 
-        return ret;
+out:
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
+    return ret;
 }
 
-
 static int
-gf_cli_log_rotate_cbk (struct rpc_req *req, struct iovec *iov,
-                          int count, void *myframe)
+print_quota_list_usage_output(cli_local_t *local, char *path, int64_t avail,
+                              char *sl_str, quota_limits_t *limits,
+                              quota_meta_t *used_space, gf_boolean_t sl,
+                              gf_boolean_t hl, double sl_num,
+                              gf_boolean_t limit_set)
 {
-        gf_cli_rsp             rsp   = {0,};
-        int                    ret   = -1;
-        char                   msg[1024] = {0,};
-
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
-
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
+    int32_t ret = -1;
+    char *used_str = NULL;
+    char *avail_str = NULL;
+    char *hl_str = NULL;
+    char *sl_val = NULL;
+
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_quota_xml_output(local, path, limits->hl, sl_str, sl_num,
+                                   used_space->size, avail, sl ? "Yes" : "No",
+                                   hl ? "Yes" : "No", limit_set);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to output in xml format for quota list command");
         }
+        goto out;
+    }
 
-        gf_log ("cli", GF_LOG_DEBUG, "Received resp to log rotate");
+    used_str = gf_uint64_2human_readable(used_space->size);
 
-        if (rsp.op_ret && strcmp (rsp.op_errstr, ""))
-                snprintf (msg, sizeof (msg), "%s", rsp.op_errstr);
-        else
-                snprintf (msg, sizeof (msg), "log rotate %s",
-                          (rsp.op_ret) ? "unsuccessful": "successful");
+    if (limit_set) {
+        hl_str = gf_uint64_2human_readable(limits->hl);
+        sl_val = gf_uint64_2human_readable(sl_num);
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_str ("volLogRotate", msg, rsp.op_ret,
-                                          rsp.op_errno, rsp.op_errstr);
-                if (ret)
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
-                goto out;
+        if (!used_str) {
+            cli_out("%-40s %7s %7s(%s) %8" PRIu64 "%9" PRIu64
+                    ""
+                    "%15s %18s",
+                    path, hl_str, sl_str, sl_val, used_space->size, avail,
+                    sl ? "Yes" : "No", hl ? "Yes" : "No");
+        } else {
+            avail_str = gf_uint64_2human_readable(avail);
+            cli_out("%-40s %7s %7s(%s) %8s %7s %15s %20s", path, hl_str, sl_str,
+                    sl_val, used_str, avail_str, sl ? "Yes" : "No",
+                    hl ? "Yes" : "No");
         }
+    } else {
+        cli_out("%-36s %10s %10s %14s %9s %15s %18s", path, "N/A", "N/A",
+                used_str, "N/A", "N/A", "N/A");
+    }
 
-        if (rsp.op_ret)
-                cli_err ("volume log-rotate: failed: %s", msg);
-        else
-                cli_out ("volume log-rotate: success");
-        ret = rsp.op_ret;
-
+    ret = 0;
 out:
-        cli_cmd_broadcast_response (ret);
-        free (rsp.dict.dict_val);
+    GF_FREE(hl_str);
+    GF_FREE(used_str);
+    GF_FREE(avail_str);
+    GF_FREE(sl_val);
 
-        return ret;
+    return ret;
 }
 
 static int
-gf_cli_sync_volume_cbk (struct rpc_req *req, struct iovec *iov,
-                           int count, void *myframe)
+print_quota_list_object_output(cli_local_t *local, char *path, int64_t avail,
+                               char *sl_str, quota_limits_t *limits,
+                               quota_meta_t *used_space, gf_boolean_t sl,
+                               gf_boolean_t hl, double sl_num,
+                               gf_boolean_t limit_set)
 {
-        gf_cli_rsp                     rsp   = {0,};
-        int                            ret   = -1;
-        char                           msg[1024] = {0,};
+    int32_t ret = -1;
+    int64_t sl_val = sl_num;
 
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
-
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_quota_object_xml_output(local, path, sl_str, sl_val, limits,
+                                          used_space, avail, sl ? "Yes" : "No",
+                                          hl ? "Yes" : "No", limit_set);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to output in xml format for quota list command");
+        }
+        goto out;
+    }
+
+    if (limit_set) {
+        cli_out("%-40s %9" PRIu64 " %9s(%" PRId64 ") %10" PRIu64
+                ""
+                "%10" PRIu64 " %11" PRIu64 " %15s %20s",
+                path, limits->hl, sl_str, sl_val, used_space->file_count,
+                used_space->dir_count, avail, sl ? "Yes" : "No",
+                hl ? "Yes" : "No");
+    } else {
+        cli_out("%-40s %9s %9s %10" PRIu64 " %10" PRIu64 " %11s %15s %20s",
+                path, "N/A", "N/A", used_space->file_count,
+                used_space->dir_count, "N/A", "N/A", "N/A");
+    }
+    ret = 0;
 
-        gf_log ("cli", GF_LOG_DEBUG, "Received resp to sync");
+out:
 
-        if (rsp.op_ret && strcmp (rsp.op_errstr, ""))
-                snprintf (msg, sizeof (msg), "volume sync: failed: %s",
-                          rsp.op_errstr);
-        else
-                snprintf (msg, sizeof (msg), "volume sync: %s",
-                          (rsp.op_ret) ? "failed": "success");
+    return ret;
+}
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_str ("volSync", msg, rsp.op_ret,
-                                          rsp.op_errno, rsp.op_errstr);
-                if (ret)
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
+static int
+print_quota_list_output(cli_local_t *local, char *path, char *default_sl,
+                        quota_limits_t *limits, quota_meta_t *used_space,
+                        int type, gf_boolean_t limit_set)
+{
+    int64_t avail = 0;
+    char percent_str[20] = {0};
+    char *sl_final = NULL;
+    int ret = -1;
+    double sl_num = 0;
+    gf_boolean_t sl = _gf_false;
+    gf_boolean_t hl = _gf_false;
+    int64_t used_size = 0;
+
+    GF_ASSERT(local);
+    GF_ASSERT(path);
+
+    if (limit_set) {
+        if (limits->sl < 0) {
+            ret = gf_string2percent(default_sl, &sl_num);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "could not convert default soft limit to percent");
+                goto out;
+            }
+            sl_num = (sl_num * limits->hl) / 100;
+            sl_final = default_sl;
+        } else {
+            sl_num = (limits->sl * limits->hl) / 100;
+            ret = snprintf(percent_str, sizeof(percent_str), "%" PRIu64 "%%",
+                           limits->sl);
+            if (ret < 0)
                 goto out;
+            sl_final = percent_str;
         }
-
-        if (rsp.op_ret)
-                cli_err ("%s", msg);
+        if (type == GF_QUOTA_OPTION_TYPE_LIST)
+            used_size = used_space->size;
         else
-                cli_out ("%s", msg);
-        ret = rsp.op_ret;
-
+            used_size = used_space->file_count + used_space->dir_count;
+
+        if (limits->hl > used_size) {
+            avail = limits->hl - used_size;
+            hl = _gf_false;
+            if (used_size > sl_num)
+                sl = _gf_true;
+            else
+                sl = _gf_false;
+        } else {
+            avail = 0;
+            hl = sl = _gf_true;
+        }
+    }
+
+    if (type == GF_QUOTA_OPTION_TYPE_LIST)
+        ret = print_quota_list_usage_output(local, path, avail, sl_final,
+                                            limits, used_space, sl, hl, sl_num,
+                                            limit_set);
+    else
+        ret = print_quota_list_object_output(local, path, avail, sl_final,
+                                             limits, used_space, sl, hl, sl_num,
+                                             limit_set);
 out:
-        cli_cmd_broadcast_response (ret);
-        return ret;
+    return ret;
 }
 
 static int
-print_quota_list_output (char *mountdir, char *default_sl, char *path)
-{
-        int64_t used_space       = 0;
-        int64_t avail            = 0;
-        char    *used_str         = NULL;
-        char    *avail_str        = NULL;
-        int     ret               = -1;
-        char    *sl_final         = NULL;
-        char    *hl_str           = NULL;
-        double  sl_num           = 0;
-        gf_boolean_t sl          = _gf_false;
-        gf_boolean_t hl          = _gf_false;
-        char percent_str[20]     = {0};
-
-        struct quota_limit {
-                int64_t hl;
-                int64_t sl;
-        } __attribute__ ((__packed__)) existing_limits;
-
-        ret = sys_lgetxattr (mountdir, "trusted.glusterfs.quota.limit-set",
-                             (void *)&existing_limits,
-                             sizeof (existing_limits));
-        if (ret < 0) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get the xattr "
-                        "trusted.glusterfs.quota.limit-set on %s. Reason : %s",
-                        mountdir, strerror (errno));
-                switch (errno) {
+print_quota_list_from_mountdir(cli_local_t *local, char *mountdir,
+                               char *default_sl, char *path, int type)
+{
+    int ret = -1;
+    ssize_t xattr_size = 0;
+    quota_limits_t limits = {
+        0,
+    };
+    quota_meta_t used_space = {
+        0,
+    };
+    char *key = NULL;
+    gf_boolean_t limit_set = _gf_true;
+
+    GF_ASSERT(local);
+    GF_ASSERT(mountdir);
+    GF_ASSERT(path);
+
+    if (type == GF_QUOTA_OPTION_TYPE_LIST)
+        key = QUOTA_LIMIT_KEY;
+    else
+        key = QUOTA_LIMIT_OBJECTS_KEY;
+
+    ret = sys_lgetxattr(mountdir, key, (void *)&limits, sizeof(limits));
+    if (ret < 0) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Failed to get the xattr %s on %s. Reason : %s", key, mountdir,
+               strerror(errno));
+
+        switch (errno) {
 #if defined(ENODATA)
-                case ENODATA:
+            case ENODATA:
 #endif
 #if defined(ENOATTR) && (ENOATTR != ENODATA)
-                case ENOATTR:
+            case ENOATTR:
 #endif
-                        cli_err ("%-40s %s", path, "Limit not set");
-                        break;
-                default:
-                        cli_err ("%-40s %s", path, strerror (errno));
-                        break;
-                }
-
-                goto out;
-        }
-
-        existing_limits.hl = ntoh64 (existing_limits.hl);
-        existing_limits.sl = ntoh64 (existing_limits.sl);
+                /* If it's an ENOATTR, quota/inode-quota is
+                 * configured(limit is set at least for one directory).
+                 * The user is trying to issue 'list/list-objects'
+                 * command for a directory on which quota limit is
+                 * not set and we are showing the used-space in case
+                 * of list-usage and showing (dir_count, file_count)
+                 * in case of list-objects. Other labels are
+                 * shown "N/A".
+                 */
 
-        hl_str = gf_uint64_2human_readable (existing_limits.hl);
+                limit_set = _gf_false;
+                goto enoattr;
+                break;
 
-        if (existing_limits.sl < 0) {
-                ret = gf_string2percent (default_sl, &sl_num);
-                sl_num = (sl_num * existing_limits.hl) / 100;
-                sl_final = default_sl;
-        } else {
-                sl_num = (existing_limits.sl * existing_limits.hl) / 100;
-                snprintf (percent_str, sizeof (percent_str), "%"PRIu64"%%",
-                          existing_limits.sl);
-                sl_final = percent_str;
+            default:
+                cli_err("%-40s %s", path, strerror(errno));
+                break;
         }
 
-        ret = sys_lgetxattr (mountdir, "trusted.glusterfs.quota.size",
-                             &used_space, sizeof (used_space));
+        goto out;
+    }
 
-        if (ret < 0) {
-                cli_out ("%-40s %7s %9s %11s %7s %15s %20s",
-                         path, hl_str, sl_final,
-                         "N/A", "N/A", "N/A", "N/A");
-        } else {
-                used_space = ntoh64 (used_space);
-
-                used_str = gf_uint64_2human_readable (used_space);
-
-                if (existing_limits.hl > used_space) {
-                        avail = existing_limits.hl - used_space;
-                        hl = _gf_false;
-                        if (used_space > sl_num)
-                                sl = _gf_true;
-                        else
-                                sl = _gf_false;
-                } else {
-                        avail = 0;
-                        hl = sl = _gf_true;
-                }
+    limits.hl = ntoh64(limits.hl);
+    limits.sl = ntoh64(limits.sl);
 
-                avail_str = gf_uint64_2human_readable (avail);
-                if (used_str == NULL) {
-                        cli_out ("%-40s %7s %9s %11"PRIu64
-                                 "%9"PRIu64" %15s %18s", path, hl_str,
-                                  sl_final, used_space, avail, sl? "Yes" : "No",
-                                  hl? "Yes" : "No");
-                } else {
-                        cli_out ("%-40s %7s %9s %11s %7s %15s %20s", path, hl_str,
-                                 sl_final, used_str, avail_str, sl? "Yes" : "No",
-                                 hl? "Yes" : "No");
-                }
-        }
+enoattr:
+    xattr_size = sys_lgetxattr(mountdir, QUOTA_SIZE_KEY, NULL, 0);
+    if (xattr_size < (sizeof(int64_t) * 2) &&
+        type == GF_QUOTA_OPTION_TYPE_LIST_OBJECTS) {
+        ret = -1;
 
+        /* This can happen when glusterfs is upgraded from 3.6 to 3.7
+         * and the xattr healing is not completed.
+         */
+    } else if (xattr_size > (sizeof(int64_t) * 2)) {
+        ret = sys_lgetxattr(mountdir, QUOTA_SIZE_KEY, &used_space,
+                            sizeof(used_space));
+    } else if (xattr_size > 0) {
+        /* This is for compatibility.
+         * Older version had only file usage
+         */
+        ret = sys_lgetxattr(mountdir, QUOTA_SIZE_KEY, &(used_space.size),
+                            sizeof(used_space.size));
+        used_space.file_count = 0;
+        used_space.dir_count = 0;
+    } else {
+        ret = -1;
+    }
+
+    if (ret < 0) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get quota size on path %s: %s",
+               mountdir, strerror(errno));
+        print_quota_list_empty(path, type);
+        goto out;
+    }
+
+    used_space.size = ntoh64(used_space.size);
+    used_space.file_count = ntoh64(used_space.file_count);
+    used_space.dir_count = ntoh64(used_space.dir_count);
+
+    ret = print_quota_list_output(local, path, default_sl, &limits, &used_space,
+                                  type, limit_set);
 out:
-        GF_FREE (used_str);
-        GF_FREE (avail_str);
-        GF_FREE (hl_str);
-        return ret;
+    return ret;
 }
 
-int
-gf_cli_print_limit_list_from_dict (char *volname, dict_t *dict,
-                                   char *default_sl, int count, char *op_errstr)
+static int
+gluster_remove_auxiliary_mount(char *volname)
 {
-        int  ret               = -1;
-        int  i                 = 0;
-        char key[1024]         = {0,};
-        char mountdir[PATH_MAX] = {0,};
-        char *path              = NULL;
+    int ret = -1;
+    char mountdir[PATH_MAX] = {
+        0,
+    };
+    xlator_t *this = NULL;
 
-        if (!dict|| count <= 0)
-                goto out;
+    this = THIS;
+    GF_ASSERT(this);
 
-        /* Need to check if any quota limits are set on the volume before trying
-         * to list them
-         */
-        if (!_limits_set_on_volume (volname)) {
-                ret = 0;
-                cli_out ("quota: No quota configured on volume %s", volname);
-                goto out;
-        }
+    GLUSTERD_GET_QUOTA_LIST_MOUNT_PATH(mountdir, volname, "/");
+    ret = gf_umount_lazy(this->name, mountdir, 1);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "umount on %s failed, reason : %s",
+               mountdir, strerror(errno));
+    }
 
-        /* Check if the mount is online before doing any listing */
-        if (!_quota_aux_mount_online (volname)) {
-                ret = -1;
-                goto out;
-        }
+    return ret;
+}
 
-        cli_out ("                  Path                   Hard-limit "
-                 "Soft-limit   Used  Available  Soft-limit exceeded?"
-                 "  Hard-limit exceeded?");
-        cli_out ("--------------------------------------------------------"
-                 "--------------------------------------------------------"
-                 "-----------");
+static int
+gf_cli_print_limit_list_from_dict(cli_local_t *local, char *volname,
+                                  dict_t *dict, char *default_sl, int count,
+                                  int op_ret, int op_errno, char *op_errstr)
+{
+    int ret = -1;
+    int i = 0;
+    char key[32] = {
+        0,
+    };
+    int keylen;
+    char mountdir[PATH_MAX] = {
+        0,
+    };
+    char *path = NULL;
+    int type = -1;
+
+    if (!dict || count <= 0)
+        goto out;
+
+    ret = dict_get_int32_sizen(dict, "type", &type);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get quota type");
+        goto out;
+    }
+
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_vol_quota_limit_list_begin(local, op_ret, op_errno,
+                                                        op_errstr);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+            goto out;
+        }
+    } else {
+        print_quota_list_header(type);
+    }
 
-        while (count--) {
-                snprintf (key, sizeof (key), "path%d", i++);
+    while (count--) {
+        keylen = snprintf(key, sizeof(key), "path%d", i++);
 
-                ret = dict_get_str (dict, key, &path);
-                if (ret < 0) {
-                        gf_log ("cli", GF_LOG_DEBUG, "Path not present in limit"
-                                " list");
-                        continue;
-                }
+        ret = dict_get_strn(dict, key, keylen, &path);
+        if (ret < 0) {
+            gf_log("cli", GF_LOG_DEBUG, "Path not present in limit list");
+            continue;
+        }
 
-                ret = gf_canonicalize_path (path);
-                if (ret)
-                        goto out;
-                GLUSTERD_GET_QUOTA_AUX_MOUNT_PATH (mountdir, volname, path);
-                ret = print_quota_list_output (mountdir, default_sl, path);
+        ret = gf_canonicalize_path(path);
+        if (ret)
+            goto out;
+        GLUSTERD_GET_QUOTA_LIST_MOUNT_PATH(mountdir, volname, path);
+        ret = print_quota_list_from_mountdir(local, mountdir, default_sl, path,
+                                             type);
+    }
 
-        }
 out:
-        return ret;
+    return ret;
 }
 
-int
-print_quota_list_from_quotad (call_frame_t *frame, dict_t *rsp_dict)
-{
-        int64_t used_space    = 0;
-        int64_t avail         = 0;
-        int64_t *limit         = NULL;
-        char    *used_str      = NULL;
-        char    *avail_str     = NULL;
-        char    *hl_str        = NULL;
-        char    *sl_final      = NULL;
-        char    *path          = NULL;
-        char    *default_sl    = NULL;
-        int     ret            = -1;
-        cli_local_t *local     = NULL;
-        dict_t *gd_rsp_dict    = NULL;
-        double sl_num          = 0;
-        gf_boolean_t sl        = _gf_false;
-        gf_boolean_t hl        = _gf_false;
-        char  percent_str[20]  = {0,};
-
-        local = frame->local;
-        gd_rsp_dict = local->dict;
-
-        struct quota_limit {
-                int64_t hl;
-                int64_t sl;
-        } __attribute__ ((__packed__)) *existing_limits = NULL;
-
-        ret = dict_get_str (rsp_dict, GET_ANCESTRY_PATH_KEY, &path);
-        if (ret) {
-                gf_log ("cli", GF_LOG_WARNING, "path key is not present "
-                        "in dict");
-                goto out;
-        }
-
-        ret = dict_get_bin (rsp_dict, QUOTA_LIMIT_KEY, (void**)&limit);
+static int
+print_quota_list_from_quotad(call_frame_t *frame, dict_t *rsp_dict)
+{
+    char *path = NULL;
+    char *default_sl = NULL;
+    int ret = -1;
+    cli_local_t *local = NULL;
+    dict_t *gd_rsp_dict = NULL;
+    quota_meta_t used_space = {
+        0,
+    };
+    quota_limits_t limits = {
+        0,
+    };
+    quota_limits_t *size_limits = NULL;
+    int32_t type = 0;
+    int32_t success_count = 0;
+
+    GF_ASSERT(frame);
+
+    local = frame->local;
+    gd_rsp_dict = local->dict;
+
+    ret = dict_get_int32_sizen(rsp_dict, "type", &type);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get type");
+        goto out;
+    }
+
+    ret = dict_get_str_sizen(rsp_dict, GET_ANCESTRY_PATH_KEY, &path);
+    if (ret) {
+        gf_log("cli", GF_LOG_WARNING, "path key is not present in dict");
+        goto out;
+    }
+
+    ret = dict_get_str_sizen(gd_rsp_dict, "default-soft-limit", &default_sl);
+    if (ret) {
+        gf_log(frame->this->name, GF_LOG_ERROR,
+               "failed to get default soft limit");
+        goto out;
+    }
+
+    if (type == GF_QUOTA_OPTION_TYPE_LIST) {
+        ret = dict_get_bin(rsp_dict, QUOTA_LIMIT_KEY, (void **)&size_limits);
         if (ret) {
-                gf_log ("cli", GF_LOG_WARNING,
-                        "limit key not present in dict");
-                goto out;
+            gf_log("cli", GF_LOG_WARNING, "limit key not present in dict on %s",
+                   path);
+            goto out;
         }
-
-        ret = dict_get_str (gd_rsp_dict, "default-soft-limit", &default_sl);
+    } else {
+        ret = dict_get_bin(rsp_dict, QUOTA_LIMIT_OBJECTS_KEY,
+                           (void **)&size_limits);
         if (ret) {
-                gf_log (frame->this->name, GF_LOG_ERROR, "failed to "
-                        "get default soft limit");
+            gf_log("cli", GF_LOG_WARNING,
+                   "object limit key not present in dict on %s", path);
+            goto out;
+        }
+    }
+
+    limits.hl = ntoh64(size_limits->hl);
+    limits.sl = ntoh64(size_limits->sl);
+
+    if (type == GF_QUOTA_OPTION_TYPE_LIST)
+        ret = quota_dict_get_meta(rsp_dict, QUOTA_SIZE_KEY,
+                                  SLEN(QUOTA_SIZE_KEY), &used_space);
+    else
+        ret = quota_dict_get_inode_meta(rsp_dict, QUOTA_SIZE_KEY,
+                                        SLEN(QUOTA_SIZE_KEY), &used_space);
+
+    if (ret < 0) {
+        gf_log("cli", GF_LOG_WARNING, "size key not present in dict");
+        print_quota_list_empty(path, type);
+        goto out;
+    }
+
+    LOCK(&local->lock);
+    {
+        ret = dict_get_int32_sizen(gd_rsp_dict, "quota-list-success-count",
+                                   &success_count);
+        if (ret)
+            success_count = 0;
+
+        ret = dict_set_int32_sizen(gd_rsp_dict, "quota-list-success-count",
+                                   success_count + 1);
+    }
+    UNLOCK(&local->lock);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Failed to set quota-list-success-count in dict");
+        goto out;
+    }
+
+    if (success_count == 0) {
+        if (!(global_state->mode & GLUSTER_MODE_XML)) {
+            print_quota_list_header(type);
+        } else {
+            ret = cli_xml_output_vol_quota_limit_list_begin(local, 0, 0, NULL);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR, XML_ERROR);
                 goto out;
+            }
         }
-        existing_limits = (struct quota_limit *)limit;
-        existing_limits->hl = ntoh64 (existing_limits->hl);
-        existing_limits->sl = ntoh64 (existing_limits->sl);
+    }
 
-        hl_str = gf_uint64_2human_readable (existing_limits->hl);
+    ret = print_quota_list_output(local, path, default_sl, &limits, &used_space,
+                                  type, _gf_true);
+out:
+    return ret;
+}
 
-        if (existing_limits->sl < 0) {
-                ret = gf_string2percent (default_sl, &sl_num);
-                sl_num = (sl_num * existing_limits->hl) / 100;
-                sl_final = default_sl;
-        } else {
-                sl_num = (existing_limits->sl * existing_limits->hl) / 100;
-                snprintf (percent_str, sizeof (percent_str), "%"PRIu64"%%",
-                          existing_limits->sl);
-                sl_final =  percent_str;
-        }
+static void *
+cli_cmd_broadcast_response_detached(void *opaque)
+{
+    int32_t ret = 0;
 
-        ret = dict_get_bin (rsp_dict, QUOTA_SIZE_KEY, (void**)&limit);
-        if (ret < 0) {
-                gf_log ("cli", GF_LOG_WARNING,
-                        "size key not present in dict");
-                cli_out ("%-40s %7s %9s %11s %7s %15s %20s", path, hl_str,
-                         sl_final, "N/A", "N/A", "N/A", "N/A");
-        } else {
-                used_space = *limit;
-                used_space = ntoh64 (used_space);
-                used_str = gf_uint64_2human_readable (used_space);
-
-                if (existing_limits->hl > used_space) {
-                        avail = existing_limits->hl - used_space;
-                        hl = _gf_false;
-                        if (used_space > sl_num)
-                                sl = _gf_true;
-                        else
-                                sl = _gf_false;
-                } else {
-                        avail = 0;
-                        hl = sl = _gf_true;
-                }
-                avail_str = gf_uint64_2human_readable (avail);
-                if (used_str == NULL)
-                        cli_out ("%-40s %7s %9s %11"PRIu64
-                                 "%9"PRIu64" %15s %20s", path, hl_str,
-                                 sl_final, used_space, avail, sl? "Yes" : "No",
-                                 hl? "Yes" : "No");
-                else
-                        cli_out ("%-40s %7s %9s %11s %7s %15s %20s", path,
-                                 hl_str, sl_final, used_str, avail_str,
-                                 sl? "Yes" : "No", hl? "Yes" : "No");
-        }
+    ret = (intptr_t)opaque;
+    cli_cmd_broadcast_response(ret);
 
-        ret = 0;
-out:
-        GF_FREE (used_str);
-        GF_FREE (avail_str);
-        GF_FREE (hl_str);
-        return ret;
+    return NULL;
 }
 
-int
-cli_quotad_getlimit_cbk (struct rpc_req *req, struct iovec *iov,
-                          int count, void *myframe)
+static int32_t
+cli_quota_compare_path(struct list_head *list1, struct list_head *list2)
 {
-    //TODO: we need to gather the path, hard-limit, soft-limit and used space
-        gf_cli_rsp         rsp         = {0,};
-        int                ret         = -1;
-        dict_t            *dict        = NULL;
-        call_frame_t      *frame       = NULL;
+    struct list_node *node1 = NULL;
+    struct list_node *node2 = NULL;
+    dict_t *dict1 = NULL;
+    dict_t *dict2 = NULL;
+    char *path1 = NULL;
+    char *path2 = NULL;
+    int ret = 0;
 
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
+    node1 = list_entry(list1, struct list_node, list);
+    node2 = list_entry(list2, struct list_node, list);
+
+    dict1 = node1->ptr;
+    dict2 = node2->ptr;
+
+    ret = dict_get_str_sizen(dict1, GET_ANCESTRY_PATH_KEY, &path1);
+    if (ret < 0)
+        return 0;
+
+    ret = dict_get_str_sizen(dict2, GET_ANCESTRY_PATH_KEY, &path2);
+    if (ret < 0)
+        return 0;
+
+    return strcmp(path1, path2);
+}
 
-        frame = myframe;
+static int
+cli_quotad_getlimit_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                        void *myframe)
+{
+    /*TODO: we need to gather the path, hard-limit, soft-limit and used space*/
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    dict_t *dict = NULL;
+    struct list_node *node = NULL;
+    struct list_node *tmpnode = NULL;
+    call_frame_t *frame = NULL;
+    cli_local_t *local = NULL;
+    int32_t list_count = 0;
+    pthread_t th_id = {
+        0,
+    };
+    int32_t max_count = 0;
+
+    GF_ASSERT(myframe);
+
+    frame = myframe;
+
+    GF_ASSERT(frame->local);
+
+    local = frame->local;
+
+    LOCK(&local->lock);
+    {
+        ret = dict_get_int32_sizen(local->dict, "quota-list-count",
+                                   &list_count);
+        if (ret)
+            list_count = 0;
+
+        list_count++;
+        ret = dict_set_int32_sizen(local->dict, "quota-list-count", list_count);
+    }
+    UNLOCK(&local->lock);
+
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to set quota-list-count in dict");
+        goto out;
+    }
+
+    if (-1 == req->rpc_status) {
+        if (list_count == 0)
+            cli_err(
+                "Connection failed. Please check if quota "
+                "daemon is operational.");
+        ret = -1;
+        goto out;
+    }
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(frame->this->name, GF_LOG_ERROR, XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    if (rsp.op_ret) {
+        ret = -1;
+        if (strcmp(rsp.op_errstr, ""))
+            cli_err("quota command failed : %s", rsp.op_errstr);
+        else
+            cli_err("quota command : failed");
+        goto out;
+    }
 
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (rsp.dict.dict_len) {
+        /* Unserialize the dictionary */
+        dict = dict_new();
+
+        ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &dict);
         if (ret < 0) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
+            gf_log("cli", GF_LOG_ERROR, DICT_UNSERIALIZE_FAIL);
+            goto out;
         }
 
-        if (rsp.op_ret) {
-                ret = -1;
-                if (strcmp (rsp.op_errstr, ""))
-                        cli_err ("quota command failed : %s", rsp.op_errstr);
-                else
-                        cli_err ("quota command : failed");
-                goto out;
+        node = list_node_add_order(dict, &local->dict_list,
+                                   cli_quota_compare_path);
+        if (node == NULL) {
+            gf_log("cli", GF_LOG_ERROR, "failed to add node to the list");
+            dict_unref(dict);
+            ret = -1;
+            goto out;
         }
+    }
 
-        if (rsp.dict.dict_len) {
-                /* Unserialize the dictionary */
-                dict  = dict_new ();
+    ret = dict_get_int32_sizen(local->dict, "max_count", &max_count);
+    if (ret < 0) {
+        gf_log("cli", GF_LOG_ERROR, "failed to get max_count");
+        goto out;
+    }
 
-                ret = dict_unserialize (rsp.dict.dict_val,
-                                        rsp.dict.dict_len,
-                                        &dict);
-                if (ret < 0) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "failed to "
-                                "unserialize req-buffer to dictionary");
-                        goto out;
-                }
-                print_quota_list_from_quotad (frame, dict);
+    if (list_count == max_count) {
+        list_for_each_entry_safe(node, tmpnode, &local->dict_list, list)
+        {
+            dict = node->ptr;
+            print_quota_list_from_quotad(frame, dict);
+            list_node_del(node);
+            dict_unref(dict);
         }
+    }
 
 out:
-        cli_cmd_broadcast_response (ret);
-        if (dict)
-                dict_unref (dict);
+    /* Bad Fix: CLI holds the lock to process a command.
+     * When processing quota list command, below sequence of steps executed
+     * in the same thread and causing deadlock
+     *
+     * 1) CLI holds the lock
+     * 2) Send rpc_clnt_submit request to quotad for quota usage
+     * 3) If quotad is down, rpc_clnt_submit invokes cbk function with error
+     * 4) cbk function cli_quotad_getlimit_cbk invokes
+     *    cli_cmd_broadcast_response which tries to hold lock to broadcast
+     *    the results and hangs, because same thread has already holding
+     *    the lock
+     *
+     * Broadcasting response in a separate thread which is not a
+     * good fix. This needs to be re-visted with better solution
+     */
+    if (ret == -1) {
+        ret = pthread_create(&th_id, NULL, cli_cmd_broadcast_response_detached,
+                             (void *)-1);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, "pthread_create failed: %s",
+                   strerror(errno));
+    } else {
+        cli_cmd_broadcast_response(ret);
+    }
+    gf_free_xdr_cli_rsp(rsp);
 
-        free (rsp.dict.dict_val);
-        return ret;
+    return ret;
 }
 
-int
-cli_quotad_getlimit (call_frame_t *frame, xlator_t *this, void *data)
+static int
+cli_quotad_getlimit(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf_cli_req          req = {{0,}};
-        int                 ret = 0;
-        dict_t             *dict = NULL;
+    gf_cli_req req = {{
+        0,
+    }};
+    int ret = 0;
+    dict_t *dict = NULL;
 
-        if (!frame || !this ||  !data) {
-                ret = -1;
-                goto out;
-        }
+    if (!frame || !this || !data) {
+        ret = -1;
+        goto out;
+    }
 
-        dict = data;
-        ret = dict_allocate_and_serialize (dict, &req.dict.dict_val,
-                                           &req.dict.dict_len);
-        if (ret < 0) {
-                gf_log (this->name, GF_LOG_ERROR,
-                        "failed to serialize the data");
+    dict = data;
+    ret = add_cli_cmd_timeout_to_dict(dict);
 
-                goto out;
-        }
+    ret = dict_allocate_and_serialize(dict, &req.dict.dict_val,
+                                      &req.dict.dict_len);
+    if (ret < 0) {
+        gf_log(this->name, GF_LOG_ERROR, DICT_SERIALIZE_FAIL);
 
-        ret = cli_cmd_submit (global_quotad_rpc, &req, frame, &cli_quotad_clnt,
-                              GF_AGGREGATOR_GETLIMIT, NULL,
-                              this, cli_quotad_getlimit_cbk,
-                              (xdrproc_t) xdr_gf_cli_req);
-
-out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+        goto out;
+    }
 
+    ret = cli_cmd_submit(global_quotad_rpc, &req, frame, &cli_quotad_clnt,
+                         GF_AGGREGATOR_GETLIMIT, NULL, this,
+                         cli_quotad_getlimit_cbk, (xdrproc_t)xdr_gf_cli_req);
 
+out:
+    GF_FREE(req.dict.dict_val);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
+    return ret;
 }
 
-void
-gf_cli_quota_list (char *volname, dict_t *dict, int count, char *op_errstr,
-                   char *default_sl)
+static void
+gf_cli_quota_list(cli_local_t *local, char *volname, dict_t *dict,
+                  char *default_sl, int count, int op_ret, int op_errno,
+                  char *op_errstr)
 {
-        GF_VALIDATE_OR_GOTO ("cli", volname, out);
+    if (!cli_cmd_connected())
+        goto out;
 
-        if (!connected)
-                goto out;
+    if (count > 0) {
+        GF_VALIDATE_OR_GOTO("cli", volname, out);
 
-        if (count > 0)
-                gf_cli_print_limit_list_from_dict (volname, dict, default_sl,
-                                                   count, op_errstr);
+        gf_cli_print_limit_list_from_dict(local, volname, dict, default_sl,
+                                          count, op_ret, op_errno, op_errstr);
+    }
 out:
-        return;
+    return;
 }
 
-int
-gf_cli_quota_cbk (struct rpc_req *req, struct iovec *iov,
-                     int count, void *myframe)
-{
-        gf_cli_rsp         rsp         = {0,};
-        int                ret         = -1;
-        dict_t            *dict        = NULL;
-        char              *volname     = NULL;
-        int32_t            type        = 0;
-        call_frame_t      *frame       = NULL;
-        char              *default_sl  = NULL;
-        char              *limit_list  = NULL;
-        cli_local_t       *local       = NULL;
-        dict_t            *aggr        = NULL;
-        char              *default_sl_dup  = NULL;
-        int32_t            entry_count      = 0;
-        if (-1 == req->rpc_status) {
-                goto out;
+static int
+gf_cli_quota_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                 void *myframe)
+{
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    dict_t *dict = NULL;
+    char *volname = NULL;
+    int32_t type = 0;
+    call_frame_t *frame = NULL;
+    char *default_sl = NULL;
+    cli_local_t *local = NULL;
+    char *default_sl_dup = NULL;
+    int32_t entry_count = 0;
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
+
+    frame = myframe;
+
+    GF_ASSERT(frame->local);
+
+    local = frame->local;
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(frame->this->name, GF_LOG_ERROR, XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    if (rsp.op_ret) {
+        ret = -1;
+        if (global_state->mode & GLUSTER_MODE_XML)
+            goto xml_output;
+
+        if (strcmp(rsp.op_errstr, "")) {
+            cli_err("quota command failed : %s", rsp.op_errstr);
+            if (rsp.op_ret == -ENOENT)
+                cli_err("please enter the path relative to the volume");
+        } else {
+            cli_err("quota command : failed");
         }
 
-        frame = myframe;
-        local = frame->local;
-        aggr  = local->dict;
+        goto out;
+    }
+
+    if (rsp.dict.dict_len) {
+        /* Unserialize the dictionary */
+        dict = dict_new();
 
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+        ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &dict);
         if (ret < 0) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
+            gf_log("cli", GF_LOG_ERROR, DICT_UNSERIALIZE_FAIL);
+            goto out;
         }
+    }
 
-        if (rsp.op_ret) {
-                ret = -1;
-                if (global_state->mode & GLUSTER_MODE_XML)
-                        goto xml_output;
+    gf_log("cli", GF_LOG_DEBUG, "Received resp to quota command");
 
-                if (strcmp (rsp.op_errstr, ""))
-                        cli_err ("quota command failed : %s", rsp.op_errstr);
-                else
-                        cli_err ("quota command : failed");
+    ret = dict_get_str_sizen(dict, "default-soft-limit", &default_sl);
+    if (ret)
+        gf_log(frame->this->name, GF_LOG_TRACE,
+               "failed to get default soft limit");
 
-                goto out;
+    // default-soft-limit is part of rsp_dict only iff we sent
+    // GLUSTER_CLI_QUOTA with type being GF_QUOTA_OPTION_TYPE_LIST
+    if (default_sl) {
+        default_sl_dup = gf_strdup(default_sl);
+        if (!default_sl_dup) {
+            ret = -1;
+            goto out;
         }
-
-        if (rsp.dict.dict_len) {
-                /* Unserialize the dictionary */
-                dict  = dict_new ();
-
-                ret = dict_unserialize (rsp.dict.dict_val,
-                                        rsp.dict.dict_len,
-                                        &dict);
-                if (ret < 0) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "failed to "
-                                "unserialize req-buffer to dictionary");
-                        goto out;
-                }
+        ret = dict_set_dynstr_sizen(local->dict, "default-soft-limit",
+                                    default_sl_dup);
+        if (ret) {
+            gf_log(frame->this->name, GF_LOG_TRACE,
+                   "failed to set default soft limit");
+            GF_FREE(default_sl_dup);
         }
+    }
 
-        gf_log ("cli", GF_LOG_DEBUG, "Received resp to quota command");
+    ret = dict_get_str_sizen(dict, "volname", &volname);
+    if (ret)
+        gf_log(frame->this->name, GF_LOG_ERROR, "failed to get volname");
 
-        ret = dict_get_str (dict, "volname", &volname);
-        if (ret)
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "failed to get volname");
+    ret = dict_get_int32_sizen(dict, "type", &type);
+    if (ret)
+        gf_log(frame->this->name, GF_LOG_TRACE, "failed to get type");
 
-        ret = dict_get_str (dict, "default-soft-limit", &default_sl);
-        if (ret)
-                gf_log (frame->this->name, GF_LOG_TRACE, "failed to get "
-                        "default soft limit");
+    ret = dict_get_int32_sizen(dict, "count", &entry_count);
+    if (ret)
+        gf_log(frame->this->name, GF_LOG_TRACE, "failed to get count");
 
-        // default-soft-limit is part of rsp_dict only iff we sent
-        // GLUSTER_CLI_QUOTA with type being GF_QUOTA_OPTION_TYPE_LIST
-        if (default_sl) {
-                default_sl_dup = gf_strdup (default_sl);
-                if (!default_sl_dup) {
-                        ret = -1;
-                        goto out;
-                }
-                ret = dict_set_dynstr (aggr, "default-soft-limit",
-                                       default_sl_dup);
-                if (ret) {
-                        gf_log (frame->this->name, GF_LOG_TRACE,
-                                "failed to set default soft limit");
-                        GF_FREE (default_sl_dup);
-                }
+    if ((type == GF_QUOTA_OPTION_TYPE_LIST) ||
+        (type == GF_QUOTA_OPTION_TYPE_LIST_OBJECTS)) {
+        gf_cli_quota_list(local, volname, dict, default_sl, entry_count,
+                          rsp.op_ret, rsp.op_errno, rsp.op_errstr);
+
+        if (global_state->mode & GLUSTER_MODE_XML) {
+            ret = cli_xml_output_vol_quota_limit_list_end(local);
+            if (ret < 0) {
+                ret = -1;
+                gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+            }
+            goto out;
         }
+    }
 
-        ret = dict_get_int32 (dict, "type", &type);
-        if (ret)
-                gf_log (frame->this->name, GF_LOG_TRACE,
-                        "failed to get type");
+xml_output:
 
-        ret = dict_get_int32 (dict, "count", &entry_count);
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_str("volQuota", NULL, rsp.op_ret, rsp.op_errno,
+                                 rsp.op_errstr);
         if (ret)
-                gf_log (frame->this->name, GF_LOG_TRACE, "failed to get count");
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
 
-        if (type == GF_QUOTA_OPTION_TYPE_LIST) {
-                if (global_state->mode & GLUSTER_MODE_XML) {
-                        ret = cli_xml_output_vol_quota_limit_list
-                                (volname, limit_list, rsp.op_ret,
-                                 rsp.op_errno, rsp.op_errstr);
-                        if (ret)
-                                gf_log ("cli", GF_LOG_ERROR,
-                                        "Error outputting to xml");
-                        goto out;
-                }
-
-                gf_cli_quota_list (volname, dict, entry_count, rsp.op_errstr,
-                                   default_sl);
-        }
+    if (!rsp.op_ret && type != GF_QUOTA_OPTION_TYPE_LIST &&
+        type != GF_QUOTA_OPTION_TYPE_LIST_OBJECTS)
+        cli_out("volume quota : success");
 
-xml_output:
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_str ("volQuota", NULL, rsp.op_ret,
-                                          rsp.op_errno, rsp.op_errstr);
-                if (ret)
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
-                goto out;
-        }
+    ret = rsp.op_ret;
+out:
 
-        if (!rsp.op_ret && type != GF_QUOTA_OPTION_TYPE_LIST)
-                cli_out ("volume quota : success");
+    if ((type == GF_QUOTA_OPTION_TYPE_LIST) ||
+        (type == GF_QUOTA_OPTION_TYPE_LIST_OBJECTS)) {
+        gluster_remove_auxiliary_mount(volname);
+    }
 
-        ret = rsp.op_ret;
-out:
-        cli_cmd_broadcast_response (ret);
-        if (dict)
-                dict_unref (dict);
+    cli_cmd_broadcast_response(ret);
+    if (dict)
+        dict_unref(dict);
 
-        free (rsp.dict.dict_val);
+    gf_free_xdr_cli_rsp(rsp);
 
-        return ret;
+    return ret;
 }
 
-int
-gf_cli_getspec_cbk (struct rpc_req *req, struct iovec *iov,
-                       int count, void *myframe)
+static int
+gf_cli_getspec_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                   void *myframe)
 {
-        gf_getspec_rsp          rsp   = {0,};
-        int                     ret   = -1;
-        char                   *spec  = NULL;
-        call_frame_t           *frame = NULL;
-
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
-
-        frame = myframe;
-
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_getspec_rsp);
-        if (ret < 0) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
-
-        if (rsp.op_ret == -1) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "getspec failed");
-                goto out;
-        }
-
-        gf_log ("cli", GF_LOG_INFO, "Received resp to getspec");
-
-        spec = GF_MALLOC (rsp.op_ret + 1, cli_mt_char);
-        if (!spec) {
-                gf_log("", GF_LOG_ERROR, "out of memory");
-                goto out;
-        }
-        memcpy (spec, rsp.spec, rsp.op_ret);
-        spec[rsp.op_ret] = '\0';
-        cli_out ("%s", spec);
-        GF_FREE (spec);
-
-        ret = 0;
+    gf_getspec_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    char *spec = NULL;
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_getspec_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    if (rsp.op_ret == -1) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               "getspec failed");
+        ret = -1;
+        goto out;
+    }
+
+    gf_log("cli", GF_LOG_INFO, "Received resp to getspec");
+
+    spec = GF_MALLOC(rsp.op_ret + 1, cli_mt_char);
+    if (!spec) {
+        gf_log("", GF_LOG_ERROR, "out of memory");
+        ret = -1;
+        goto out;
+    }
+    memcpy(spec, rsp.spec, rsp.op_ret);
+    spec[rsp.op_ret] = '\0';
+    cli_out("%s", spec);
+    GF_FREE(spec);
+
+    ret = 0;
 
 out:
-        cli_cmd_broadcast_response (ret);
-        return ret;
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_getspec_rsp(rsp);
+    return ret;
 }
 
-int
-gf_cli_pmap_b2p_cbk (struct rpc_req *req, struct iovec *iov,
-                        int count, void *myframe)
+static int
+gf_cli_pmap_b2p_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                    void *myframe)
 {
-        pmap_port_by_brick_rsp rsp = {0,};
-        int                     ret   = -1;
-        char                   *spec  = NULL;
-        call_frame_t           *frame = NULL;
+    pmap_port_by_brick_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    char *spec = NULL;
 
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
+    GF_ASSERT(myframe);
 
-        frame = myframe;
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
 
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_pmap_port_by_brick_rsp);
-        if (ret < 0) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_pmap_port_by_brick_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
 
-        if (rsp.op_ret == -1) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "pump_b2p failed");
-                goto out;
-        }
+    if (rsp.op_ret == -1) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               "pump_b2p failed");
+        ret = -1;
+        goto out;
+    }
 
-        gf_log ("cli", GF_LOG_INFO, "Received resp to pmap b2p");
+    gf_log("cli", GF_LOG_INFO, "Received resp to pmap b2p");
 
-        cli_out ("%d", rsp.port);
-        GF_FREE (spec);
+    cli_out("%d", rsp.port);
+    GF_FREE(spec);
 
-        ret = rsp.op_ret;
+    ret = rsp.op_ret;
 
 out:
-        cli_cmd_broadcast_response (ret);
-        return ret;
+    cli_cmd_broadcast_response(ret);
+    return ret;
 }
 
-
-int32_t
-gf_cli_probe (call_frame_t *frame, xlator_t *this,
-                 void *data)
+static int32_t
+gf_cli_probe(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf_cli_req         req      = {{0,},};
-        int                ret      = 0;
-        dict_t            *dict     = NULL;
-        int                port     = 0;
-
-        if (!frame || !this ||  !data) {
-                ret = -1;
-                goto out;
-        }
-
-        dict = data;
-
-        ret = dict_get_int32 (dict, "port", &port);
-        if (ret) {
-                ret = dict_set_int32 (dict, "port", CLI_GLUSTERD_PORT);
-                if (ret)
-                        goto out;
-        }
+    gf_cli_req req = {
+        {
+            0,
+        },
+    };
+    int ret = 0;
+    dict_t *dict = NULL;
+    int port = 0;
+
+    if (!frame || !this || !data) {
+        ret = -1;
+        goto out;
+    }
+
+    dict = data;
+
+    ret = dict_get_int32_sizen(dict, "port", &port);
+    if (ret) {
+        ret = dict_set_int32_sizen(dict, "port", CLI_GLUSTERD_PORT);
+        if (ret)
+            goto out;
+    }
 
-        ret = cli_to_glusterd (&req, frame, gf_cli_probe_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, dict,
-                               GLUSTER_CLI_PROBE, this, cli_rpc_prog, NULL);
+    ret = cli_to_glusterd(&req, frame, gf_cli_probe_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict, GLUSTER_CLI_PROBE,
+                          this, cli_rpc_prog, NULL);
 
 out:
-        GF_FREE (req.dict.dict_val);
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    GF_FREE(req.dict.dict_val);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
 
-        return ret;
+    return ret;
 }
 
-int32_t
-gf_cli_deprobe (call_frame_t *frame, xlator_t *this,
-                   void *data)
+static int32_t
+gf_cli_deprobe(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf_cli_req           req      = {{0,},};
-        int                  ret      = 0;
-        dict_t              *dict     = NULL;
-        int                  port     = 0;
-        int                  flags    = 0;
-
-        if (!frame || !this ||  !data) {
-                ret = -1;
-                goto out;
-        }
-
-        dict = data;
-        ret = dict_get_int32 (dict, "port", &port);
-        if (ret) {
-                ret = dict_set_int32 (dict, "port", CLI_GLUSTERD_PORT);
-                if (ret)
-                        goto out;
-        }
+    gf_cli_req req = {
+        {
+            0,
+        },
+    };
+    int ret = 0;
+    dict_t *dict = NULL;
+    int port = 0;
+    int flags = 0;
+
+    if (!frame || !this || !data) {
+        ret = -1;
+        goto out;
+    }
+
+    dict = data;
+    ret = dict_get_int32_sizen(dict, "port", &port);
+    if (ret) {
+        ret = dict_set_int32_sizen(dict, "port", CLI_GLUSTERD_PORT);
+        if (ret)
+            goto out;
+    }
 
-        ret = dict_get_int32 (dict, "flags", &flags);
-        if (ret) {
-                ret = dict_set_int32 (dict, "flags", 0);
-                if (ret)
-                        goto out;
-        }
+    ret = dict_get_int32_sizen(dict, "flags", &flags);
+    if (ret) {
+        ret = dict_set_int32_sizen(dict, "flags", 0);
+        if (ret)
+            goto out;
+    }
 
-        ret = cli_to_glusterd (&req, frame, gf_cli_deprobe_cbk,
-                               (xdrproc_t)xdr_gf_cli_req, dict,
-                              GLUSTER_CLI_DEPROBE, this, cli_rpc_prog, NULL);
+    ret = cli_to_glusterd(&req, frame, gf_cli_deprobe_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict, GLUSTER_CLI_DEPROBE,
+                          this, cli_rpc_prog, NULL);
 
 out:
-        GF_FREE (req.dict.dict_val);
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    GF_FREE(req.dict.dict_val);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
 
-        return ret;
+    return ret;
 }
 
-int32_t
-gf_cli_list_friends (call_frame_t *frame, xlator_t *this,
-                     void *data)
+static int32_t
+gf_cli_list_friends(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf1_cli_peer_list_req   req = {0,};
-        int                     ret = 0;
-        unsigned long           flags = 0;
+    gf1_cli_peer_list_req req = {
+        0,
+    };
+    int ret = 0;
+    unsigned long flags = 0;
+
+    if (!frame || !this) {
+        ret = -1;
+        goto out;
+    }
+
+    GF_ASSERT(frame->local == NULL);
+
+    flags = (long)data;
+    req.flags = flags;
+    frame->local = (void *)flags;
+    ret = cli_cmd_submit(
+        NULL, &req, frame, cli_rpc_prog, GLUSTER_CLI_LIST_FRIENDS, NULL, this,
+        gf_cli_list_friends_cbk, (xdrproc_t)xdr_gf1_cli_peer_list_req);
 
-        if (!frame || !this) {
-                ret = -1;
-                goto out;
-        }
+out:
+    if (ret && frame) {
+        /*
+         * If everything goes fine, gf_cli_list_friends_cbk()
+         * [invoked through cli_cmd_submit()]resets the
+         * frame->local to NULL. In case cli_cmd_submit()
+         * fails in between, RESET frame->local here.
+         */
+        frame->local = NULL;
+    }
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
+    return ret;
+}
+
+static int32_t
+gf_cli_get_state(call_frame_t *frame, xlator_t *this, void *data)
+{
+    gf_cli_req req = {
+        {
+            0,
+        },
+    };
+    int ret = 0;
+    dict_t *dict = NULL;
 
-        GF_ASSERT (frame->local == NULL);
+    dict = data;
 
-        flags = (long)data;
-        req.flags = flags;
-        frame->local = (void*)flags;
-        ret = cli_cmd_submit (NULL, &req, frame, cli_rpc_prog,
-                              GLUSTER_CLI_LIST_FRIENDS, NULL,
-                              this, gf_cli_list_friends_cbk,
-                              (xdrproc_t) xdr_gf1_cli_peer_list_req);
+    ret = cli_to_glusterd(&req, frame, gf_cli_get_state_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict,
+                          GLUSTER_CLI_GET_STATE, this, cli_rpc_prog, NULL);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
+    GF_FREE(req.dict.dict_val);
 
-out:
-        if (ret) {
-                /*
-                 * If everything goes fine, gf_cli_list_friends_cbk()
-                 * [invoked through cli_cmd_submit()]resets the
-                 * frame->local to NULL. In case cli_cmd_submit()
-                 * fails in between, RESET frame->local here.
-                 */
-                frame->local = NULL;
-        }
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    return ret;
 }
 
-int32_t
-gf_cli_get_next_volume (call_frame_t *frame, xlator_t *this,
-                           void *data)
+static int32_t
+gf_cli_get_next_volume(call_frame_t *frame, xlator_t *this, void *data)
 {
+    int ret = 0;
+    cli_cmd_volume_get_ctx_t *ctx = NULL;
+    cli_local_t *local = NULL;
 
-        int                             ret = 0;
-        cli_cmd_volume_get_ctx_t        *ctx = NULL;
-        cli_local_t                     *local = NULL;
-
-        if (!frame || !this || !data) {
-                ret = -1;
-                goto out;
-        }
+    if (!frame || !this || !data) {
+        ret = -1;
+        goto out;
+    }
 
-        ctx = data;
-        local = frame->local;
+    ctx = data;
+    local = frame->local;
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_vol_info_begin (local, 0, 0, "");
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Error outputting to xml");
-                        goto out;
-                }
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_vol_info_begin(local, 0, 0, "");
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+            goto out;
         }
+    }
 
-        ret = gf_cli_get_volume (frame, this, data);
+    ret = gf_cli_get_volume(frame, this, data);
 
+    if (!local || !local->get_vol.volname) {
+        if ((global_state->mode & GLUSTER_MODE_XML))
+            goto end_xml;
 
-        if (!local || !local->get_vol.volname) {
-                if ((global_state->mode & GLUSTER_MODE_XML))
-                        goto end_xml;
-
-                cli_err ("No volumes present");
-                goto out;
-        }
+        cli_err("No volumes present");
+        goto out;
+    }
 
+    ctx->volname = local->get_vol.volname;
 
+    while (ctx->volname) {
+        ret = gf_cli_get_volume(frame, this, ctx);
+        if (ret)
+            goto out;
         ctx->volname = local->get_vol.volname;
-
-        while (ctx->volname) {
-                ret = gf_cli_get_volume (frame, this, ctx);
-                if (ret)
-                        goto out;
-                ctx->volname = local->get_vol.volname;
-        }
+    }
 
 end_xml:
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_vol_info_end (local);
-                if (ret)
-                        gf_log ("cli", GF_LOG_ERROR, "Error outputting to xml");
-        }
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_vol_info_end(local);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+    }
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
+    return ret;
 }
 
-int32_t
-gf_cli_get_volume (call_frame_t *frame, xlator_t *this,
-                      void *data)
+static int32_t
+gf_cli_get_volume(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf_cli_req                      req = {{0,}};
-        int                             ret = 0;
-        cli_cmd_volume_get_ctx_t        *ctx = NULL;
-        dict_t                          *dict = NULL;
-        int32_t                         flags = 0;
-
-        if (!frame || !this || !data) {
-                ret = -1;
-                goto out;
-        }
-
-        ctx = data;
-
-        dict = dict_new ();
-        if (!dict)
-                goto out;
-
-        if (ctx->volname) {
-                ret = dict_set_str (dict, "volname", ctx->volname);
-                if (ret)
-                        goto out;
-        }
-
-        flags = ctx->flags;
-        ret = dict_set_int32 (dict, "flags", flags);
-        if (ret) {
-                gf_log (frame->this->name, GF_LOG_ERROR, "failed to set flags");
-                goto out;
-        }
-
-        ret = dict_allocate_and_serialize (dict, &req.dict.dict_val,
-                                           &req.dict.dict_len);
-
-        ret = cli_cmd_submit (NULL, &req, frame, cli_rpc_prog,
-                              GLUSTER_CLI_GET_VOLUME, NULL,
-                              this, gf_cli_get_volume_cbk,
-                              (xdrproc_t) xdr_gf_cli_req);
+    gf_cli_req req = {{
+        0,
+    }};
+    int ret = 0;
+    cli_cmd_volume_get_ctx_t *ctx = NULL;
+    dict_t *dict = NULL;
+    int32_t flags = 0;
+
+    if (!this || !data) {
+        ret = -1;
+        goto out;
+    }
+
+    ctx = data;
+
+    dict = dict_new();
+    if (!dict) {
+        gf_log(THIS->name, GF_LOG_ERROR, "Failed to create the dict");
+        ret = -1;
+        goto out;
+    }
+
+    if (ctx->volname) {
+        ret = dict_set_str_sizen(dict, "volname", ctx->volname);
+        if (ret)
+            goto out;
+    }
+
+    flags = ctx->flags;
+    ret = dict_set_int32_sizen(dict, "flags", flags);
+    if (ret) {
+        gf_log(frame->this->name, GF_LOG_ERROR, "failed to set flags");
+        goto out;
+    }
+
+    ret = dict_allocate_and_serialize(dict, &req.dict.dict_val,
+                                      &req.dict.dict_len);
+    if (ret) {
+        gf_log(frame->this->name, GF_LOG_ERROR, DICT_SERIALIZE_FAIL);
+        goto out;
+    }
+
+    ret = cli_cmd_submit(NULL, &req, frame, cli_rpc_prog,
+                         GLUSTER_CLI_GET_VOLUME, NULL, this,
+                         gf_cli_get_volume_cbk, (xdrproc_t)xdr_gf_cli_req);
 
 out:
-        if (dict)
-                dict_unref (dict);
+    if (dict)
+        dict_unref(dict);
 
-        GF_FREE (req.dict.dict_val);
+    GF_FREE(req.dict.dict_val);
 
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
+    return ret;
 }
 
-int32_t
-gf_cli3_1_uuid_get (call_frame_t *frame, xlator_t *this,
-                      void *data)
+static int32_t
+gf_cli3_1_uuid_get(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf_cli_req                      req = {{0,}};
-        int                             ret = 0;
-        dict_t                          *dict = NULL;
-
-        if (!frame || !this || !data) {
-                ret = -1;
-                goto out;
-        }
+    gf_cli_req req = {{
+        0,
+    }};
+    int ret = 0;
+    dict_t *dict = NULL;
 
-        dict = data;
-        ret = cli_to_glusterd (&req, frame, gf_cli3_1_uuid_get_cbk,
-                               (xdrproc_t)xdr_gf_cli_req, dict,
-                               GLUSTER_CLI_UUID_GET, this, cli_rpc_prog,
-                               NULL);
-out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    dict = data;
+    ret = cli_to_glusterd(&req, frame, gf_cli3_1_uuid_get_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict, GLUSTER_CLI_UUID_GET,
+                          this, cli_rpc_prog, NULL);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
+    GF_FREE(req.dict.dict_val);
+    return ret;
 }
 
-int32_t
-gf_cli3_1_uuid_reset (call_frame_t *frame, xlator_t *this,
-                      void *data)
+static int32_t
+gf_cli3_1_uuid_reset(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf_cli_req                      req = {{0,}};
-        int                             ret = 0;
-        dict_t                          *dict = NULL;
+    gf_cli_req req = {{
+        0,
+    }};
+    int ret = 0;
+    dict_t *dict = NULL;
 
-        if (!frame || !this || !data) {
-                ret = -1;
-                goto out;
-        }
-
-        dict = data;
-        ret = cli_to_glusterd (&req, frame, gf_cli3_1_uuid_reset_cbk,
-                               (xdrproc_t)xdr_gf_cli_req, dict,
-                               GLUSTER_CLI_UUID_RESET, this, cli_rpc_prog,
-                               NULL);
-out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    dict = data;
+    ret = cli_to_glusterd(&req, frame, gf_cli3_1_uuid_reset_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict,
+                          GLUSTER_CLI_UUID_RESET, this, cli_rpc_prog, NULL);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
+    GF_FREE(req.dict.dict_val);
+    return ret;
 }
 
-int32_t
-gf_cli_create_volume (call_frame_t *frame, xlator_t *this,
-                         void *data)
+static int32_t
+gf_cli_create_volume(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf_cli_req              req = {{0,}};
-        int                     ret = 0;
-        dict_t                  *dict = NULL;
-
-        if (!frame || !this ||  !data) {
-                ret = -1;
-                goto out;
-        }
-
-        dict = data;
+    gf_cli_req req = {{
+        0,
+    }};
+    int ret = 0;
+    dict_t *dict = NULL;
 
-        ret = cli_to_glusterd (&req, frame, gf_cli_create_volume_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, dict,
-                               GLUSTER_CLI_CREATE_VOLUME, this, cli_rpc_prog,
-                               NULL);
+    dict = data;
 
-out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    ret = cli_to_glusterd(&req, frame, gf_cli_create_volume_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict,
+                          GLUSTER_CLI_CREATE_VOLUME, this, cli_rpc_prog, NULL);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
 
-        GF_FREE (req.dict.dict_val);
+    GF_FREE(req.dict.dict_val);
 
-        return ret;
+    return ret;
 }
 
-int32_t
-gf_cli_delete_volume (call_frame_t *frame, xlator_t *this,
-                         void *data)
+static int32_t
+gf_cli_delete_volume(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf_cli_req              req = {{0,}};
-        int                     ret = 0;
-        dict_t                  *dict = NULL;
-
-        if (!frame || !this ||  !data) {
-                ret = -1;
-                goto out;
-        }
+    gf_cli_req req = {{
+        0,
+    }};
+    int ret = 0;
+    dict_t *dict = NULL;
 
-        dict = data;
+    dict = data;
 
-        ret = cli_to_glusterd (&req, frame, gf_cli_delete_volume_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, dict,
-                               GLUSTER_CLI_DELETE_VOLUME, this, cli_rpc_prog,
-                               NULL);
+    ret = cli_to_glusterd(&req, frame, gf_cli_delete_volume_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict,
+                          GLUSTER_CLI_DELETE_VOLUME, this, cli_rpc_prog, NULL);
+    GF_FREE(req.dict.dict_val);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
 
-out:
-        GF_FREE (req.dict.dict_val);
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-
-        return ret;
+    return ret;
 }
 
-int32_t
-gf_cli_start_volume (call_frame_t *frame, xlator_t *this,
-                         void *data)
+static int32_t
+gf_cli_start_volume(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf_cli_req              req = {{0,}};
-        int                     ret = 0;
-        dict_t                  *dict = NULL;
-
-        if (!frame || !this ||  !data) {
-                ret = -1;
-                goto out;
-        }
+    gf_cli_req req = {{
+        0,
+    }};
+    int ret = 0;
+    dict_t *dict = NULL;
 
-        dict = data;
+    dict = data;
 
-        ret = cli_to_glusterd (&req, frame, gf_cli_start_volume_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, dict,
-                               GLUSTER_CLI_START_VOLUME, this, cli_rpc_prog,
-                               NULL);
+    ret = cli_to_glusterd(&req, frame, gf_cli_start_volume_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict,
+                          GLUSTER_CLI_START_VOLUME, this, cli_rpc_prog, NULL);
 
-out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
+    GF_FREE(req.dict.dict_val);
 
-        return ret;
+    return ret;
 }
 
-int32_t
-gf_cli_stop_volume (call_frame_t *frame, xlator_t *this,
-                         void *data)
+static int32_t
+gf_cli_stop_volume(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf_cli_req             req = {{0,}};
-        int                    ret = 0;
-        dict_t                 *dict = data;
+    gf_cli_req req = {{
+        0,
+    }};
+    int ret = 0;
+    dict_t *dict = data;
 
-        if (!frame || !this ||  !data) {
-                ret = -1;
-                goto out;
-        }
+    dict = data;
 
-        dict = data;
+    ret = cli_to_glusterd(&req, frame, gf_cli_stop_volume_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict,
+                          GLUSTER_CLI_STOP_VOLUME, this, cli_rpc_prog, NULL);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
+    GF_FREE(req.dict.dict_val);
 
-        ret = cli_to_glusterd (&req, frame, gf_cli_stop_volume_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, dict,
-                               GLUSTER_CLI_STOP_VOLUME, this, cli_rpc_prog,
-                               NULL);
-
-out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-
-        return ret;
+    return ret;
 }
 
-int32_t
-gf_cli_defrag_volume (call_frame_t *frame, xlator_t *this,
-                         void *data)
+static int32_t
+gf_cli_defrag_volume(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf_cli_req              req     =  {{0,}};
-        int                     ret     = 0;
-        dict_t                 *dict    = NULL;
-
-        if (!frame || !this ||  !data) {
-                ret = -1;
-                goto out;
-        }
+    gf_cli_req req = {{
+        0,
+    }};
+    int ret = 0;
+    dict_t *dict = NULL;
 
-        dict = data;
+    dict = data;
 
-        ret = cli_to_glusterd (&req, frame, gf_cli_defrag_volume_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, dict,
-                               GLUSTER_CLI_DEFRAG_VOLUME, this, cli_rpc_prog,
-                               NULL);
+    ret = cli_to_glusterd(&req, frame, gf_cli_defrag_volume_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict,
+                          GLUSTER_CLI_DEFRAG_VOLUME, this, cli_rpc_prog, NULL);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
+    GF_FREE(req.dict.dict_val);
 
-out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-
-        return ret;
+    return ret;
 }
 
-int32_t
-gf_cli_rename_volume (call_frame_t *frame, xlator_t *this,
-                         void *data)
+static int32_t
+gf_cli_rename_volume(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf_cli_req              req = {{0,}};
-        int                     ret = 0;
-        dict_t                  *dict = NULL;
+    gf_cli_req req = {{
+        0,
+    }};
+    int ret = 0;
+    dict_t *dict = NULL;
 
-        if (!frame || !this ||  !data) {
-                ret = -1;
-                goto out;
-        }
+    if (!frame || !this || !data) {
+        ret = -1;
+        goto out;
+    }
 
-        dict = data;
+    dict = data;
 
-        ret = dict_allocate_and_serialize (dict, &req.dict.dict_val,
-                                           &req.dict.dict_len);
-        if (ret < 0) {
-                gf_log (this->name, GF_LOG_ERROR,
-                        "failed to serialize the data");
-
-                goto out;
-        }
+    ret = dict_allocate_and_serialize(dict, &req.dict.dict_val,
+                                      &req.dict.dict_len);
+    if (ret < 0) {
+        gf_log(this->name, GF_LOG_ERROR, DICT_SERIALIZE_FAIL);
 
+        goto out;
+    }
 
-        ret = cli_cmd_submit (NULL, &req, frame, cli_rpc_prog,
-                              GLUSTER_CLI_RENAME_VOLUME, NULL,
-                              this, gf_cli_rename_volume_cbk,
-                              (xdrproc_t) xdr_gf_cli_req);
+    ret = cli_cmd_submit(NULL, &req, frame, cli_rpc_prog,
+                         GLUSTER_CLI_RENAME_VOLUME, NULL, this,
+                         gf_cli_rename_volume_cbk, (xdrproc_t)xdr_gf_cli_req);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    GF_FREE(req.dict.dict_val);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
 
-        return ret;
+    return ret;
 }
 
-int32_t
-gf_cli_reset_volume (call_frame_t *frame, xlator_t *this,
-                        void *data)
+static int32_t
+gf_cli_reset_volume(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf_cli_req              req =  {{0,}};
-        int                     ret = 0;
-        dict_t                  *dict = NULL;
+    gf_cli_req req = {{
+        0,
+    }};
+    int ret = 0;
+    dict_t *dict = NULL;
 
-        if (!frame || !this ||  !data) {
-                ret = -1;
-                goto out;
-        }
-
-        dict = data;
+    dict = data;
 
-        ret = cli_to_glusterd (&req, frame, gf_cli_reset_volume_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, dict,
-                               GLUSTER_CLI_RESET_VOLUME, this, cli_rpc_prog,
-                               NULL);
-
-out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    ret = cli_to_glusterd(&req, frame, gf_cli_reset_volume_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict,
+                          GLUSTER_CLI_RESET_VOLUME, this, cli_rpc_prog, NULL);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
+    GF_FREE(req.dict.dict_val);
+    return ret;
 }
 
-int32_t
-gf_cli_set_volume (call_frame_t *frame, xlator_t *this,
-                         void *data)
+static int32_t
+gf_cli_ganesha(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf_cli_req              req =  {{0,}};
-        int                     ret = 0;
-        dict_t                  *dict = NULL;
-
-        if (!frame || !this ||  !data) {
-                ret = -1;
-                goto out;
-        }
+    gf_cli_req req = {{
+        0,
+    }};
+    int ret = 0;
+    dict_t *dict = NULL;
 
-        dict = data;
+    dict = data;
 
-        ret = cli_to_glusterd (&req, frame, gf_cli_set_volume_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, dict,
-                               GLUSTER_CLI_SET_VOLUME, this, cli_rpc_prog,
-                               NULL);
+    ret = cli_to_glusterd(&req, frame, gf_cli_ganesha_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict, GLUSTER_CLI_GANESHA,
+                          this, cli_rpc_prog, NULL);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
 
-out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-
-        return ret;
+    return ret;
 }
 
-int32_t
-gf_cli_add_brick (call_frame_t *frame, xlator_t *this,
-                         void *data)
+static int32_t
+gf_cli_set_volume(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf_cli_req              req =  {{0,}};
-        int                     ret = 0;
-        dict_t                  *dict = NULL;
-        char                    *volname = NULL;
-        int32_t                 count = 0;
-
-        if (!frame || !this ||  !data) {
-                ret = -1;
-                goto out;
-        }
-
-        dict = data;
-
-        ret = dict_get_str (dict, "volname", &volname);
-
-        if (ret)
-                goto out;
-
-        ret = dict_get_int32 (dict, "count", &count);
-        if (ret)
-                goto out;
-
-        ret = cli_to_glusterd (&req, frame, gf_cli_add_brick_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, dict,
-                               GLUSTER_CLI_ADD_BRICK, this, cli_rpc_prog, NULL);
+    gf_cli_req req = {{
+        0,
+    }};
+    int ret = 0;
+    dict_t *dict = NULL;
 
-out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    dict = data;
 
-        GF_FREE (req.dict.dict_val);
+    ret = cli_to_glusterd(&req, frame, gf_cli_set_volume_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict,
+                          GLUSTER_CLI_SET_VOLUME, this, cli_rpc_prog, NULL);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
+    GF_FREE(req.dict.dict_val);
 
-        return ret;
+    return ret;
 }
 
 int32_t
-gf_cli_remove_brick (call_frame_t *frame, xlator_t *this,
-                         void *data)
-{
-        gf_cli_req                req =  {{0,}};;
-        gf_cli_req                status_req =  {{0,}};;
-        int                       ret = 0;
-        dict_t                   *dict = NULL;
-        int32_t                   command = 0;
-        char                     *volname = NULL;
-        dict_t                   *req_dict = NULL;
-        int32_t                   cmd = 0;
-
-        if (!frame || !this ||  !data) {
-                ret = -1;
-                goto out;
-        }
-
-        dict = data;
-
-        ret = dict_get_str (dict, "volname", &volname);
-        if (ret)
-                goto out;
+gf_cli_add_brick(call_frame_t *frame, xlator_t *this, void *data)
+{
+    gf_cli_req req = {{
+        0,
+    }};
+    int ret = 0;
+    dict_t *dict = NULL;
 
-        ret = dict_get_int32 (dict, "command", &command);
-        if (ret)
-                goto out;
+    dict = data;
 
-        if ((command != GF_OP_CMD_STATUS) &&
-            (command != GF_OP_CMD_STOP)) {
+    ret = cli_to_glusterd(&req, frame, gf_cli_add_brick_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict,
+                          GLUSTER_CLI_ADD_BRICK, this, cli_rpc_prog, NULL);
 
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
 
-                ret = cli_to_glusterd (&req, frame, gf_cli_remove_brick_cbk,
-                                       (xdrproc_t) xdr_gf_cli_req, dict,
-                                       GLUSTER_CLI_REMOVE_BRICK, this,
-                                       cli_rpc_prog, NULL);
-        } else {
-                /* Need rebalance status to be sent :-) */
-                req_dict = dict_new ();
-                if (!req_dict) {
-                        ret = -1;
-                        goto out;
-                }
+    GF_FREE(req.dict.dict_val);
 
-                ret = dict_set_str (req_dict, "volname", volname);
-                if (ret) {
-                        gf_log (this->name, GF_LOG_ERROR,
-                                "Failed to set dict");
-                        goto out;
-                }
-
-                if (command == GF_OP_CMD_STATUS)
-                        cmd |= GF_DEFRAG_CMD_STATUS;
-                else
-                        cmd |= GF_DEFRAG_CMD_STOP;
+    return ret;
+}
 
-                ret = dict_set_int32 (req_dict, "rebalance-command", (int32_t) cmd);
-                if (ret) {
-                        gf_log (this->name, GF_LOG_ERROR,
-                                "Failed to set dict");
-                        goto out;
-                }
+static int32_t
+gf_cli_remove_brick(call_frame_t *frame, xlator_t *this, void *data)
+{
+    gf_cli_req req = {{
+        0,
+    }};
+    ;
+    gf_cli_req status_req = {{
+        0,
+    }};
+    ;
+    int ret = 0;
+    dict_t *dict = NULL;
+    int32_t command = 0;
+    int32_t cmd = 0;
+
+    if (!frame || !this) {
+        ret = -1;
+        goto out;
+    }
+
+    dict = data;
+
+    ret = dict_get_int32_sizen(dict, "command", &command);
+    if (ret)
+        goto out;
+
+    if ((command != GF_OP_CMD_STATUS) && (command != GF_OP_CMD_STOP)) {
+        ret = cli_to_glusterd(
+            &req, frame, gf_cli_remove_brick_cbk, (xdrproc_t)xdr_gf_cli_req,
+            dict, GLUSTER_CLI_REMOVE_BRICK, this, cli_rpc_prog, NULL);
+    } else {
+        /* Need rebalance status to be sent :-) */
+        if (command == GF_OP_CMD_STATUS)
+            cmd |= GF_DEFRAG_CMD_STATUS;
+        else
+            cmd |= GF_DEFRAG_CMD_STOP;
 
-                ret = cli_to_glusterd (&status_req, frame,
-                                       gf_cli3_remove_brick_status_cbk,
-                                       (xdrproc_t) xdr_gf_cli_req, req_dict,
-                                       GLUSTER_CLI_DEFRAG_VOLUME, this,
-                                       cli_rpc_prog, NULL);
+        ret = dict_set_int32_sizen(dict, "rebalance-command", (int32_t)cmd);
+        if (ret) {
+            gf_log(this->name, GF_LOG_ERROR, "Failed to set dict");
+            goto out;
+        }
 
-                }
+        ret = cli_to_glusterd(
+            &status_req, frame, gf_cli3_remove_brick_status_cbk,
+            (xdrproc_t)xdr_gf_cli_req, dict, GLUSTER_CLI_DEFRAG_VOLUME, this,
+            cli_rpc_prog, NULL);
+    }
 
 out:
-        if (req_dict)
-                dict_unref (req_dict);
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
 
-        GF_FREE (req.dict.dict_val);
+    GF_FREE(req.dict.dict_val);
 
-        GF_FREE (status_req.dict.dict_val);
+    GF_FREE(status_req.dict.dict_val);
 
-        return ret;
+    return ret;
 }
 
-int32_t
-gf_cli_replace_brick (call_frame_t *frame, xlator_t *this,
-                         void *data)
-{
-        gf_cli_req                  req        =  {{0,}};
-        int                         ret        = 0;
-        dict_t                     *dict       = NULL;
-        char                       *src_brick  = NULL;
-        char                       *dst_brick  = NULL;
-        char                       *volname    = NULL;
-        int32_t                     op         = 0;
-
-        if (!frame || !this ||  !data) {
-                ret = -1;
-                goto out;
-        }
-
-        dict = data;
-
-        ret = dict_get_int32 (dict, "operation", &op);
-        if (ret) {
-                gf_log (this->name, GF_LOG_DEBUG,
-                        "dict_get on operation failed");
-                goto out;
-        }
-        ret = dict_get_str (dict, "volname", &volname);
-        if (ret) {
-                gf_log (this->name, GF_LOG_DEBUG,
-                        "dict_get on volname failed");
-                goto out;
-        }
-
-        ret = dict_get_str (dict, "src-brick", &src_brick);
-        if (ret) {
-                gf_log (this->name, GF_LOG_DEBUG,
-                        "dict_get on src-brick failed");
-                goto out;
-        }
-
-        ret = dict_get_str (dict, "dst-brick", &dst_brick);
+static int32_t
+gf_cli_reset_brick(call_frame_t *frame, xlator_t *this, void *data)
+{
+    gf_cli_req req = {{
+        0,
+    }};
+    int ret = 0;
+    dict_t *dict = NULL;
+    char *dst_brick = NULL;
+    char *op = NULL;
+
+    if (!frame || !this || !data) {
+        ret = -1;
+        goto out;
+    }
+
+    dict = data;
+
+    ret = dict_get_str_sizen(dict, "operation", &op);
+    if (ret) {
+        gf_log(this->name, GF_LOG_DEBUG, "dict_get on operation failed");
+        goto out;
+    }
+
+    if (!strcmp(op, "GF_RESET_OP_COMMIT") ||
+        !strcmp(op, "GF_RESET_OP_COMMIT_FORCE")) {
+        ret = dict_get_str_sizen(dict, "dst-brick", &dst_brick);
         if (ret) {
-                gf_log (this->name, GF_LOG_DEBUG,
-                        "dict_get on dst-brick failed");
-                goto out;
+            gf_log(this->name, GF_LOG_DEBUG, "dict_get on dst-brick failed");
+            goto out;
         }
+    }
 
-        gf_log (this->name, GF_LOG_DEBUG,
-                "Received command replace-brick %s with "
-                "%s with operation=%d", src_brick,
-                dst_brick, op);
-
-        ret = cli_to_glusterd (&req, frame, gf_cli_replace_brick_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, dict,
-                               GLUSTER_CLI_REPLACE_BRICK, this, cli_rpc_prog,
-                               NULL);
+    ret = cli_to_glusterd(&req, frame, gf_cli_reset_brick_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict,
+                          GLUSTER_CLI_RESET_BRICK, this, cli_rpc_prog, NULL);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
 
-        GF_FREE (req.dict.dict_val);
+    GF_FREE(req.dict.dict_val);
 
-        return ret;
+    return ret;
 }
 
-
-int32_t
-gf_cli_log_rotate (call_frame_t *frame, xlator_t *this,
-                      void *data)
+static int32_t
+gf_cli_replace_brick(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf_cli_req                req = {{0,}};
-        int                       ret = 0;
-        dict_t                   *dict = NULL;
-
-        if (!frame || !this ||  !data) {
-                ret = -1;
-                goto out;
-        }
-
-        dict = data;
-
-        ret = cli_to_glusterd (&req, frame, gf_cli_log_rotate_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, dict,
-                               GLUSTER_CLI_LOG_ROTATE, this, cli_rpc_prog,
-                               NULL);
+    gf_cli_req req = {{
+        0,
+    }};
+    int ret = 0;
+    dict_t *dict = NULL;
+    int32_t op = 0;
+
+    if (!frame || !this || !data) {
+        ret = -1;
+        goto out;
+    }
+
+    dict = data;
+
+    ret = dict_get_int32_sizen(dict, "operation", &op);
+    if (ret) {
+        gf_log(this->name, GF_LOG_DEBUG, "dict_get on operation failed");
+        goto out;
+    }
+
+    ret = cli_to_glusterd(&req, frame, gf_cli_replace_brick_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict,
+                          GLUSTER_CLI_REPLACE_BRICK, this, cli_rpc_prog, NULL);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
+
+    GF_FREE(req.dict.dict_val);
 
-        GF_FREE (req.dict.dict_val);
-        return ret;
+    return ret;
 }
 
-int32_t
-gf_cli_sync_volume (call_frame_t *frame, xlator_t *this,
-                       void *data)
+static int32_t
+gf_cli_log_rotate(call_frame_t *frame, xlator_t *this, void *data)
 {
-        int               ret = 0;
-        gf_cli_req        req = {{0,}};
-        dict_t            *dict = NULL;
-
-        if (!frame || !this || !data) {
-                ret = -1;
-                goto out;
-        }
-
-        dict = data;
+    gf_cli_req req = {{
+        0,
+    }};
+    int ret = 0;
+    dict_t *dict = NULL;
 
-        ret = cli_to_glusterd (&req, frame, gf_cli_sync_volume_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, dict,
-                               GLUSTER_CLI_SYNC_VOLUME, this, cli_rpc_prog,
-                               NULL);
+    dict = data;
 
-out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        GF_FREE (req.dict.dict_val);
+    ret = cli_to_glusterd(&req, frame, gf_cli_log_rotate_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict,
+                          GLUSTER_CLI_LOG_ROTATE, this, cli_rpc_prog, NULL);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
 
-        return ret;
+    GF_FREE(req.dict.dict_val);
+    return ret;
 }
 
-int32_t
-gf_cli_getspec (call_frame_t *frame, xlator_t *this,
-                         void *data)
+static int32_t
+gf_cli_sync_volume(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf_getspec_req          req = {0,};
-        int                     ret = 0;
-        dict_t                  *dict = NULL;
-        dict_t                  *op_dict = NULL;
-
-        if (!frame || !this ||  !data) {
-                ret = -1;
-                goto out;
-        }
+    int ret = 0;
+    gf_cli_req req = {{
+        0,
+    }};
+    dict_t *dict = NULL;
 
-        dict = data;
+    dict = data;
 
-        ret = dict_get_str (dict, "volid", &req.key);
-        if (ret)
-                goto out;
+    ret = cli_to_glusterd(&req, frame, gf_cli_sync_volume_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict,
+                          GLUSTER_CLI_SYNC_VOLUME, this, cli_rpc_prog, NULL);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
+    GF_FREE(req.dict.dict_val);
 
-        op_dict = dict_new ();
-        if (!dict) {
-                ret = -1;
-                goto out;
-        }
-
-        // Set the supported min and max op-versions, so glusterd can make a
-        // decision
-        ret = dict_set_int32 (op_dict, "min-op-version", GD_OP_VERSION_MIN);
-        if (ret) {
-                gf_log (THIS->name, GF_LOG_ERROR, "Failed to set min-op-version"
-                        " in request dict");
-                goto out;
-        }
-
-        ret = dict_set_int32 (op_dict, "max-op-version", GD_OP_VERSION_MAX);
-        if (ret) {
-                gf_log (THIS->name, GF_LOG_ERROR, "Failed to set max-op-version"
-                        " in request dict");
-                goto out;
-        }
-
-        ret = dict_allocate_and_serialize (op_dict, &req.xdata.xdata_val,
-                                           &req.xdata.xdata_len);
-        if (ret < 0) {
-                gf_log (THIS->name, GF_LOG_ERROR,
-                        "Failed to serialize dictionary");
-                goto out;
-        }
+    return ret;
+}
 
-        ret = cli_cmd_submit (NULL, &req, frame, &cli_handshake_prog,
-                              GF_HNDSK_GETSPEC, NULL,
-                              this, gf_cli_getspec_cbk,
-                              (xdrproc_t) xdr_gf_getspec_req);
+static int32_t
+gf_cli_getspec(call_frame_t *frame, xlator_t *this, void *data)
+{
+    gf_getspec_req req = {
+        0,
+    };
+    int ret = 0;
+    dict_t *dict = NULL;
+    dict_t *op_dict = NULL;
+
+    if (!frame || !this) {
+        ret = -1;
+        goto out;
+    }
+
+    dict = data;
+
+    ret = dict_get_str_sizen(dict, "volid", &req.key);
+    if (ret)
+        goto out;
+
+    op_dict = dict_new();
+    if (!op_dict) {
+        ret = -1;
+        goto out;
+    }
+
+    // Set the supported min and max op-versions, so glusterd can make a
+    // decision
+    ret = dict_set_int32_sizen(op_dict, "min-op-version", GD_OP_VERSION_MIN);
+    if (ret) {
+        gf_log(THIS->name, GF_LOG_ERROR,
+               "Failed to set min-op-version in request dict");
+        goto out;
+    }
+
+    ret = dict_set_int32_sizen(op_dict, "max-op-version", GD_OP_VERSION_MAX);
+    if (ret) {
+        gf_log(THIS->name, GF_LOG_ERROR,
+               "Failed to set max-op-version in request dict");
+        goto out;
+    }
+
+    ret = dict_allocate_and_serialize(op_dict, &req.xdata.xdata_val,
+                                      &req.xdata.xdata_len);
+    if (ret < 0) {
+        gf_log(THIS->name, GF_LOG_ERROR, DICT_SERIALIZE_FAIL);
+        goto out;
+    }
+
+    ret = cli_cmd_submit(NULL, &req, frame, &cli_handshake_prog,
+                         GF_HNDSK_GETSPEC, NULL, this, gf_cli_getspec_cbk,
+                         (xdrproc_t)xdr_gf_getspec_req);
 
 out:
-        if (op_dict) {
-                dict_unref(op_dict);
-        }
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    if (op_dict) {
+        dict_unref(op_dict);
+    }
+    GF_FREE(req.xdata.xdata_val);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
 
-        return ret;
+    return ret;
 }
 
-int32_t
-gf_cli_quota (call_frame_t *frame, xlator_t *this,
-                 void *data)
+static int32_t
+gf_cli_quota(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf_cli_req          req = {{0,}};
-        int                 ret = 0;
-        dict_t             *dict = NULL;
+    gf_cli_req req = {{
+        0,
+    }};
+    int ret = 0;
+    dict_t *dict = NULL;
 
-        if (!frame || !this ||  !data) {
-                ret = -1;
-                goto out;
-        }
-
-        dict = data;
-
-        ret = cli_to_glusterd (&req, frame, gf_cli_quota_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, dict,
-                               GLUSTER_CLI_QUOTA, this, cli_rpc_prog, NULL);
+    dict = data;
 
-out:
-        GF_FREE (req.dict.dict_val);
-
-        return ret;
+    ret = cli_to_glusterd(&req, frame, gf_cli_quota_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict, GLUSTER_CLI_QUOTA,
+                          this, cli_rpc_prog, NULL);
+    GF_FREE(req.dict.dict_val);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
+    return ret;
 }
 
-int32_t
-gf_cli_pmap_b2p (call_frame_t *frame, xlator_t *this, void *data)
+static int32_t
+gf_cli_pmap_b2p(call_frame_t *frame, xlator_t *this, void *data)
 {
-        pmap_port_by_brick_req  req = {0,};
-        int                     ret = 0;
-        dict_t                  *dict = NULL;
+    pmap_port_by_brick_req req = {
+        0,
+    };
+    int ret = 0;
+    dict_t *dict = NULL;
 
-        if (!frame || !this ||  !data) {
-                ret = -1;
-                goto out;
-        }
+    if (!frame || !this) {
+        ret = -1;
+        goto out;
+    }
 
-        dict = data;
+    dict = data;
 
-        ret = dict_get_str (dict, "brick", &req.brick);
-        if (ret)
-                goto out;
+    ret = dict_get_str_sizen(dict, "brick", &req.brick);
+    if (ret)
+        goto out;
 
-        ret = cli_cmd_submit (NULL, &req, frame, &cli_pmap_prog,
-                              GF_PMAP_PORTBYBRICK, NULL,
-                              this, gf_cli_pmap_b2p_cbk,
-                              (xdrproc_t) xdr_pmap_port_by_brick_req);
+    ret = cli_cmd_submit(NULL, &req, frame, &cli_pmap_prog, GF_PMAP_PORTBYBRICK,
+                         NULL, this, gf_cli_pmap_b2p_cbk,
+                         (xdrproc_t)xdr_pmap_port_by_brick_req);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
 
-        return ret;
+    return ret;
 }
 
 static int
-gf_cli_fsm_log_cbk (struct rpc_req *req, struct iovec *iov,
-                       int count, void *myframe)
-{
-        gf1_cli_fsm_log_rsp        rsp   = {0,};
-        int                        ret   = -1;
-        dict_t                     *dict = NULL;
-        int                        tr_count = 0;
-        char                       key[256] = {0};
-        int                        i = 0;
-        char                       *old_state = NULL;
-        char                       *new_state = NULL;
-        char                       *event = NULL;
-        char                       *time = NULL;
-
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
-
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf1_cli_fsm_log_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
-
-        if (rsp.op_ret) {
-                if (strcmp (rsp.op_errstr, ""))
-                        cli_err ("%s", rsp.op_errstr);
-                cli_err ("fsm log unsuccessful");
-                ret = rsp.op_ret;
-                goto out;
-        }
-
-        dict = dict_new ();
-        if (!dict) {
-                ret = -1;
-                goto out;
-        }
-
-        ret = dict_unserialize (rsp.fsm_log.fsm_log_val,
-                                rsp.fsm_log.fsm_log_len,
-                                &dict);
-
-        if (ret) {
-                cli_err ("bad response");
-                goto out;
-        }
-
-        ret = dict_get_int32 (dict, "count", &tr_count);
-        if (tr_count)
-                cli_out("number of transitions: %d", tr_count);
-        else
-                cli_err("No transitions");
-        for (i = 0; i < tr_count; i++) {
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "log%d-old-state", i);
-                ret = dict_get_str (dict, key, &old_state);
-                if (ret)
-                        goto out;
+gf_cli_fsm_log_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                   void *myframe)
+{
+    gf1_cli_fsm_log_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    dict_t *dict = NULL;
+    int tr_count = 0;
+    char key[64] = {0};
+    int keylen;
+    int i = 0;
+    char *old_state = NULL;
+    char *new_state = NULL;
+    char *event = NULL;
+    char *time = NULL;
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf1_cli_fsm_log_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    if (rsp.op_ret) {
+        if (strcmp(rsp.op_errstr, ""))
+            cli_err("%s", rsp.op_errstr);
+        cli_err("fsm log unsuccessful");
+        ret = rsp.op_ret;
+        goto out;
+    }
+
+    dict = dict_new();
+    if (!dict) {
+        ret = -1;
+        goto out;
+    }
+
+    ret = dict_unserialize(rsp.fsm_log.fsm_log_val, rsp.fsm_log.fsm_log_len,
+                           &dict);
+
+    if (ret) {
+        cli_err(DICT_UNSERIALIZE_FAIL);
+        goto out;
+    }
+
+    ret = dict_get_int32_sizen(dict, "count", &tr_count);
+    if (!ret && tr_count)
+        cli_out("number of transitions: %d", tr_count);
+    else
+        cli_err("No transitions");
+    for (i = 0; i < tr_count; i++) {
+        keylen = snprintf(key, sizeof(key), "log%d-old-state", i);
+        ret = dict_get_strn(dict, key, keylen, &old_state);
+        if (ret)
+            goto out;
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "log%d-event", i);
-                ret = dict_get_str (dict, key, &event);
-                if (ret)
-                        goto out;
+        keylen = snprintf(key, sizeof(key), "log%d-event", i);
+        ret = dict_get_strn(dict, key, keylen, &event);
+        if (ret)
+            goto out;
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "log%d-new-state", i);
-                ret = dict_get_str (dict, key, &new_state);
-                if (ret)
-                        goto out;
+        keylen = snprintf(key, sizeof(key), "log%d-new-state", i);
+        ret = dict_get_strn(dict, key, keylen, &new_state);
+        if (ret)
+            goto out;
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "log%d-time", i);
-                ret = dict_get_str (dict, key, &time);
-                if (ret)
-                        goto out;
-                cli_out ("Old State: [%s]\n"
-                         "New State: [%s]\n"
-                         "Event    : [%s]\n"
-                         "timestamp: [%s]\n", old_state, new_state, event, time);
-        }
+        keylen = snprintf(key, sizeof(key), "log%d-time", i);
+        ret = dict_get_strn(dict, key, keylen, &time);
+        if (ret)
+            goto out;
+        cli_out(
+            "Old State: [%s]\n"
+            "New State: [%s]\n"
+            "Event    : [%s]\n"
+            "timestamp: [%s]\n",
+            old_state, new_state, event, time);
+    }
 
-        ret = rsp.op_ret;
+    ret = rsp.op_ret;
 
 out:
-        cli_cmd_broadcast_response (ret);
-        return ret;
+    cli_cmd_broadcast_response(ret);
+    if (dict) {
+        dict_unref(dict);
+    }
+    gf_free_xdr_fsm_log_rsp(rsp);
+
+    return ret;
 }
 
-int32_t
-gf_cli_fsm_log (call_frame_t *frame, xlator_t *this, void *data)
+static int32_t
+gf_cli_fsm_log(call_frame_t *frame, xlator_t *this, void *data)
 {
-        int                        ret = -1;
-        gf1_cli_fsm_log_req        req = {0,};
-
-        GF_ASSERT (frame);
-        GF_ASSERT (this);
-        GF_ASSERT (data);
-
-        if (!frame || !this || !data)
-                goto out;
-        req.name = data;
-        ret = cli_cmd_submit (NULL, &req, frame, cli_rpc_prog,
-                              GLUSTER_CLI_FSM_LOG, NULL,
-                              this, gf_cli_fsm_log_cbk,
-                              (xdrproc_t) xdr_gf1_cli_fsm_log_req);
+    int ret = -1;
+    gf1_cli_fsm_log_req req = {
+        0,
+    };
+
+    GF_ASSERT(frame);
+    GF_ASSERT(this);
+    GF_ASSERT(data);
+
+    if (!frame || !this || !data)
+        goto out;
+    req.name = data;
+    ret = cli_cmd_submit(NULL, &req, frame, cli_rpc_prog, GLUSTER_CLI_FSM_LOG,
+                         NULL, this, gf_cli_fsm_log_cbk,
+                         (xdrproc_t)xdr_gf1_cli_fsm_log_req);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
 
-        return ret;
+    return ret;
 }
 
-int
-gf_cli_gsync_config_command (dict_t *dict)
-{
-        runner_t runner     = {0,};
-        char *subop         = NULL;
-        char *gwd           = NULL;
-        char *slave         = NULL;
-        char *confpath      = NULL;
-        char *master        = NULL;
-        char *op_name       = NULL;
-        int   ret           = -1;
-        char  conf_path[PATH_MAX] = "";
-
-        if (dict_get_str (dict, "subop", &subop) != 0)
-                return -1;
-
-        if (strcmp (subop, "get") != 0 && strcmp (subop, "get-all") != 0) {
-                cli_out (GEOREP" config updated successfully");
-                return 0;
-        }
-
-        if (dict_get_str (dict, "glusterd_workdir", &gwd) != 0 ||
-            dict_get_str (dict, "slave", &slave) != 0)
-                return -1;
-
-        if (dict_get_str (dict, "master", &master) != 0)
-                master = NULL;
-        if (dict_get_str (dict, "op_name", &op_name) != 0)
-                op_name = NULL;
-
-        ret = dict_get_str (dict, "conf_path", &confpath);
-        if (!confpath) {
-                ret = snprintf (conf_path, sizeof(conf_path) - 1,
-                                "%s/"GEOREP"/gsyncd_template.conf", gwd);
-                conf_path[ret] = '\0';
-                confpath = conf_path;
-        }
-
-        runinit (&runner);
-        runner_add_args (&runner, GSYNCD_PREFIX"/gsyncd", "-c", NULL);
-        runner_argprintf (&runner, "%s", confpath);
-        runner_argprintf (&runner, "--iprefix=%s", DATADIR);
-        if (master)
-                runner_argprintf (&runner, ":%s", master);
-        runner_add_arg (&runner, slave);
-        runner_argprintf (&runner, "--config-%s", subop);
-        if (op_name)
-                runner_add_arg (&runner, op_name);
-
-        return runner_run (&runner);
-}
-
-char*
-get_struct_variable (int mem_num, gf_gsync_status_t *sts_val)
-{
-        switch (mem_num) {
-        case 0:  return (sts_val->node);
-        case 1:  return (sts_val->master);
-        case 2:  return (sts_val->brick);
-        case 3:  return (sts_val->slave_node);
-        case 4:  return (sts_val->worker_status);
-        case 5:  return (sts_val->checkpoint_status);
-        case 6:  return (sts_val->crawl_status);
-        case 7:  return (sts_val->files_syncd);
-        case 8:  return (sts_val->files_remaining);
-        case 9:  return (sts_val->bytes_remaining);
-        case 10: return (sts_val->purges_remaining);
-        case 11: return (sts_val->total_files_skipped);
-        case 12: return (sts_val->brick_host_uuid);
-        case 13: return (sts_val->slavekey);
-        case 14: return (sts_val->session_slave);
-        default:
-                 goto out;
-        }
-
-out:
-        return NULL;
+static int
+gf_cli_gsync_config_command(dict_t *dict)
+{
+    runner_t runner = {
+        0,
+    };
+    char *subop = NULL;
+    char *gwd = NULL;
+    char *slave = NULL;
+    char *confpath = NULL;
+    char *master = NULL;
+    char *op_name = NULL;
+    int ret = -1;
+    char conf_path[PATH_MAX] = "";
+
+    if (dict_get_str_sizen(dict, "subop", &subop) != 0)
+        return -1;
+
+    if (strcmp(subop, "get") != 0 && strcmp(subop, "get-all") != 0) {
+        cli_out(GEOREP " config updated successfully");
+        return 0;
+    }
+
+    if (dict_get_str_sizen(dict, "glusterd_workdir", &gwd) != 0 ||
+        dict_get_str_sizen(dict, "slave", &slave) != 0)
+        return -1;
+
+    if (dict_get_str_sizen(dict, "master", &master) != 0)
+        master = NULL;
+    if (dict_get_str_sizen(dict, "op_name", &op_name) != 0)
+        op_name = NULL;
+
+    ret = dict_get_str_sizen(dict, "conf_path", &confpath);
+    if (ret || !confpath) {
+        ret = snprintf(conf_path, sizeof(conf_path) - 1,
+                       "%s/" GEOREP "/gsyncd_template.conf", gwd);
+        conf_path[ret] = '\0';
+        confpath = conf_path;
+    }
+
+    runinit(&runner);
+    runner_add_args(&runner, GSYNCD_PREFIX "/gsyncd", "-c", NULL);
+    runner_argprintf(&runner, "%s", confpath);
+    runner_argprintf(&runner, "--iprefix=%s", DATADIR);
+    if (master)
+        runner_argprintf(&runner, ":%s", master);
+    runner_add_arg(&runner, slave);
+    runner_argprintf(&runner, "--config-%s", subop);
+    if (op_name)
+        runner_add_arg(&runner, op_name);
+
+    return runner_run(&runner);
 }
 
-int
-gf_cli_print_status (char **title_values,
-                     gf_gsync_status_t **sts_vals,
-                     int *spacing, int gsync_count,
-                     int number_of_fields, int is_detail)
-{
-        int     i                        = 0;
-        int     j                        = 0;
-        int     ret                      = 0;
-        int     status_fields            = 6; /* Indexed at 0 */
-        int     total_spacing            = 0;
-        char  **output_values            = NULL;
-        char   *tmp                      = NULL;
-        char   *hyphens                  = NULL;
-
-        /* calculating spacing for hyphens */
-        for (i = 0; i < number_of_fields; i++) {
-                /* Suppressing detail output for status */
-                if ((!is_detail) && (i > status_fields)) {
-                       /* Suppressing detailed output for
-                        * status */
-                        continue;
-                }
-                spacing[i] += 3; /* Adding extra space to
-                                    distinguish between fields */
-                total_spacing += spacing[i];
-        }
-        total_spacing += 4; /* For the spacing between the fields */
-
-        /* char pointers for each field */
-        output_values = GF_CALLOC (number_of_fields, sizeof (char *),
-                                   gf_common_mt_char);
-        if (!output_values) {
-                ret = -1;
-                goto out;
-        }
-        for (i = 0; i < number_of_fields; i++) {
-                output_values[i] = GF_CALLOC (spacing[i] + 1, sizeof (char),
-                                              gf_common_mt_char);
-                if (!output_values[i]) {
-                        ret = -1;
-                        goto out;
-                }
-        }
-
-        hyphens = GF_CALLOC (total_spacing + 1, sizeof (char),
-                             gf_common_mt_char);
-        if (!hyphens) {
+static int
+gf_cli_print_status(char **title_values, gf_gsync_status_t **sts_vals,
+                    int *spacing, int gsync_count, int number_of_fields,
+                    int is_detail)
+{
+    int i = 0;
+    int j = 0;
+    int ret = 0;
+    int status_fields = 8; /* Indexed at 0 */
+    int total_spacing = 0;
+    char **output_values = NULL;
+    char *tmp = NULL;
+    char *hyphens = NULL;
+
+    /* calculating spacing for hyphens */
+    for (i = 0; i < number_of_fields; i++) {
+        /* Suppressing detail output for status */
+        if ((!is_detail) && (i > status_fields)) {
+            /* Suppressing detailed output for
+             * status */
+            continue;
+        }
+        spacing[i] += 3; /* Adding extra space to
+                            distinguish between fields */
+        total_spacing += spacing[i];
+    }
+    total_spacing += 4; /* For the spacing between the fields */
+
+    /* char pointers for each field */
+    output_values = GF_MALLOC(number_of_fields * sizeof(char *),
+                              gf_common_mt_char);
+    if (!output_values) {
+        ret = -1;
+        goto out;
+    }
+    for (i = 0; i < number_of_fields; i++) {
+        output_values[i] = GF_CALLOC(spacing[i] + 1, sizeof(char),
+                                     gf_common_mt_char);
+        if (!output_values[i]) {
+            ret = -1;
+            goto out;
+        }
+    }
+
+    cli_out(" ");
+
+    /* setting the title "NODE", "MASTER", etc. from title_values[]
+       and printing the same */
+    for (j = 0; j < number_of_fields; j++) {
+        if ((!is_detail) && (j > status_fields)) {
+            /* Suppressing detailed output for
+             * status */
+            output_values[j][0] = '\0';
+            continue;
+        }
+        memset(output_values[j], ' ', spacing[j]);
+        memcpy(output_values[j], title_values[j], strlen(title_values[j]));
+        output_values[j][spacing[j]] = '\0';
+    }
+    cli_out("%s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s", output_values[0],
+            output_values[1], output_values[2], output_values[3],
+            output_values[4], output_values[5], output_values[6],
+            output_values[7], output_values[8], output_values[9],
+            output_values[10], output_values[11], output_values[12],
+            output_values[13], output_values[14], output_values[15]);
+
+    hyphens = GF_MALLOC((total_spacing + 1) * sizeof(char), gf_common_mt_char);
+    if (!hyphens) {
+        ret = -1;
+        goto out;
+    }
+
+    /* setting and printing the hyphens */
+    memset(hyphens, '-', total_spacing);
+    hyphens[total_spacing] = '\0';
+    cli_out("%s", hyphens);
+
+    for (i = 0; i < gsync_count; i++) {
+        for (j = 0; j < number_of_fields; j++) {
+            if ((!is_detail) && (j > status_fields)) {
+                /* Suppressing detailed output for
+                 * status */
+                output_values[j][0] = '\0';
+                continue;
+            }
+            tmp = get_struct_variable(j, sts_vals[i]);
+            if (!tmp) {
+                gf_log("", GF_LOG_ERROR, "struct member empty.");
                 ret = -1;
                 goto out;
+            }
+            memset(output_values[j], ' ', spacing[j]);
+            memcpy(output_values[j], tmp, strlen(tmp));
+            output_values[j][spacing[j]] = '\0';
         }
 
-        cli_out (" ");
-
-        /* setting the title "NODE", "MASTER", etc. from title_values[]
-           and printing the same */
-        for (j = 0; j < number_of_fields; j++) {
-                if ((!is_detail) && (j > status_fields)) {
-                       /* Suppressing detailed output for
-                        * status */
-                       output_values[j][0] = '\0';
-                       continue;
-                }
-                memset (output_values[j], ' ', spacing[j]);
-                memcpy (output_values[j], title_values[j],
-                        strlen(title_values[j]));
-                output_values[j][spacing[j]] = '\0';
-        }
-        cli_out ("%s %s %s %s %s %s %s %s %s %s %s %s",
-                 output_values[0], output_values[1],
-                 output_values[2], output_values[3],
-                 output_values[4], output_values[5],
-                 output_values[6], output_values[7],
-                 output_values[8], output_values[9],
-                 output_values[10], output_values[11]);
-
-        /* setting and printing the hyphens */
-        memset (hyphens, '-', total_spacing);
-        hyphens[total_spacing] = '\0';
-        cli_out ("%s", hyphens);
-
-        for (i = 0; i < gsync_count; i++) {
-                for (j = 0; j < number_of_fields; j++) {
-                        if ((!is_detail) && (j > status_fields)) {
-                                /* Suppressing detailed output for
-                                 * status */
-                                output_values[j][0] = '\0';
-                                continue;
-                        }
-                        tmp = get_struct_variable(j, sts_vals[i]);
-                        if (!tmp) {
-                                gf_log ("", GF_LOG_ERROR,
-                                        "struct member empty.");
-                                ret = -1;
-                                goto out;
-                        }
-                        memset (output_values[j], ' ', spacing[j]);
-                        memcpy (output_values[j], tmp, strlen (tmp));
-                        output_values[j][spacing[j]] = '\0';
-                }
-
-                cli_out ("%s %s %s %s %s %s %s %s %s %s %s %s",
-                         output_values[0], output_values[1],
-                         output_values[2], output_values[3],
-                         output_values[4], output_values[5],
-                         output_values[6], output_values[7],
-                         output_values[8], output_values[9],
-                         output_values[10], output_values[11]);
-        }
+        cli_out("%s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s",
+                output_values[0], output_values[1], output_values[2],
+                output_values[3], output_values[4], output_values[5],
+                output_values[6], output_values[7], output_values[8],
+                output_values[9], output_values[10], output_values[11],
+                output_values[12], output_values[13], output_values[14],
+                output_values[15]);
+    }
 
 out:
-        if (output_values) {
-                for (i = 0; i < number_of_fields; i++) {
-                        if (output_values[i])
-                                GF_FREE (output_values[i]);
-                }
-                GF_FREE (output_values);
+    if (output_values) {
+        for (i = 0; i < number_of_fields; i++) {
+            if (output_values[i])
+                GF_FREE(output_values[i]);
         }
+        GF_FREE(output_values);
+    }
 
-        if (hyphens)
-                GF_FREE (hyphens);
+    if (hyphens)
+        GF_FREE(hyphens);
 
-        return ret;
+    return ret;
 }
 
 int
-gf_cli_read_status_data (dict_t *dict,
-                         gf_gsync_status_t **sts_vals,
-                         int *spacing, int gsync_count,
-                         int number_of_fields)
-{
-        char   *tmp                    = NULL;
-        char    sts_val_name[PATH_MAX] = "";
-        int     ret                    = 0;
-        int     i                      = 0;
-        int     j                      = 0;
-
-        /* Storing per node status info in each object */
-        for (i = 0; i < gsync_count; i++) {
-                snprintf (sts_val_name, sizeof(sts_val_name), "status_value%d", i);
-
-                /* Fetching the values from dict, and calculating
-                   the max length for each field */
-                ret = dict_get_bin (dict, sts_val_name, (void **)&(sts_vals[i]));
-                if (ret)
-                        goto out;
-
-                for (j = 0; j < number_of_fields; j++) {
-                        tmp = get_struct_variable(j, sts_vals[i]);
-                        if (!tmp) {
-                                gf_log ("", GF_LOG_ERROR,
-                                        "struct member empty.");
-                                ret = -1;
-                                goto out;
-                        }
-                        if (strlen (tmp) > spacing[j])
-                                spacing[j] = strlen (tmp);
-                }
-        }
+gf_gsync_status_t_comparator(const void *p, const void *q)
+{
+    char *slavekey1 = NULL;
+    char *slavekey2 = NULL;
 
-out:
-        return ret;
-}
+    slavekey1 = get_struct_variable(20, (*(gf_gsync_status_t **)p));
+    slavekey2 = get_struct_variable(20, (*(gf_gsync_status_t **)q));
+    if (!slavekey1 || !slavekey2) {
+        gf_log("cli", GF_LOG_ERROR, "struct member empty.");
+        return 0;
+    }
 
-int
-gf_cli_gsync_status_output (dict_t *dict, gf_boolean_t is_detail)
-{
-        int                     gsync_count    = 0;
-        int                     i              = 0;
-        int                     ret            = 0;
-        int                     spacing[13]    = {0};
-        int                     num_of_fields  = 12;
-        char                    errmsg[1024]   = "";
-        char                   *master         = NULL;
-        char                   *slave          = NULL;
-        char                   *title_values[] = {"MASTER NODE", "MASTER VOL",
-                                                  "MASTER BRICK", "SLAVE",
-                                                  "STATUS", "CHECKPOINT STATUS",
-                                                  "CRAWL STATUS", "FILES SYNCD",
-                                                  "FILES PENDING", "BYTES PENDING",
-                                                  "DELETES PENDING", "FILES SKIPPED"};
-        gf_gsync_status_t     **sts_vals       = NULL;
-
-        /* Checks if any session is active or not */
-        ret = dict_get_int32 (dict, "gsync-count", &gsync_count);
-        if (ret) {
-                ret = dict_get_str (dict, "master", &master);
-
-                ret = dict_get_str (dict, "slave", &slave);
-
-                if (master) {
-                        if (slave)
-                                snprintf (errmsg, sizeof(errmsg), "No active "
-                                          "geo-replication sessions between %s"
-                                          " and %s", master, slave);
-                        else
-                                snprintf (errmsg, sizeof(errmsg), "No active "
-                                          "geo-replication sessions for %s",
-                                          master);
-                } else
-                        snprintf (errmsg, sizeof(errmsg), "No active "
-                                  "geo-replication sessions");
-
-                gf_log ("cli", GF_LOG_INFO, "%s", errmsg);
-                cli_out ("%s", errmsg);
-                ret = 0;
-                goto out;
-        }
+    return strcmp(slavekey1, slavekey2);
+}
 
-        for (i = 0; i < num_of_fields; i++)
-                spacing[i] = strlen(title_values[i]);
+static int
+gf_cli_read_status_data(dict_t *dict, gf_gsync_status_t **sts_vals,
+                        int *spacing, int gsync_count, int number_of_fields)
+{
+    char *tmp = NULL;
+    char sts_val_name[PATH_MAX] = "";
+    int ret = 0;
+    int i = 0;
+    int j = 0;
+
+    /* Storing per node status info in each object */
+    for (i = 0; i < gsync_count; i++) {
+        snprintf(sts_val_name, sizeof(sts_val_name), "status_value%d", i);
+
+        /* Fetching the values from dict, and calculating
+           the max length for each field */
+        ret = dict_get_bin(dict, sts_val_name, (void **)&(sts_vals[i]));
+        if (ret)
+            goto out;
 
-        /* gsync_count = number of nodes reporting output.
-           each sts_val object will store output of each
-           node */
-        sts_vals = GF_CALLOC (gsync_count, sizeof (gf_gsync_status_t *),
-                              gf_common_mt_char);
-        if (!sts_vals) {
+        for (j = 0; j < number_of_fields; j++) {
+            tmp = get_struct_variable(j, sts_vals[i]);
+            if (!tmp) {
+                gf_log("", GF_LOG_ERROR, "struct member empty.");
                 ret = -1;
                 goto out;
+            }
+            if (strlen(tmp) > spacing[j])
+                spacing[j] = strlen(tmp);
         }
-        for (i = 0; i < gsync_count; i++) {
-                sts_vals[i] = GF_CALLOC (1, sizeof (gf_gsync_status_t),
-                                         gf_common_mt_char);
-                if (!sts_vals[i]) {
-                        ret = -1;
-                        goto out;
-                }
-        }
+    }
 
-        ret = gf_cli_read_status_data (dict, sts_vals, spacing,
-                                       gsync_count, num_of_fields);
-        if (ret) {
-                gf_log ("", GF_LOG_ERROR, "Unable to read status data");
-                goto out;
-        }
+    /* Sort based on Session Slave */
+    qsort(sts_vals, gsync_count, sizeof(gf_gsync_status_t *),
+          gf_gsync_status_t_comparator);
 
-        ret = gf_cli_print_status (title_values, sts_vals, spacing, gsync_count,
-                                   num_of_fields, is_detail);
-        if (ret) {
-                gf_log ("", GF_LOG_ERROR, "Unable to print status output");
-                goto out;
-        }
+out:
+    return ret;
+}
+
+static int
+gf_cli_gsync_status_output(dict_t *dict, gf_boolean_t is_detail)
+{
+    int gsync_count = 0;
+    int i = 0;
+    int ret = 0;
+    int spacing[16] = {0};
+    int num_of_fields = 16;
+    char errmsg[1024] = "";
+    char *master = NULL;
+    char *slave = NULL;
+    static char *title_values[] = {"MASTER NODE",
+                                   "MASTER VOL",
+                                   "MASTER BRICK",
+                                   "SLAVE USER",
+                                   "SLAVE",
+                                   "SLAVE NODE",
+                                   "STATUS",
+                                   "CRAWL STATUS",
+                                   "LAST_SYNCED",
+                                   "ENTRY",
+                                   "DATA",
+                                   "META",
+                                   "FAILURES",
+                                   "CHECKPOINT TIME",
+                                   "CHECKPOINT COMPLETED",
+                                   "CHECKPOINT COMPLETION TIME"};
+    gf_gsync_status_t **sts_vals = NULL;
+
+    /* Checks if any session is active or not */
+    ret = dict_get_int32_sizen(dict, "gsync-count", &gsync_count);
+    if (ret) {
+        ret = dict_get_str_sizen(dict, "master", &master);
+
+        ret = dict_get_str_sizen(dict, "slave", &slave);
+
+        if (master) {
+            if (slave)
+                snprintf(errmsg, sizeof(errmsg),
+                         "No active geo-replication sessions between %s"
+                         " and %s",
+                         master, slave);
+            else
+                snprintf(errmsg, sizeof(errmsg),
+                         "No active geo-replication sessions for %s", master);
+        } else
+            snprintf(errmsg, sizeof(errmsg),
+                     "No active geo-replication sessions");
+
+        gf_log("cli", GF_LOG_INFO, "%s", errmsg);
+        cli_out("%s", errmsg);
+        ret = -1;
+        goto out;
+    }
+
+    for (i = 0; i < num_of_fields; i++)
+        spacing[i] = strlen(title_values[i]);
+
+    /* gsync_count = number of nodes reporting output.
+       each sts_val object will store output of each
+       node */
+    sts_vals = GF_MALLOC(gsync_count * sizeof(gf_gsync_status_t *),
+                         gf_common_mt_char);
+    if (!sts_vals) {
+        ret = -1;
+        goto out;
+    }
+
+    ret = gf_cli_read_status_data(dict, sts_vals, spacing, gsync_count,
+                                  num_of_fields);
+    if (ret) {
+        gf_log("", GF_LOG_ERROR, "Unable to read status data");
+        goto out;
+    }
+
+    ret = gf_cli_print_status(title_values, sts_vals, spacing, gsync_count,
+                              num_of_fields, is_detail);
+    if (ret) {
+        gf_log("", GF_LOG_ERROR, "Unable to print status output");
+        goto out;
+    }
 
 out:
-        if (sts_vals)
-                GF_FREE (sts_vals);
+    if (sts_vals)
+        GF_FREE(sts_vals);
 
-        return ret;
+    return ret;
 }
 
 static int32_t
-write_contents_to_common_pem_file (dict_t *dict, int output_count)
-{
-        char            *workdir                   = NULL;
-        char             common_pem_file[PATH_MAX] = "";
-        char            *output                    = NULL;
-        char             output_name[PATH_MAX]     = "";
-        int              bytes_written             = 0;
-        int              fd                        = -1;
-        int              ret                       = -1;
-        int              i                         = -1;
-
-        ret = dict_get_str (dict, "glusterd_workdir", &workdir);
-        if (ret || !workdir) {
-                gf_log ("", GF_LOG_ERROR, "Unable to fetch workdir");
+write_contents_to_common_pem_file(dict_t *dict, int output_count)
+{
+    char *workdir = NULL;
+    char common_pem_file[PATH_MAX] = "";
+    char *output = NULL;
+    char output_name[32] = "";
+    int bytes_written = 0;
+    int fd = -1;
+    int ret = -1;
+    int i = -1;
+
+    ret = dict_get_str_sizen(dict, "glusterd_workdir", &workdir);
+    if (ret || !workdir) {
+        gf_log("", GF_LOG_ERROR, "Unable to fetch workdir");
+        ret = -1;
+        goto out;
+    }
+
+    snprintf(common_pem_file, sizeof(common_pem_file),
+             "%s/geo-replication/common_secret.pem.pub", workdir);
+
+    sys_unlink(common_pem_file);
+
+    fd = open(common_pem_file, O_WRONLY | O_CREAT, 0600);
+    if (fd == -1) {
+        gf_log("", GF_LOG_ERROR, "Failed to open %s Error : %s",
+               common_pem_file, strerror(errno));
+        ret = -1;
+        goto out;
+    }
+
+    for (i = 1; i <= output_count; i++) {
+        ret = snprintf(output_name, sizeof(output_name), "output_%d", i);
+        ret = dict_get_strn(dict, output_name, ret, &output);
+        if (ret) {
+            gf_log("", GF_LOG_ERROR, "Failed to get %s.", output_name);
+            cli_out("Unable to fetch output.");
+        }
+        if (output) {
+            bytes_written = sys_write(fd, output, strlen(output));
+            if (bytes_written != strlen(output)) {
+                gf_log("", GF_LOG_ERROR, "Failed to write to %s",
+                       common_pem_file);
                 ret = -1;
                 goto out;
-        }
-
-        snprintf (common_pem_file, sizeof(common_pem_file),
-                  "%s/geo-replication/common_secret.pem.pub",
-                  workdir);
-
-        unlink (common_pem_file);
-
-        fd = open (common_pem_file, O_WRONLY | O_CREAT, 0600);
-        if (fd == -1) {
-                gf_log ("", GF_LOG_ERROR, "Failed to open %s"
-                        " Error : %s", common_pem_file,
-                        strerror (errno));
+            }
+            /* Adding the new line character */
+            bytes_written = sys_write(fd, "\n", 1);
+            if (bytes_written != 1) {
+                gf_log("", GF_LOG_ERROR, "Failed to add new line char");
                 ret = -1;
                 goto out;
+            }
+            output = NULL;
         }
+    }
 
-        for (i = 1; i <= output_count; i++) {
-                memset (output_name, '\0', sizeof (output_name));
-                snprintf (output_name, sizeof (output_name),
-                          "output_%d", i);
-                ret = dict_get_str (dict, output_name, &output);
-                if (ret) {
-                        gf_log ("", GF_LOG_ERROR, "Failed to get %s.",
-                                output_name);
-                        cli_out ("Unable to fetch output.");
-                }
-                if (output) {
-                        bytes_written = write (fd, output, strlen(output));
-                        if (bytes_written != strlen(output)) {
-                                gf_log ("", GF_LOG_ERROR, "Failed to write "
-                                        "to %s", common_pem_file);
-                                ret = -1;
-                                goto out;
-                        }
-                        /* Adding the new line character */
-                        bytes_written = write (fd, "\n", strlen("\n"));
-                        if (bytes_written != strlen("\n")) {
-                                gf_log ("", GF_LOG_ERROR,
-                                        "Failed to add new line char");
-                                ret = -1;
-                                goto out;
-                        }
-                        output = NULL;
-                }
-        }
-
-        cli_out ("Common secret pub file present at %s", common_pem_file);
-        ret = 0;
+    cli_out("Common secret pub file present at %s", common_pem_file);
+    ret = 0;
 out:
-        if (fd)
-                close (fd);
+    if (fd >= 0)
+        sys_close(fd);
 
-        gf_log ("", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("", GF_LOG_DEBUG, RETURNING, ret);
+    return ret;
 }
 
-int
-gf_cli_sys_exec_cbk (struct rpc_req *req, struct iovec *iov,
-                     int count, void *myframe)
-{
-        int                     ret     = -1;
-        int                     output_count     = -1;
-        int                     i     = -1;
-        char                   *output  = NULL;
-        char                   *command = NULL;
-        char                    output_name[PATH_MAX] = "";
-        gf_cli_rsp              rsp     = {0, };
-        dict_t                  *dict   = NULL;
-        call_frame_t            *frame  = NULL;
-
-        if (req->rpc_status == -1) {
-                ret = -1;
-                goto out;
-        }
-
-        frame = myframe;
-
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-       }
-
-        dict = dict_new ();
-
-        if (!dict) {
-                ret = -1;
-                goto out;
-        }
-
-        ret = dict_unserialize (rsp.dict.dict_val, rsp.dict.dict_len, &dict);
+static int
+gf_cli_sys_exec_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                    void *myframe)
+{
+    int ret = -1;
+    int output_count = -1;
+    int i = -1;
+    char *output = NULL;
+    char *command = NULL;
+    char output_name[32] = "";
+    gf_cli_rsp rsp = {
+        0,
+    };
+    dict_t *dict = NULL;
+
+    GF_ASSERT(myframe);
+
+    if (req->rpc_status == -1) {
+        goto out;
+    }
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    dict = dict_new();
+
+    if (!dict) {
+        ret = -1;
+        goto out;
+    }
+
+    ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &dict);
+
+    if (ret)
+        goto out;
+
+    if (rsp.op_ret) {
+        cli_err("%s", rsp.op_errstr ? rsp.op_errstr : "Command failed.");
+        ret = rsp.op_ret;
+        goto out;
+    }
 
-        if (ret)
-                goto out;
+    ret = dict_get_int32_sizen(dict, "output_count", &output_count);
+    if (ret) {
+        cli_out("Command executed successfully.");
+        ret = 0;
+        goto out;
+    }
 
-        if (rsp.op_ret) {
-                cli_err ("%s", rsp.op_errstr ? rsp.op_errstr :
-                         "Command failed.");
-                ret = rsp.op_ret;
-                goto out;
-        }
+    ret = dict_get_str_sizen(dict, "command", &command);
+    if (ret) {
+        gf_log("", GF_LOG_ERROR, "Unable to get command from dict");
+        goto out;
+    }
 
-        ret = dict_get_int32 (dict, "output_count", &output_count);
-        if (ret) {
-                cli_out ("Command executed successfully.");
-                ret = 0;
-                goto out;
-        }
+    if (!strcmp(command, "gsec_create")) {
+        ret = write_contents_to_common_pem_file(dict, output_count);
+        if (!ret)
+            goto out;
+    }
 
-        ret = dict_get_str (dict, "command", &command);
+    for (i = 1; i <= output_count; i++) {
+        ret = snprintf(output_name, sizeof(output_name), "output_%d", i);
+        ret = dict_get_strn(dict, output_name, ret, &output);
         if (ret) {
-                gf_log ("", GF_LOG_ERROR,
-                        "Unable to get command from dict");
-                goto out;
+            gf_log("", GF_LOG_ERROR, "Failed to get %s.", output_name);
+            cli_out("Unable to fetch output.");
         }
-
-        if (!strcmp (command, "gsec_create")) {
-                ret = write_contents_to_common_pem_file (dict, output_count);
-                if (!ret)
-                        goto out;
+        if (output) {
+            cli_out("%s", output);
+            output = NULL;
         }
+    }
 
-        for (i = 1; i <= output_count; i++) {
-                memset (output_name, '\0', sizeof (output_name));
-                snprintf (output_name, sizeof (output_name),
-                          "output_%d", i);
-                ret = dict_get_str (dict, output_name, &output);
-                if (ret) {
-                        gf_log ("", GF_LOG_ERROR, "Failed to get %s.",
-                                output_name);
-                        cli_out ("Unable to fetch output.");
-                }
-                if (output) {
-                        cli_out ("%s", output);
-                        output = NULL;
-                }
-        }
-
-        ret = 0;
+    ret = 0;
 out:
-        if (dict)
-                dict_unref (dict);
-        cli_cmd_broadcast_response (ret);
-
-        free (rsp.dict.dict_val);
-
-        return ret;
+    if (dict)
+        dict_unref(dict);
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
+    return ret;
 }
 
-int
-gf_cli_copy_file_cbk (struct rpc_req *req, struct iovec *iov,
-                      int count, void *myframe)
+static int
+gf_cli_copy_file_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                     void *myframe)
 {
-        int                     ret     = -1;
-        gf_cli_rsp              rsp     = {0, };
-        dict_t                  *dict   = NULL;
-        call_frame_t            *frame  = NULL;
+    int ret = -1;
+    gf_cli_rsp rsp = {
+        0,
+    };
+    dict_t *dict = NULL;
 
-        if (req->rpc_status == -1) {
-                ret = -1;
-                goto out;
-        }
+    GF_ASSERT(myframe);
 
-        frame = myframe;
+    if (req->rpc_status == -1) {
+        goto out;
+    }
 
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-       }
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
 
-        dict = dict_new ();
+    dict = dict_new();
 
-        if (!dict) {
-                ret = -1;
-                goto out;
-        }
+    if (!dict) {
+        ret = -1;
+        goto out;
+    }
 
-        ret = dict_unserialize (rsp.dict.dict_val, rsp.dict.dict_len, &dict);
+    ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &dict);
 
-       if (ret)
-                goto out;
+    if (ret)
+        goto out;
 
-        if (rsp.op_ret) {
-                cli_err ("%s", rsp.op_errstr ? rsp.op_errstr :
-                         "Copy unsuccessful");
-                ret = rsp.op_ret;
-                goto out;
-        }
+    if (rsp.op_ret) {
+        cli_err("%s", rsp.op_errstr ? rsp.op_errstr : "Copy unsuccessful");
+        ret = rsp.op_ret;
+        goto out;
+    }
 
-        cli_out ("Successfully copied file.");
+    cli_out("Successfully copied file.");
 
 out:
-        if (dict)
-                dict_unref (dict);
-        cli_cmd_broadcast_response (ret);
-
-        free (rsp.dict.dict_val);
-
-        return ret;
+    if (dict)
+        dict_unref(dict);
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
+    return ret;
 }
 
-int
-gf_cli_gsync_set_cbk (struct rpc_req *req, struct iovec *iov,
-                         int count, void *myframe)
+static int
+gf_cli_gsync_set_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                     void *myframe)
 {
-        int                     ret     = -1;
-        gf_cli_rsp              rsp     = {0, };
-        dict_t                  *dict   = NULL;
-        char                    *gsync_status = NULL;
-        char                    *master = NULL;
-        char                    *slave  = NULL;
-        int32_t                  type   = 0;
-        call_frame_t            *frame  = NULL;
-        gf_boolean_t             status_detail = _gf_false;
-
-
-        if (req->rpc_status == -1) {
-                ret = -1;
-                goto out;
-        }
-
-        frame = myframe;
-
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
-
-        dict = dict_new ();
-
-        if (!dict) {
-                ret = -1;
-                goto out;
-        }
-
-        ret = dict_unserialize (rsp.dict.dict_val, rsp.dict.dict_len, &dict);
-
+    int ret = -1;
+    gf_cli_rsp rsp = {
+        0,
+    };
+    dict_t *dict = NULL;
+    char *gsync_status = NULL;
+    char *master = NULL;
+    char *slave = NULL;
+    int32_t type = 0;
+    gf_boolean_t status_detail = _gf_false;
+
+    GF_ASSERT(myframe);
+
+    if (req->rpc_status == -1) {
+        goto out;
+    }
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    dict = dict_new();
+
+    if (!dict) {
+        ret = -1;
+        goto out;
+    }
+
+    ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &dict);
+
+    if (ret)
+        goto out;
+
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_vol_gsync(dict, rsp.op_ret, rsp.op_errno,
+                                       rsp.op_errstr);
         if (ret)
-                goto out;
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_vol_gsync (dict, rsp.op_ret, rsp.op_errno,
-                                                rsp.op_errstr);
-                if (ret)
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
-                goto out;
-        }
-
-        if (rsp.op_ret) {
-                cli_err ("%s", rsp.op_errstr ? rsp.op_errstr :
-                         GEOREP" command unsuccessful");
-                ret = rsp.op_ret;
-                goto out;
-        }
-
-        ret = dict_get_str (dict, "gsync-status", &gsync_status);
-        if (!ret)
-                cli_out ("%s", gsync_status);
-        else
-                ret = 0;
-
-        ret = dict_get_int32 (dict, "type", &type);
-        if (ret) {
-                gf_log (frame->this->name, GF_LOG_ERROR, "failed to get type");
-                goto out;
-        }
-
-        switch (type) {
-                case GF_GSYNC_OPTION_TYPE_START:
-                case GF_GSYNC_OPTION_TYPE_STOP:
-                        if (dict_get_str (dict, "master", &master) != 0)
-                                master = "???";
-                        if (dict_get_str (dict, "slave", &slave) != 0)
-                                slave = "???";
-
-                        cli_out ("%s " GEOREP " session between %s & %s"
-                                 " has been successful",
-                                 type == GF_GSYNC_OPTION_TYPE_START ?
-                                  "Starting" : "Stopping",
-                                 master, slave);
-                break;
-
-                case GF_GSYNC_OPTION_TYPE_PAUSE:
-                case GF_GSYNC_OPTION_TYPE_RESUME:
-                        if (dict_get_str (dict, "master", &master) != 0)
-                                master = "???";
-                        if (dict_get_str (dict, "slave", &slave) != 0)
-                                slave = "???";
-
-                        cli_out ("%s " GEOREP " session between %s & %s"
-                                 " has been successful",
-                                 type == GF_GSYNC_OPTION_TYPE_PAUSE ?
-                                  "Pausing" : "Resuming",
-                                 master, slave);
-                break;
-
-                case GF_GSYNC_OPTION_TYPE_CONFIG:
-                        ret = gf_cli_gsync_config_command (dict);
-                break;
-
-                case GF_GSYNC_OPTION_TYPE_STATUS:
-                        status_detail = dict_get_str_boolean (dict,
-                                                              "status-detail",
-                                                              _gf_false);
-                        if (status_detail)
-                                ret = gf_cli_gsync_status_output (dict, status_detail);
-                        else
-                                ret = gf_cli_gsync_status_output (dict, status_detail);
-                break;
-
-                case GF_GSYNC_OPTION_TYPE_DELETE:
-                        if (dict_get_str (dict, "master", &master) != 0)
-                                master = "???";
-                        if (dict_get_str (dict, "slave", &slave) != 0)
-                                slave = "???";
-                        cli_out ("Deleting " GEOREP " session between %s & %s"
-                                 " has been successful", master, slave);
-                break;
-
-                case GF_GSYNC_OPTION_TYPE_CREATE:
-                        if (dict_get_str (dict, "master", &master) != 0)
-                                master = "???";
-                        if (dict_get_str (dict, "slave", &slave) != 0)
-                                slave = "???";
-                        cli_out ("Creating " GEOREP " session between %s & %s"
-                                 " has been successful", master, slave);
-                break;
+    if (rsp.op_ret) {
+        cli_err("%s",
+                rsp.op_errstr ? rsp.op_errstr : GEOREP " command unsuccessful");
+        ret = rsp.op_ret;
+        goto out;
+    }
+
+    ret = dict_get_str_sizen(dict, "gsync-status", &gsync_status);
+    if (!ret)
+        cli_out("%s", gsync_status);
+
+    ret = dict_get_int32_sizen(dict, "type", &type);
+    if (ret) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               "failed to get type");
+        goto out;
+    }
+
+    switch (type) {
+        case GF_GSYNC_OPTION_TYPE_START:
+        case GF_GSYNC_OPTION_TYPE_STOP:
+            if (dict_get_str_sizen(dict, "master", &master) != 0)
+                master = "???";
+            if (dict_get_str_sizen(dict, "slave", &slave) != 0)
+                slave = "???";
+
+            cli_out(
+                "%s " GEOREP " session between %s & %s has been successful",
+                type == GF_GSYNC_OPTION_TYPE_START ? "Starting" : "Stopping",
+                master, slave);
+            break;
+
+        case GF_GSYNC_OPTION_TYPE_PAUSE:
+        case GF_GSYNC_OPTION_TYPE_RESUME:
+            if (dict_get_str_sizen(dict, "master", &master) != 0)
+                master = "???";
+            if (dict_get_str_sizen(dict, "slave", &slave) != 0)
+                slave = "???";
+
+            cli_out("%s " GEOREP " session between %s & %s has been successful",
+                    type == GF_GSYNC_OPTION_TYPE_PAUSE ? "Pausing" : "Resuming",
+                    master, slave);
+            break;
+
+        case GF_GSYNC_OPTION_TYPE_CONFIG:
+            ret = gf_cli_gsync_config_command(dict);
+            break;
+
+        case GF_GSYNC_OPTION_TYPE_STATUS:
+            status_detail = dict_get_str_boolean(dict, "status-detail",
+                                                 _gf_false);
+            ret = gf_cli_gsync_status_output(dict, status_detail);
+            break;
+
+        case GF_GSYNC_OPTION_TYPE_DELETE:
+            if (dict_get_str_sizen(dict, "master", &master) != 0)
+                master = "???";
+            if (dict_get_str_sizen(dict, "slave", &slave) != 0)
+                slave = "???";
+            cli_out("Deleting " GEOREP
+                    " session between %s & %s has been successful",
+                    master, slave);
+            break;
+
+        case GF_GSYNC_OPTION_TYPE_CREATE:
+            if (dict_get_str_sizen(dict, "master", &master) != 0)
+                master = "???";
+            if (dict_get_str_sizen(dict, "slave", &slave) != 0)
+                slave = "???";
+            cli_out("Creating " GEOREP
+                    " session between %s & %s has been successful",
+                    master, slave);
+            break;
 
-                default:
-                        cli_out (GEOREP" command executed successfully");
-        }
+        default:
+            cli_out(GEOREP " command executed successfully");
+    }
 
 out:
-        if (dict)
-                dict_unref (dict);
-        cli_cmd_broadcast_response (ret);
-
-        free (rsp.dict.dict_val);
-
-        return ret;
+    if (dict)
+        dict_unref(dict);
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
+    return ret;
 }
 
-int32_t
-gf_cli_sys_exec (call_frame_t *frame, xlator_t *this, void *data)
+static int32_t
+gf_cli_sys_exec(call_frame_t *frame, xlator_t *this, void *data)
 {
-        int                      ret    = 0;
-        dict_t                  *dict   = NULL;
-        gf_cli_req               req = {{0,}};
-
+    int ret = 0;
+    dict_t *dict = NULL;
+    gf_cli_req req = {{
+        0,
+    }};
+
+    dict = data;
+
+    ret = cli_to_glusterd(&req, frame, gf_cli_sys_exec_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict, GLUSTER_CLI_SYS_EXEC,
+                          this, cli_rpc_prog, NULL);
+    if (ret)
         if (!frame || !this || !data) {
-                ret = -1;
-                gf_log ("cli", GF_LOG_ERROR, "Invalid data");
-                goto out;
+            ret = -1;
+            gf_log("cli", GF_LOG_ERROR, "Invalid data");
         }
 
-        dict = data;
-
-        ret = cli_to_glusterd (&req, frame, gf_cli_sys_exec_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, dict,
-                               GLUSTER_CLI_SYS_EXEC, this, cli_rpc_prog,
-                               NULL);
-out:
-        GF_FREE (req.dict.dict_val);
-        return ret;
+    GF_FREE(req.dict.dict_val);
+    return ret;
 }
 
-int32_t
-gf_cli_copy_file (call_frame_t *frame, xlator_t *this, void *data)
+static int32_t
+gf_cli_copy_file(call_frame_t *frame, xlator_t *this, void *data)
 {
-        int                      ret    = 0;
-        dict_t                  *dict   = NULL;
-        gf_cli_req               req = {{0,}};
-
+    int ret = 0;
+    dict_t *dict = NULL;
+    gf_cli_req req = {{
+        0,
+    }};
+
+    dict = data;
+
+    ret = cli_to_glusterd(&req, frame, gf_cli_copy_file_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict,
+                          GLUSTER_CLI_COPY_FILE, this, cli_rpc_prog, NULL);
+    if (ret)
         if (!frame || !this || !data) {
-                ret = -1;
-                gf_log ("cli", GF_LOG_ERROR, "Invalid data");
-                goto out;
+            ret = -1;
+            gf_log("cli", GF_LOG_ERROR, "Invalid data");
         }
 
-        dict = data;
-
-        ret = cli_to_glusterd (&req, frame, gf_cli_copy_file_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, dict,
-                               GLUSTER_CLI_COPY_FILE, this, cli_rpc_prog,
-                               NULL);
-out:
-        GF_FREE (req.dict.dict_val);
-        return ret;
+    GF_FREE(req.dict.dict_val);
+    return ret;
 }
 
-int32_t
-gf_cli_gsync_set (call_frame_t *frame, xlator_t *this,
-                     void *data)
+static int32_t
+gf_cli_gsync_set(call_frame_t *frame, xlator_t *this, void *data)
 {
-        int                      ret    = 0;
-        dict_t                  *dict   = NULL;
-        gf_cli_req               req = {{0,}};
+    int ret = 0;
+    dict_t *dict = NULL;
+    gf_cli_req req = {{
+        0,
+    }};
 
-        if (!frame || !this || !data) {
-                ret = -1;
-                goto out;
-        }
-
-        dict = data;
+    dict = data;
 
-        ret = cli_to_glusterd (&req, frame, gf_cli_gsync_set_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, dict,
-                               GLUSTER_CLI_GSYNC_SET, this, cli_rpc_prog,
-                               NULL);
+    ret = cli_to_glusterd(&req, frame, gf_cli_gsync_set_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict,
+                          GLUSTER_CLI_GSYNC_SET, this, cli_rpc_prog, NULL);
+    GF_FREE(req.dict.dict_val);
 
-out:
-        GF_FREE (req.dict.dict_val);
-
-        return ret;
+    return ret;
 }
 
-
 int
-cli_profile_info_percentage_cmp (void *a, void *b)
+cli_profile_info_percentage_cmp(void *a, void *b)
 {
-        cli_profile_info_t *ia = NULL;
-        cli_profile_info_t *ib = NULL;
-        int                ret = 0;
+    cli_profile_info_t *ia = NULL;
+    cli_profile_info_t *ib = NULL;
 
-        ia = a;
-        ib = b;
-        if (ia->percentage_avg_latency < ib->percentage_avg_latency)
-                ret = -1;
-        else if (ia->percentage_avg_latency > ib->percentage_avg_latency)
-                ret = 1;
-        else
-                ret = 0;
-        return ret;
-}
+    ia = a;
+    ib = b;
+    if (ia->percentage_avg_latency < ib->percentage_avg_latency)
+        return -1;
+    else if (ia->percentage_avg_latency > ib->percentage_avg_latency)
+        return 1;
 
+    return 0;
+}
 
-void
-cmd_profile_volume_brick_out (dict_t *dict, int count, int interval)
-{
-        char                    key[256] = {0};
-        int                     i = 0;
-        uint64_t                sec = 0;
-        uint64_t                r_count = 0;
-        uint64_t                w_count = 0;
-        uint64_t                rb_counts[32] = {0};
-        uint64_t                wb_counts[32] = {0};
-        cli_profile_info_t      profile_info[GF_FOP_MAXVALUE] = {{0}};
-        char                    output[128] = {0};
-        int                     per_line = 0;
-        char                    read_blocks[128] = {0};
-        char                    write_blocks[128] = {0};
-        int                     index = 0;
-        int                     is_header_printed = 0;
-        int                     ret = 0;
-        double                  total_percentage_latency = 0;
-
-        for (i = 0; i < 32; i++) {
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%d-%d-read-%d", count,
-                          interval, (1 << i));
-                ret = dict_get_uint64 (dict, key, &rb_counts[i]);
-        }
-
-        for (i = 0; i < 32; i++) {
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%d-%d-write-%d", count, interval,
-                          (1<<i));
-                ret = dict_get_uint64 (dict, key, &wb_counts[i]);
+static void
+cmd_profile_volume_brick_out(dict_t *dict, int count, int interval)
+{
+    char key[256] = {0};
+    int i = 0;
+    uint64_t sec = 0;
+    uint64_t r_count = 0;
+    uint64_t w_count = 0;
+    uint64_t rb_counts[32] = {0};
+    uint64_t wb_counts[32] = {0};
+    cli_profile_info_t profile_info[GF_FOP_MAXVALUE] = {{0}};
+    cli_profile_info_t upcall_info[GF_UPCALL_FLAGS_MAXVALUE] = {
+        {0},
+    };
+    char output[128] = {0};
+    int per_line = 0;
+    char read_blocks[128] = {0};
+    char write_blocks[128] = {0};
+    int index = 0;
+    int is_header_printed = 0;
+    int ret = 0;
+    double total_percentage_latency = 0;
+
+    for (i = 0; i < 32; i++) {
+        snprintf(key, sizeof(key), "%d-%d-read-%" PRIu32, count, interval,
+                 (1U << i));
+        ret = dict_get_uint64(dict, key, &rb_counts[i]);
+        if (ret) {
+            gf_log("cli", GF_LOG_DEBUG, "failed to get %s from dict", key);
         }
+    }
 
-        for (i = 0; i < GF_FOP_MAXVALUE; i++) {
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%d-%d-%d-hits", count,
-                          interval, i);
-                ret = dict_get_uint64 (dict, key, &profile_info[i].fop_hits);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%d-%d-%d-avglatency", count,
-                          interval, i);
-                ret = dict_get_double (dict, key, &profile_info[i].avg_latency);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%d-%d-%d-minlatency", count,
-                          interval, i);
-                ret = dict_get_double (dict, key, &profile_info[i].min_latency);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%d-%d-%d-maxlatency", count,
-                          interval, i);
-                ret = dict_get_double (dict, key, &profile_info[i].max_latency);
-                profile_info[i].fop_name = (char *)gf_fop_list[i];
-
-                total_percentage_latency +=
-                       (profile_info[i].fop_hits * profile_info[i].avg_latency);
-        }
-        if (total_percentage_latency) {
-                for (i = 0; i < GF_FOP_MAXVALUE; i++) {
-                        profile_info[i].percentage_avg_latency = 100 * (
-                     (profile_info[i].avg_latency* profile_info[i].fop_hits) /
-                                total_percentage_latency);
-                }
-                gf_array_insertionsort (profile_info, 1, GF_FOP_MAXVALUE - 1,
-                                    sizeof (cli_profile_info_t),
-                                    cli_profile_info_percentage_cmp);
+    for (i = 0; i < 32; i++) {
+        snprintf(key, sizeof(key), "%d-%d-write-%" PRIu32, count, interval,
+                 (1U << i));
+        ret = dict_get_uint64(dict, key, &wb_counts[i]);
+        if (ret) {
+            gf_log("cli", GF_LOG_DEBUG, "failed to get %s from dict", key);
         }
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%d-%d-duration", count, interval);
-        ret = dict_get_uint64 (dict, key, &sec);
+    }
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%d-%d-total-read", count, interval);
-        ret = dict_get_uint64 (dict, key, &r_count);
+    for (i = 0; i < GF_UPCALL_FLAGS_MAXVALUE; i++) {
+        snprintf(key, sizeof(key), "%d-%d-%d-upcall-hits", count, interval, i);
+        ret = dict_get_uint64(dict, key, &upcall_info[i].fop_hits);
+        if (ret) {
+            gf_log("cli", GF_LOG_DEBUG, "failed to get %s from dict", key);
+        }
+        upcall_info[i].fop_name = (char *)gf_upcall_list[i];
+    }
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%d-%d-total-write", count, interval);
-        ret = dict_get_uint64 (dict, key, &w_count);
+    for (i = 0; i < GF_FOP_MAXVALUE; i++) {
+        snprintf(key, sizeof(key), "%d-%d-%d-hits", count, interval, i);
+        ret = dict_get_uint64(dict, key, &profile_info[i].fop_hits);
+        if (ret) {
+            gf_log("cli", GF_LOG_DEBUG, "failed to get %s from dict", key);
+        }
 
-        if (ret == 0) {
+        snprintf(key, sizeof(key), "%d-%d-%d-avglatency", count, interval, i);
+        ret = dict_get_double(dict, key, &profile_info[i].avg_latency);
+        if (ret) {
+            gf_log("cli", GF_LOG_DEBUG, "failed to get %s from dict", key);
         }
 
-        if (interval == -1)
-                cli_out ("Cumulative Stats:");
-        else
-                cli_out ("Interval %d Stats:", interval);
-        snprintf (output, sizeof (output), "%14s", "Block Size:");
-        snprintf (read_blocks, sizeof (read_blocks), "%14s", "No. of Reads:");
-        snprintf (write_blocks, sizeof (write_blocks), "%14s", "No. of Writes:");
-        index = 14;
-        for (i = 0; i < 32; i++) {
-                if ((rb_counts[i] == 0) && (wb_counts[i] == 0))
-                        continue;
-                per_line++;
-                snprintf (output+index, sizeof (output)-index, "%19db+ ", (1<<i));
-                if (rb_counts[i]) {
-                        snprintf (read_blocks+index, sizeof (read_blocks)-index,
-                                  "%21"PRId64" ", rb_counts[i]);
-                } else {
-                        snprintf (read_blocks+index, sizeof (read_blocks)-index,
-                                  "%21s ", "0");
-                }
-                if (wb_counts[i]) {
-                        snprintf (write_blocks+index, sizeof (write_blocks)-index,
-                                  "%21"PRId64" ", wb_counts[i]);
-                } else {
-                        snprintf (write_blocks+index, sizeof (write_blocks)-index,
-                                  "%21s ", "0");
-                }
-                index += 22;
-                if (per_line == 3) {
-                        cli_out ("%s", output);
-                        cli_out ("%s", read_blocks);
-                        cli_out ("%s", write_blocks);
-                        cli_out (" ");
-                        per_line = 0;
-                        memset (output, 0, sizeof (output));
-                        memset (read_blocks, 0, sizeof (read_blocks));
-                        memset (write_blocks, 0, sizeof (write_blocks));
-                        snprintf (output, sizeof (output), "%14s", "Block Size:");
-                        snprintf (read_blocks, sizeof (read_blocks), "%14s",
-                                  "No. of Reads:");
-                        snprintf (write_blocks, sizeof (write_blocks), "%14s",
-                                  "No. of Writes:");
-                        index = 14;
-                }
+        snprintf(key, sizeof(key), "%d-%d-%d-minlatency", count, interval, i);
+        ret = dict_get_double(dict, key, &profile_info[i].min_latency);
+        if (ret) {
+            gf_log("cli", GF_LOG_DEBUG, "failed to get %s from dict", key);
         }
 
-        if (per_line != 0) {
-                cli_out ("%s", output);
-                cli_out ("%s", read_blocks);
-                cli_out ("%s", write_blocks);
+        snprintf(key, sizeof(key), "%d-%d-%d-maxlatency", count, interval, i);
+        ret = dict_get_double(dict, key, &profile_info[i].max_latency);
+        if (ret) {
+            gf_log("cli", GF_LOG_DEBUG, "failed to get %s from dict", key);
         }
+        profile_info[i].fop_name = (char *)gf_fop_list[i];
+
+        total_percentage_latency += (profile_info[i].fop_hits *
+                                     profile_info[i].avg_latency);
+    }
+    if (total_percentage_latency) {
         for (i = 0; i < GF_FOP_MAXVALUE; i++) {
-                if (profile_info[i].fop_hits == 0)
-                        continue;
-                if (is_header_printed == 0) {
-                        cli_out ("%10s %13s %13s %13s %14s %11s", "%-latency",
-                                 "Avg-latency", "Min-Latency", "Max-Latency",
-                                 "No. of calls", "Fop");
-                        cli_out ("%10s %13s %13s %13s %14s %11s", "---------",
-                                 "-----------", "-----------", "-----------",
-                                 "------------", "----");
-                        is_header_printed = 1;
-                }
-                if (profile_info[i].fop_hits) {
-                        cli_out ("%10.2lf %10.2lf us %10.2lf us %10.2lf us"
-                                 " %14"PRId64" %11s",
-                                 profile_info[i].percentage_avg_latency,
-                                 profile_info[i].avg_latency,
-                                 profile_info[i].min_latency,
-                                 profile_info[i].max_latency,
-                                 profile_info[i].fop_hits,
-                                 profile_info[i].fop_name);
-                }
+            profile_info[i]
+                .percentage_avg_latency = 100 * ((profile_info[i].avg_latency *
+                                                  profile_info[i].fop_hits) /
+                                                 total_percentage_latency);
+        }
+        gf_array_insertionsort(profile_info, 1, GF_FOP_MAXVALUE - 1,
+                               sizeof(cli_profile_info_t),
+                               cli_profile_info_percentage_cmp);
+    }
+    snprintf(key, sizeof(key), "%d-%d-duration", count, interval);
+    ret = dict_get_uint64(dict, key, &sec);
+    if (ret) {
+        gf_log("cli", GF_LOG_DEBUG, "failed to get %s from dict", key);
+    }
+
+    snprintf(key, sizeof(key), "%d-%d-total-read", count, interval);
+    ret = dict_get_uint64(dict, key, &r_count);
+    if (ret) {
+        gf_log("cli", GF_LOG_DEBUG, "failed to get %s from dict", key);
+    }
+
+    snprintf(key, sizeof(key), "%d-%d-total-write", count, interval);
+    ret = dict_get_uint64(dict, key, &w_count);
+    if (ret) {
+        gf_log("cli", GF_LOG_DEBUG, "failed to get %s from dict", key);
+    }
+
+    if (interval == -1)
+        cli_out("Cumulative Stats:");
+    else
+        cli_out("Interval %d Stats:", interval);
+    snprintf(output, sizeof(output), "%14s", "Block Size:");
+    snprintf(read_blocks, sizeof(read_blocks), "%14s", "No. of Reads:");
+    snprintf(write_blocks, sizeof(write_blocks), "%14s", "No. of Writes:");
+    index = 14;
+    for (i = 0; i < 32; i++) {
+        if ((rb_counts[i] == 0) && (wb_counts[i] == 0))
+            continue;
+        per_line++;
+        snprintf(output + index, sizeof(output) - index, "%19" PRIu32 "b+ ",
+                 (1U << i));
+        if (rb_counts[i]) {
+            snprintf(read_blocks + index, sizeof(read_blocks) - index,
+                     "%21" PRId64 " ", rb_counts[i]);
+        } else {
+            snprintf(read_blocks + index, sizeof(read_blocks) - index, "%21s ",
+                     "0");
         }
-        cli_out (" ");
-        cli_out ("%12s: %"PRId64" seconds", "Duration", sec);
-        cli_out ("%12s: %"PRId64" bytes", "Data Read", r_count);
-        cli_out ("%12s: %"PRId64" bytes", "Data Written", w_count);
-        cli_out (" ");
+        if (wb_counts[i]) {
+            snprintf(write_blocks + index, sizeof(write_blocks) - index,
+                     "%21" PRId64 " ", wb_counts[i]);
+        } else {
+            snprintf(write_blocks + index, sizeof(write_blocks) - index,
+                     "%21s ", "0");
+        }
+        index += 22;
+        if (per_line == 3) {
+            cli_out("%s", output);
+            cli_out("%s", read_blocks);
+            cli_out("%s", write_blocks);
+            cli_out(" ");
+            per_line = 0;
+            snprintf(output, sizeof(output), "%14s", "Block Size:");
+            snprintf(read_blocks, sizeof(read_blocks), "%14s", "No. of Reads:");
+            snprintf(write_blocks, sizeof(write_blocks), "%14s",
+                     "No. of Writes:");
+            index = 14;
+        }
+    }
+
+    if (per_line != 0) {
+        cli_out("%s", output);
+        cli_out("%s", read_blocks);
+        cli_out("%s", write_blocks);
+    }
+    for (i = 0; i < GF_FOP_MAXVALUE; i++) {
+        if (profile_info[i].fop_hits == 0)
+            continue;
+        if (is_header_printed == 0) {
+            cli_out("%10s %13s %13s %13s %14s %11s", "%-latency", "Avg-latency",
+                    "Min-Latency", "Max-Latency", "No. of calls", "Fop");
+            cli_out("%10s %13s %13s %13s %14s %11s", "---------", "-----------",
+                    "-----------", "-----------", "------------", "----");
+            is_header_printed = 1;
+        }
+        if (profile_info[i].fop_hits) {
+            cli_out(
+                "%10.2lf %10.2lf us %10.2lf us %10.2lf us"
+                " %14" PRId64 " %11s",
+                profile_info[i].percentage_avg_latency,
+                profile_info[i].avg_latency, profile_info[i].min_latency,
+                profile_info[i].max_latency, profile_info[i].fop_hits,
+                profile_info[i].fop_name);
+        }
+    }
+
+    for (i = 0; i < GF_UPCALL_FLAGS_MAXVALUE; i++) {
+        if (upcall_info[i].fop_hits == 0)
+            continue;
+        if (upcall_info[i].fop_hits) {
+            cli_out(
+                "%10.2lf %10.2lf us %10.2lf us %10.2lf us"
+                " %14" PRId64 " %11s",
+                upcall_info[i].percentage_avg_latency,
+                upcall_info[i].avg_latency, upcall_info[i].min_latency,
+                upcall_info[i].max_latency, upcall_info[i].fop_hits,
+                upcall_info[i].fop_name);
+        }
+    }
+
+    cli_out(" ");
+    cli_out("%12s: %" PRId64 " seconds", "Duration", sec);
+    cli_out("%12s: %" PRId64 " bytes", "Data Read", r_count);
+    cli_out("%12s: %" PRId64 " bytes", "Data Written", w_count);
+    cli_out(" ");
 }
 
-int32_t
-gf_cli_profile_volume_cbk (struct rpc_req *req, struct iovec *iov,
-                              int count, void *myframe)
-{
-        gf_cli_rsp                        rsp   = {0,};
-        int                               ret   = -1;
-        dict_t                            *dict = NULL;
-        gf1_cli_stats_op                  op = GF_CLI_STATS_NONE;
-        char                              key[256] = {0};
-        int                               interval = 0;
-        int                               i = 1;
-        int32_t                           brick_count = 0;
-        char                              *volname = NULL;
-        char                              *brick = NULL;
-        char                              str[1024] = {0,};
-        int                               stats_cleared = 0;
-        gf1_cli_info_op                   info_op = GF_CLI_INFO_NONE;
-
-        if (-1 == req->rpc_status) {
-                goto out;
+static int32_t
+gf_cli_profile_volume_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                          void *myframe)
+{
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    dict_t *dict = NULL;
+    gf1_cli_stats_op op = GF_CLI_STATS_NONE;
+    char key[64] = {0};
+    int len;
+    int interval = 0;
+    int i = 1;
+    int32_t brick_count = 0;
+    char *volname = NULL;
+    char *brick = NULL;
+    char str[1024] = {
+        0,
+    };
+    int stats_cleared = 0;
+    gf1_cli_info_op info_op = GF_CLI_INFO_NONE;
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
+
+    gf_log("cli", GF_LOG_DEBUG, "Received resp to profile");
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    dict = dict_new();
+
+    if (!dict) {
+        ret = -1;
+        goto out;
+    }
+
+    ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &dict);
+
+    if (ret) {
+        gf_log("", GF_LOG_ERROR, DICT_UNSERIALIZE_FAIL);
+        goto out;
+    }
+
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_vol_profile(dict, rsp.op_ret, rsp.op_errno,
+                                         rsp.op_errstr);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
+
+    ret = dict_get_int32_sizen(dict, "op", (int32_t *)&op);
+    if (ret)
+        goto out;
+
+    ret = dict_get_str_sizen(dict, "volname", &volname);
+    if (ret)
+        goto out;
+
+    if (rsp.op_ret && strcmp(rsp.op_errstr, "")) {
+        cli_err("%s", rsp.op_errstr);
+    } else {
+        switch (op) {
+            case GF_CLI_STATS_START:
+                cli_out("Starting volume profile on %s has been %s ", volname,
+                        (rsp.op_ret) ? "unsuccessful" : "successful");
+                break;
+            case GF_CLI_STATS_STOP:
+                cli_out("Stopping volume profile on %s has been %s ", volname,
+                        (rsp.op_ret) ? "unsuccessful" : "successful");
+                break;
+            case GF_CLI_STATS_INFO:
+                break;
+            default:
+                cli_out("volume profile on %s has been %s ", volname,
+                        (rsp.op_ret) ? "unsuccessful" : "successful");
+                break;
         }
+    }
 
-        gf_log ("cli", GF_LOG_DEBUG, "Received resp to profile");
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
+    if (rsp.op_ret) {
+        ret = rsp.op_ret;
+        goto out;
+    }
 
-        dict = dict_new ();
+    if (GF_CLI_STATS_INFO != op) {
+        ret = 0;
+        goto out;
+    }
 
-        if (!dict) {
-                ret = -1;
-                goto out;
-        }
+    ret = dict_get_int32_sizen(dict, "count", &brick_count);
+    if (ret)
+        goto out;
 
-        ret = dict_unserialize (rsp.dict.dict_val,
-                                rsp.dict.dict_len,
-                                &dict);
+    if (!brick_count) {
+        cli_err("All bricks of volume %s are down.", volname);
+        goto out;
+    }
 
-        if (ret) {
-                gf_log ("", GF_LOG_ERROR,
-                                "Unable to allocate memory");
-                goto out;
-        } else {
-                dict->extra_stdfree = rsp.dict.dict_val;
-        }
+    ret = dict_get_int32_sizen(dict, "info-op", (int32_t *)&info_op);
+    if (ret)
+        goto out;
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_vol_profile (dict, rsp.op_ret,
-                                                  rsp.op_errno,
-                                                  rsp.op_errstr);
-                if (ret)
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
-                goto out;
+    while (i <= brick_count) {
+        len = snprintf(key, sizeof(key), "%d-brick", i);
+        ret = dict_get_strn(dict, key, len, &brick);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, "Couldn't get brick name");
+            goto out;
         }
 
-        ret = dict_get_str (dict, "volname", &volname);
-        if (ret)
-                goto out;
+        ret = dict_get_str_boolean(dict, "nfs", _gf_false);
 
-        ret = dict_get_int32 (dict, "op", (int32_t*)&op);
         if (ret)
-                goto out;
-
-        if (rsp.op_ret && strcmp (rsp.op_errstr, "")) {
-                cli_err ("%s", rsp.op_errstr);
+            len = snprintf(str, sizeof(str), "NFS Server : %s", brick);
+        else
+            len = snprintf(str, sizeof(str), "Brick: %s", brick);
+        cli_out("%s", str);
+        memset(str, '-', len);
+        cli_out("%s", str);
+
+        if (GF_CLI_INFO_CLEAR == info_op) {
+            len = snprintf(key, sizeof(key), "%d-stats-cleared", i);
+            ret = dict_get_int32n(dict, key, len, &stats_cleared);
+            if (ret)
+                goto out;
+            cli_out(stats_cleared ? "Cleared stats."
+                                  : "Failed to clear stats.");
         } else {
-                switch (op) {
-                case GF_CLI_STATS_START:
-                        cli_out ("Starting volume profile on %s has been %s ",
-                                 volname,
-                                 (rsp.op_ret) ? "unsuccessful": "successful");
-                        break;
-                case GF_CLI_STATS_STOP:
-                        cli_out ("Stopping volume profile on %s has been %s ",
-                                 volname,
-                                 (rsp.op_ret) ? "unsuccessful": "successful");
-                        break;
-                case GF_CLI_STATS_INFO:
-                        break;
-                default:
-                        cli_out ("volume profile on %s has been %s ",
-                                 volname,
-                                 (rsp.op_ret) ? "unsuccessful": "successful");
-                        break;
-                }
-        }
-
-        if (rsp.op_ret) {
-                ret = rsp.op_ret;
-                goto out;
-        }
+            len = snprintf(key, sizeof(key), "%d-cumulative", i);
+            ret = dict_get_int32n(dict, key, len, &interval);
+            if (ret == 0)
+                cmd_profile_volume_brick_out(dict, i, interval);
 
-        if (GF_CLI_STATS_INFO != op) {
-                ret = 0;
-                goto out;
+            len = snprintf(key, sizeof(key), "%d-interval", i);
+            ret = dict_get_int32n(dict, key, len, &interval);
+            if (ret == 0)
+                cmd_profile_volume_brick_out(dict, i, interval);
         }
+        i++;
+    }
+    ret = rsp.op_ret;
 
-        ret = dict_get_int32 (dict, "info-op", (int32_t*)&info_op);
-        if (ret)
-                goto out;
-
-        ret = dict_get_int32 (dict, "count", &brick_count);
-        if (ret)
-                goto out;
+out:
+    if (dict)
+        dict_unref(dict);
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
+    return ret;
+}
 
-        if (!brick_count) {
-                cli_err ("All bricks of volume %s are down.", volname);
-                goto out;
-        }
+static int32_t
+gf_cli_profile_volume(call_frame_t *frame, xlator_t *this, void *data)
+{
+    int ret = -1;
+    gf_cli_req req = {{
+        0,
+    }};
+    dict_t *dict = NULL;
 
-        while (i <= brick_count) {
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%d-brick", i);
-                ret = dict_get_str (dict, key, &brick);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Couldn't get brick name");
-                        goto out;
-                }
+    GF_ASSERT(frame);
+    GF_ASSERT(this);
+    GF_ASSERT(data);
 
-                ret = dict_get_str_boolean (dict, "nfs", _gf_false);
+    dict = data;
 
-                if (ret)
-                        snprintf (str, sizeof (str), "NFS Server : %s", brick);
-                else
-                        snprintf (str, sizeof (str), "Brick: %s", brick);
-                cli_out ("%s", str);
-                memset (str, '-', strlen (str));
-                cli_out ("%s", str);
-
-                if (GF_CLI_INFO_CLEAR == info_op) {
-                        snprintf (key, sizeof (key), "%d-stats-cleared", i);
-                        ret = dict_get_int32 (dict, key, &stats_cleared);
-                        if (ret)
-                                goto out;
-                        cli_out (stats_cleared ? "Cleared stats." :
-                                                 "Failed to clear stats.");
-                } else {
-                        snprintf (key, sizeof (key), "%d-cumulative", i);
-                        ret = dict_get_int32 (dict, key, &interval);
-                        if (ret == 0)
-                                cmd_profile_volume_brick_out (dict, i,
-                                                              interval);
-
-                        snprintf (key, sizeof (key), "%d-interval", i);
-                        ret = dict_get_int32 (dict, key, &interval);
-                        if (ret == 0)
-                                cmd_profile_volume_brick_out (dict, i,
-                                                              interval);
-                }
-                i++;
-        }
-        ret = rsp.op_ret;
+    ret = cli_to_glusterd(&req, frame, gf_cli_profile_volume_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict,
+                          GLUSTER_CLI_PROFILE_VOLUME, this, cli_rpc_prog, NULL);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
 
-out:
-        if (dict)
-                dict_unref (dict);
-        free (rsp.op_errstr);
-        cli_cmd_broadcast_response (ret);
-        return ret;
+    GF_FREE(req.dict.dict_val);
+    return ret;
 }
 
-int32_t
-gf_cli_profile_volume (call_frame_t *frame, xlator_t *this, void *data)
+static int32_t
+gf_cli_top_volume_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                      void *myframe)
 {
-        int                        ret   = -1;
-        gf_cli_req                 req   = {{0,}};
-        dict_t                     *dict = NULL;
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    dict_t *dict = NULL;
+    gf1_cli_stats_op op = GF_CLI_STATS_NONE;
+    char key[256] = {0};
+    int keylen;
+    int i = 0;
+    int32_t brick_count = 0;
+    char brick[1024];
+    int32_t members = 0;
+    char *filename;
+    char *bricks;
+    uint64_t value = 0;
+    int32_t j = 0;
+    gf1_cli_top_op top_op = GF_CLI_TOP_NONE;
+    uint64_t nr_open = 0;
+    uint64_t max_nr_open = 0;
+    double throughput = 0;
+    double time = 0;
+    int32_t time_sec = 0;
+    long int time_usec = 0;
+    char timestr[GF_TIMESTR_SIZE] = {
+        0,
+    };
+    char *openfd_str = NULL;
+    gf_boolean_t nfs = _gf_false;
+    gf_boolean_t clear_stats = _gf_false;
+    int stats_cleared = 0;
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
+
+    gf_log("cli", GF_LOG_DEBUG, "Received resp to top");
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    if (rsp.op_ret) {
+        if (strcmp(rsp.op_errstr, ""))
+            cli_err("%s", rsp.op_errstr);
+        cli_err("volume top unsuccessful");
+        ret = rsp.op_ret;
+        goto out;
+    }
 
-        GF_ASSERT (frame);
-        GF_ASSERT (this);
-        GF_ASSERT (data);
+    dict = dict_new();
 
-        if (!frame || !this || !data)
-                goto out;
-        dict = data;
+    if (!dict) {
+        ret = -1;
+        goto out;
+    }
 
-        ret = cli_to_glusterd (&req, frame, gf_cli_profile_volume_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, dict,
-                               GLUSTER_CLI_PROFILE_VOLUME, this, cli_rpc_prog,
-                               NULL);
+    ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &dict);
 
-out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    if (ret) {
+        gf_log("", GF_LOG_ERROR, DICT_UNSERIALIZE_FAIL);
+        goto out;
+    }
 
-        GF_FREE (req.dict.dict_val);
-        return ret;
-}
+    ret = dict_get_int32_sizen(dict, "op", (int32_t *)&op);
 
-int32_t
-gf_cli_top_volume_cbk (struct rpc_req *req, struct iovec *iov,
-                              int count, void *myframe)
-{
-        gf_cli_rsp                        rsp   = {0,};
-        int                               ret   = -1;
-        dict_t                           *dict = NULL;
-        gf1_cli_stats_op                  op = GF_CLI_STATS_NONE;
-        char                              key[256] = {0};
-        int                               i = 0;
-        int32_t                           brick_count = 0;
-        char                              brick[1024];
-        int32_t                           members = 0;
-        char                             *filename;
-        char                             *bricks;
-        uint64_t                          value = 0;
-        int32_t                           j = 0;
-        gf1_cli_top_op                    top_op = GF_CLI_TOP_NONE;
-        uint64_t                          nr_open = 0;
-        uint64_t                          max_nr_open = 0;
-        double                            throughput = 0;
-        double                            time = 0;
-        int32_t                           time_sec = 0;
-        long int                          time_usec = 0;
-        char                              timestr[256] = {0, };
-        char                             *openfd_str = NULL;
-        gf_boolean_t                      nfs = _gf_false;
-        gf_boolean_t                      clear_stats = _gf_false;
-        int                               stats_cleared = 0;
-
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
+    if (op != GF_CLI_STATS_TOP) {
+        ret = 0;
+        goto out;
+    }
 
-        gf_log ("cli", GF_LOG_DEBUG, "Received resp to top");
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_vol_top(dict, rsp.op_ret, rsp.op_errno,
+                                     rsp.op_errstr);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
         }
+        goto out;
+    }
 
-        if (rsp.op_ret) {
-                if (strcmp (rsp.op_errstr, ""))
-                        cli_err ("%s", rsp.op_errstr);
-                cli_err ("volume top unsuccessful");
-                ret = rsp.op_ret;
-                goto out;
-        }
+    ret = dict_get_int32_sizen(dict, "count", &brick_count);
+    if (ret)
+        goto out;
+    keylen = snprintf(key, sizeof(key), "%d-top-op", 1);
+    ret = dict_get_int32n(dict, key, keylen, (int32_t *)&top_op);
+    if (ret)
+        goto out;
 
-        dict = dict_new ();
+    clear_stats = dict_get_str_boolean(dict, "clear-stats", _gf_false);
 
-        if (!dict) {
-                ret = -1;
-                goto out;
-        }
+    while (i < brick_count) {
+        i++;
+        keylen = snprintf(brick, sizeof(brick), "%d-brick", i);
+        ret = dict_get_strn(dict, brick, keylen, &bricks);
+        if (ret)
+            goto out;
 
-        ret = dict_unserialize (rsp.dict.dict_val,
-                                rsp.dict.dict_len,
-                                &dict);
+        nfs = dict_get_str_boolean(dict, "nfs", _gf_false);
 
-        if (ret) {
-                gf_log ("", GF_LOG_ERROR,
-                                "Unable to allocate memory");
+        if (clear_stats) {
+            keylen = snprintf(key, sizeof(key), "%d-stats-cleared", i);
+            ret = dict_get_int32n(dict, key, keylen, &stats_cleared);
+            if (ret)
                 goto out;
+            cli_out(stats_cleared ? "Cleared stats for %s %s"
+                                  : "Failed to clear stats for %s %s",
+                    nfs ? "NFS server on" : "brick", bricks);
+            continue;
         }
 
-        ret = dict_get_int32 (dict, "op", (int32_t*)&op);
+        if (nfs)
+            cli_out("NFS Server : %s", bricks);
+        else
+            cli_out("Brick: %s", bricks);
 
-        if (op != GF_CLI_STATS_TOP) {
-                ret = 0;
-                goto out;
-        }
+        keylen = snprintf(key, sizeof(key), "%d-members", i);
+        ret = dict_get_int32n(dict, key, keylen, &members);
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_vol_top (dict, rsp.op_ret,
-                                              rsp.op_errno,
-                                              rsp.op_errstr);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
+        switch (top_op) {
+            case GF_CLI_TOP_OPEN:
+                snprintf(key, sizeof(key), "%d-current-open", i);
+                ret = dict_get_uint64(dict, key, &nr_open);
+                if (ret)
+                    break;
+                snprintf(key, sizeof(key), "%d-max-open", i);
+                ret = dict_get_uint64(dict, key, &max_nr_open);
+                if (ret)
+                    goto out;
+                keylen = snprintf(key, sizeof(key), "%d-max-openfd-time", i);
+                ret = dict_get_strn(dict, key, keylen, &openfd_str);
+                if (ret)
+                    goto out;
+                cli_out("Current open fds: %" PRIu64 ", Max open fds: %" PRIu64
+                        ", Max openfd time: %s",
+                        nr_open, max_nr_open, openfd_str);
+            case GF_CLI_TOP_READ:
+            case GF_CLI_TOP_WRITE:
+            case GF_CLI_TOP_OPENDIR:
+            case GF_CLI_TOP_READDIR:
+                if (!members) {
+                    continue;
+                }
+                cli_out("Count\t\tfilename\n=======================");
+                break;
+            case GF_CLI_TOP_READ_PERF:
+            case GF_CLI_TOP_WRITE_PERF:
+                snprintf(key, sizeof(key), "%d-throughput", i);
+                ret = dict_get_double(dict, key, &throughput);
+                if (!ret) {
+                    snprintf(key, sizeof(key), "%d-time", i);
+                    ret = dict_get_double(dict, key, &time);
                 }
+                if (!ret)
+                    cli_out("Throughput %.2f MBps time %.4f secs", throughput,
+                            time / 1e6);
+
+                if (!members) {
+                    continue;
+                }
+                cli_out("%*s %-*s %-*s", VOL_TOP_PERF_SPEED_WIDTH, "MBps",
+                        VOL_TOP_PERF_FILENAME_DEF_WIDTH, "Filename",
+                        VOL_TOP_PERF_TIME_WIDTH, "Time");
+                cli_out("%*s %-*s %-*s", VOL_TOP_PERF_SPEED_WIDTH,
+                        "====", VOL_TOP_PERF_FILENAME_DEF_WIDTH,
+                        "========", VOL_TOP_PERF_TIME_WIDTH, "====");
+                break;
+            default:
                 goto out;
         }
 
-        ret = dict_get_int32 (dict, "count", &brick_count);
-        if (ret)
-                goto out;
-        snprintf (key, sizeof (key), "%d-top-op", 1);
-        ret = dict_get_int32 (dict, key, (int32_t*)&top_op);
-        if (ret)
-                goto out;
-
-        clear_stats = dict_get_str_boolean (dict, "clear-stats", _gf_false);
-
-        while (i < brick_count) {
-                i++;
-                snprintf (brick, sizeof (brick), "%d-brick", i);
-                ret = dict_get_str (dict, brick, &bricks);
+        for (j = 1; j <= members; j++) {
+            keylen = snprintf(key, sizeof(key), "%d-filename-%d", i, j);
+            ret = dict_get_strn(dict, key, keylen, &filename);
+            if (ret)
+                break;
+            snprintf(key, sizeof(key), "%d-value-%d", i, j);
+            ret = dict_get_uint64(dict, key, &value);
+            if (ret)
+                goto out;
+            if (top_op == GF_CLI_TOP_READ_PERF ||
+                top_op == GF_CLI_TOP_WRITE_PERF) {
+                keylen = snprintf(key, sizeof(key), "%d-time-sec-%d", i, j);
+                ret = dict_get_int32n(dict, key, keylen, (int32_t *)&time_sec);
                 if (ret)
-                        goto out;
-
-                nfs = dict_get_str_boolean (dict, "nfs", _gf_false);
-
-                if (clear_stats) {
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key), "%d-stats-cleared", i);
-                        ret = dict_get_int32 (dict, key, &stats_cleared);
-                        if (ret)
-                                goto out;
-                        cli_out (stats_cleared ? "Cleared stats for %s %s" :
-                                 "Failed to clear stats for %s %s",
-                                 nfs ? "NFS server on" : "brick", bricks);
-                        continue;
-                }
-
-                if (nfs)
-                        cli_out ("NFS Server : %s", bricks);
+                    goto out;
+                keylen = snprintf(key, sizeof(key), "%d-time-usec-%d", i, j);
+                ret = dict_get_int32n(dict, key, keylen, (int32_t *)&time_usec);
+                if (ret)
+                    goto out;
+                gf_time_fmt(timestr, sizeof timestr, time_sec, gf_timefmt_FT);
+                snprintf(timestr + strlen(timestr),
+                         sizeof timestr - strlen(timestr), ".%ld", time_usec);
+                if (strlen(filename) < VOL_TOP_PERF_FILENAME_DEF_WIDTH)
+                    cli_out("%*" PRIu64 " %-*s %-*s", VOL_TOP_PERF_SPEED_WIDTH,
+                            value, VOL_TOP_PERF_FILENAME_DEF_WIDTH, filename,
+                            VOL_TOP_PERF_TIME_WIDTH, timestr);
                 else
-                        cli_out ("Brick: %s", bricks);
-
-                snprintf(key, sizeof (key), "%d-members", i);
-                ret = dict_get_int32 (dict, key, &members);
-
-                switch (top_op) {
-                case GF_CLI_TOP_OPEN:
-                        snprintf (key, sizeof (key), "%d-current-open", i);
-                        ret = dict_get_uint64 (dict, key, &nr_open);
-                        if (ret)
-                                break;
-                        snprintf (key, sizeof (key), "%d-max-open", i);
-                        ret = dict_get_uint64 (dict, key, &max_nr_open);
-                        if (ret)
-                                goto out;
-                        snprintf (key, sizeof (key), "%d-max-openfd-time", i);
-                        ret = dict_get_str (dict, key, &openfd_str);
-                        if (ret)
-                                goto out;
-                        cli_out ("Current open fds: %"PRIu64", Max open"
-                                " fds: %"PRIu64", Max openfd time: %s", nr_open,
-                                 max_nr_open, openfd_str);
-                case GF_CLI_TOP_READ:
-                case GF_CLI_TOP_WRITE:
-                case GF_CLI_TOP_OPENDIR:
-                case GF_CLI_TOP_READDIR:
-                        if (!members) {
-                                continue;
-                        }
-                        cli_out ("Count\t\tfilename\n=======================");
-                        break;
-                case GF_CLI_TOP_READ_PERF:
-                case GF_CLI_TOP_WRITE_PERF:
-                        snprintf (key, sizeof (key), "%d-throughput", i);
-                        ret = dict_get_double (dict, key, &throughput);
-                        if (!ret) {
-                                snprintf (key, sizeof (key), "%d-time", i);
-                                ret = dict_get_double (dict, key, &time);
-                        }
-                        if (!ret)
-                                cli_out ("Throughput %.2f MBps time %.4f secs", throughput,
-                                          time / 1e6);
-
-                        if (!members) {
-                                continue;
-                        }
-                        cli_out ("%*s %-*s %-*s",
-                                 VOL_TOP_PERF_SPEED_WIDTH, "MBps",
-                                 VOL_TOP_PERF_FILENAME_DEF_WIDTH, "Filename",
-                                 VOL_TOP_PERF_TIME_WIDTH, "Time");
-                        cli_out ("%*s %-*s %-*s",
-                                 VOL_TOP_PERF_SPEED_WIDTH, "====",
-                                 VOL_TOP_PERF_FILENAME_DEF_WIDTH, "========",
-                                 VOL_TOP_PERF_TIME_WIDTH, "====");
-                        break;
-                default:
-                        goto out;
-                }
-
-                for (j = 1; j <= members; j++) {
-                        snprintf (key, sizeof (key), "%d-filename-%d", i, j);
-                        ret = dict_get_str (dict, key, &filename);
-                        if (ret)
-                                break;
-                        snprintf (key, sizeof (key), "%d-value-%d", i, j);
-                        ret = dict_get_uint64 (dict, key, &value);
-                        if (ret)
-                                goto out;
-                        if ( top_op == GF_CLI_TOP_READ_PERF ||
-                                top_op == GF_CLI_TOP_WRITE_PERF) {
-                                snprintf (key, sizeof (key), "%d-time-sec-%d", i, j);
-                                ret = dict_get_int32 (dict, key, (int32_t *)&time_sec);
-                                if (ret)
-                                        goto out;
-                                snprintf (key, sizeof (key), "%d-time-usec-%d", i, j);
-                                ret = dict_get_int32 (dict, key, (int32_t *)&time_usec);
-                                if (ret)
-                                        goto out;
-                                gf_time_fmt (timestr, sizeof timestr,
-                                             time_sec, gf_timefmt_FT);
-                                snprintf (timestr + strlen (timestr), sizeof timestr - strlen (timestr),
-                                  ".%ld", time_usec);
-                                if (strlen (filename) < VOL_TOP_PERF_FILENAME_DEF_WIDTH)
-                                        cli_out ("%*"PRIu64" %-*s %-*s",
-                                                 VOL_TOP_PERF_SPEED_WIDTH,
-                                                 value,
-                                                 VOL_TOP_PERF_FILENAME_DEF_WIDTH,
-                                                 filename,
-                                                 VOL_TOP_PERF_TIME_WIDTH,
-                                                 timestr);
-                                else
-                                        cli_out ("%*"PRIu64" ...%-*s %-*s",
-                                                 VOL_TOP_PERF_SPEED_WIDTH,
-                                                 value,
-                                                 VOL_TOP_PERF_FILENAME_ALT_WIDTH ,
-                                                 filename + strlen (filename) -
-                                                 VOL_TOP_PERF_FILENAME_ALT_WIDTH,
-                                                 VOL_TOP_PERF_TIME_WIDTH,
-                                                 timestr);
-                        } else {
-                                cli_out ("%"PRIu64"\t\t%s", value, filename);
-                        }
-                }
-        }
-        ret = rsp.op_ret;
+                    cli_out("%*" PRIu64 " ...%-*s %-*s",
+                            VOL_TOP_PERF_SPEED_WIDTH, value,
+                            VOL_TOP_PERF_FILENAME_ALT_WIDTH,
+                            filename + strlen(filename) -
+                                VOL_TOP_PERF_FILENAME_ALT_WIDTH,
+                            VOL_TOP_PERF_TIME_WIDTH, timestr);
+            } else {
+                cli_out("%" PRIu64 "\t\t%s", value, filename);
+            }
+        }
+    }
+    ret = rsp.op_ret;
 
 out:
-        cli_cmd_broadcast_response (ret);
+    cli_cmd_broadcast_response(ret);
 
-        if (dict)
-                dict_unref (dict);
+    if (dict)
+        dict_unref(dict);
 
-        free (rsp.dict.dict_val);
-        return ret;
+    gf_free_xdr_cli_rsp(rsp);
+    return ret;
 }
 
-int32_t
-gf_cli_top_volume (call_frame_t *frame, xlator_t *this, void *data)
+static int32_t
+gf_cli_top_volume(call_frame_t *frame, xlator_t *this, void *data)
 {
-        int                        ret   = -1;
-        gf_cli_req                 req   = {{0,}};
-        dict_t                     *dict = NULL;
-
-        GF_ASSERT (frame);
-        GF_ASSERT (this);
-        GF_ASSERT (data);
+    int ret = -1;
+    gf_cli_req req = {{
+        0,
+    }};
+    dict_t *dict = NULL;
 
-        if (!frame || !this || !data)
-                goto out;
-        dict = data;
+    GF_ASSERT(frame);
+    GF_ASSERT(this);
+    GF_ASSERT(data);
 
-        ret = cli_to_glusterd (&req, frame, gf_cli_top_volume_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, dict,
-                               GLUSTER_CLI_PROFILE_VOLUME, this, cli_rpc_prog,
-                               NULL);
+    dict = data;
 
-out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        GF_FREE (req.dict.dict_val);
-        return ret;
+    ret = cli_to_glusterd(&req, frame, gf_cli_top_volume_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict,
+                          GLUSTER_CLI_PROFILE_VOLUME, this, cli_rpc_prog, NULL);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
+    GF_FREE(req.dict.dict_val);
+    return ret;
 }
 
-
-int
-gf_cli_getwd_cbk (struct rpc_req *req, struct iovec *iov,
-                       int count, void *myframe)
+static int
+gf_cli_getwd_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                 void *myframe)
 {
-        gf1_cli_getwd_rsp rsp   = {0,};
-        int               ret   = -1;
-
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
-
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf1_cli_getwd_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
-
-        if (rsp.op_ret == -1) {
-                cli_err ("getwd failed");
-                ret = rsp.op_ret;
-                goto out;
-        }
+    gf1_cli_getwd_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf1_cli_getwd_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    if (rsp.op_ret == -1) {
+        cli_err("getwd failed");
+        ret = rsp.op_ret;
+        goto out;
+    }
 
-        gf_log ("cli", GF_LOG_INFO, "Received resp to getwd");
+    gf_log("cli", GF_LOG_INFO, "Received resp to getwd");
 
-        cli_out ("%s", rsp.wd);
+    cli_out("%s", rsp.wd);
 
-        ret = 0;
+    ret = 0;
 
 out:
-        cli_cmd_broadcast_response (ret);
-        return ret;
+    cli_cmd_broadcast_response(ret);
+    if (rsp.wd) {
+        free(rsp.wd);
+    }
+
+    return ret;
 }
 
-int32_t
-gf_cli_getwd (call_frame_t *frame, xlator_t *this, void *data)
+static int32_t
+gf_cli_getwd(call_frame_t *frame, xlator_t *this, void *data)
 {
-        int                      ret = -1;
-        gf1_cli_getwd_req        req = {0,};
+    int ret = -1;
+    gf1_cli_getwd_req req = {
+        0,
+    };
 
-        GF_ASSERT (frame);
-        GF_ASSERT (this);
+    GF_ASSERT(frame);
+    GF_ASSERT(this);
 
-        if (!frame || !this)
-                goto out;
+    if (!frame || !this)
+        goto out;
 
-        ret = cli_cmd_submit (NULL, &req, frame, cli_rpc_prog,
-                              GLUSTER_CLI_GETWD, NULL,
-                              this, gf_cli_getwd_cbk,
-                              (xdrproc_t) xdr_gf1_cli_getwd_req);
+    ret = cli_cmd_submit(NULL, &req, frame, cli_rpc_prog, GLUSTER_CLI_GETWD,
+                         NULL, this, gf_cli_getwd_cbk,
+                         (xdrproc_t)xdr_gf1_cli_getwd_req);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
 
-        return ret;
+    return ret;
 }
 
-void
-cli_print_volume_status_mempool (dict_t *dict, char *prefix)
-{
-        int             ret = -1;
-        int32_t         mempool_count = 0;
-        char            *name = NULL;
-        int32_t         hotcount = 0;
-        int32_t         coldcount = 0;
-        uint64_t        paddedsizeof = 0;
-        uint64_t        alloccount = 0;
-        int32_t         maxalloc = 0;
-        uint64_t        pool_misses = 0;
-        int32_t         maxstdalloc = 0;
-        char            key[1024] = {0,};
-        int             i = 0;
-
-        GF_ASSERT (dict);
-        GF_ASSERT (prefix);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.mempool-count",prefix);
-        ret = dict_get_int32 (dict, key, &mempool_count);
+static void
+cli_print_volume_status_mempool(dict_t *dict, char *prefix)
+{
+    int ret = -1;
+    int32_t mempool_count = 0;
+    char *name = NULL;
+    int32_t hotcount = 0;
+    int32_t coldcount = 0;
+    uint64_t paddedsizeof = 0;
+    uint64_t alloccount = 0;
+    int32_t maxalloc = 0;
+    uint64_t pool_misses = 0;
+    int32_t maxstdalloc = 0;
+    char key[128] = {
+        /* prefix is really small 'brick%d' really */
+        0,
+    };
+    int keylen;
+    int i = 0;
+
+    GF_ASSERT(dict);
+    GF_ASSERT(prefix);
+
+    keylen = snprintf(key, sizeof(key), "%s.mempool-count", prefix);
+    ret = dict_get_int32n(dict, key, keylen, &mempool_count);
+    if (ret)
+        goto out;
+
+    cli_out("Mempool Stats\n-------------");
+    cli_out("%-30s %9s %9s %12s %10s %8s %8s %12s", "Name", "HotCount",
+            "ColdCount", "PaddedSizeof", "AllocCount", "MaxAlloc", "Misses",
+            "Max-StdAlloc");
+    cli_out("%-30s %9s %9s %12s %10s %8s %8s %12s", "----", "--------",
+            "---------", "------------", "----------", "--------", "--------",
+            "------------");
+
+    for (i = 0; i < mempool_count; i++) {
+        keylen = snprintf(key, sizeof(key), "%s.pool%d.name", prefix, i);
+        ret = dict_get_strn(dict, key, keylen, &name);
         if (ret)
-                goto out;
-
-        cli_out ("Mempool Stats\n-------------");
-        cli_out ("%-30s %9s %9s %12s %10s %8s %8s %12s", "Name", "HotCount",
-                 "ColdCount", "PaddedSizeof", "AllocCount", "MaxAlloc",
-                 "Misses", "Max-StdAlloc");
-        cli_out ("%-30s %9s %9s %12s %10s %8s %8s %12s", "----", "--------",
-                 "---------", "------------", "----------",
-                 "--------", "--------", "------------");
-
-        for (i = 0; i < mempool_count; i++) {
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.pool%d.name", prefix, i);
-                ret = dict_get_str (dict, key, &name);
-                if (ret)
-                        goto out;
+            goto out;
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.pool%d.hotcount", prefix, i);
-                ret = dict_get_int32 (dict, key, &hotcount);
-                if (ret)
-                        goto out;
+        keylen = snprintf(key, sizeof(key), "%s.pool%d.hotcount", prefix, i);
+        ret = dict_get_int32n(dict, key, keylen, &hotcount);
+        if (ret)
+            goto out;
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.pool%d.coldcount", prefix, i);
-                ret = dict_get_int32 (dict, key, &coldcount);
-                if (ret)
-                        goto out;
+        keylen = snprintf(key, sizeof(key), "%s.pool%d.coldcount", prefix, i);
+        ret = dict_get_int32n(dict, key, keylen, &coldcount);
+        if (ret)
+            goto out;
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.pool%d.paddedsizeof",
-                          prefix, i);
-                ret = dict_get_uint64 (dict, key, &paddedsizeof);
-                if (ret)
-                        goto out;
+        snprintf(key, sizeof(key), "%s.pool%d.paddedsizeof", prefix, i);
+        ret = dict_get_uint64(dict, key, &paddedsizeof);
+        if (ret)
+            goto out;
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.pool%d.alloccount", prefix, i);
-                ret = dict_get_uint64 (dict, key, &alloccount);
-                if (ret)
-                        goto out;
+        snprintf(key, sizeof(key), "%s.pool%d.alloccount", prefix, i);
+        ret = dict_get_uint64(dict, key, &alloccount);
+        if (ret)
+            goto out;
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.pool%d.max_alloc", prefix, i);
-                ret = dict_get_int32 (dict, key, &maxalloc);
-                if (ret)
-                        goto out;
+        keylen = snprintf(key, sizeof(key), "%s.pool%d.max_alloc", prefix, i);
+        ret = dict_get_int32n(dict, key, keylen, &maxalloc);
+        if (ret)
+            goto out;
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.pool%d.max-stdalloc", prefix, i);
-                ret = dict_get_int32 (dict, key, &maxstdalloc);
-                if (ret)
-                        goto out;
+        keylen = snprintf(key, sizeof(key), "%s.pool%d.max-stdalloc", prefix,
+                          i);
+        ret = dict_get_int32n(dict, key, keylen, &maxstdalloc);
+        if (ret)
+            goto out;
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.pool%d.pool-misses", prefix, i);
-                ret = dict_get_uint64 (dict, key, &pool_misses);
-                if (ret)
-                        goto out;
+        snprintf(key, sizeof(key), "%s.pool%d.pool-misses", prefix, i);
+        ret = dict_get_uint64(dict, key, &pool_misses);
+        if (ret)
+            goto out;
 
-                cli_out ("%-30s %9d %9d %12"PRIu64" %10"PRIu64" %8d %8"PRIu64
-                         " %12d", name, hotcount, coldcount, paddedsizeof,
-                         alloccount, maxalloc, pool_misses, maxstdalloc);
-        }
+        cli_out("%-30s %9d %9d %12" PRIu64 " %10" PRIu64 " %8d %8" PRIu64
+                " %12d",
+                name, hotcount, coldcount, paddedsizeof, alloccount, maxalloc,
+                pool_misses, maxstdalloc);
+    }
 
 out:
-        return;
-
+    return;
 }
 
-void
-cli_print_volume_status_mem (dict_t *dict, gf_boolean_t notbrick)
+static void
+cli_print_volume_status_mem(dict_t *dict, gf_boolean_t notbrick)
 {
-        int             ret = -1;
-        char            *volname = NULL;
-        char            *hostname = NULL;
-        char            *path = NULL;
-        int             online = -1;
-        char            key[1024] = {0,};
-        int             brick_index_max = -1;
-        int             other_count = 0;
-        int             index_max = 0;
-        int             val = 0;
-        int             i = 0;
-
-        GF_ASSERT (dict);
-
-        ret = dict_get_str (dict, "volname", &volname);
-        if (ret)
-                goto out;
-        cli_out ("Memory status for volume : %s", volname);
-
-        ret = dict_get_int32 (dict, "brick-index-max", &brick_index_max);
+    int ret = -1;
+    char *volname = NULL;
+    char *hostname = NULL;
+    char *path = NULL;
+    int online = -1;
+    char key[64] = {
+        0,
+    };
+    int brick_index_max = -1;
+    int other_count = 0;
+    int index_max = 0;
+    int val = 0;
+    int i = 0;
+
+    GF_ASSERT(dict);
+
+    ret = dict_get_str_sizen(dict, "volname", &volname);
+    if (ret)
+        goto out;
+    cli_out("Memory status for volume : %s", volname);
+
+    ret = dict_get_int32_sizen(dict, "brick-index-max", &brick_index_max);
+    if (ret)
+        goto out;
+    ret = dict_get_int32_sizen(dict, "other-count", &other_count);
+    if (ret)
+        goto out;
+
+    index_max = brick_index_max + other_count;
+
+    for (i = 0; i <= index_max; i++) {
+        cli_out("----------------------------------------------");
+
+        snprintf(key, sizeof(key), "brick%d.hostname", i);
+        ret = dict_get_str(dict, key, &hostname);
         if (ret)
-                goto out;
-        ret = dict_get_int32 (dict, "other-count", &other_count);
+            continue;
+        snprintf(key, sizeof(key), "brick%d.path", i);
+        ret = dict_get_str(dict, key, &path);
         if (ret)
-                goto out;
-
-        index_max = brick_index_max + other_count;
-
-        for (i = 0; i <= index_max; i++) {
-                cli_out ("----------------------------------------------");
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.hostname", i);
-                ret = dict_get_str (dict, key, &hostname);
-                if (ret)
-                        continue;
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.path", i);
-                ret = dict_get_str (dict, key, &path);
-                if (ret)
-                        continue;
-                if (notbrick)
-                        cli_out ("%s : %s", hostname, path);
-                else
-                        cli_out ("Brick : %s:%s", hostname, path);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.status", i);
-                ret = dict_get_int32 (dict, key, &online);
-                if (ret)
-                        goto out;
-                if (!online) {
-                        if (notbrick)
-                                cli_out ("%s is offline", hostname);
-                        else
-                                cli_out ("Brick is offline");
-                        continue;
-                }
+            continue;
+        if (notbrick)
+            cli_out("%s : %s", hostname, path);
+        else
+            cli_out("Brick : %s:%s", hostname, path);
 
-                cli_out ("Mallinfo\n--------");
+        snprintf(key, sizeof(key), "brick%d.status", i);
+        ret = dict_get_int32(dict, key, &online);
+        if (ret)
+            goto out;
+        if (!online) {
+            if (notbrick)
+                cli_out("%s is offline", hostname);
+            else
+                cli_out("Brick is offline");
+            continue;
+        }
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.mallinfo.arena", i);
-                ret = dict_get_int32 (dict, key, &val);
-                if (ret)
-                        goto out;
-                cli_out ("%-8s : %d","Arena", val);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.mallinfo.ordblks", i);
-                ret = dict_get_int32 (dict, key, &val);
-                if(ret)
-                        goto out;
-                cli_out ("%-8s : %d","Ordblks", val);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.mallinfo.smblks", i);
-                ret = dict_get_int32 (dict, key, &val);
-                if(ret)
-                        goto out;
-                cli_out ("%-8s : %d","Smblks", val);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.mallinfo.hblks", i);
-                ret = dict_get_int32 (dict, key, &val);
-                if(ret)
-                        goto out;
-                cli_out ("%-8s : %d", "Hblks", val);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.mallinfo.hblkhd", i);
-                ret = dict_get_int32 (dict, key, &val);
-                if (ret)
-                        goto out;
-                cli_out ("%-8s : %d", "Hblkhd", val);
+        cli_out("Mallinfo\n--------");
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.mallinfo.usmblks", i);
-                ret = dict_get_int32 (dict, key, &val);
-                if (ret)
-                        goto out;
-                cli_out ("%-8s : %d", "Usmblks", val);
+        snprintf(key, sizeof(key), "brick%d.mallinfo.arena", i);
+        ret = dict_get_int32(dict, key, &val);
+        if (ret)
+            goto out;
+        cli_out("%-8s : %d", "Arena", val);
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.mallinfo.fsmblks", i);
-                ret = dict_get_int32 (dict, key, &val);
-                if (ret)
-                        goto out;
-                cli_out ("%-8s : %d", "Fsmblks", val);
+        snprintf(key, sizeof(key), "brick%d.mallinfo.ordblks", i);
+        ret = dict_get_int32(dict, key, &val);
+        if (ret)
+            goto out;
+        cli_out("%-8s : %d", "Ordblks", val);
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.mallinfo.uordblks", i);
-                ret = dict_get_int32 (dict, key, &val);
-                if (ret)
-                        goto out;
-                cli_out ("%-8s : %d", "Uordblks", val);
+        snprintf(key, sizeof(key), "brick%d.mallinfo.smblks", i);
+        ret = dict_get_int32(dict, key, &val);
+        if (ret)
+            goto out;
+        cli_out("%-8s : %d", "Smblks", val);
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.mallinfo.fordblks", i);
-                ret = dict_get_int32 (dict, key, &val);
-                if (ret)
-                        goto out;
-                cli_out ("%-8s : %d", "Fordblks", val);
+        snprintf(key, sizeof(key), "brick%d.mallinfo.hblks", i);
+        ret = dict_get_int32(dict, key, &val);
+        if (ret)
+            goto out;
+        cli_out("%-8s : %d", "Hblks", val);
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.mallinfo.keepcost", i);
-                ret = dict_get_int32 (dict, key, &val);
-                if (ret)
-                        goto out;
-                cli_out ("%-8s : %d", "Keepcost", val);
+        snprintf(key, sizeof(key), "brick%d.mallinfo.hblkhd", i);
+        ret = dict_get_int32(dict, key, &val);
+        if (ret)
+            goto out;
+        cli_out("%-8s : %d", "Hblkhd", val);
 
-                cli_out (" ");
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d", i);
-                cli_print_volume_status_mempool (dict, key);
-        }
-out:
-        cli_out ("----------------------------------------------\n");
-        return;
-}
+        snprintf(key, sizeof(key), "brick%d.mallinfo.usmblks", i);
+        ret = dict_get_int32(dict, key, &val);
+        if (ret)
+            goto out;
+        cli_out("%-8s : %d", "Usmblks", val);
 
-void
-cli_print_volume_status_clients (dict_t *dict, gf_boolean_t notbrick)
-{
-        int             ret = -1;
-        char            *volname = NULL;
-        int             brick_index_max = -1;
-        int             other_count = 0;
-        int             index_max = 0;
-        char            *hostname = NULL;
-        char            *path = NULL;
-        int             online = -1;
-        int             client_count = 0;
-        char            *clientname = NULL;
-        uint64_t        bytesread = 0;
-        uint64_t        byteswrite = 0;
-        char            key[1024] = {0,};
-        int             i = 0;
-        int             j = 0;
-
-        GF_ASSERT (dict);
-
-        ret = dict_get_str (dict, "volname", &volname);
+        snprintf(key, sizeof(key), "brick%d.mallinfo.fsmblks", i);
+        ret = dict_get_int32(dict, key, &val);
         if (ret)
-                goto out;
-        cli_out ("Client connections for volume %s", volname);
+            goto out;
+        cli_out("%-8s : %d", "Fsmblks", val);
 
-        ret = dict_get_int32 (dict, "brick-index-max", &brick_index_max);
+        snprintf(key, sizeof(key), "brick%d.mallinfo.uordblks", i);
+        ret = dict_get_int32(dict, key, &val);
         if (ret)
-                goto out;
-        ret = dict_get_int32 (dict, "other-count", &other_count);
+            goto out;
+        cli_out("%-8s : %d", "Uordblks", val);
+
+        snprintf(key, sizeof(key), "brick%d.mallinfo.fordblks", i);
+        ret = dict_get_int32(dict, key, &val);
         if (ret)
-                goto out;
+            goto out;
+        cli_out("%-8s : %d", "Fordblks", val);
 
-        index_max = brick_index_max + other_count;
+        snprintf(key, sizeof(key), "brick%d.mallinfo.keepcost", i);
+        ret = dict_get_int32(dict, key, &val);
+        if (ret)
+            goto out;
+        cli_out("%-8s : %d", "Keepcost", val);
 
-        for (i = 0; i <= index_max; i++) {
-                cli_out ("----------------------------------------------");
+        cli_out(" ");
+        snprintf(key, sizeof(key), "brick%d", i);
+        cli_print_volume_status_mempool(dict, key);
+    }
+out:
+    cli_out("----------------------------------------------\n");
+    return;
+}
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.hostname", i);
-                ret = dict_get_str (dict, key, &hostname);
+static void
+cli_print_volume_status_client_list(dict_t *dict, gf_boolean_t notbrick)
+{
+    int ret = -1;
+    char *volname = NULL;
+    int client_count = 0;
+    int current_count = 0;
+    char key[64] = {
+        0,
+    };
+    int i = 0;
+    int total = 0;
+    char *name = NULL;
+    gf_boolean_t is_fuse_done = _gf_false;
+    gf_boolean_t is_gfapi_done = _gf_false;
+    gf_boolean_t is_rebalance_done = _gf_false;
+    gf_boolean_t is_glustershd_done = _gf_false;
+    gf_boolean_t is_quotad_done = _gf_false;
+    gf_boolean_t is_snapd_done = _gf_false;
+
+    GF_ASSERT(dict);
+
+    ret = dict_get_str_sizen(dict, "volname", &volname);
+    if (ret)
+        goto out;
+    cli_out("Client connections for volume %s", volname);
+
+    ret = dict_get_int32_sizen(dict, "client-count", &client_count);
+    if (ret)
+        goto out;
+
+    cli_out("%-48s %15s", "Name", "count");
+    cli_out("%-48s %15s", "-----", "------");
+    for (i = 0; i < client_count; i++) {
+        name = NULL;
+        snprintf(key, sizeof(key), "client%d.name", i);
+        ret = dict_get_str(dict, key, &name);
+
+        if (!strncmp(name, "fuse", 4)) {
+            if (!is_fuse_done) {
+                is_fuse_done = _gf_true;
+                ret = dict_get_int32_sizen(dict, "fuse-count", &current_count);
                 if (ret)
-                        goto out;
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.path", i);
-                ret = dict_get_str (dict, key, &path);
+                    goto out;
+                total = total + current_count;
+                goto print;
+            }
+            continue;
+        } else if (!strncmp(name, "gfapi", 5)) {
+            if (!is_gfapi_done) {
+                is_gfapi_done = _gf_true;
+                ret = dict_get_int32_sizen(dict, "gfapi-count", &current_count);
                 if (ret)
-                        goto out;
-
-                if (notbrick)
-                        cli_out ("%s : %s", hostname, path);
-                else
-                        cli_out ("Brick : %s:%s", hostname, path);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.status", i);
-                ret = dict_get_int32 (dict, key, &online);
+                    goto out;
+                total = total + current_count;
+                goto print;
+            }
+            continue;
+        } else if (!strcmp(name, "rebalance")) {
+            if (!is_rebalance_done) {
+                is_rebalance_done = _gf_true;
+                ret = dict_get_int32_sizen(dict, "rebalance-count",
+                                           &current_count);
                 if (ret)
-                        goto out;
-                if (!online) {
-                        if (notbrick)
-                                cli_out ("%s is offline", hostname);
-                        else
-                                cli_out ("Brick is offline");
-                        continue;
-                }
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.clientcount", i);
-                ret = dict_get_int32 (dict, key, &client_count);
+                    goto out;
+                total = total + current_count;
+                goto print;
+            }
+            continue;
+        } else if (!strcmp(name, "glustershd")) {
+            if (!is_glustershd_done) {
+                is_glustershd_done = _gf_true;
+                ret = dict_get_int32_sizen(dict, "glustershd-count",
+                                           &current_count);
                 if (ret)
-                        goto out;
-
-                cli_out ("Clients connected : %d", client_count);
-                if (client_count == 0)
-                        continue;
-
-                cli_out ("%-48s %15s %15s", "Hostname", "BytesRead",
-                         "BytesWritten");
-                cli_out ("%-48s %15s %15s", "--------", "---------",
-                         "------------");
-                for (j =0; j < client_count; j++) {
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key),
-                                  "brick%d.client%d.hostname", i, j);
-                        ret = dict_get_str (dict, key, &clientname);
-                        if (ret)
-                                goto out;
-
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key),
-                                  "brick%d.client%d.bytesread", i, j);
-                        ret = dict_get_uint64 (dict, key, &bytesread);
-                        if (ret)
-                                goto out;
-
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key),
-                                 "brick%d.client%d.byteswrite", i, j);
-                        ret = dict_get_uint64 (dict, key, &byteswrite);
-                        if (ret)
-                                goto out;
-
-                        cli_out ("%-48s %15"PRIu64" %15"PRIu64,
-                                 clientname, bytesread, byteswrite);
-                }
+                    goto out;
+                total = total + current_count;
+                goto print;
+            }
+            continue;
+        } else if (!strcmp(name, "quotad")) {
+            if (!is_quotad_done) {
+                is_quotad_done = _gf_true;
+                ret = dict_get_int32_sizen(dict, "quotad-count",
+                                           &current_count);
+                if (ret)
+                    goto out;
+                total = total + current_count;
+                goto print;
+            }
+            continue;
+        } else if (!strcmp(name, "snapd")) {
+            if (!is_snapd_done) {
+                is_snapd_done = _gf_true;
+                ret = dict_get_int32_sizen(dict, "snapd-count", &current_count);
+                if (ret)
+                    goto out;
+                total = total + current_count;
+                goto print;
+            }
+            continue;
         }
+
+    print:
+        cli_out("%-48s %15d", name, current_count);
+    }
 out:
-        cli_out ("----------------------------------------------\n");
-        return;
+    cli_out("\ntotal clients for volume %s : %d ", volname, total);
+    cli_out(
+        "-----------------------------------------------------------------\n");
+    return;
 }
 
-void
-cli_print_volume_status_inode_entry (dict_t *dict, char *prefix)
+static void
+cli_print_volume_status_clients(dict_t *dict, gf_boolean_t notbrick)
 {
-        int             ret = -1;
-        char            key[1024] = {0,};
-        char            *gfid = NULL;
-        uint64_t        nlookup = 0;
-        uint32_t        ref = 0;
-        int             ia_type = 0;
-        char            inode_type;
-
-        GF_ASSERT (dict);
-        GF_ASSERT (prefix);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.gfid", prefix);
-        ret = dict_get_str (dict, key, &gfid);
-        if (ret)
-                goto out;
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.nlookup", prefix);
-        ret = dict_get_uint64 (dict, key, &nlookup);
-        if (ret)
-                goto out;
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.ref", prefix);
-        ret = dict_get_uint32 (dict, key, &ref);
-        if (ret)
-                goto out;
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.ia_type", prefix);
-        ret = dict_get_int32 (dict, key, &ia_type);
-        if (ret)
-                goto out;
+    int ret = -1;
+    char *volname = NULL;
+    int brick_index_max = -1;
+    int other_count = 0;
+    int index_max = 0;
+    char *hostname = NULL;
+    char *path = NULL;
+    int online = -1;
+    int client_count = 0;
+    char *clientname = NULL;
+    uint64_t bytesread = 0;
+    uint64_t byteswrite = 0;
+    uint32_t opversion = 0;
+    char key[128] = {
+        0,
+    };
+    int i = 0;
+    int j = 0;
+
+    GF_ASSERT(dict);
+
+    ret = dict_get_str_sizen(dict, "volname", &volname);
+    if (ret)
+        goto out;
+    cli_out("Client connections for volume %s", volname);
+
+    ret = dict_get_int32_sizen(dict, "brick-index-max", &brick_index_max);
+    if (ret)
+        goto out;
+    ret = dict_get_int32_sizen(dict, "other-count", &other_count);
+    if (ret)
+        goto out;
+
+    index_max = brick_index_max + other_count;
+
+    for (i = 0; i <= index_max; i++) {
+        hostname = NULL;
+        path = NULL;
+        online = -1;
+        client_count = 0;
+        clientname = NULL;
+        bytesread = 0;
+        byteswrite = 0;
+
+        cli_out("----------------------------------------------");
+
+        snprintf(key, sizeof(key), "brick%d.hostname", i);
+        ret = dict_get_str(dict, key, &hostname);
+
+        snprintf(key, sizeof(key), "brick%d.path", i);
+        ret = dict_get_str(dict, key, &path);
+
+        if (hostname && path) {
+            if (notbrick)
+                cli_out("%s : %s", hostname, path);
+            else
+                cli_out("Brick : %s:%s", hostname, path);
+        }
+        snprintf(key, sizeof(key), "brick%d.status", i);
+        ret = dict_get_int32(dict, key, &online);
+        if (!online) {
+            if (notbrick)
+                cli_out("%s is offline", hostname);
+            else
+                cli_out("Brick is offline");
+            continue;
+        }
+
+        snprintf(key, sizeof(key), "brick%d.clientcount", i);
+        ret = dict_get_int32(dict, key, &client_count);
+
+        if (hostname && path)
+            cli_out("Clients connected : %d", client_count);
+        if (client_count == 0)
+            continue;
+
+        cli_out("%-48s %15s %15s %15s", "Hostname", "BytesRead", "BytesWritten",
+                "OpVersion");
+        cli_out("%-48s %15s %15s %15s", "--------", "---------", "------------",
+                "---------");
+        for (j = 0; j < client_count; j++) {
+            snprintf(key, sizeof(key), "brick%d.client%d.hostname", i, j);
+            ret = dict_get_str(dict, key, &clientname);
+
+            snprintf(key, sizeof(key), "brick%d.client%d.bytesread", i, j);
+            ret = dict_get_uint64(dict, key, &bytesread);
+
+            snprintf(key, sizeof(key), "brick%d.client%d.byteswrite", i, j);
+            ret = dict_get_uint64(dict, key, &byteswrite);
+
+            snprintf(key, sizeof(key), "brick%d.client%d.opversion", i, j);
+            ret = dict_get_uint32(dict, key, &opversion);
+
+            cli_out("%-48s %15" PRIu64 " %15" PRIu64 " %15" PRIu32, clientname,
+                    bytesread, byteswrite, opversion);
+        }
+    }
+out:
+    cli_out("----------------------------------------------\n");
+    return;
+}
 
-        switch (ia_type) {
+#ifdef DEBUG /* this function is only used in debug */
+static void
+cli_print_volume_status_inode_entry(dict_t *dict, char *prefix)
+{
+    int ret = -1;
+    char key[1024] = {
+        0,
+    };
+    char *gfid = NULL;
+    uint64_t nlookup = 0;
+    uint32_t ref = 0;
+    int ia_type = 0;
+    char inode_type;
+
+    GF_ASSERT(dict);
+    GF_ASSERT(prefix);
+
+    snprintf(key, sizeof(key), "%s.gfid", prefix);
+    ret = dict_get_str(dict, key, &gfid);
+    if (ret)
+        goto out;
+
+    snprintf(key, sizeof(key), "%s.nlookup", prefix);
+    ret = dict_get_uint64(dict, key, &nlookup);
+    if (ret)
+        goto out;
+
+    snprintf(key, sizeof(key), "%s.ref", prefix);
+    ret = dict_get_uint32(dict, key, &ref);
+    if (ret)
+        goto out;
+
+    snprintf(key, sizeof(key), "%s.ia_type", prefix);
+    ret = dict_get_int32(dict, key, &ia_type);
+    if (ret)
+        goto out;
+
+    switch (ia_type) {
         case IA_IFREG:
-                inode_type = 'R';
-                break;
+            inode_type = 'R';
+            break;
         case IA_IFDIR:
-                inode_type = 'D';
-                break;
+            inode_type = 'D';
+            break;
         case IA_IFLNK:
-                inode_type = 'L';
-                break;
+            inode_type = 'L';
+            break;
         case IA_IFBLK:
-                inode_type = 'B';
-                break;
+            inode_type = 'B';
+            break;
         case IA_IFCHR:
-                inode_type = 'C';
-                break;
+            inode_type = 'C';
+            break;
         case IA_IFIFO:
-                inode_type = 'F';
-                break;
+            inode_type = 'F';
+            break;
         case IA_IFSOCK:
-                inode_type = 'S';
-                break;
+            inode_type = 'S';
+            break;
         default:
-                inode_type = 'I';
-                break;
-        }
+            inode_type = 'I';
+            break;
+    }
 
-        cli_out ("%-40s %14"PRIu64" %14"PRIu32" %9c",
-                 gfid, nlookup, ref, inode_type);
+    cli_out("%-40s %14" PRIu64 " %14" PRIu32 " %9c", gfid, nlookup, ref,
+            inode_type);
 
 out:
-        return;
-
+    return;
 }
+#endif
 
-void
-cli_print_volume_status_itables (dict_t *dict, char *prefix)
+static void
+cli_print_volume_status_itables(dict_t *dict, char *prefix)
 {
-        int             ret = -1;
-        char            key[1024] = {0,};
-        uint32_t        active_size = 0;
-        uint32_t        lru_size = 0;
-        uint32_t        purge_size = 0;
-        int             i =0;
-
-        GF_ASSERT (dict);
-        GF_ASSERT (prefix);
+    int ret = -1;
+    char key[1024] = {
+        0,
+    };
+    uint32_t active_size = 0;
+    uint32_t lru_size = 0;
+    uint32_t purge_size = 0;
+    uint32_t lru_limit = 0;
+#ifdef DEBUG
+    int i = 0;
+#endif
+    GF_ASSERT(dict);
+    GF_ASSERT(prefix);
+
+    snprintf(key, sizeof(key), "%s.lru_limit", prefix);
+    ret = dict_get_uint32(dict, key, &lru_limit);
+    if (ret)
+        goto out;
+    cli_out("LRU limit     : %u", lru_limit);
+
+    snprintf(key, sizeof(key), "%s.active_size", prefix);
+    ret = dict_get_uint32(dict, key, &active_size);
+    if (ret)
+        goto out;
+
+#ifdef DEBUG
+    if (active_size != 0) {
+        cli_out("Active inodes:");
+        cli_out("%-40s %14s %14s %9s", "GFID", "Lookups", "Ref", "IA type");
+        cli_out("%-40s %14s %14s %9s", "----", "-------", "---", "-------");
+    }
+    for (i = 0; i < active_size; i++) {
+        snprintf(key, sizeof(key), "%s.active%d", prefix, i);
+        cli_print_volume_status_inode_entry(dict, key);
+    }
+    cli_out(" ");
+#else
+    cli_out("Active Inodes : %u", active_size);
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.active_size", prefix);
-        ret = dict_get_uint32 (dict, key, &active_size);
-        if (ret)
-                goto out;
-        if (active_size != 0) {
-                cli_out ("Active inodes:");
-                cli_out ("%-40s %14s %14s %9s", "GFID", "Lookups", "Ref",
-                         "IA type");
-                cli_out ("%-40s %14s %14s %9s", "----", "-------", "---",
-                         "-------");
-        }
-        for (i = 0; i < active_size; i++) {
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.active%d", prefix, i);
-                cli_print_volume_status_inode_entry (dict, key);
-        }
-        cli_out (" ");
+#endif
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.lru_size", prefix);
-        ret = dict_get_uint32 (dict, key, &lru_size);
-        if (ret)
-                goto out;
-        if (lru_size != 0) {
-                cli_out ("LRU inodes:");
-                cli_out ("%-40s %14s %14s %9s", "GFID", "Lookups", "Ref",
-                         "IA type");
-                cli_out ("%-40s %14s %14s %9s", "----", "-------", "---",
-                         "-------");
-        }
-        for (i = 0; i < lru_size; i++) {
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.lru%d", prefix, i);
-                cli_print_volume_status_inode_entry (dict, key);
-        }
-        cli_out (" ");
+    snprintf(key, sizeof(key), "%s.lru_size", prefix);
+    ret = dict_get_uint32(dict, key, &lru_size);
+    if (ret)
+        goto out;
+
+#ifdef DEBUG
+    if (lru_size != 0) {
+        cli_out("LRU inodes:");
+        cli_out("%-40s %14s %14s %9s", "GFID", "Lookups", "Ref", "IA type");
+        cli_out("%-40s %14s %14s %9s", "----", "-------", "---", "-------");
+    }
+    for (i = 0; i < lru_size; i++) {
+        snprintf(key, sizeof(key), "%s.lru%d", prefix, i);
+        cli_print_volume_status_inode_entry(dict, key);
+    }
+    cli_out(" ");
+#else
+    cli_out("LRU Inodes    : %u", lru_size);
+#endif
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.purge_size", prefix);
-        ret = dict_get_uint32 (dict, key, &purge_size);
-        if (ret)
-                goto out;
-        if (purge_size != 0) {
-                cli_out ("Purged inodes:");
-                cli_out ("%-40s %14s %14s %9s", "GFID", "Lookups", "Ref",
-                         "IA type");
-                cli_out ("%-40s %14s %14s %9s", "----", "-------", "---",
-                         "-------");
-        }
-        for (i = 0; i < purge_size; i++) {
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.purge%d", prefix, i);
-                cli_print_volume_status_inode_entry (dict, key);
-        }
+    snprintf(key, sizeof(key), "%s.purge_size", prefix);
+    ret = dict_get_uint32(dict, key, &purge_size);
+    if (ret)
+        goto out;
+#ifdef DEBUG
+    if (purge_size != 0) {
+        cli_out("Purged inodes:");
+        cli_out("%-40s %14s %14s %9s", "GFID", "Lookups", "Ref", "IA type");
+        cli_out("%-40s %14s %14s %9s", "----", "-------", "---", "-------");
+    }
+    for (i = 0; i < purge_size; i++) {
+        snprintf(key, sizeof(key), "%s.purge%d", prefix, i);
+        cli_print_volume_status_inode_entry(dict, key);
+    }
+#else
+    cli_out("Purge Inodes  : %u", purge_size);
+#endif
 
 out:
-        return;
+    return;
 }
 
-void
-cli_print_volume_status_inode (dict_t *dict, gf_boolean_t notbrick)
+static void
+cli_print_volume_status_inode(dict_t *dict, gf_boolean_t notbrick)
 {
-        int             ret = -1;
-        char            *volname = NULL;
-        int             brick_index_max = -1;
-        int             other_count = 0;
-        int             index_max = 0;
-        char            *hostname = NULL;
-        char            *path = NULL;
-        int             online = -1;
-        int             conn_count = 0;
-        char            key[1024] = {0,};
-        int             i = 0;
-        int             j = 0;
-
-        GF_ASSERT (dict);
-
-        ret = dict_get_str (dict, "volname", &volname);
+    int ret = -1;
+    char *volname = NULL;
+    int brick_index_max = -1;
+    int other_count = 0;
+    int index_max = 0;
+    char *hostname = NULL;
+    char *path = NULL;
+    int online = -1;
+    int conn_count = 0;
+    char key[64] = {
+        0,
+    };
+    int i = 0;
+    int j = 0;
+
+    GF_ASSERT(dict);
+
+    ret = dict_get_str_sizen(dict, "volname", &volname);
+    if (ret)
+        goto out;
+    cli_out("Inode tables for volume %s", volname);
+
+    ret = dict_get_int32_sizen(dict, "brick-index-max", &brick_index_max);
+    if (ret)
+        goto out;
+    ret = dict_get_int32_sizen(dict, "other-count", &other_count);
+    if (ret)
+        goto out;
+
+    index_max = brick_index_max + other_count;
+
+    for (i = 0; i <= index_max; i++) {
+        cli_out("----------------------------------------------");
+
+        snprintf(key, sizeof(key), "brick%d.hostname", i);
+        ret = dict_get_str(dict, key, &hostname);
         if (ret)
-                goto out;
-        cli_out ("Inode tables for volume %s", volname);
-
-        ret = dict_get_int32 (dict, "brick-index-max", &brick_index_max);
+            goto out;
+        snprintf(key, sizeof(key), "brick%d.path", i);
+        ret = dict_get_str(dict, key, &path);
         if (ret)
-                goto out;
-        ret = dict_get_int32 (dict, "other-count", &other_count);
-        if (ret)
-                goto out;
-
-        index_max = brick_index_max + other_count;
-
-        for ( i = 0; i <= index_max; i++) {
-                cli_out ("----------------------------------------------");
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.hostname", i);
-                ret = dict_get_str (dict, key, &hostname);
-                if (ret)
-                        goto out;
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.path", i);
-                ret = dict_get_str (dict, key, &path);
-                if (ret)
-                        goto out;
-                if (notbrick)
-                        cli_out ("%s : %s", hostname, path);
-                else
-                        cli_out ("Brick : %s:%s", hostname, path);
+            goto out;
+        if (notbrick)
+            cli_out("%s : %s", hostname, path);
+        else
+            cli_out("Brick : %s:%s", hostname, path);
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.status", i);
-                ret = dict_get_int32 (dict, key, &online);
-                if (ret)
-                        goto out;
-                if (!online) {
-                        if (notbrick)
-                                cli_out ("%s is offline", hostname);
-                        else
-                                cli_out ("Brick is offline");
-                        continue;
-                }
+        snprintf(key, sizeof(key), "brick%d.status", i);
+        ret = dict_get_int32(dict, key, &online);
+        if (ret)
+            goto out;
+        if (!online) {
+            if (notbrick)
+                cli_out("%s is offline", hostname);
+            else
+                cli_out("Brick is offline");
+            continue;
+        }
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.conncount", i);
-                ret = dict_get_int32 (dict, key, &conn_count);
-                if (ret)
-                        goto out;
+        snprintf(key, sizeof(key), "brick%d.conncount", i);
+        ret = dict_get_int32(dict, key, &conn_count);
+        if (ret)
+            goto out;
 
-                for (j = 0; j < conn_count; j++) {
-                        if (conn_count > 1)
-                                cli_out ("Connection %d:", j+1);
+        for (j = 0; j < conn_count; j++) {
+            if (conn_count > 1)
+                cli_out("Connection %d:", j + 1);
 
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key), "brick%d.conn%d.itable",
-                                  i, j);
-                        cli_print_volume_status_itables (dict, key);
-                        cli_out (" ");
-                }
+            snprintf(key, sizeof(key), "brick%d.conn%d.itable", i, j);
+            cli_print_volume_status_itables(dict, key);
+            cli_out(" ");
         }
+    }
 out:
-        cli_out ("----------------------------------------------");
-        return;
+    cli_out("----------------------------------------------");
+    return;
 }
 
 void
-cli_print_volume_status_fdtable (dict_t *dict, char *prefix)
-{
-        int             ret = -1;
-        char            key[1024] = {0,};
-        int             refcount = 0;
-        uint32_t        maxfds = 0;
-        int             firstfree = 0;
-        int             openfds = 0;
-        int             fd_pid = 0;
-        int             fd_refcount = 0;
-        int             fd_flags = 0;
-        int             i = 0;
-
-        GF_ASSERT (dict);
-        GF_ASSERT (prefix);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.refcount", prefix);
-        ret = dict_get_int32 (dict, key, &refcount);
+cli_print_volume_status_fdtable(dict_t *dict, char *prefix)
+{
+    int ret = -1;
+    char key[256] = {
+        0,
+    };
+    int refcount = 0;
+    uint32_t maxfds = 0;
+    int firstfree = 0;
+    int openfds = 0;
+    int fd_pid = 0;
+    int fd_refcount = 0;
+    int fd_flags = 0;
+    int i = 0;
+
+    GF_ASSERT(dict);
+    GF_ASSERT(prefix);
+
+    snprintf(key, sizeof(key), "%s.refcount", prefix);
+    ret = dict_get_int32(dict, key, &refcount);
+    if (ret)
+        goto out;
+
+    snprintf(key, sizeof(key), "%s.maxfds", prefix);
+    ret = dict_get_uint32(dict, key, &maxfds);
+    if (ret)
+        goto out;
+
+    snprintf(key, sizeof(key), "%s.firstfree", prefix);
+    ret = dict_get_int32(dict, key, &firstfree);
+    if (ret)
+        goto out;
+
+    cli_out("RefCount = %d  MaxFDs = %d  FirstFree = %d", refcount, maxfds,
+            firstfree);
+
+    snprintf(key, sizeof(key), "%s.openfds", prefix);
+    ret = dict_get_int32(dict, key, &openfds);
+    if (ret)
+        goto out;
+    if (0 == openfds) {
+        cli_err("No open fds");
+        goto out;
+    }
+
+    cli_out("%-19s %-19s %-19s %-19s", "FD Entry", "PID", "RefCount", "Flags");
+    cli_out("%-19s %-19s %-19s %-19s", "--------", "---", "--------", "-----");
+
+    for (i = 0; i < maxfds; i++) {
+        snprintf(key, sizeof(key), "%s.fdentry%d.pid", prefix, i);
+        ret = dict_get_int32(dict, key, &fd_pid);
         if (ret)
-                goto out;
+            continue;
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.maxfds", prefix);
-        ret = dict_get_uint32 (dict, key, &maxfds);
+        snprintf(key, sizeof(key), "%s.fdentry%d.refcount", prefix, i);
+        ret = dict_get_int32(dict, key, &fd_refcount);
         if (ret)
-                goto out;
+            continue;
 
-        memset (key, 0 ,sizeof (key));
-        snprintf (key, sizeof (key), "%s.firstfree", prefix);
-        ret = dict_get_int32 (dict, key, &firstfree);
+        snprintf(key, sizeof(key), "%s.fdentry%d.flags", prefix, i);
+        ret = dict_get_int32(dict, key, &fd_flags);
         if (ret)
-                goto out;
+            continue;
 
-        cli_out ("RefCount = %d  MaxFDs = %d  FirstFree = %d",
-                 refcount, maxfds, firstfree);
+        cli_out("%-19d %-19d %-19d %-19d", i, fd_pid, fd_refcount, fd_flags);
+    }
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.openfds", prefix);
-        ret = dict_get_int32 (dict, key, &openfds);
+out:
+    return;
+}
+
+static void
+cli_print_volume_status_fd(dict_t *dict, gf_boolean_t notbrick)
+{
+    int ret = -1;
+    char *volname = NULL;
+    int brick_index_max = -1;
+    int other_count = 0;
+    int index_max = 0;
+    char *hostname = NULL;
+    char *path = NULL;
+    int online = -1;
+    int conn_count = 0;
+    char key[64] = {
+        0,
+    };
+    int i = 0;
+    int j = 0;
+
+    GF_ASSERT(dict);
+
+    ret = dict_get_str_sizen(dict, "volname", &volname);
+    if (ret)
+        goto out;
+    cli_out("FD tables for volume %s", volname);
+
+    ret = dict_get_int32_sizen(dict, "brick-index-max", &brick_index_max);
+    if (ret)
+        goto out;
+    ret = dict_get_int32_sizen(dict, "other-count", &other_count);
+    if (ret)
+        goto out;
+
+    index_max = brick_index_max + other_count;
+
+    for (i = 0; i <= index_max; i++) {
+        cli_out("----------------------------------------------");
+
+        snprintf(key, sizeof(key), "brick%d.hostname", i);
+        ret = dict_get_str(dict, key, &hostname);
         if (ret)
-                goto out;
-        if (0 == openfds) {
-                cli_err ("No open fds");
-                goto out;
-        }
+            goto out;
+        snprintf(key, sizeof(key), "brick%d.path", i);
+        ret = dict_get_str(dict, key, &path);
+        if (ret)
+            goto out;
 
-        cli_out ("%-19s %-19s %-19s %-19s", "FD Entry", "PID",
-                 "RefCount", "Flags");
-        cli_out ("%-19s %-19s %-19s %-19s", "--------", "---",
-                 "--------", "-----");
+        if (notbrick)
+            cli_out("%s : %s", hostname, path);
+        else
+            cli_out("Brick : %s:%s", hostname, path);
 
-        for (i = 0; i < maxfds ; i++) {
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.fdentry%d.pid", prefix, i);
-                ret = dict_get_int32 (dict, key, &fd_pid);
-                if (ret)
-                        continue;
+        snprintf(key, sizeof(key), "brick%d.status", i);
+        ret = dict_get_int32(dict, key, &online);
+        if (ret)
+            goto out;
+        if (!online) {
+            if (notbrick)
+                cli_out("%s is offline", hostname);
+            else
+                cli_out("Brick is offline");
+            continue;
+        }
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.fdentry%d.refcount",
-                          prefix, i);
-                ret = dict_get_int32 (dict, key, &fd_refcount);
-                if (ret)
-                        continue;
+        snprintf(key, sizeof(key), "brick%d.conncount", i);
+        ret = dict_get_int32(dict, key, &conn_count);
+        if (ret)
+            goto out;
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.fdentry%d.flags", prefix, i);
-                ret = dict_get_int32 (dict, key, &fd_flags);
-                if (ret)
-                        continue;
+        for (j = 0; j < conn_count; j++) {
+            cli_out("Connection %d:", j + 1);
 
-                cli_out ("%-19d %-19d %-19d %-19d", i, fd_pid, fd_refcount,
-                         fd_flags);
+            snprintf(key, sizeof(key), "brick%d.conn%d.fdtable", i, j);
+            cli_print_volume_status_fdtable(dict, key);
+            cli_out(" ");
         }
-
+    }
 out:
-        return;
+    cli_out("----------------------------------------------");
+    return;
 }
 
-void
-cli_print_volume_status_fd (dict_t *dict, gf_boolean_t notbrick)
+static void
+cli_print_volume_status_call_frame(dict_t *dict, char *prefix)
 {
-        int             ret = -1;
-        char            *volname = NULL;
-        int             brick_index_max = -1;
-        int             other_count = 0;
-        int             index_max = 0;
-        char            *hostname = NULL;
-        char            *path = NULL;
-        int             online = -1;
-        int             conn_count = 0;
-        char            key[1024] = {0,};
-        int             i = 0;
-        int             j = 0;
-
-        GF_ASSERT (dict);
-
-        ret = dict_get_str (dict, "volname", &volname);
-        if (ret)
-                goto out;
-        cli_out ("FD tables for volume %s", volname);
+    int ret = -1;
+    char key[1024] = {
+        0,
+    };
+    int ref_count = 0;
+    char *translator = 0;
+    int complete = 0;
+    char *parent = NULL;
+    char *wind_from = NULL;
+    char *wind_to = NULL;
+    char *unwind_from = NULL;
+    char *unwind_to = NULL;
+
+    if (!dict || !prefix)
+        return;
 
-        ret = dict_get_int32 (dict, "brick-index-max", &brick_index_max);
-        if (ret)
-                goto out;
-        ret = dict_get_int32 (dict, "other-count", &other_count);
-        if (ret)
-                goto out;
+    snprintf(key, sizeof(key), "%s.refcount", prefix);
+    ret = dict_get_int32(dict, key, &ref_count);
+    if (ret)
+        return;
 
-        index_max = brick_index_max + other_count;
+    snprintf(key, sizeof(key), "%s.translator", prefix);
+    ret = dict_get_str(dict, key, &translator);
+    if (ret)
+        return;
 
-        for (i = 0; i <= index_max; i++) {
-                cli_out ("----------------------------------------------");
+    snprintf(key, sizeof(key), "%s.complete", prefix);
+    ret = dict_get_int32(dict, key, &complete);
+    if (ret)
+        return;
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.hostname", i);
-                ret = dict_get_str (dict, key, &hostname);
-                if (ret)
-                        goto out;
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.path", i);
-                ret = dict_get_str (dict, key, &path);
-                if (ret)
-                        goto out;
+    cli_out("  Ref Count   = %d", ref_count);
+    cli_out("  Translator  = %s", translator);
+    cli_out("  Completed   = %s", (complete ? "Yes" : "No"));
 
-                if (notbrick)
-                        cli_out ("%s : %s", hostname, path);
-                else
-                        cli_out ("Brick : %s:%s", hostname, path);
+    snprintf(key, sizeof(key), "%s.parent", prefix);
+    ret = dict_get_str(dict, key, &parent);
+    if (!ret)
+        cli_out("  Parent      = %s", parent);
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.status", i);
-                ret = dict_get_int32 (dict, key, &online);
-                if (ret)
-                        goto out;
-                if (!online) {
-                        if (notbrick)
-                                cli_out ("%s is offline", hostname);
-                        else
-                                cli_out ("Brick is offline");
-                        continue;
-                }
+    snprintf(key, sizeof(key), "%s.windfrom", prefix);
+    ret = dict_get_str(dict, key, &wind_from);
+    if (!ret)
+        cli_out("  Wind From   = %s", wind_from);
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.conncount", i);
-                ret = dict_get_int32 (dict, key, &conn_count);
-                if (ret)
-                        goto out;
+    snprintf(key, sizeof(key), "%s.windto", prefix);
+    ret = dict_get_str(dict, key, &wind_to);
+    if (!ret)
+        cli_out("  Wind To     = %s", wind_to);
 
-                for (j = 0; j < conn_count; j++) {
-                        cli_out ("Connection %d:", j+1);
+    snprintf(key, sizeof(key), "%s.unwindfrom", prefix);
+    ret = dict_get_str(dict, key, &unwind_from);
+    if (!ret)
+        cli_out("  Unwind From = %s", unwind_from);
 
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key), "brick%d.conn%d.fdtable",
-                                  i, j);
-                        cli_print_volume_status_fdtable (dict, key);
-                        cli_out (" ");
-                }
-        }
-out:
-        cli_out ("----------------------------------------------");
-        return;
+    snprintf(key, sizeof(key), "%s.unwindto", prefix);
+    ret = dict_get_str(dict, key, &unwind_to);
+    if (!ret)
+        cli_out("  Unwind To   = %s", unwind_to);
 }
 
-void
-cli_print_volume_status_call_frame (dict_t *dict, char *prefix)
+static void
+cli_print_volume_status_call_stack(dict_t *dict, char *prefix)
 {
-        int             ret = -1;
-        char            key[1024] = {0,};
-        int             ref_count = 0;
-        char            *translator = 0;
-        int             complete = 0;
-        char            *parent = NULL;
-        char            *wind_from = NULL;
-        char            *wind_to = NULL;
-        char            *unwind_from = NULL;
-        char            *unwind_to = NULL;
+    int ret = -1;
+    char key[256] = {
+        0,
+    };
+    int uid = 0;
+    int gid = 0;
+    int pid = 0;
+    uint64_t unique = 0;
+    // char            *op = NULL;
+    int count = 0;
+    int i = 0;
+
+    if (!prefix)
+        return;
 
-        if (!dict || !prefix)
-                return;
+    snprintf(key, sizeof(key), "%s.uid", prefix);
+    ret = dict_get_int32(dict, key, &uid);
+    if (ret)
+        return;
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.refcount", prefix);
-        ret = dict_get_int32 (dict, key, &ref_count);
-        if (ret)
-                return;
+    snprintf(key, sizeof(key), "%s.gid", prefix);
+    ret = dict_get_int32(dict, key, &gid);
+    if (ret)
+        return;
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.translator", prefix);
-        ret = dict_get_str (dict, key, &translator);
-        if (ret)
-                return;
+    snprintf(key, sizeof(key), "%s.pid", prefix);
+    ret = dict_get_int32(dict, key, &pid);
+    if (ret)
+        return;
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.complete", prefix);
-        ret = dict_get_int32 (dict, key, &complete);
-        if (ret)
-                return;
+    snprintf(key, sizeof(key), "%s.unique", prefix);
+    ret = dict_get_uint64(dict, key, &unique);
+    if (ret)
+        return;
 
-        cli_out ("  Ref Count   = %d", ref_count);
-        cli_out ("  Translator  = %s", translator);
-        cli_out ("  Completed   = %s", (complete ? "Yes" : "No"));
+    /*
+    snprintf (key, sizeof (key), "%s.op", prefix);
+    ret = dict_get_str (dict, key, &op);
+    if (ret)
+            return;
+    */
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.parent", prefix);
-        ret = dict_get_str (dict, key, &parent);
-        if (!ret)
-                cli_out ("  Parent      = %s", parent);
+    snprintf(key, sizeof(key), "%s.count", prefix);
+    ret = dict_get_int32(dict, key, &count);
+    if (ret)
+        return;
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.windfrom", prefix);
-        ret = dict_get_str (dict, key, &wind_from);
-        if (!ret)
-                cli_out ("  Wind From   = %s", wind_from);
+    cli_out(" UID    : %d", uid);
+    cli_out(" GID    : %d", gid);
+    cli_out(" PID    : %d", pid);
+    cli_out(" Unique : %" PRIu64, unique);
+    // cli_out ("\tOp     : %s", op);
+    cli_out(" Frames : %d", count);
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.windto", prefix);
-        ret = dict_get_str (dict, key, &wind_to);
-        if (!ret)
-                cli_out ("  Wind To     = %s", wind_to);
+    for (i = 0; i < count; i++) {
+        cli_out(" Frame %d", i + 1);
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.unwindfrom", prefix);
-        ret = dict_get_str (dict, key, &unwind_from);
-        if (!ret)
-                cli_out ("  Unwind From = %s", unwind_from);
+        snprintf(key, sizeof(key), "%s.frame%d", prefix, i);
+        cli_print_volume_status_call_frame(dict, key);
+    }
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.unwindto", prefix);
-        ret = dict_get_str (dict, key, &unwind_to);
-        if (!ret)
-                cli_out ("  Unwind To   = %s", unwind_to);
+    cli_out(" ");
 }
 
-void
-cli_print_volume_status_call_stack (dict_t *dict, char *prefix)
+static void
+cli_print_volume_status_callpool(dict_t *dict, gf_boolean_t notbrick)
 {
-        int             ret = -1;
-        char            key[1024] = {0,};
-        int             uid = 0;
-        int             gid = 0;
-        int             pid = 0;
-        uint64_t        unique = 0;
-        //char            *op = NULL;
-        int             count = 0;
-        int             i = 0;
-
-        if (!dict || !prefix)
-                return;
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.uid", prefix);
-        ret = dict_get_int32 (dict, key, &uid);
-        if (ret)
-                return;
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.gid", prefix);
-        ret = dict_get_int32 (dict, key, &gid);
+    int ret = -1;
+    char *volname = NULL;
+    int brick_index_max = -1;
+    int other_count = 0;
+    int index_max = 0;
+    char *hostname = NULL;
+    char *path = NULL;
+    int online = -1;
+    int call_count = 0;
+    char key[64] = {
+        0,
+    };
+    int i = 0;
+    int j = 0;
+
+    GF_ASSERT(dict);
+
+    ret = dict_get_str_sizen(dict, "volname", &volname);
+    if (ret)
+        goto out;
+    cli_out("Pending calls for volume %s", volname);
+
+    ret = dict_get_int32_sizen(dict, "brick-index-max", &brick_index_max);
+    if (ret)
+        goto out;
+    ret = dict_get_int32_sizen(dict, "other-count", &other_count);
+    if (ret)
+        goto out;
+
+    index_max = brick_index_max + other_count;
+
+    for (i = 0; i <= index_max; i++) {
+        cli_out("----------------------------------------------");
+
+        snprintf(key, sizeof(key), "brick%d.hostname", i);
+        ret = dict_get_str(dict, key, &hostname);
         if (ret)
-                return;
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.pid", prefix);
-        ret = dict_get_int32 (dict, key, &pid);
+            goto out;
+        snprintf(key, sizeof(key), "brick%d.path", i);
+        ret = dict_get_str(dict, key, &path);
         if (ret)
-                return;
+            goto out;
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.unique", prefix);
-        ret = dict_get_uint64 (dict, key, &unique);
-        if (ret)
-                return;
+        if (notbrick)
+            cli_out("%s : %s", hostname, path);
+        else
+            cli_out("Brick : %s:%s", hostname, path);
 
-        /*
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.op", prefix);
-        ret = dict_get_str (dict, key, &op);
+        snprintf(key, sizeof(key), "brick%d.status", i);
+        ret = dict_get_int32(dict, key, &online);
         if (ret)
-                return;
-        */
-
+            goto out;
+        if (!online) {
+            if (notbrick)
+                cli_out("%s is offline", hostname);
+            else
+                cli_out("Brick is offline");
+            continue;
+        }
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.count", prefix);
-        ret = dict_get_int32 (dict, key, &count);
+        snprintf(key, sizeof(key), "brick%d.callpool.count", i);
+        ret = dict_get_int32(dict, key, &call_count);
         if (ret)
-                return;
+            goto out;
+        cli_out("Pending calls: %d", call_count);
 
-        cli_out (" UID    : %d", uid);
-        cli_out (" GID    : %d", gid);
-        cli_out (" PID    : %d", pid);
-        cli_out (" Unique : %"PRIu64, unique);
-        //cli_out ("\tOp     : %s", op);
-        cli_out (" Frames : %d", count);
+        if (0 == call_count)
+            continue;
 
-        for (i = 0; i < count; i++) {
-                cli_out (" Frame %d", i+1);
+        for (j = 0; j < call_count; j++) {
+            cli_out("Call Stack%d", j + 1);
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.frame%d", prefix, i);
-                cli_print_volume_status_call_frame (dict, key);
+            snprintf(key, sizeof(key), "brick%d.callpool.stack%d", i, j);
+            cli_print_volume_status_call_stack(dict, key);
         }
+    }
 
-        cli_out (" ");
+out:
+    cli_out("----------------------------------------------");
+    return;
 }
 
-void
-cli_print_volume_status_callpool (dict_t *dict, gf_boolean_t notbrick)
+static void
+cli_print_volume_status_tasks(dict_t *dict)
 {
-        int             ret = -1;
-        char            *volname = NULL;
-        int             brick_index_max = -1;
-        int             other_count = 0;
-        int             index_max = 0;
-        char            *hostname = NULL;
-        char            *path = NULL;
-        int             online = -1;
-        int             call_count = 0;
-        char            key[1024] = {0,};
-        int             i = 0;
-        int             j = 0;
+    int ret = -1;
+    int i = 0;
+    int j = 0;
+    int count = 0;
+    int task_count = 0;
+    int status = 0;
+    char *op = NULL;
+    char *task_id_str = NULL;
+    char *volname = NULL;
+    char key[64] = {
+        0,
+    };
+    char task[32] = {
+        0,
+    };
+    char *brick = NULL;
+
+    ret = dict_get_int32_sizen(dict, "tasks", &task_count);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get tasks count");
+        return;
+    }
 
-        GF_ASSERT (dict);
+    ret = dict_get_str_sizen(dict, "volname", &volname);
+    if (ret)
+        goto out;
 
-        ret = dict_get_str (dict, "volname", &volname);
-        if (ret)
-                goto out;
-        cli_out ("Pending calls for volume %s", volname);
+    cli_out("Task Status of Volume %s", volname);
+    cli_print_line(CLI_BRICK_STATUS_LINE_LEN);
 
-        ret = dict_get_int32 (dict, "brick-index-max", &brick_index_max);
-        if (ret)
-                goto out;
-        ret = dict_get_int32 (dict, "other-count", &other_count);
+    if (task_count == 0) {
+        cli_out("There are no active volume tasks");
+        cli_out(" ");
+        return;
+    }
+
+    for (i = 0; i < task_count; i++) {
+        snprintf(key, sizeof(key), "task%d.type", i);
+        ret = dict_get_str(dict, key, &op);
         if (ret)
-                goto out;
+            return;
+        cli_out("%-20s : %-20s", "Task", op);
 
-        index_max = brick_index_max + other_count;
+        snprintf(key, sizeof(key), "task%d.id", i);
+        ret = dict_get_str(dict, key, &task_id_str);
+        if (ret)
+            return;
+        cli_out("%-20s : %-20s", "ID", task_id_str);
 
-        for (i = 0; i <= index_max; i++) {
-                cli_out ("----------------------------------------------");
+        snprintf(key, sizeof(key), "task%d.status", i);
+        ret = dict_get_int32(dict, key, &status);
+        if (ret)
+            return;
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.hostname", i);
-                ret = dict_get_str (dict, key, &hostname);
-                if (ret)
-                        goto out;
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.path", i);
-                ret = dict_get_str (dict, key, &path);
-                if (ret)
-                        goto out;
+        snprintf(task, sizeof(task), "task%d", i);
 
-                if (notbrick)
-                        cli_out ("%s : %s", hostname, path);
-                else
-                        cli_out ("Brick : %s:%s", hostname, path);
+        if (!strcmp(op, "Remove brick")) {
+            snprintf(key, sizeof(key), "%s.count", task);
+            ret = dict_get_int32(dict, key, &count);
+            if (ret)
+                goto out;
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.status", i);
-                ret = dict_get_int32 (dict, key, &online);
-                if (ret)
-                        goto out;
-                if (!online) {
-                        if (notbrick)
-                                cli_out ("%s is offline", hostname);
-                        else
-                                cli_out ("Brick is offline");
-                        continue;
-                }
+            cli_out("%-20s", "Removed bricks:");
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.callpool.count", i);
-                ret = dict_get_int32 (dict, key, &call_count);
+            for (j = 1; j <= count; j++) {
+                snprintf(key, sizeof(key), "%s.brick%d", task, j);
+                ret = dict_get_str(dict, key, &brick);
                 if (ret)
-                        goto out;
-                cli_out ("Pending calls: %d", call_count);
+                    goto out;
 
-                if (0 == call_count)
-                        continue;
-
-                for (j = 0; j < call_count; j++) {
-                        cli_out ("Call Stack%d", j+1);
-
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key),
-                                  "brick%d.callpool.stack%d", i, j);
-                        cli_print_volume_status_call_stack (dict, key);
-                }
+                cli_out("%-20s", brick);
+            }
         }
+        cli_out("%-20s : %-20s", "Status", cli_vol_task_status_str[status]);
+        cli_out(" ");
+    }
 
 out:
-        cli_out ("----------------------------------------------");
-        return;
+    return;
 }
 
-static void
-cli_print_volume_status_tasks (dict_t *dict)
-{
-        int             ret         = -1;
-        int             i           = 0;
-        int             j           = 0;
-        int             count       = 0;
-        int             task_count  = 0;
-        int             status      = 0;
-        char           *op          = NULL;
-        char           *task_id_str = NULL;
-        char           *volname     = NULL;
-        char            key[1024]   = {0,};
-        char            task[1024]  = {0,};
-        char           *brick       = NULL;
-        char           *src_brick   = NULL;
-        char           *dest_brick  = NULL;
-
-        ret = dict_get_str (dict, "volname", &volname);
-        if (ret)
-                goto out;
-
-        ret = dict_get_int32 (dict, "tasks", &task_count);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get tasks count");
-                return;
+static int
+gf_cli_status_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                  void *myframe)
+{
+    int ret = -1;
+    int brick_index_max = -1;
+    int other_count = 0;
+    int index_max = 0;
+    int i = 0;
+    int type = -1;
+    int hot_brick_count = -1;
+    int pid = -1;
+    uint32_t cmd = 0;
+    gf_boolean_t notbrick = _gf_false;
+    char key[64] = {
+        0,
+    };
+    char *hostname = NULL;
+    char *path = NULL;
+    char *volname = NULL;
+    dict_t *dict = NULL;
+    gf_cli_rsp rsp = {
+        0,
+    };
+    cli_volume_status_t status = {0};
+    cli_local_t *local = NULL;
+    gf_boolean_t wipe_local = _gf_false;
+    char msg[1024] = {
+        0,
+    };
+
+    GF_ASSERT(myframe);
+
+    if (req->rpc_status == -1)
+        goto out;
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    gf_log("cli", GF_LOG_DEBUG, "Received response to status cmd");
+
+    local = ((call_frame_t *)myframe)->local;
+    if (!local) {
+        local = cli_local_get();
+        if (!local) {
+            ret = -1;
+            gf_log("cli", GF_LOG_ERROR, "Failed to get local");
+            goto out;
         }
+        wipe_local = _gf_true;
+    }
 
-        cli_out ("Task Status of Volume %s", volname);
-        cli_print_line (CLI_BRICK_STATUS_LINE_LEN);
+    if (rsp.op_ret) {
+        if (strcmp(rsp.op_errstr, ""))
+            snprintf(msg, sizeof(msg), "%s", rsp.op_errstr);
+        else
+            snprintf(msg, sizeof(msg),
+                     "Unable to obtain volume status information.");
 
-        if (task_count == 0) {
-                cli_out ("There are no active volume tasks");
-                cli_out (" ");
-                return;
+        if (global_state->mode & GLUSTER_MODE_XML) {
+            if (!local->all)
+                cli_xml_output_str("volStatus", msg, rsp.op_ret, rsp.op_errno,
+                                   rsp.op_errstr);
+            ret = 0;
+            goto out;
         }
 
-        for (i = 0; i < task_count; i++) {
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "task%d.type", i);
-                ret = dict_get_str(dict, key, &op);
-                if (ret)
-                        return;
-                cli_out ("%-20s : %-20s", "Task", op);
+        cli_err("%s", msg);
+        if (local && local->all) {
+            ret = 0;
+            cli_out(" ");
+        } else
+            ret = -1;
+
+        goto out;
+    }
+
+    dict = dict_new();
+    if (!dict) {
+        gf_log(THIS->name, GF_LOG_ERROR, "Failed to create the dict");
+        ret = -1;
+        goto out;
+    }
+
+    ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &dict);
+    if (ret)
+        goto out;
+
+    ret = dict_get_uint32(dict, "cmd", &cmd);
+    if (ret)
+        goto out;
+
+    if ((cmd & GF_CLI_STATUS_ALL)) {
+        if (local && local->dict) {
+            dict_ref(dict);
+            ret = dict_set_static_ptr(local->dict, "rsp-dict", dict);
+            ret = 0;
+        } else {
+            gf_log("cli", GF_LOG_ERROR, "local not found");
+            ret = -1;
+        }
+        goto out;
+    }
+
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        if (!local->all) {
+            ret = cli_xml_output_vol_status_begin(local, rsp.op_ret,
+                                                  rsp.op_errno, rsp.op_errstr);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+                goto xml_end;
+            }
+        }
+        if (cmd & GF_CLI_STATUS_TASKS) {
+            ret = cli_xml_output_vol_status_tasks_detail(local, dict);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR, "Error outputting to xml");
+                goto xml_end;
+            }
+        } else {
+            ret = cli_xml_output_vol_status(local, dict);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+                goto xml_end;
+            }
+        }
+
+    xml_end:
+        if (!local->all) {
+            ret = cli_xml_output_vol_status_end(local);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+            }
+        }
+        goto out;
+    }
+
+    if ((cmd & GF_CLI_STATUS_NFS) || (cmd & GF_CLI_STATUS_SHD) ||
+        (cmd & GF_CLI_STATUS_QUOTAD) || (cmd & GF_CLI_STATUS_SNAPD) ||
+        (cmd & GF_CLI_STATUS_BITD) || (cmd & GF_CLI_STATUS_SCRUB))
+        notbrick = _gf_true;
+
+    switch (cmd & GF_CLI_STATUS_MASK) {
+        case GF_CLI_STATUS_MEM:
+            cli_print_volume_status_mem(dict, notbrick);
+            goto cont;
+            break;
+        case GF_CLI_STATUS_CLIENTS:
+            cli_print_volume_status_clients(dict, notbrick);
+            goto cont;
+            break;
+        case GF_CLI_STATUS_CLIENT_LIST:
+            cli_print_volume_status_client_list(dict, notbrick);
+            goto cont;
+            break;
+        case GF_CLI_STATUS_INODE:
+            cli_print_volume_status_inode(dict, notbrick);
+            goto cont;
+            break;
+        case GF_CLI_STATUS_FD:
+            cli_print_volume_status_fd(dict, notbrick);
+            goto cont;
+            break;
+        case GF_CLI_STATUS_CALLPOOL:
+            cli_print_volume_status_callpool(dict, notbrick);
+            goto cont;
+            break;
+        case GF_CLI_STATUS_TASKS:
+            cli_print_volume_status_tasks(dict);
+            goto cont;
+            break;
+        default:
+            break;
+    }
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "task%d.id", i);
-                ret = dict_get_str (dict, key, &task_id_str);
-                if (ret)
-                        return;
-                cli_out ("%-20s : %-20s", "ID", task_id_str);
+    ret = dict_get_str_sizen(dict, "volname", &volname);
+    if (ret)
+        goto out;
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "task%d.status", i);
-                ret = dict_get_int32 (dict, key, &status);
-                if (ret)
-                        return;
-
-                snprintf (task, sizeof (task), "task%d", i);
-
-                /*
-                   Replace brick only has two states - In progress and Complete
-                   Ref: xlators/mgmt/glusterd/src/glusterd-replace-brick.c
-                */
-
-                if (!strcmp (op, "Replace brick")) {
-                        if (status)
-                                status = GF_DEFRAG_STATUS_COMPLETE;
-                        else
-                                status = GF_DEFRAG_STATUS_STARTED;
-
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key), "%s.src-brick", task);
-                        ret = dict_get_str (dict, key, &src_brick);
-                        if (ret)
-                                goto out;
-
-                        cli_out ("%-20s : %-20s", "Source Brick", src_brick);
-
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key), "%s.dst-brick", task);
-                        ret = dict_get_str (dict, key, &dest_brick);
-                        if (ret)
-                                goto out;
-
-                        cli_out ("%-20s : %-20s", "Destination Brick",
-                                 dest_brick);
-
-                } else if (!strcmp (op, "Remove brick")) {
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key), "%s.count", task);
-                        ret = dict_get_int32 (dict, key, &count);
-                        if (ret)
-                                goto out;
-
-                        cli_out ("%-20s", "Removed bricks:");
-
-                        for (j = 1; j <= count; j++) {
-                                memset (key, 0, sizeof (key));
-                                snprintf (key, sizeof (key),"%s.brick%d",
-                                          task, j);
-                                ret = dict_get_str (dict, key, &brick);
-                                if (ret)
-                                        goto out;
-
-                                cli_out ("%-20s", brick);
-                        }
-                }
-                cli_out ("%-20s : %-20s", "Status",
-                         cli_vol_task_status_str[status]);
-                cli_out (" ");
-        }
+    ret = dict_get_int32_sizen(dict, "brick-index-max", &brick_index_max);
+    if (ret)
+        goto out;
 
-out:
-        return;
-}
+    ret = dict_get_int32_sizen(dict, "other-count", &other_count);
+    if (ret)
+        goto out;
 
-static int
-gf_cli_status_cbk (struct rpc_req *req, struct iovec *iov,
-                      int count, void *myframe)
-{
-        int                             ret             = -1;
-        int                             brick_index_max = -1;
-        int                             other_count     = 0;
-        int                             index_max       = 0;
-        int                             i               = 0;
-        int                             pid             = -1;
-        uint32_t                        cmd             = 0;
-        gf_boolean_t                    notbrick        = _gf_false;
-        char                            key[1024]       = {0,};
-        char                           *hostname        = NULL;
-        char                           *path            = NULL;
-        char                           *volname         = NULL;
-        dict_t                         *dict            = NULL;
-        gf_cli_rsp                      rsp             = {0,};
-        cli_volume_status_t             status          = {0};
-        cli_local_t                    *local           = NULL;
-        gf_boolean_t                    wipe_local      = _gf_false;
-        char                            msg[1024]       = {0,};
-
-        if (req->rpc_status == -1)
-                goto out;
+    index_max = brick_index_max + other_count;
 
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
+    ret = dict_get_int32_sizen(dict, "type", &type);
+    if (ret)
+        goto out;
 
-        gf_log ("cli", GF_LOG_DEBUG, "Received response to status cmd");
+    ret = dict_get_int32_sizen(dict, "hot_brick_count", &hot_brick_count);
+    if (ret)
+        goto out;
 
-        local = ((call_frame_t *)myframe)->local;
-        if (!local) {
-                local = cli_local_get ();
-                if (!local) {
-                        ret = -1;
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to get local");
-                        goto out;
-                }
-                wipe_local = _gf_true;
-        }
+    cli_out("Status of volume: %s", volname);
 
-        if (rsp.op_ret) {
-                if (strcmp (rsp.op_errstr, ""))
-                        snprintf (msg, sizeof (msg), "%s", rsp.op_errstr);
-                else
-                        snprintf (msg, sizeof (msg), "Unable to obtain volume "
-                                  "status information.");
-
-                if (global_state->mode & GLUSTER_MODE_XML) {
-                        if (!local->all)
-                                cli_xml_output_str ("volStatus", msg,
-                                                    rsp.op_ret, rsp.op_errno,
-                                                    rsp.op_errstr);
-                        ret = 0;
-                        goto out;
-                }
+    if ((cmd & GF_CLI_STATUS_DETAIL) == 0) {
+        cli_out("%-*s %s  %s  %s  %s", CLI_VOL_STATUS_BRICK_LEN,
+                "Gluster process", "TCP Port", "RDMA Port", "Online", "Pid");
+        cli_print_line(CLI_BRICK_STATUS_LINE_LEN);
+    }
 
-                cli_err ("%s", msg);
-                if (local && local->all) {
-                        ret = 0;
-                        cli_out (" ");
-                } else
-                        ret = -1;
+    status.brick = GF_MALLOC(PATH_MAX + 256, gf_common_mt_strdup);
+    if (!status.brick) {
+        errno = ENOMEM;
+        ret = -1;
+        goto out;
+    }
 
-                goto out;
-        }
-
-        dict = dict_new ();
-        if (!dict)
-                goto out;
+    for (i = 0; i <= index_max; i++) {
+        status.rdma_port = 0;
 
-        ret = dict_unserialize (rsp.dict.dict_val,
-                                rsp.dict.dict_len,
-                                &dict);
+        snprintf(key, sizeof(key), "brick%d.hostname", i);
+        ret = dict_get_str(dict, key, &hostname);
         if (ret)
-                goto out;
+            continue;
 
-        ret = dict_get_uint32 (dict, "cmd", &cmd);
+        snprintf(key, sizeof(key), "brick%d.path", i);
+        ret = dict_get_str(dict, key, &path);
         if (ret)
-                goto out;
-
-        if ((cmd & GF_CLI_STATUS_ALL)) {
-                if (local && local->dict) {
-                        dict_ref (dict);
-                        ret = dict_set_static_ptr (local->dict, "rsp-dict", dict);
-                        ret = 0;
-                } else {
-                        gf_log ("cli", GF_LOG_ERROR, "local not found");
-                        ret = -1;
-                }
-                goto out;
-        }
-
-        if ((cmd & GF_CLI_STATUS_NFS) || (cmd & GF_CLI_STATUS_SHD) ||
-            (cmd & GF_CLI_STATUS_QUOTAD) || (cmd & GF_CLI_STATUS_SNAPD))
-                notbrick = _gf_true;
+            continue;
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                if (!local->all) {
-                        ret = cli_xml_output_vol_status_begin (local,
-                                                               rsp.op_ret,
-                                                               rsp.op_errno,
-                                                               rsp.op_errstr);
-                        if (ret) {
-                                gf_log ("cli", GF_LOG_ERROR,
-                                        "Error outputting to xml");
-                                goto out;
-                        }
-                }
-                if (cmd & GF_CLI_STATUS_TASKS) {
-                        ret = cli_xml_output_vol_status_tasks_detail (local,
-                                                                      dict);
-                        if (ret) {
-                                gf_log ("cli", GF_LOG_ERROR,"Error outputting "
-                                        "to xml");
-                                goto out;
-                        }
-                } else {
-                        ret = cli_xml_output_vol_status (local, dict);
-                        if (ret) {
-                                gf_log ("cli", GF_LOG_ERROR,
-                                        "Error outputting to xml");
-                                goto out;
-                        }
-                }
-
-                if (!local->all) {
-                        ret = cli_xml_output_vol_status_end (local);
-                        if (ret) {
-                                gf_log ("cli", GF_LOG_ERROR,
-                                        "Error outputting to xml");
-                        }
-                }
-                goto out;
+        /* Brick/not-brick is handled separately here as all
+         * types of nodes are contained in the default output
+         */
+        status.brick[0] = '\0';
+        if (!strcmp(hostname, "NFS Server") ||
+            !strcmp(hostname, "Self-heal Daemon") ||
+            !strcmp(hostname, "Quota Daemon") ||
+            !strcmp(hostname, "Snapshot Daemon") ||
+            !strcmp(hostname, "Scrubber Daemon") ||
+            !strcmp(hostname, "Bitrot Daemon"))
+            snprintf(status.brick, PATH_MAX + 255, "%s on %s", hostname, path);
+        else {
+            snprintf(key, sizeof(key), "brick%d.rdma_port", i);
+            ret = dict_get_int32(dict, key, &(status.rdma_port));
+            if (ret)
+                continue;
+            snprintf(status.brick, PATH_MAX + 255, "Brick %s:%s", hostname,
+                     path);
         }
 
-        status.brick = GF_CALLOC (1, PATH_MAX + 256, gf_common_mt_strdup);
-
-        switch (cmd & GF_CLI_STATUS_MASK) {
-                case GF_CLI_STATUS_MEM:
-                        cli_print_volume_status_mem (dict, notbrick);
-                        goto cont;
-                        break;
-                case GF_CLI_STATUS_CLIENTS:
-                        cli_print_volume_status_clients (dict, notbrick);
-                        goto cont;
-                        break;
-                case GF_CLI_STATUS_INODE:
-                        cli_print_volume_status_inode (dict, notbrick);
-                        goto cont;
-                        break;
-                case GF_CLI_STATUS_FD:
-                        cli_print_volume_status_fd (dict, notbrick);
-                        goto cont;
-                        break;
-                case GF_CLI_STATUS_CALLPOOL:
-                        cli_print_volume_status_callpool (dict, notbrick);
-                        goto cont;
-                        break;
-                case GF_CLI_STATUS_TASKS:
-                        cli_print_volume_status_tasks (dict);
-                        goto cont;
-                        break;
-                default:
-                        break;
-        }
-
-        ret = dict_get_str (dict, "volname", &volname);
+        snprintf(key, sizeof(key), "brick%d.port", i);
+        ret = dict_get_int32(dict, key, &(status.port));
         if (ret)
-                goto out;
+            continue;
 
-        ret = dict_get_int32 (dict, "brick-index-max", &brick_index_max);
+        snprintf(key, sizeof(key), "brick%d.status", i);
+        ret = dict_get_int32(dict, key, &(status.online));
         if (ret)
-                goto out;
+            continue;
 
-        ret = dict_get_int32 (dict, "other-count", &other_count);
+        snprintf(key, sizeof(key), "brick%d.pid", i);
+        ret = dict_get_int32(dict, key, &pid);
         if (ret)
-                goto out;
-
-        index_max = brick_index_max + other_count;
-
+            continue;
+        if (pid == -1)
+            ret = gf_asprintf(&(status.pid_str), "%s", "N/A");
+        else
+            ret = gf_asprintf(&(status.pid_str), "%d", pid);
 
-        cli_out ("Status of volume: %s", volname);
+        if (ret == -1)
+            goto out;
 
-        if ((cmd & GF_CLI_STATUS_DETAIL) == 0) {
-                cli_out ("Gluster process\t\t\t\t\t\tPort\tOnline\tPid");
-                cli_print_line (CLI_BRICK_STATUS_LINE_LEN);
+        if ((cmd & GF_CLI_STATUS_DETAIL)) {
+            ret = cli_get_detail_status(dict, i, &status);
+            if (ret)
+                goto out;
+            cli_print_line(CLI_BRICK_STATUS_LINE_LEN);
+            cli_print_detailed_status(&status);
+        } else {
+            cli_print_brick_status(&status);
         }
 
-        for (i = 0; i <= index_max; i++) {
-
+        /* Allocatated memory using gf_asprintf*/
+        GF_FREE(status.pid_str);
+    }
+    cli_out(" ");
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.hostname", i);
-                ret = dict_get_str (dict, key, &hostname);
-                if (ret)
-                        continue;
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.path", i);
-                ret = dict_get_str (dict, key, &path);
-                if (ret)
-                        continue;
-
-                /* Brick/not-brick is handled separately here as all
-                 * types of nodes are contained in the default output
-                 */
-                memset (status.brick, 0, PATH_MAX + 255);
-                if (!strcmp (hostname, "NFS Server") ||
-                    !strcmp (hostname, "Self-heal Daemon") ||
-                    !strcmp (hostname, "Quota Daemon") ||
-                    !strcmp (hostname, "Snapshot Daemon"))
-                        snprintf (status.brick, PATH_MAX + 255, "%s on %s",
-                                  hostname, path);
-                else
-                        snprintf (status.brick, PATH_MAX + 255, "Brick %s:%s",
-                                  hostname, path);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.port", i);
-                ret = dict_get_int32 (dict, key, &(status.port));
-                if (ret)
-                        continue;
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.status", i);
-                ret = dict_get_int32 (dict, key, &(status.online));
-                if (ret)
-                        continue;
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.pid", i);
-                ret = dict_get_int32 (dict, key, &pid);
-                if (ret)
-                        continue;
-                if (pid == -1)
-                        ret = gf_asprintf (&(status.pid_str), "%s", "N/A");
-                else
-                        ret = gf_asprintf (&(status.pid_str), "%d", pid);
-
-                if (ret == -1)
-                        goto out;
-
-                if ((cmd & GF_CLI_STATUS_DETAIL)) {
-                        ret = cli_get_detail_status (dict, i, &status);
-                        if (ret)
-                                goto out;
-                        cli_print_line (CLI_BRICK_STATUS_LINE_LEN);
-                        cli_print_detailed_status (&status);
-
-                } else {
-                        cli_print_brick_status (&status);
-                }
-        }
-        cli_out (" ");
-
-        if ((cmd & GF_CLI_STATUS_MASK) == GF_CLI_STATUS_NONE)
-                cli_print_volume_status_tasks (dict);
+    if ((cmd & GF_CLI_STATUS_MASK) == GF_CLI_STATUS_NONE)
+        cli_print_volume_status_tasks(dict);
 cont:
-        ret = rsp.op_ret;
+    ret = rsp.op_ret;
 
 out:
-        if (dict)
-                dict_unref (dict);
-        GF_FREE (status.brick);
-        if (local && wipe_local) {
-                cli_local_wipe (local);
-        }
+    if (dict)
+        dict_unref(dict);
+    GF_FREE(status.brick);
+    if (local && wipe_local) {
+        cli_local_wipe(local);
+    }
 
-        cli_cmd_broadcast_response (ret);
-        return ret;
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
+    return ret;
 }
 
-int32_t
-gf_cli_status_volume (call_frame_t *frame, xlator_t *this,
-                         void *data)
+static int32_t
+gf_cli_status_volume(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf_cli_req                      req  = {{0,}};
-        int                             ret  = -1;
-        dict_t                         *dict = NULL;
+    gf_cli_req req = {{
+        0,
+    }};
+    int ret = -1;
+    dict_t *dict = NULL;
 
-        if (!frame || !this || !data)
-                goto out;
-
-        dict = data;
+    dict = data;
 
-        ret = cli_to_glusterd (&req, frame, gf_cli_status_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, dict,
-                               GLUSTER_CLI_STATUS_VOLUME, this, cli_rpc_prog,
-                               NULL);
- out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning: %d", ret);
-        return ret;
+    ret = cli_to_glusterd(&req, frame, gf_cli_status_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict,
+                          GLUSTER_CLI_STATUS_VOLUME, this, cli_rpc_prog, NULL);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
+    GF_FREE(req.dict.dict_val);
+    return ret;
 }
 
-int
-gf_cli_status_volume_all (call_frame_t *frame, xlator_t *this, void *data)
-{
-        int              i            = 0;
-        int              ret          = -1;
-        int              vol_count    = -1;
-        uint32_t         cmd          = 0;
-        char             key[1024]    = {0};
-        char            *volname      = NULL;
-        void            *vol_dict     = NULL;
-        dict_t          *dict         = NULL;
-        cli_local_t     *local        = NULL;
-
-        if (frame->local) {
-                local = frame->local;
-                local->all = _gf_true;
-        } else
-                goto out;
+static int
+gf_cli_status_volume_all(call_frame_t *frame, xlator_t *this, void *data)
+{
+    int i = 0;
+    int ret = -1;
+    int vol_count = -1;
+    uint32_t cmd = 0;
+    char key[1024] = {0};
+    char *volname = NULL;
+    void *vol_dict = NULL;
+    dict_t *dict = NULL;
+    cli_local_t *local = NULL;
 
-        ret = dict_get_uint32 (local->dict, "cmd", &cmd);
-        if (ret)
-                goto out;
+    if (!frame)
+        goto out;
 
+    if (!frame->local)
+        goto out;
 
-        ret = gf_cli_status_volume (frame, this, data);
-        if (ret)
-                goto out;
+    local = frame->local;
 
-        ret = dict_get_ptr (local->dict, "rsp-dict", &vol_dict);
-        if (ret)
-                goto out;
+    ret = dict_get_uint32(local->dict, "cmd", &cmd);
+    if (ret)
+        goto out;
 
-        ret = dict_get_int32 ((dict_t *)vol_dict, "vol_count", &vol_count);
-        if (ret) {
-                cli_err ("Failed to get names of volumes");
-                goto out;
-        }
+    local->all = _gf_true;
 
-        /* remove the "all" flag in cmd */
-        cmd &= ~GF_CLI_STATUS_ALL;
-        cmd |= GF_CLI_STATUS_VOL;
+    ret = gf_cli_status_volume(frame, this, data);
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                //TODO: Pass proper op_* values
-                ret = cli_xml_output_vol_status_begin (local, 0,0, NULL);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
-                        goto out;
-                }
-        }
+    if (ret)
+        goto out;
 
-        if (vol_count == 0 && !(global_state->mode & GLUSTER_MODE_XML)) {
-                cli_err ("No volumes present");
-                ret = 0;
-                goto out;
-        }
+    ret = dict_get_ptr(local->dict, "rsp-dict", &vol_dict);
+    if (ret)
+        goto out;
 
-        for (i = 0; i < vol_count; i++) {
+    ret = dict_get_int32_sizen((dict_t *)vol_dict, "vol_count", &vol_count);
+    if (ret) {
+        cli_err("Failed to get names of volumes");
+        goto out;
+    }
 
-                dict = dict_new ();
-                if (!dict)
-                        goto out;
+    /* remove the "all" flag in cmd */
+    cmd &= ~GF_CLI_STATUS_ALL;
+    cmd |= GF_CLI_STATUS_VOL;
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "vol%d", i);
-                ret = dict_get_str (vol_dict, key, &volname);
-                if (ret)
-                        goto out;
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        // TODO: Pass proper op_* values
+        ret = cli_xml_output_vol_status_begin(local, 0, 0, NULL);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+            goto xml_end;
+        }
+    }
 
-                ret = dict_set_str (dict, "volname", volname);
-                if (ret)
-                        goto out;
+    if (vol_count == 0 && !(global_state->mode & GLUSTER_MODE_XML)) {
+        cli_err("No volumes present");
+        ret = 0;
+        goto out;
+    }
 
-                ret = dict_set_uint32 (dict, "cmd", cmd);
-                if (ret)
-                        goto out;
+    for (i = 0; i < vol_count; i++) {
+        dict = dict_new();
+        if (!dict)
+            goto out;
 
-                ret = gf_cli_status_volume (frame, this, dict);
-                if (ret)
-                        goto out;
+        ret = snprintf(key, sizeof(key), "vol%d", i);
+        ret = dict_get_strn(vol_dict, key, ret, &volname);
+        if (ret)
+            goto out;
 
-                dict_unref (dict);
-        }
+        ret = dict_set_str_sizen(dict, "volname", volname);
+        if (ret)
+            goto out;
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_vol_status_end (local);
-        }
+        ret = dict_set_uint32(dict, "cmd", cmd);
+        if (ret)
+            goto out;
 
- out:
+        ret = gf_cli_status_volume(frame, this, dict);
         if (ret)
-                gf_log ("cli", GF_LOG_ERROR, "status all failed");
+            goto out;
+
+        dict_unref(dict);
+    }
 
-        if (vol_dict)
-                dict_unref (vol_dict);
+xml_end:
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_vol_status_end(local);
+    }
 
-        if (ret && dict)
-                dict_unref (dict);
+out:
+    if (ret)
+        gf_log("cli", GF_LOG_ERROR, "status all failed");
+
+    if (vol_dict)
+        dict_unref(vol_dict);
+
+    if (ret && dict)
+        dict_unref(dict);
 
-        if (local)
-                cli_local_wipe (local);
+    if (local)
+        cli_local_wipe(local);
 
-        if (frame)
-                frame->local = NULL;
+    if (frame)
+        frame->local = NULL;
 
-        return ret;
+    return ret;
 }
 
 static int
-gf_cli_mount_cbk (struct rpc_req *req, struct iovec *iov,
-                  int count, void *myframe)
+gf_cli_mount_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                 void *myframe)
 {
-        gf1_cli_mount_rsp rsp   = {0,};
-        int               ret   = -1;
+    gf1_cli_mount_rsp rsp = {
+        0,
+    };
+    int ret = -1;
 
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
+    GF_ASSERT(myframe);
 
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf1_cli_mount_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
 
-        gf_log ("cli", GF_LOG_INFO, "Received resp to mount");
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf1_cli_mount_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
 
-        if (rsp.op_ret == 0) {
-                ret = 0;
-                cli_out ("%s", rsp.path);
-        } else {
-                /* weird sounding but easy to parse... */
-                cli_err ("%d : failed with this errno (%s)",
-                         rsp.op_errno, strerror (rsp.op_errno));
-                ret = -1;
-        }
+    gf_log("cli", GF_LOG_INFO, "Received resp to mount");
+
+    if (rsp.op_ret == 0) {
+        ret = 0;
+        cli_out("%s", rsp.path);
+    } else {
+        /* weird sounding but easy to parse... */
+        cli_err("%d : failed with this errno (%s)", rsp.op_errno,
+                strerror(rsp.op_errno));
+        ret = -1;
+    }
 
 out:
-        cli_cmd_broadcast_response (ret);
-        return ret;
+    cli_cmd_broadcast_response(ret);
+    if (rsp.path) {
+        free(rsp.path);
+    }
+    return ret;
 }
 
-int32_t
-gf_cli_mount (call_frame_t *frame, xlator_t *this, void *data)
+static int32_t
+gf_cli_mount(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf1_cli_mount_req  req  = {0,};
-        int                ret  = -1;
-        void            **dataa = data;
-        char             *label = NULL;
-        dict_t            *dict = NULL;
-
-        if (!frame || !this || !data)
-                goto out;
-
-        label = dataa[0];
-        dict  = dataa[1];
-
-        req.label = label;
-        ret = dict_allocate_and_serialize (dict, &req.dict.dict_val,
-                                           &req.dict.dict_len);
-        if (ret) {
-                ret = -1;
-                goto out;
-        }
-
-        ret = cli_cmd_submit (NULL, &req, frame, cli_rpc_prog,
-                              GLUSTER_CLI_MOUNT, NULL,
-                              this, gf_cli_mount_cbk,
-                              (xdrproc_t)xdr_gf1_cli_mount_req);
+    gf1_cli_mount_req req = {
+        0,
+    };
+    int ret = -1;
+    void **dataa = data;
+    char *label = NULL;
+    dict_t *dict = NULL;
+
+    if (!frame || !this || !data)
+        goto out;
+
+    label = dataa[0];
+    dict = dataa[1];
+
+    req.label = label;
+    ret = dict_allocate_and_serialize(dict, &req.dict.dict_val,
+                                      &req.dict.dict_len);
+    if (ret) {
+        ret = -1;
+        goto out;
+    }
+
+    ret = cli_cmd_submit(NULL, &req, frame, cli_rpc_prog, GLUSTER_CLI_MOUNT,
+                         NULL, this, gf_cli_mount_cbk,
+                         (xdrproc_t)xdr_gf1_cli_mount_req);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    GF_FREE(req.dict.dict_val);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
+    return ret;
 }
 
 static int
-gf_cli_umount_cbk (struct rpc_req *req, struct iovec *iov,
-                   int count, void *myframe)
+gf_cli_umount_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                  void *myframe)
 {
-        gf1_cli_umount_rsp rsp   = {0,};
-        int               ret   = -1;
+    gf1_cli_umount_rsp rsp = {
+        0,
+    };
+    int ret = -1;
 
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
+    GF_ASSERT(myframe);
 
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf1_cli_umount_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
 
-        gf_log ("cli", GF_LOG_INFO, "Received resp to mount");
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf1_cli_umount_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
 
-        if (rsp.op_ret == 0)
-                ret = 0;
-        else {
-                cli_err ("umount failed");
-                ret = -1;
-        }
+    gf_log("cli", GF_LOG_INFO, "Received resp to mount");
+
+    if (rsp.op_ret == 0)
+        ret = 0;
+    else {
+        cli_err("umount failed");
+        ret = -1;
+    }
 
 out:
-        cli_cmd_broadcast_response (ret);
-        return ret;
+    cli_cmd_broadcast_response(ret);
+    return ret;
 }
 
-int32_t
-gf_cli_umount (call_frame_t *frame, xlator_t *this, void *data)
+static int32_t
+gf_cli_umount(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf1_cli_umount_req  req  = {0,};
-        int                ret  = -1;
-        dict_t            *dict = NULL;
+    gf1_cli_umount_req req = {
+        0,
+    };
+    int ret = -1;
+    dict_t *dict = NULL;
 
-        if (!frame || !this || !data)
-                goto out;
+    if (!frame || !this || !data)
+        goto out;
 
-        dict = data;
+    dict = data;
 
-        ret = dict_get_str (dict, "path", &req.path);
-        if (ret == 0)
-                ret = dict_get_int32 (dict, "lazy", &req.lazy);
+    ret = dict_get_str_sizen(dict, "path", &req.path);
+    if (ret == 0)
+        ret = dict_get_int32_sizen(dict, "lazy", &req.lazy);
 
-        if (ret) {
-                ret = -1;
-                goto out;
-        }
+    if (ret) {
+        ret = -1;
+        goto out;
+    }
 
-        ret = cli_cmd_submit (NULL, &req, frame, cli_rpc_prog,
-                              GLUSTER_CLI_UMOUNT, NULL,
-                              this, gf_cli_umount_cbk,
-                              (xdrproc_t)xdr_gf1_cli_umount_req);
+    ret = cli_cmd_submit(NULL, &req, frame, cli_rpc_prog, GLUSTER_CLI_UMOUNT,
+                         NULL, this, gf_cli_umount_cbk,
+                         (xdrproc_t)xdr_gf1_cli_umount_req);
 
- out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+out:
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
+    return ret;
 }
 
-void
-cmd_heal_volume_statistics_out (dict_t *dict, int  brick)
+static void
+cmd_heal_volume_statistics_out(dict_t *dict, int brick)
 {
-
-        uint64_t        num_entries = 0;
-        int             ret = 0;
-        char            key[256] = {0};
-        char            *hostname = NULL;
-        uint64_t        i = 0;
-        uint64_t        healed_count = 0;
-        uint64_t        split_brain_count = 0;
-        uint64_t        heal_failed_count = 0;
-        char            *start_time_str = NULL;
-        char            *end_time_str = NULL;
-        char            *crawl_type = NULL;
-        int             progress = -1;
-
-        snprintf (key, sizeof key, "%d-hostname", brick);
-        ret = dict_get_str (dict, key, &hostname);
+    uint64_t num_entries = 0;
+    int ret = 0;
+    char key[256] = {0};
+    char *hostname = NULL;
+    uint64_t i = 0;
+    uint64_t healed_count = 0;
+    uint64_t split_brain_count = 0;
+    uint64_t heal_failed_count = 0;
+    char *start_time_str = NULL;
+    char *end_time_str = NULL;
+    char *crawl_type = NULL;
+    int progress = -1;
+
+    snprintf(key, sizeof key, "%d-hostname", brick);
+    ret = dict_get_str(dict, key, &hostname);
+    if (ret)
+        goto out;
+    cli_out("------------------------------------------------");
+    cli_out("\nCrawl statistics for brick no %d", brick);
+    cli_out("Hostname of brick %s", hostname);
+
+    snprintf(key, sizeof key, "statistics-%d-count", brick);
+    ret = dict_get_uint64(dict, key, &num_entries);
+    if (ret)
+        goto out;
+
+    for (i = 0; i < num_entries; i++) {
+        snprintf(key, sizeof key, "statistics_crawl_type-%d-%" PRIu64, brick,
+                 i);
+        ret = dict_get_str(dict, key, &crawl_type);
         if (ret)
-                goto out;
-        cli_out ("------------------------------------------------");
-        cli_out ("\nCrawl statistics for brick no %d", brick);
-        cli_out ("Hostname of brick %s", hostname);
+            goto out;
 
-        snprintf (key, sizeof key, "statistics-%d-count", brick);
-        ret = dict_get_uint64 (dict, key, &num_entries);
+        snprintf(key, sizeof key, "statistics_healed_cnt-%d-%" PRIu64, brick,
+                 i);
+        ret = dict_get_uint64(dict, key, &healed_count);
         if (ret)
-                goto out;
+            goto out;
 
-        for (i = 0; i < num_entries; i++)
-        {
-                snprintf (key, sizeof key, "statistics_crawl_type-%d-%"PRIu64,
-                          brick, i);
-                ret = dict_get_str (dict, key, &crawl_type);
-                if (ret)
-                        goto out;
-
-                snprintf (key, sizeof key, "statistics_healed_cnt-%d-%"PRIu64,
-                          brick,i);
-                ret = dict_get_uint64 (dict, key, &healed_count);
-                if (ret)
-                        goto out;
-
-                snprintf (key, sizeof key, "statistics_sb_cnt-%d-%"PRIu64,
-                          brick, i);
-                ret = dict_get_uint64 (dict, key, &split_brain_count);
-                if (ret)
-                        goto out;
-                snprintf (key, sizeof key, "statistics_heal_failed_cnt-%d-%"PRIu64,
-                          brick, i);
-                ret = dict_get_uint64 (dict, key, &heal_failed_count);
-                if (ret)
-                        goto out;
-                snprintf (key, sizeof key, "statistics_strt_time-%d-%"PRIu64,
-                          brick, i);
-                ret = dict_get_str (dict, key,  &start_time_str);
-                if (ret)
-                        goto out;
-                snprintf (key, sizeof key, "statistics_end_time-%d-%"PRIu64,
-                          brick, i);
-                ret = dict_get_str (dict, key, &end_time_str);
-                if (ret)
-                        goto out;
-                snprintf (key, sizeof key, "statistics_inprogress-%d-%"PRIu64,
-                          brick, i);
-                ret = dict_get_int32 (dict, key, &progress);
-                if (ret)
-                        goto out;
-
-                cli_out ("\nStarting time of crawl: %s", start_time_str);
-                if (progress == 1)
-                        cli_out ("Crawl is in progress");
-                else
-                        cli_out ("Ending time of crawl: %s", end_time_str);
-
-                cli_out ("Type of crawl: %s", crawl_type);
-                cli_out ("No. of entries healed: %"PRIu64,
-                         healed_count);
-                cli_out ("No. of entries in split-brain: %"PRIu64,
-                        split_brain_count);
-                cli_out ("No. of heal failed entries: %"PRIu64,
-                         heal_failed_count);
+        snprintf(key, sizeof key, "statistics_sb_cnt-%d-%" PRIu64, brick, i);
+        ret = dict_get_uint64(dict, key, &split_brain_count);
+        if (ret)
+            goto out;
+        snprintf(key, sizeof key, "statistics_heal_failed_cnt-%d-%" PRIu64,
+                 brick, i);
+        ret = dict_get_uint64(dict, key, &heal_failed_count);
+        if (ret)
+            goto out;
+        snprintf(key, sizeof key, "statistics_strt_time-%d-%" PRIu64, brick, i);
+        ret = dict_get_str(dict, key, &start_time_str);
+        if (ret)
+            goto out;
+        snprintf(key, sizeof key, "statistics_end_time-%d-%" PRIu64, brick, i);
+        ret = dict_get_str(dict, key, &end_time_str);
+        if (ret)
+            goto out;
+        snprintf(key, sizeof key, "statistics_inprogress-%d-%" PRIu64, brick,
+                 i);
+        ret = dict_get_int32(dict, key, &progress);
+        if (ret)
+            goto out;
 
-        }
+        cli_out("\nStarting time of crawl: %s", start_time_str);
+        if (progress == 1)
+            cli_out("Crawl is in progress");
+        else
+            cli_out("Ending time of crawl: %s", end_time_str);
 
+        cli_out("Type of crawl: %s", crawl_type);
+        cli_out("No. of entries healed: %" PRIu64, healed_count);
+        cli_out("No. of entries in split-brain: %" PRIu64, split_brain_count);
+        cli_out("No. of heal failed entries: %" PRIu64, heal_failed_count);
+    }
 
 out:
-        return;
+    return;
 }
 
-void
-cmd_heal_volume_brick_out (dict_t *dict, int brick)
-{
-        uint64_t        num_entries = 0;
-        int             ret = 0;
-        char            key[256] = {0};
-        char           *hostname = NULL;
-        char           *path = NULL;
-        char           *status = NULL;
-        uint64_t        i = 0;
-        uint32_t        time = 0;
-        char            timestr[32] = {0};
-        char            *shd_status = NULL;
-
-        snprintf (key, sizeof key, "%d-hostname", brick);
-        ret = dict_get_str (dict, key, &hostname);
-        if (ret)
-                goto out;
-        snprintf (key, sizeof key, "%d-path", brick);
-        ret = dict_get_str (dict, key, &path);
-        if (ret)
-                goto out;
-        cli_out ("\nBrick %s:%s", hostname, path);
-
-        snprintf (key, sizeof key, "%d-status", brick);
-        ret = dict_get_str (dict, key, &status);
-        if (status && strlen (status))
-                cli_out ("Status: %s", status);
-
-        snprintf (key, sizeof key, "%d-shd-status",brick);
-        ret = dict_get_str (dict, key, &shd_status);
-
-        if(!shd_status)
-        {
-                snprintf (key, sizeof key, "%d-count", brick);
-                ret = dict_get_uint64 (dict, key, &num_entries);
-                cli_out ("Number of entries: %"PRIu64, num_entries);
-
-
-                for (i = 0; i < num_entries; i++) {
-                        snprintf (key, sizeof key, "%d-%"PRIu64, brick, i);
-                        ret = dict_get_str (dict, key, &path);
-                        if (ret)
-                                continue;
-                        time = 0;
-                        snprintf (key, sizeof key, "%d-%"PRIu64"-time",
-                                  brick, i);
-                        ret = dict_get_uint32 (dict, key, &time);
-                        if (!time) {
-                                cli_out ("%s", path);
-                        } else {
-                                gf_time_fmt (timestr, sizeof timestr,
-                                             time, gf_timefmt_FT);
-                                if (i == 0) {
-                                cli_out ("at                    path on brick");
-                                cli_out ("-----------------------------------");
-                                }
-                                cli_out ("%s %s", timestr, path);
-                        }
-                }
-        }
+static void
+cmd_heal_volume_brick_out(dict_t *dict, int brick)
+{
+    uint64_t num_entries = 0;
+    int ret = 0;
+    char key[64] = {0};
+    char *hostname = NULL;
+    char *path = NULL;
+    char *status = NULL;
+    uint64_t i = 0;
+    uint32_t time = 0;
+    char timestr[GF_TIMESTR_SIZE] = {0};
+    char *shd_status = NULL;
+
+    snprintf(key, sizeof key, "%d-hostname", brick);
+    ret = dict_get_str(dict, key, &hostname);
+    if (ret)
+        goto out;
+    snprintf(key, sizeof key, "%d-path", brick);
+    ret = dict_get_str(dict, key, &path);
+    if (ret)
+        goto out;
+    cli_out("\nBrick %s:%s", hostname, path);
+
+    snprintf(key, sizeof key, "%d-status", brick);
+    ret = dict_get_str(dict, key, &status);
+    if (status && status[0] != '\0')
+        cli_out("Status: %s", status);
+
+    snprintf(key, sizeof key, "%d-shd-status", brick);
+    ret = dict_get_str(dict, key, &shd_status);
+
+    if (!shd_status) {
+        snprintf(key, sizeof key, "%d-count", brick);
+        ret = dict_get_uint64(dict, key, &num_entries);
+        cli_out("Number of entries: %" PRIu64, num_entries);
+
+        for (i = 0; i < num_entries; i++) {
+            snprintf(key, sizeof key, "%d-%" PRIu64, brick, i);
+            ret = dict_get_str(dict, key, &path);
+            if (ret)
+                continue;
+            time = 0;
+            snprintf(key, sizeof key, "%d-%" PRIu64 "-time", brick, i);
+            ret = dict_get_uint32(dict, key, &time);
+            if (ret || !time) {
+                cli_out("%s", path);
+            } else {
+                gf_time_fmt(timestr, sizeof timestr, time, gf_timefmt_FT);
+                if (i == 0) {
+                    cli_out("at                    path on brick");
+                    cli_out("-----------------------------------");
+                }
+                cli_out("%s %s", timestr, path);
+            }
+        }
+    }
 
 out:
-        return;
+    return;
 }
 
-
-void
-cmd_heal_volume_statistics_heal_count_out (dict_t *dict, int brick)
+static void
+cmd_heal_volume_statistics_heal_count_out(dict_t *dict, int brick)
 {
-        uint64_t        num_entries = 0;
-        int             ret = 0;
-        char            key[256] = {0};
-        char           *hostname = NULL;
-        char           *path = NULL;
-        char           *status = NULL;
-        char            *shd_status = NULL;
-
-        snprintf (key, sizeof key, "%d-hostname", brick);
-        ret = dict_get_str (dict, key, &hostname);
+    uint64_t num_entries = 0;
+    int ret = 0;
+    char key[64] = {0};
+    char *hostname = NULL;
+    char *path = NULL;
+    char *status = NULL;
+    char *shd_status = NULL;
+
+    snprintf(key, sizeof key, "%d-hostname", brick);
+    ret = dict_get_str(dict, key, &hostname);
+    if (ret)
+        goto out;
+    snprintf(key, sizeof key, "%d-path", brick);
+    ret = dict_get_str(dict, key, &path);
+    if (ret)
+        goto out;
+    cli_out("\nBrick %s:%s", hostname, path);
+
+    snprintf(key, sizeof key, "%d-status", brick);
+    ret = dict_get_str(dict, key, &status);
+    if (status && strlen(status))
+        cli_out("Status: %s", status);
+
+    snprintf(key, sizeof key, "%d-shd-status", brick);
+    ret = dict_get_str(dict, key, &shd_status);
+
+    if (!shd_status) {
+        snprintf(key, sizeof key, "%d-hardlinks", brick);
+        ret = dict_get_uint64(dict, key, &num_entries);
         if (ret)
-                goto out;
-        snprintf (key, sizeof key, "%d-path", brick);
-        ret = dict_get_str (dict, key, &path);
-        if (ret)
-                goto out;
-        cli_out ("\nBrick %s:%s", hostname, path);
-
-        snprintf (key, sizeof key, "%d-status", brick);
-        ret = dict_get_str (dict, key, &status);
-        if (status && strlen (status))
-                cli_out ("Status: %s", status);
-
-        snprintf (key, sizeof key, "%d-shd-status",brick);
-        ret = dict_get_str (dict, key, &shd_status);
-
-        if(!shd_status)
-        {
-                snprintf (key, sizeof key, "%d-hardlinks", brick);
-                ret = dict_get_uint64 (dict, key, &num_entries);
-                if (ret)
-                        cli_out ("No gathered input for this brick");
-                else
-                        cli_out ("Number of entries: %"PRIu64, num_entries);
-
-
-        }
+            cli_out("No gathered input for this brick");
+        else
+            cli_out("Number of entries: %" PRIu64, num_entries);
+    }
 
 out:
-        return;
+    return;
 }
 
+static int
+gf_is_cli_heal_get_command(gf_xl_afr_op_t heal_op)
+{
+    /* If the command is get command value is 1 otherwise 0, for
+       invalid commands -1 */
+    static int get_cmds[GF_SHD_OP_HEAL_DISABLE + 1] = {
+        [GF_SHD_OP_INVALID] = -1,
+        [GF_SHD_OP_HEAL_INDEX] = 0,
+        [GF_SHD_OP_HEAL_FULL] = 0,
+        [GF_SHD_OP_INDEX_SUMMARY] = 1,
+        [GF_SHD_OP_SPLIT_BRAIN_FILES] = 1,
+        [GF_SHD_OP_STATISTICS] = 1,
+        [GF_SHD_OP_STATISTICS_HEAL_COUNT] = 1,
+        [GF_SHD_OP_STATISTICS_HEAL_COUNT_PER_REPLICA] = 1,
+        [GF_SHD_OP_HEAL_ENABLE] = 0,
+        [GF_SHD_OP_HEAL_DISABLE] = 0,
+    };
+
+    if (heal_op > GF_SHD_OP_INVALID && heal_op <= GF_SHD_OP_HEAL_DISABLE)
+        return get_cmds[heal_op] == 1;
+    return _gf_false;
+}
 
 int
-gf_cli_heal_volume_cbk (struct rpc_req *req, struct iovec *iov,
-                             int count, void *myframe)
-{
-        gf_cli_rsp              rsp   = {0,};
-        int                     ret   = -1;
-        cli_local_t             *local = NULL;
-        char                    *volname = NULL;
-        call_frame_t            *frame = NULL;
-        dict_t                  *input_dict = NULL;
-        dict_t                  *dict = NULL;
-        int                     brick_count = 0;
-        int                     i = 0;
-        gf_xl_afr_op_t          heal_op = GF_AFR_OP_INVALID;
-        char                    *operation = NULL;
-        char                    *substr = NULL;
-        char                    *heal_op_str = NULL;
-
-        if (-1 == req->rpc_status) {
-                goto out;
-        }
-
-        frame = myframe;
-
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
-
-        if (frame)
-                local = frame->local;
-
-        if (local) {
-                input_dict = local->dict;
-                ret = dict_get_int32 (input_dict, "heal-op",
-                                      (int32_t*)&heal_op);
-        }
-//TODO: Proper XML output
-//#if (HAVE_LIB_XML)
-//        if (global_state->mode & GLUSTER_MODE_XML) {
-//                ret = cli_xml_output_dict ("volHeal", dict, rsp.op_ret,
-//                                           rsp.op_errno, rsp.op_errstr);
-//                if (ret)
-//                        gf_log ("cli", GF_LOG_ERROR,
-//                                "Error outputting to xml");
-//                goto out;
-//        }
-//#endif
-
-        ret = dict_get_str (input_dict, "volname", &volname);
-        if (ret) {
-                gf_log (frame->this->name, GF_LOG_ERROR, "failed to get volname");
-                goto out;
-        }
-
-        gf_log ("cli", GF_LOG_INFO, "Received resp to heal volume");
-
-        switch (heal_op) {
-                case    GF_AFR_OP_HEAL_INDEX:
-                        heal_op_str = "to perform index self heal";
-                        break;
-                case    GF_AFR_OP_HEAL_FULL:
-                        heal_op_str = "to perform full self heal";
-                        break;
-                case    GF_AFR_OP_INDEX_SUMMARY:
-                        heal_op_str = "list of entries to be healed";
-                        break;
-                case    GF_AFR_OP_HEALED_FILES:
-                        heal_op_str = "list of healed entries";
-                        break;
-                case    GF_AFR_OP_HEAL_FAILED_FILES:
-                        heal_op_str = "list of heal failed entries";
-                        break;
-                case    GF_AFR_OP_SPLIT_BRAIN_FILES:
-                        heal_op_str = "list of split brain entries";
-                        break;
-                case    GF_AFR_OP_STATISTICS:
-                        heal_op_str =  "crawl statistics";
-                        break;
-                case    GF_AFR_OP_STATISTICS_HEAL_COUNT:
-                        heal_op_str = "count of entries to be healed";
-                        break;
-                case    GF_AFR_OP_STATISTICS_HEAL_COUNT_PER_REPLICA:
-                        heal_op_str = "count of entries to be healed per replica";
-                        break;
-                case    GF_AFR_OP_INVALID:
-                        heal_op_str = "invalid heal op";
-                        break;
-        }
-
-        if ((heal_op == GF_AFR_OP_HEAL_FULL) ||
-            (heal_op == GF_AFR_OP_HEAL_INDEX)) {
-                operation = "Launching heal operation";
-                substr = "\nUse heal info commands to check status";
-        } else {
-                operation = "Gathering";
-                substr = "";
-        }
-
-        if (rsp.op_ret) {
-                if (strcmp (rsp.op_errstr, "")) {
-                        cli_err ("%s", rsp.op_errstr);
-                } else {
-                        cli_err ("%s %s on volume %s has been unsuccessful",
-                                 operation, heal_op_str, volname);
-                }
-
-                ret = rsp.op_ret;
-                goto out;
-        } else {
-                cli_out ("%s %s on volume %s has been successful %s", operation,
-                         heal_op_str, volname, substr);
+gf_cli_heal_volume_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                       void *myframe)
+{
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    cli_local_t *local = NULL;
+    char *volname = NULL;
+    call_frame_t *frame = NULL;
+    dict_t *dict = NULL;
+    int brick_count = 0;
+    int i = 0;
+    gf_xl_afr_op_t heal_op = GF_SHD_OP_INVALID;
+    const char *operation = NULL;
+    const char *substr = NULL;
+    const char *heal_op_str = NULL;
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status) {
+        goto out;
+    }
+
+    frame = myframe;
+
+    GF_ASSERT(frame->local);
+
+    local = frame->local;
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(frame->this->name, GF_LOG_ERROR, XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    ret = dict_get_int32_sizen(local->dict, "heal-op", (int32_t *)&heal_op);
+    // TODO: Proper XML output
+    //#if (HAVE_LIB_XML)
+    //        if (global_state->mode & GLUSTER_MODE_XML) {
+    //                ret = cli_xml_output_dict ("volHeal", dict, rsp.op_ret,
+    //                                           rsp.op_errno, rsp.op_errstr);
+    //                if (ret)
+    //                        gf_log ("cli", GF_LOG_ERROR, XML_ERROR);
+    //                goto out;
+    //        }
+    //#endif
+
+    ret = dict_get_str_sizen(local->dict, "volname", &volname);
+    if (ret) {
+        gf_log(frame->this->name, GF_LOG_ERROR, "failed to get volname");
+        goto out;
+    }
+
+    gf_log("cli", GF_LOG_INFO, "Received resp to heal volume");
+
+    operation = "Gathering ";
+    substr = "";
+    switch (heal_op) {
+        case GF_SHD_OP_HEAL_INDEX:
+            operation = "Launching heal operation ";
+            heal_op_str = "to perform index self heal";
+            substr = "\nUse heal info commands to check status.";
+            break;
+        case GF_SHD_OP_HEAL_FULL:
+            operation = "Launching heal operation ";
+            heal_op_str = "to perform full self heal";
+            substr = "\nUse heal info commands to check status.";
+            break;
+        case GF_SHD_OP_INDEX_SUMMARY:
+            heal_op_str = "list of entries to be healed";
+            break;
+        case GF_SHD_OP_SPLIT_BRAIN_FILES:
+            heal_op_str = "list of split brain entries";
+            break;
+        case GF_SHD_OP_STATISTICS:
+            heal_op_str = "crawl statistics";
+            break;
+        case GF_SHD_OP_STATISTICS_HEAL_COUNT:
+            heal_op_str = "count of entries to be healed";
+            break;
+        case GF_SHD_OP_STATISTICS_HEAL_COUNT_PER_REPLICA:
+            heal_op_str = "count of entries to be healed per replica";
+            break;
+        case GF_SHD_OP_SBRAIN_HEAL_FROM_BIGGER_FILE:
+        case GF_SHD_OP_SBRAIN_HEAL_FROM_LATEST_MTIME:
+        case GF_SHD_OP_SBRAIN_HEAL_FROM_BRICK:
+        case GF_SHD_OP_HEAL_SUMMARY:
+        case GF_SHD_OP_HEALED_FILES:
+        case GF_SHD_OP_HEAL_FAILED_FILES:
+            /* These cases are never hit; they're coded just to silence the
+             * compiler warnings.*/
+            break;
+
+        case GF_SHD_OP_INVALID:
+            heal_op_str = "invalid heal op";
+            break;
+        case GF_SHD_OP_HEAL_ENABLE:
+            operation = "";
+            heal_op_str = "Enable heal";
+            break;
+        case GF_SHD_OP_HEAL_DISABLE:
+            operation = "";
+            heal_op_str = "Disable heal";
+            break;
+        case GF_SHD_OP_GRANULAR_ENTRY_HEAL_ENABLE:
+            operation = "";
+            heal_op_str = "Enable granular entry heal";
+            break;
+        case GF_SHD_OP_GRANULAR_ENTRY_HEAL_DISABLE:
+            operation = "";
+            heal_op_str = "Disable granular entry heal";
+            break;
+    }
+
+    if (rsp.op_ret) {
+        if (strcmp(rsp.op_errstr, "")) {
+            cli_err("%s%s on volume %s has been unsuccessful:", operation,
+                    heal_op_str, volname);
+            cli_err("%s", rsp.op_errstr);
         }
-
         ret = rsp.op_ret;
-        if ((heal_op == GF_AFR_OP_HEAL_FULL) ||
-            (heal_op == GF_AFR_OP_HEAL_INDEX))
-                goto out;
-
-        dict = dict_new ();
-        if (!dict) {
-                ret = -1;
-                goto out;
-        }
-
-        ret = dict_unserialize (rsp.dict.dict_val,
-                                rsp.dict.dict_len,
-                                &dict);
-
-        if (ret) {
-                gf_log ("", GF_LOG_ERROR,
-                                "Unable to allocate memory");
-                goto out;
-        } else {
-                dict->extra_stdfree = rsp.dict.dict_val;
-        }
-        ret = dict_get_int32 (dict, "count", &brick_count);
-        if (ret)
-                goto out;
-
-        if (!brick_count) {
-                cli_err ("All bricks of volume %s are down.", volname);
-                goto out;
-        }
-
-        switch (heal_op) {
-                case GF_AFR_OP_STATISTICS:
-                        for (i = 0; i < brick_count; i++)
-                                cmd_heal_volume_statistics_out (dict, i);
-                        break;
-                case GF_AFR_OP_STATISTICS_HEAL_COUNT:
-                case GF_AFR_OP_STATISTICS_HEAL_COUNT_PER_REPLICA:
-                        for (i = 0; i < brick_count; i++)
-                                cmd_heal_volume_statistics_heal_count_out (dict,
-                                                                           i);
-                        break;
-                case GF_AFR_OP_INDEX_SUMMARY:
-                case GF_AFR_OP_HEALED_FILES:
-                case GF_AFR_OP_HEAL_FAILED_FILES:
-                case GF_AFR_OP_SPLIT_BRAIN_FILES:
-                        for (i = 0; i < brick_count; i++)
-                                cmd_heal_volume_brick_out (dict, i);
-                        break;
-                default:
-                        break;
-        }
+        goto out;
+    } else {
+        cli_out("%s%s on volume %s has been successful %s", operation,
+                heal_op_str, volname, substr);
+    }
+
+    ret = rsp.op_ret;
+    if (!gf_is_cli_heal_get_command(heal_op))
+        goto out;
+
+    dict = dict_new();
+    if (!dict) {
+        ret = -1;
+        goto out;
+    }
+
+    ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &dict);
+
+    if (ret) {
+        gf_log("", GF_LOG_ERROR, DICT_UNSERIALIZE_FAIL);
+        goto out;
+    }
+
+    ret = dict_get_int32_sizen(dict, "count", &brick_count);
+    if (ret)
+        goto out;
+
+    if (!brick_count) {
+        cli_err("All bricks of volume %s are down.", volname);
+        ret = -1;
+        goto out;
+    }
+
+    switch (heal_op) {
+        case GF_SHD_OP_STATISTICS:
+            for (i = 0; i < brick_count; i++)
+                cmd_heal_volume_statistics_out(dict, i);
+            break;
+        case GF_SHD_OP_STATISTICS_HEAL_COUNT:
+        case GF_SHD_OP_STATISTICS_HEAL_COUNT_PER_REPLICA:
+            for (i = 0; i < brick_count; i++)
+                cmd_heal_volume_statistics_heal_count_out(dict, i);
+            break;
+        case GF_SHD_OP_INDEX_SUMMARY:
+        case GF_SHD_OP_SPLIT_BRAIN_FILES:
+            for (i = 0; i < brick_count; i++)
+                cmd_heal_volume_brick_out(dict, i);
+            break;
+        default:
+            break;
+    }
 
-        ret = rsp.op_ret;
+    ret = rsp.op_ret;
 
 out:
-        cli_cmd_broadcast_response (ret);
-        free (rsp.op_errstr);
-        if (dict)
-                dict_unref (dict);
-        return ret;
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
+    if (dict)
+        dict_unref(dict);
+    return ret;
 }
 
-int32_t
-gf_cli_heal_volume (call_frame_t *frame, xlator_t *this,
-                         void *data)
+static int32_t
+gf_cli_heal_volume(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf_cli_req              req = {{0,}};
-        int                     ret = 0;
-        dict_t                  *dict = NULL;
+    gf_cli_req req = {{
+        0,
+    }};
+    int ret = 0;
+    dict_t *dict = NULL;
 
-        if (!frame || !this ||  !data) {
-                ret = -1;
-                goto out;
-        }
+    dict = data;
 
-        dict = data;
+    ret = cli_to_glusterd(&req, frame, gf_cli_heal_volume_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, dict,
+                          GLUSTER_CLI_HEAL_VOLUME, this, cli_rpc_prog, NULL);
 
-        ret = cli_to_glusterd (&req, frame, gf_cli_heal_volume_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, dict,
-                               GLUSTER_CLI_HEAL_VOLUME, this, cli_rpc_prog,
-                               NULL);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
 
-out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    GF_FREE(req.dict.dict_val);
 
-        GF_FREE (req.dict.dict_val);
-
-        return ret;
+    return ret;
 }
 
-int32_t
-gf_cli_statedump_volume_cbk (struct rpc_req *req, struct iovec *iov,
-                                int count, void *myframe)
+static int32_t
+gf_cli_statedump_volume_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                            void *myframe)
 {
-        gf_cli_rsp                      rsp = {0,};
-        int                             ret = -1;
-        char                            msg[1024] = {0,};
-
-        if (-1 == req->rpc_status)
-                goto out;
-        ret = xdr_to_generic (*iov, &rsp,
-                              (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
-        gf_log ("cli", GF_LOG_DEBUG, "Received response to statedump");
-        if (rsp.op_ret)
-                snprintf (msg, sizeof(msg), "%s", rsp.op_errstr);
-        else
-                snprintf (msg, sizeof (msg), "Volume statedump successful");
-
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_str ("volStatedump", msg, rsp.op_ret,
-                                          rsp.op_errno, rsp.op_errstr);
-                if (ret)
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
-                goto out;
-        }
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    char msg[1024] = "Volume statedump successful";
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status)
+        goto out;
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
+    gf_log("cli", GF_LOG_DEBUG, "Received response to statedump");
+    if (rsp.op_ret)
+        snprintf(msg, sizeof(msg), "%s", rsp.op_errstr);
+
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_str("volStatedump", msg, rsp.op_ret, rsp.op_errno,
+                                 rsp.op_errstr);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
 
-        if (rsp.op_ret)
-                cli_err ("volume statedump: failed: %s", msg);
-        else
-                cli_out ("volume statedump: success");
-        ret = rsp.op_ret;
+    if (rsp.op_ret)
+        cli_err("volume statedump: failed: %s", msg);
+    else
+        cli_out("volume statedump: success");
+    ret = rsp.op_ret;
 
 out:
-        cli_cmd_broadcast_response (ret);
-        return ret;
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
+    return ret;
 }
 
-int32_t
-gf_cli_statedump_volume (call_frame_t *frame, xlator_t *this,
-                            void *data)
+static int32_t
+gf_cli_statedump_volume(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf_cli_req                      req = {{0,}};
-        dict_t                          *options = NULL;
-        int                             ret = -1;
-
-        if (!frame || !this || !data)
-                goto out;
+    gf_cli_req req = {{
+        0,
+    }};
+    dict_t *options = NULL;
+    int ret = -1;
 
-        options = data;
+    options = data;
 
-        ret = cli_to_glusterd (&req, frame, gf_cli_statedump_volume_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, options,
-                               GLUSTER_CLI_STATEDUMP_VOLUME, this, cli_rpc_prog,
-                               NULL);
+    ret = cli_to_glusterd(
+        &req, frame, gf_cli_statedump_volume_cbk, (xdrproc_t)xdr_gf_cli_req,
+        options, GLUSTER_CLI_STATEDUMP_VOLUME, this, cli_rpc_prog, NULL);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
 
-out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-
-        GF_FREE (req.dict.dict_val);
-        return ret;
+    GF_FREE(req.dict.dict_val);
+    return ret;
 }
 
-int32_t
-gf_cli_list_volume_cbk (struct rpc_req *req, struct iovec *iov,
-                                int count, void *myframe)
+static int32_t
+gf_cli_list_volume_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                       void *myframe)
 {
-        int             ret = -1;
-        gf_cli_rsp      rsp = {0,};
-        dict_t          *dict = NULL;
-        int             vol_count = 0;;
-        char            *volname = NULL;
-        char            key[1024] = {0,};
-        int             i = 0;
-
-        if (-1 == req->rpc_status)
-                goto out;
-        ret = xdr_to_generic (*iov, &rsp,
-                              (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
-
-        dict = dict_new ();
-        if (!dict) {
-                ret = -1;
-                goto out;
-        }
+    int ret = -1;
+    gf_cli_rsp rsp = {
+        0,
+    };
+    dict_t *dict = NULL;
+    int vol_count = 0;
+    ;
+    char *volname = NULL;
+    char key[1024] = {
+        0,
+    };
+    int i = 0;
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status)
+        goto out;
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    dict = dict_new();
+    if (!dict) {
+        ret = -1;
+        goto out;
+    }
+
+    ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &dict);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, DICT_UNSERIALIZE_FAIL);
+        goto out;
+    }
+
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_vol_list(dict, rsp.op_ret, rsp.op_errno,
+                                      rsp.op_errstr);
+        if (ret)
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
+
+    if (rsp.op_ret)
+        cli_err("%s", rsp.op_errstr);
+    else {
+        ret = dict_get_int32_sizen(dict, "count", &vol_count);
+        if (ret)
+            goto out;
 
-        ret = dict_unserialize (rsp.dict.dict_val, rsp.dict.dict_len, &dict);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to allocate memory");
-                goto out;
+        if (vol_count == 0) {
+            cli_err("No volumes present in cluster");
+            goto out;
         }
-
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_vol_list (dict, rsp.op_ret, rsp.op_errno,
-                                               rsp.op_errstr);
-                if (ret)
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Error outputting to xml");
+        for (i = 0; i < vol_count; i++) {
+            ret = snprintf(key, sizeof(key), "volume%d", i);
+            ret = dict_get_strn(dict, key, ret, &volname);
+            if (ret)
                 goto out;
+            cli_out("%s", volname);
         }
+    }
 
-        if (rsp.op_ret)
-                cli_err ("%s", rsp.op_errstr);
-        else {
-                ret = dict_get_int32 (dict, "count", &vol_count);
-                if (ret)
-                        goto out;
-
-                if (vol_count == 0) {
-                        cli_err ("No volumes present in cluster");
-                        goto out;
-                }
-                for (i = 0; i < vol_count; i++) {
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key), "volume%d", i);
-                        ret = dict_get_str (dict, key, &volname);
-                        if (ret)
-                                goto out;
-                        cli_out ("%s", volname);
-                }
-        }
-
-        ret = rsp.op_ret;
+    ret = rsp.op_ret;
 
 out:
-        cli_cmd_broadcast_response (ret);
-        return ret;
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
+
+    if (dict)
+        dict_unref(dict);
+    return ret;
 }
 
-int32_t
-gf_cli_list_volume (call_frame_t *frame, xlator_t *this, void *data)
+static int32_t
+gf_cli_list_volume(call_frame_t *frame, xlator_t *this, void *data)
 {
-        int             ret = -1;
-        gf_cli_req      req = {{0,}};
-
-        if (!frame || !this)
-                goto out;
-
-        ret = cli_cmd_submit (NULL, &req, frame, cli_rpc_prog,
-                              GLUSTER_CLI_LIST_VOLUME, NULL,
-                              this, gf_cli_list_volume_cbk,
-                              (xdrproc_t)xdr_gf_cli_req);
+    int ret = -1;
+    gf_cli_req req = {{
+        0,
+    }};
 
-out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    ret = cli_cmd_submit(NULL, &req, frame, cli_rpc_prog,
+                         GLUSTER_CLI_LIST_VOLUME, NULL, this,
+                         gf_cli_list_volume_cbk, (xdrproc_t)xdr_gf_cli_req);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
+    return ret;
 }
 
-int32_t
-gf_cli_clearlocks_volume_cbk (struct rpc_req *req, struct iovec *iov,
-                                  int count, void *myframe)
+static int32_t
+gf_cli_clearlocks_volume_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                             void *myframe)
 {
-        gf_cli_rsp                      rsp = {0,};
-        int                             ret = -1;
-        char                            *lk_summary = NULL;
-        char                            *volname = NULL;
-        dict_t                          *dict = NULL;
-
-        if (-1 == req->rpc_status)
-                goto out;
-        ret = xdr_to_generic (*iov, &rsp,
-                              (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    char *lk_summary = NULL;
+    dict_t *dict = NULL;
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status)
+        goto out;
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
+    gf_log("cli", GF_LOG_DEBUG, "Received response to clear-locks");
+
+    if (rsp.op_ret) {
+        cli_err("Volume clear-locks unsuccessful");
+        cli_err("%s", rsp.op_errstr);
+
+    } else {
+        if (!rsp.dict.dict_len) {
+            cli_err("Possibly no locks cleared");
+            ret = 0;
+            goto out;
         }
-        gf_log ("cli", GF_LOG_DEBUG, "Received response to clear-locks");
-
-        if (rsp.op_ret) {
-                cli_err ("Volume clear-locks unsuccessful");
-                cli_err ("%s", rsp.op_errstr);
-
-        } else {
-                if (!rsp.dict.dict_len) {
-                        cli_err ("Possibly no locks cleared");
-                        ret = 0;
-                        goto out;
-                }
-
-                dict = dict_new ();
 
-                if (!dict) {
-                        ret = -1;
-                        goto out;
-                }
-
-                ret = dict_unserialize (rsp.dict.dict_val,
-                                        rsp.dict.dict_len,
-                                        &dict);
+        dict = dict_new();
 
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Unable to serialize response dictionary");
-                        goto out;
-                }
+        if (!dict) {
+            ret = -1;
+            goto out;
+        }
 
-                ret = dict_get_str (dict, "volname", &volname);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Unable to get volname "
-                                "from dictionary");
-                        goto out;
-                }
+        ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &dict);
 
-                ret = dict_get_str (dict, "lk-summary", &lk_summary);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Unable to get lock "
-                                "summary from dictionary");
-                        goto out;
-                }
-                cli_out ("Volume clear-locks successful");
-                cli_out ("%s", lk_summary);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, DICT_UNSERIALIZE_FAIL);
+            goto out;
+        }
 
+        ret = dict_get_str(dict, "lk-summary", &lk_summary);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Unable to get lock summary from dictionary");
+            goto out;
         }
+        cli_out("Volume clear-locks successful");
+        cli_out("%s", lk_summary);
+    }
 
-        ret = rsp.op_ret;
+    ret = rsp.op_ret;
 
 out:
-        if (dict)
-                dict_unref (dict);
-        cli_cmd_broadcast_response (ret);
-        return ret;
+    if (dict)
+        dict_unref(dict);
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
+    return ret;
 }
 
-int32_t
-gf_cli_clearlocks_volume (call_frame_t *frame, xlator_t *this,
-                             void *data)
+static int32_t
+gf_cli_clearlocks_volume(call_frame_t *frame, xlator_t *this, void *data)
 {
-        gf_cli_req                      req = {{0,}};
-        dict_t                          *options = NULL;
-        int                             ret = -1;
-
-        if (!frame || !this || !data)
-                goto out;
+    gf_cli_req req = {{
+        0,
+    }};
+    dict_t *options = NULL;
+    int ret = -1;
 
-        options = data;
+    options = data;
 
-        ret = cli_to_glusterd (&req, frame, gf_cli_clearlocks_volume_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, options,
-                               GLUSTER_CLI_CLRLOCKS_VOLUME, this, cli_rpc_prog,
-                               NULL);
-out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    ret = cli_to_glusterd(
+        &req, frame, gf_cli_clearlocks_volume_cbk, (xdrproc_t)xdr_gf_cli_req,
+        options, GLUSTER_CLI_CLRLOCKS_VOLUME, this, cli_rpc_prog, NULL);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
 
-        GF_FREE (req.dict.dict_val);
-        return ret;
+    GF_FREE(req.dict.dict_val);
+    return ret;
 }
 
-int32_t
-cli_snapshot_remove_reply (gf_cli_rsp *rsp, dict_t *dict, call_frame_t *frame)
+static int32_t
+cli_snapshot_remove_reply(gf_cli_rsp *rsp, dict_t *dict, call_frame_t *frame)
 {
-        int32_t         ret             = -1;
-        char            *snap_name      = NULL;
-        int32_t         delete_cmd      = -1;
-        cli_local_t     *local          = NULL;
-
-        GF_ASSERT (frame);
-        GF_ASSERT (rsp);
-        GF_ASSERT (dict);
-
-        if (rsp->op_ret) {
-                cli_err ("snapshot delete: failed: %s",
-                        rsp->op_errstr ? rsp->op_errstr :
-                        "Please check log file for details");
-                ret = rsp->op_ret;
-                goto out;
-        }
-
-        ret = dict_get_int32 (dict, "delete-cmd", &delete_cmd);
+    int32_t ret = -1;
+    char *snap_name = NULL;
+    int32_t delete_cmd = -1;
+    cli_local_t *local = NULL;
+
+    GF_ASSERT(frame);
+    GF_ASSERT(rsp);
+    GF_ASSERT(dict);
+
+    local = frame->local;
+
+    ret = dict_get_int32_sizen(dict, "sub-cmd", &delete_cmd);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not get sub-cmd");
+        goto end;
+    }
+
+    if ((global_state->mode & GLUSTER_MODE_XML) &&
+        (delete_cmd == GF_SNAP_DELETE_TYPE_SNAP)) {
+        ret = cli_xml_output_snap_delete_begin(local, rsp->op_ret,
+                                               rsp->op_errno, rsp->op_errstr);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not get delete-cmd");
-                goto out;
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to create xml output for delete");
+            goto end;
+        }
+    }
+
+    if (rsp->op_ret && !(global_state->mode & GLUSTER_MODE_XML)) {
+        cli_err("snapshot delete: failed: %s",
+                rsp->op_errstr ? rsp->op_errstr
+                               : "Please check log file for details");
+        ret = rsp->op_ret;
+        goto out;
+    }
+
+    if (delete_cmd == GF_SNAP_DELETE_TYPE_ALL ||
+        delete_cmd == GF_SNAP_DELETE_TYPE_VOL) {
+        local = ((call_frame_t *)frame)->local;
+        if (!local) {
+            ret = -1;
+            gf_log("cli", GF_LOG_ERROR, "frame->local is NULL");
+            goto out;
         }
 
-        if (delete_cmd == GF_SNAP_DELETE_TYPE_ALL ||
-            delete_cmd == GF_SNAP_DELETE_TYPE_VOL) {
-                local = ((call_frame_t *) frame) -> local;
-                if (!local) {
-                        ret = -1;
-                        gf_log ("cli", GF_LOG_ERROR, "frame->local is NULL");
-                        goto out;
-                }
+        /* During first call back of snapshot delete of type
+         * ALL and VOL, We will get the snapcount and snapnames.
+         * Hence to make the subsequent rpc calls for individual
+         * snapshot delete, We need to save it in local dictionary.
+         */
+        dict_copy(dict, local->dict);
+        ret = 0;
+        goto out;
+    }
 
-                /* During first call back of snapshot delete of type
-                 * ALL and VOL, We will get the snapcount and snapnames.
-                 * Hence to make the subsequent rpc calls for individual
-                 * snapshot delete, We need to save it in local dictionary.
-                 */
-                dict_copy (dict, local->dict);
-                ret = 0;
-                goto out;
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_snapshot_delete(local, dict, rsp);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to create xml output for snapshot delete command");
+            goto out;
         }
-
-        ret = dict_get_str (dict, "snapname", &snap_name);
+        /* Error out in case of the op already failed */
+        if (rsp->op_ret) {
+            ret = rsp->op_ret;
+            goto out;
+        }
+    } else {
+        ret = dict_get_str_sizen(dict, "snapname", &snap_name);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get snapname");
-                goto out;
+            gf_log("cli", GF_LOG_ERROR, "Failed to get snapname");
+            goto out;
         }
 
-        cli_out ("snapshot delete: %s: snap removed successfully",
-                 snap_name);
-        ret = 0;
+        cli_out("snapshot delete: %s: snap removed successfully", snap_name);
+    }
+    ret = 0;
 
 out:
-        return ret;
+    if ((global_state->mode & GLUSTER_MODE_XML) &&
+        (delete_cmd == GF_SNAP_DELETE_TYPE_SNAP)) {
+        ret = cli_xml_output_snap_delete_end(local);
+    }
+end:
+    return ret;
 }
 
-int
-cli_snapshot_config_display (dict_t *dict, gf_cli_rsp *rsp)
-{
-        char                buf[PATH_MAX]        = "";
-        char               *volname              = NULL;
-        int                 ret                  = -1;
-        int                 config_command       = 0;
-        uint64_t            value                = 0;
-        uint64_t            hard_limit           = 0;
-        uint64_t            soft_limit           = 0;
-        uint64_t            i                    = 0;
-        uint64_t            voldisplaycount      = 0;
-        char               *auto_delete          = NULL;
-        char               *snap_activate        = NULL;
-
-        GF_ASSERT (dict);
-        GF_ASSERT (rsp);
+static int
+cli_snapshot_config_display(dict_t *dict, gf_cli_rsp *rsp)
+{
+    char buf[PATH_MAX] = "";
+    char *volname = NULL;
+    int ret = -1;
+    int config_command = 0;
+    uint64_t value = 0;
+    uint64_t hard_limit = 0;
+    uint64_t soft_limit = 0;
+    uint64_t i = 0;
+    uint64_t voldisplaycount = 0;
+    char *auto_delete = NULL;
+    char *snap_activate = NULL;
+
+    GF_ASSERT(dict);
+    GF_ASSERT(rsp);
+
+    if (rsp->op_ret) {
+        cli_err("Snapshot Config : failed: %s",
+                rsp->op_errstr ? rsp->op_errstr
+                               : "Please check log file for details");
+        ret = rsp->op_ret;
+        goto out;
+    }
+
+    ret = dict_get_int32_sizen(dict, "config-command", &config_command);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not fetch config type");
+        goto out;
+    }
+
+    ret = dict_get_uint64(dict, "snap-max-hard-limit", &hard_limit);
+    /* Ignore the error, as the key specified is optional */
+    ret = dict_get_uint64(dict, "snap-max-soft-limit", &soft_limit);
+
+    ret = dict_get_str_sizen(dict, "auto-delete", &auto_delete);
+
+    ret = dict_get_str_sizen(dict, "snap-activate-on-create", &snap_activate);
+
+    if (!hard_limit && !soft_limit &&
+        config_command != GF_SNAP_CONFIG_DISPLAY && !auto_delete &&
+        !snap_activate) {
+        ret = -1;
+        gf_log(THIS->name, GF_LOG_ERROR, "Could not fetch config-key");
+        goto out;
+    }
+
+    ret = dict_get_str_sizen(dict, "volname", &volname);
+    /* Ignore the error, as volname is optional */
+
+    if (!volname) {
+        volname = "System";
+    }
+
+    switch (config_command) {
+        case GF_SNAP_CONFIG_TYPE_SET:
+            if (hard_limit && soft_limit) {
+                cli_out(
+                    "snapshot config: snap-max-hard-limit "
+                    "& snap-max-soft-limit for system set successfully");
+            } else if (hard_limit) {
+                cli_out(
+                    "snapshot config: snap-max-hard-limit "
+                    "for %s set successfully",
+                    volname);
+            } else if (soft_limit) {
+                cli_out(
+                    "snapshot config: snap-max-soft-limit "
+                    "for %s set successfully",
+                    volname);
+            } else if (auto_delete) {
+                cli_out("snapshot config: auto-delete successfully set");
+            } else if (snap_activate) {
+                cli_out("snapshot config: activate-on-create successfully set");
+            }
+            break;
 
-        if (rsp->op_ret) {
-                cli_err ("Snapshot Config : failed: %s",
-                         rsp->op_errstr ? rsp->op_errstr :
-                         "Please check log file for details");
-                ret = rsp->op_ret;
+        case GF_SNAP_CONFIG_DISPLAY:
+            cli_out("\nSnapshot System Configuration:");
+            ret = dict_get_uint64(dict, "snap-max-hard-limit", &value);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Could not fetch snap_max_hard_limit for %s", volname);
+                ret = -1;
                 goto out;
-        }
+            }
+            cli_out("snap-max-hard-limit : %" PRIu64, value);
 
-        ret = dict_get_int32 (dict, "config-command", &config_command);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not fetch config type");
+            ret = dict_get_uint64(dict, "snap-max-soft-limit", &soft_limit);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Could not fetch snap-max-soft-limit for %s", volname);
+                ret = -1;
                 goto out;
-        }
-
-        ret = dict_get_str (dict, "volname", &volname);
-        /* Ignore the error, as volname is optional */
-
-        if (!volname) {
-                volname = "System";
-        }
+            }
+            cli_out("snap-max-soft-limit : %" PRIu64 "%%", soft_limit);
 
-        ret = dict_get_uint64 (dict, "snap-max-hard-limit", &hard_limit);
-        /* Ignore the error, as the key specified is optional */
-        ret = dict_get_uint64 (dict, "snap-max-soft-limit", &soft_limit);
+            cli_out("auto-delete : %s", auto_delete);
 
-        ret = dict_get_str (dict, "auto-delete", &auto_delete);
+            cli_out("activate-on-create : %s\n", snap_activate);
 
-        ret = dict_get_str (dict, "snap-activate-on-create", &snap_activate);
+            cli_out("Snapshot Volume Configuration:");
 
-        if (!hard_limit && !soft_limit
-                        && config_command != GF_SNAP_CONFIG_DISPLAY
-                        && !auto_delete && !snap_activate) {
+            ret = dict_get_uint64(dict, "voldisplaycount", &voldisplaycount);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR, "Could not fetch voldisplaycount");
                 ret = -1;
-                gf_log(THIS->name, GF_LOG_ERROR,
-                       "Could not fetch config-key");
                 goto out;
-        }
-
-        switch (config_command) {
-        case GF_SNAP_CONFIG_TYPE_SET:
-                if (hard_limit && soft_limit) {
-                        cli_out ("snapshot config: snap-max-hard-limit "
-                                "& snap-max-soft-limit for system set "
-                                 "successfully");
-                } else if (hard_limit) {
-                        cli_out ("snapshot config: snap-max-hard-limit "
-                                 "for %s set successfully",
-                                 volname);
-                } else if (soft_limit) {
-                        cli_out ("snapshot config: snap-max-soft-limit "
-                                 "for %s set successfully",
-                                 volname);
-                } else if (auto_delete) {
-                        cli_out ("snapshot config: auto-delete "
-                                 "successfully set");
-                } else if (snap_activate) {
-                        cli_out ("snapshot config: activate-on-create "
-                                 "successfully set");
-                }
-                break;
+            }
 
-        case GF_SNAP_CONFIG_DISPLAY:
-                cli_out ("\nSnapshot System Configuration:");
-                ret = dict_get_uint64 (dict, "snap-max-hard-limit",
-                                       &value);
+            for (i = 0; i < voldisplaycount; i++) {
+                ret = snprintf(buf, sizeof(buf), "volume%" PRIu64 "-volname",
+                               i);
+                ret = dict_get_strn(dict, buf, ret, &volname);
                 if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Could not fetch "
-                                "snap_max_hard_limit for %s", volname);
-                        ret = -1;
-                        goto out;
+                    gf_log("cli", GF_LOG_ERROR, "Could not fetch %s", buf);
+                    ret = -1;
+                    goto out;
                 }
-                cli_out ("snap-max-hard-limit : %"PRIu64, value);
+                cli_out("\nVolume : %s", volname);
 
-                ret = dict_get_uint64 (dict, "snap-max-soft-limit",
-                                       &soft_limit);
+                snprintf(buf, sizeof(buf),
+                         "volume%" PRIu64 "-snap-max-hard-limit", i);
+                ret = dict_get_uint64(dict, buf, &value);
                 if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Could not fetch "
-                               "snap-max-soft-limit for %s", volname);
-                        ret = -1;
-                        goto out;
+                    gf_log("cli", GF_LOG_ERROR, "Could not fetch %s", buf);
+                    ret = -1;
+                    goto out;
                 }
-                cli_out ("snap-max-soft-limit : %"PRIu64"%%",
-                         soft_limit);
-
-                cli_out ("auto-delete : %s", auto_delete);
-
-                cli_out ("activate-on-create : %s\n", snap_activate);
+                cli_out("snap-max-hard-limit : %" PRIu64, value);
 
-                cli_out ("Snapshot Volume Configuration:");
-
-                ret = dict_get_uint64 (dict, "voldisplaycount",
-                                       &voldisplaycount);
+                snprintf(buf, sizeof(buf),
+                         "volume%" PRIu64 "-active-hard-limit", i);
+                ret = dict_get_uint64(dict, buf, &value);
                 if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                               "Could not fetch voldisplaycount");
-                        ret = -1;
-                        goto out;
-                }
-
-                for (i = 0; i < voldisplaycount; i++) {
-                        snprintf (buf, sizeof(buf), "volume%"PRIu64"-volname", i);
-                        ret = dict_get_str (dict, buf, &volname);
-                        if (ret) {
-                                gf_log ("cli", GF_LOG_ERROR, "Could not fetch "
-                                       " %s", buf);
-                                ret = -1;
-                                goto out;
-                        }
-                        cli_out ("\nVolume : %s", volname);
-
-                        snprintf (buf, sizeof(buf),
-                                  "volume%"PRIu64"-snap-max-hard-limit", i);
-                        ret = dict_get_uint64 (dict, buf, &value);
-                        if (ret) {
-                                gf_log ("cli", GF_LOG_ERROR, "Could not fetch "
-                                       " %s", buf);
-                                ret = -1;
-                                goto out;
-                        }
-                        cli_out ("snap-max-hard-limit : %"PRIu64, value);
-
-                        snprintf (buf, sizeof(buf),
-                                  "volume%"PRIu64"-active-hard-limit", i);
-                        ret = dict_get_uint64 (dict, buf, &value);
-                        if (ret) {
-                                 gf_log ("cli", GF_LOG_ERROR, "Could not fetch"
-                                        " effective snap_max_hard_limit for "
-                                        "%s", volname);
-                                ret = -1;
-                                goto out;
-                        }
-                        cli_out ("Effective snap-max-hard-limit : %"PRIu64,
-                                 value);
-
-                        snprintf (buf, sizeof(buf),
-                                      "volume%"PRIu64"-snap-max-soft-limit", i);
-                        ret = dict_get_uint64 (dict, buf, &value);
-                        if (ret) {
-                                gf_log ("cli", GF_LOG_ERROR, "Could not fetch "
-                                       " %s", buf);
-                                ret = -1;
-                                goto out;
-                        }
-                        cli_out ("Effective snap-max-soft-limit : %"PRIu64" "
-                                 "(%"PRIu64"%%)", value, soft_limit);
-                }
-                break;
+                    gf_log("cli", GF_LOG_ERROR,
+                           "Could not fetch"
+                           " effective snap_max_hard_limit for %s",
+                           volname);
+                    ret = -1;
+                    goto out;
+                }
+                cli_out("Effective snap-max-hard-limit : %" PRIu64, value);
+
+                snprintf(buf, sizeof(buf),
+                         "volume%" PRIu64 "-snap-max-soft-limit", i);
+                ret = dict_get_uint64(dict, buf, &value);
+                if (ret) {
+                    gf_log("cli", GF_LOG_ERROR, "Could not fetch %s", buf);
+                    ret = -1;
+                    goto out;
+                }
+                cli_out("Effective snap-max-soft-limit : %" PRIu64
+                        " "
+                        "(%" PRIu64 "%%)",
+                        value, soft_limit);
+            }
+            break;
         default:
-                break;
-        }
+            break;
+    }
 
-        ret = 0;
+    ret = 0;
 out:
-        return ret;
+    return ret;
 }
 
 /* This function is used to print the volume related information
@@ -7975,1627 +8651,2299 @@ out:
  * arg - 0, dict       : Response Dictionary.
  * arg - 1, prefix str : snaplist.snap{0..}.vol{0..}.*
  */
-int
-cli_get_each_volinfo_in_snap (dict_t *dict, char *keyprefix,
-                              gf_boolean_t snap_driven) {
-        char             key[PATH_MAX]  =  "";
-        char            *get_buffer     =  NULL;
-        int              value          =  0;
-        int              ret            = -1;
-        char             indent[5]      = "\t";
-        char            *volname        =  NULL;
-
-        GF_ASSERT (dict);
-        GF_ASSERT (keyprefix);
-
-        if (snap_driven) {
-                ret = snprintf (key, sizeof (key), "%s.volname", keyprefix);
-                if (ret < 0) {
-                        goto out;
-                }
-
-                ret = dict_get_str (dict, key, &get_buffer);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to get %s", key);
-                        goto out;
-                }
-                cli_out ("%s" INDENT_MAIN_HEAD "%s", indent,
-                        "Snap Volume Name", ":", get_buffer);
-
-                ret = snprintf (key, sizeof (key),
-                                "%s.origin-volname", keyprefix);
-                if (ret < 0) {
-                        goto out;
-                }
-
-                ret = dict_get_str (dict, key, &volname);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_WARNING, "Failed to get %s", key);
-                        cli_out ("%-12s", "Origin:");
-                }
-                cli_out ("%s" INDENT_MAIN_HEAD "%s", indent,
-                        "Origin Volume name", ":", volname);
-
+static int
+cli_get_each_volinfo_in_snap(dict_t *dict, char *keyprefix,
+                             gf_boolean_t snap_driven)
+{
+    char key[PATH_MAX] = "";
+    char *get_buffer = NULL;
+    int value = 0;
+    int ret = -1;
+    char indent[5] = "\t";
+    char *volname = NULL;
+
+    GF_ASSERT(dict);
+    GF_ASSERT(keyprefix);
+
+    if (snap_driven) {
+        ret = snprintf(key, sizeof(key), "%s.volname", keyprefix);
+        if (ret < 0) {
+            goto out;
+        }
 
-                ret = snprintf (key, sizeof (key), "%s.snapcount",
-                                keyprefix);
-                if (ret < 0) {
-                        goto out;
-                }
+        ret = dict_get_str(dict, key, &get_buffer);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, "Failed to get %s", key);
+            goto out;
+        }
+        cli_out("%s" INDENT_MAIN_HEAD "%s", indent, "Snap Volume Name", ":",
+                get_buffer);
 
-                ret = dict_get_int32 (dict, key, &value);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to get %s", key);
-                        goto out;
-                }
-                cli_out ("%s%s %s      %s %d", indent, "Snaps taken for",
-                        volname, ":", value);
+        ret = snprintf(key, sizeof(key), "%s.origin-volname", keyprefix);
+        if (ret < 0) {
+            goto out;
+        }
 
-                ret = snprintf (key, sizeof (key), "%s.snaps-available",
-                                keyprefix);
-                if (ret < 0) {
-                        goto out;
-                }
+        ret = dict_get_str(dict, key, &volname);
+        if (ret) {
+            gf_log("cli", GF_LOG_WARNING, "Failed to get %s", key);
+            cli_out("%-12s", "Origin:");
+        }
+        cli_out("%s" INDENT_MAIN_HEAD "%s", indent, "Origin Volume name", ":",
+                volname);
 
-                ret = dict_get_int32 (dict, key, &value);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to get %s", key);
-                        goto out;
-                }
-                cli_out ("%s%s %s  %s %d", indent, "Snaps available for",
-                         volname, ":", value);
+        ret = snprintf(key, sizeof(key), "%s.snapcount", keyprefix);
+        if (ret < 0) {
+            goto out;
         }
 
+        ret = dict_get_int32(dict, key, &value);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, "Failed to get %s", key);
+            goto out;
+        }
+        cli_out("%s%s %s      %s %d", indent, "Snaps taken for", volname, ":",
+                value);
 
-        ret = snprintf (key, sizeof (key), "%s.vol-status", keyprefix);
+        ret = snprintf(key, sizeof(key), "%s.snaps-available", keyprefix);
         if (ret < 0) {
-                goto out;
+            goto out;
         }
 
-        ret = dict_get_str (dict, key, &get_buffer);
+        ret = dict_get_int32(dict, key, &value);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get %s", key);
-                goto out;
-        }
-        cli_out ("%s" INDENT_MAIN_HEAD "%s", indent, "Status",
-                 ":", get_buffer);
+            gf_log("cli", GF_LOG_ERROR, "Failed to get %s", key);
+            goto out;
+        }
+        cli_out("%s%s %s  %s %d", indent, "Snaps available for", volname, ":",
+                value);
+    }
+
+    ret = snprintf(key, sizeof(key), "%s.vol-status", keyprefix);
+    if (ret < 0) {
+        goto out;
+    }
+
+    ret = dict_get_str(dict, key, &get_buffer);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get %s", key);
+        goto out;
+    }
+    cli_out("%s" INDENT_MAIN_HEAD "%s", indent, "Status", ":", get_buffer);
 out:
-        return ret;
+    return ret;
 }
 
 /* This function is used to print snap related information
  * arg - 0, dict       : Response dictionary.
  * arg - 1, prefix_str : snaplist.snap{0..}.*
  */
-int
-cli_get_volinfo_in_snap (dict_t *dict, char *keyprefix) {
-
-        char            key[PATH_MAX]   = "";
-        int             i               = 0;
-        int             volcount        = 0;
-        int             ret             = -1;
-
-        GF_ASSERT (dict);
-        GF_ASSERT (keyprefix);
-
-        ret = snprintf (key, sizeof (key), "%s.vol-count", keyprefix);
+static int
+cli_get_volinfo_in_snap(dict_t *dict, char *keyprefix)
+{
+    char key[PATH_MAX] = "";
+    int i = 0;
+    int volcount = 0;
+    int ret = -1;
+
+    GF_ASSERT(dict);
+    GF_ASSERT(keyprefix);
+
+    ret = snprintf(key, sizeof(key), "%s.vol-count", keyprefix);
+    if (ret < 0) {
+        goto out;
+    }
+
+    ret = dict_get_int32(dict, key, &volcount);
+    for (i = 1; i <= volcount; i++) {
+        ret = snprintf(key, sizeof(key), "%s.vol%d", keyprefix, i);
         if (ret < 0) {
-                goto out;
+            goto out;
         }
-
-        ret = dict_get_int32 (dict, key, &volcount);
-        for (i = 1 ; i <= volcount ; i++) {
-                ret = snprintf (key, sizeof (key),
-                                "%s.vol%d", keyprefix, i);
-                if (ret < 0) {
-                        goto out;
-                }
-                ret = cli_get_each_volinfo_in_snap (dict, key, _gf_true);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Could not list "
-                                "details of volume in a snap");
-                        goto out;
-                }
-                cli_out (" ");
+        ret = cli_get_each_volinfo_in_snap(dict, key, _gf_true);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Could not list details of volume in a snap");
+            goto out;
         }
+        cli_out(" ");
+    }
 
 out:
-        return ret;
+    return ret;
 }
 
-int
-cli_get_each_snap_info (dict_t *dict, char *prefix_str,
-                        gf_boolean_t snap_driven) {
-        char            key_buffer[PATH_MAX] = "";
-        char           *get_buffer           = NULL;
-        int             ret                  = -1;
-        char            indent[5]            = "";
+static int
+cli_get_each_snap_info(dict_t *dict, char *prefix_str, gf_boolean_t snap_driven)
+{
+    char key_buffer[PATH_MAX] = "";
+    char *get_buffer = NULL;
+    int ret = -1;
+    char indent[5] = "";
+
+    GF_ASSERT(dict);
+    GF_ASSERT(prefix_str);
+
+    if (!snap_driven)
+        strcat(indent, "\t");
+
+    ret = snprintf(key_buffer, sizeof(key_buffer), "%s.snapname", prefix_str);
+    if (ret < 0) {
+        goto out;
+    }
+
+    ret = dict_get_str(dict, key_buffer, &get_buffer);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to fetch snapname %s ", key_buffer);
+        goto out;
+    }
+    cli_out("%s" INDENT_MAIN_HEAD "%s", indent, "Snapshot", ":", get_buffer);
+
+    ret = snprintf(key_buffer, sizeof(key_buffer), "%s.snap-id", prefix_str);
+    if (ret < 0) {
+        goto out;
+    }
+
+    ret = dict_get_str(dict, key_buffer, &get_buffer);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to fetch snap-id %s ", key_buffer);
+        goto out;
+    }
+    cli_out("%s" INDENT_MAIN_HEAD "%s", indent, "Snap UUID", ":", get_buffer);
+
+    ret = snprintf(key_buffer, sizeof(key_buffer), "%s.snap-desc", prefix_str);
+    if (ret < 0) {
+        goto out;
+    }
+
+    ret = dict_get_str(dict, key_buffer, &get_buffer);
+    if (!ret) {
+        /* Ignore error for description */
+        cli_out("%s" INDENT_MAIN_HEAD "%s", indent, "Description", ":",
+                get_buffer);
+    }
+
+    ret = snprintf(key_buffer, sizeof(key_buffer), "%s.snap-time", prefix_str);
+    if (ret < 0) {
+        goto out;
+    }
+
+    ret = dict_get_str(dict, key_buffer, &get_buffer);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to fetch snap-time %s ",
+               prefix_str);
+        goto out;
+    }
+    cli_out("%s" INDENT_MAIN_HEAD "%s", indent, "Created", ":", get_buffer);
+
+    if (snap_driven) {
+        cli_out("%-12s", "Snap Volumes:\n");
+        ret = cli_get_volinfo_in_snap(dict, prefix_str);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, "Unable to list details of the snaps");
+            goto out;
+        }
+    }
+out:
+    return ret;
+}
+
+/* This is a generic function to print snap related information.
+ * arg - 0, dict : Response Dictionary
+ */
+static int
+cli_call_snapshot_info(dict_t *dict, gf_boolean_t bool_snap_driven)
+{
+    int snap_count = 0;
+    char key[32] = "";
+    int ret = -1;
+    int i = 0;
+
+    GF_ASSERT(dict);
 
-        GF_ASSERT (dict);
-        GF_ASSERT (prefix_str);
+    ret = dict_get_int32_sizen(dict, "snapcount", &snap_count);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to get snapcount");
+        goto out;
+    }
 
-        if (!snap_driven)
-                strcat (indent, "\t");
+    if (snap_count == 0) {
+        cli_out("No snapshots present");
+    }
 
-        ret = snprintf (key_buffer, sizeof (key_buffer), "%s.snapname",
-                        prefix_str);
+    for (i = 1; i <= snap_count; i++) {
+        ret = snprintf(key, sizeof(key), "snap%d", i);
         if (ret < 0) {
-                goto out;
+            goto out;
         }
-
-        ret = dict_get_str (dict, key_buffer, &get_buffer);
+        ret = cli_get_each_snap_info(dict, key, bool_snap_driven);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to fetch snapname %s ",
-                        key_buffer);
-                goto out;
-        }
-        cli_out ("%s" INDENT_MAIN_HEAD "%s", indent, "Snapshot",
-                ":", get_buffer);
-
-        ret = snprintf (key_buffer, sizeof (key_buffer), "%s.snap-id",
-                        prefix_str);
-        if (ret < 0) {
-                goto out;
+            gf_log("cli", GF_LOG_ERROR, "Unable to print snap details");
+            goto out;
         }
+    }
+out:
+    return ret;
+}
 
-        ret = dict_get_str (dict, key_buffer, &get_buffer);
+static int
+cli_get_snaps_in_volume(dict_t *dict)
+{
+    int ret = -1;
+    int i = 0;
+    int count = 0;
+    int avail = 0;
+    char key[32] = "";
+    char *get_buffer = NULL;
+
+    GF_ASSERT(dict);
+
+    ret = dict_get_str_sizen(dict, "origin-volname", &get_buffer);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not fetch origin-volname");
+        goto out;
+    }
+    cli_out(INDENT_MAIN_HEAD "%s", "Volume Name", ":", get_buffer);
+
+    ret = dict_get_int32_sizen(dict, "snapcount", &avail);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not fetch snapcount");
+        goto out;
+    }
+    cli_out(INDENT_MAIN_HEAD "%d", "Snaps Taken", ":", avail);
+
+    ret = dict_get_int32_sizen(dict, "snaps-available", &count);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not fetch snaps-available");
+        goto out;
+    }
+    cli_out(INDENT_MAIN_HEAD "%d", "Snaps Available", ":", count);
+
+    for (i = 1; i <= avail; i++) {
+        snprintf(key, sizeof(key), "snap%d", i);
+        ret = cli_get_each_snap_info(dict, key, _gf_false);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to fetch snap-id %s ",
-                        key_buffer);
-                goto out;
+            gf_log("cli", GF_LOG_ERROR, "Unable to print snap details");
+            goto out;
         }
-        cli_out ("%s" INDENT_MAIN_HEAD "%s", indent, "Snap UUID",
-                ":", get_buffer);
 
-        ret = snprintf (key_buffer, sizeof (key_buffer), "%s.snap-desc",
-                        prefix_str);
+        ret = snprintf(key, sizeof(key), "snap%d.vol1", i);
         if (ret < 0) {
-                goto out;
+            goto out;
         }
-
-        ret = dict_get_str (dict, key_buffer, &get_buffer);
-        if (!ret) {
-                /* Ignore error for description */
-                cli_out ("%s" INDENT_MAIN_HEAD "%s", indent,
-                         "Description", ":", get_buffer);
+        ret = cli_get_each_volinfo_in_snap(dict, key, _gf_false);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Could not get volume related information");
+            goto out;
         }
 
-        ret = snprintf (key_buffer, sizeof (key_buffer), "%s.snap-time",
-                        prefix_str);
+        cli_out(" ");
+    }
+out:
+    return ret;
+}
+
+static int
+cli_snapshot_list(dict_t *dict)
+{
+    int snapcount = 0;
+    char key[32] = "";
+    int ret = -1;
+    int i = 0;
+    char *get_buffer = NULL;
+
+    GF_ASSERT(dict);
+
+    ret = dict_get_int32_sizen(dict, "snapcount", &snapcount);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not fetch snap count");
+        goto out;
+    }
+
+    if (snapcount == 0) {
+        cli_out("No snapshots present");
+    }
+
+    for (i = 1; i <= snapcount; i++) {
+        ret = snprintf(key, sizeof(key), "snapname%d", i);
         if (ret < 0) {
-                goto out;
+            goto out;
         }
 
-        ret = dict_get_str (dict, key_buffer, &get_buffer);
+        ret = dict_get_strn(dict, key, ret, &get_buffer);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to fetch snap-time %s ",
-                        prefix_str);
-                goto out;
-        }
-        cli_out ("%s" INDENT_MAIN_HEAD "%s", indent, "Created",
-                ":", get_buffer);
-
-        if (snap_driven) {
-                cli_out ("%-12s", "Snap Volumes:\n");
-                ret = cli_get_volinfo_in_snap (dict, prefix_str);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Unable to list details "
-                                "of the snaps");
-                        goto out;
-                }
+            gf_log("cli", GF_LOG_ERROR, "Could not get %s ", key);
+            goto out;
+        } else {
+            cli_out("%s", get_buffer);
         }
+    }
 out:
-        return ret;
+    return ret;
 }
 
-/* This is a generic function to print snap related information.
- * arg - 0, dict : Response Dictionary
- */
-int
-cli_call_snapshot_info (dict_t *dict, gf_boolean_t bool_snap_driven) {
-        int             snap_count       =  0;
-        char            key[PATH_MAX]    =  "";
-        int             ret              =  -1;
-        int             i                =   0;
-
-        GF_ASSERT (dict);
-
-        ret = dict_get_int32 (dict, "snapcount", &snap_count);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to get snapcount");
-                goto out;
+static int
+cli_get_snap_volume_status(dict_t *dict, char *key_prefix)
+{
+    int ret = -1;
+    char key[PATH_MAX] = "";
+    char *buffer = NULL;
+    int brickcount = 0;
+    int i = 0;
+    int pid = 0;
+
+    GF_ASSERT(dict);
+    GF_ASSERT(key_prefix);
+
+    ret = snprintf(key, sizeof(key), "%s.brickcount", key_prefix);
+    if (ret < 0) {
+        goto out;
+    }
+    ret = dict_get_int32(dict, key, &brickcount);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to fetch brickcount");
+        goto out;
+    }
+
+    for (i = 0; i < brickcount; i++) {
+        ret = snprintf(key, sizeof(key), "%s.brick%d.path", key_prefix, i);
+        if (ret < 0) {
+            goto out;
         }
 
-        if (snap_count == 0) {
-                cli_out ("No snapshots present");
+        ret = dict_get_str(dict, key, &buffer);
+        if (ret) {
+            gf_log("cli", GF_LOG_INFO, "Unable to get Brick Path");
+            continue;
         }
+        cli_out("\n\t%-17s %s   %s", "Brick Path", ":", buffer);
 
-        for (i = 1 ; i <= snap_count ; i++) {
-                ret = snprintf (key, sizeof (key), "snap%d", i);
-                if (ret < 0) {
-                        goto out;
-                }
-                ret = cli_get_each_snap_info (dict, key, bool_snap_driven);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Unable to print snap details");
-                        goto out;
-                }
+        ret = snprintf(key, sizeof(key), "%s.brick%d.vgname", key_prefix, i);
+        if (ret < 0) {
+            goto out;
         }
-out:
-        return ret;
-}
 
-int
-cli_get_snaps_in_volume (dict_t *dict) {
-        int      ret             =  -1;
-        int      i               =  0;
-        int      count           =  0;
-        int      avail           =  0;
-        char     key[PATH_MAX]   =  "";
-        char    *get_buffer      =  NULL;
+        ret = dict_get_str(dict, key, &buffer);
+        if (ret) {
+            gf_log("cli", GF_LOG_INFO, "Unable to get Volume Group");
+            cli_out("\t%-17s %s   %s", "Volume Group", ":", "N/A");
+        } else
+            cli_out("\t%-17s %s   %s", "Volume Group", ":", buffer);
 
-        GF_ASSERT (dict);
+        ret = snprintf(key, sizeof(key), "%s.brick%d.status", key_prefix, i);
+        if (ret < 0) {
+            goto out;
+        }
 
-        ret = dict_get_str (dict, "origin-volname", &get_buffer);
+        ret = dict_get_str(dict, key, &buffer);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not fetch origin-volname");
-                goto out;
+            gf_log("cli", GF_LOG_INFO, "Unable to get Brick Running");
+            cli_out("\t%-17s %s   %s", "Brick Running", ":", "N/A");
+        } else
+            cli_out("\t%-17s %s   %s", "Brick Running", ":", buffer);
+
+        ret = snprintf(key, sizeof(key), "%s.brick%d.pid", key_prefix, i);
+        if (ret < 0) {
+            goto out;
         }
-        cli_out (INDENT_MAIN_HEAD "%s", "Volume Name", ":", get_buffer);
 
-        ret = dict_get_int32 (dict, "snapcount", &avail);
+        ret = dict_get_int32(dict, key, &pid);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not fetch snapcount");
-                goto out;
+            gf_log("cli", GF_LOG_INFO, "Unable to get pid");
+            cli_out("\t%-17s %s   %s", "Brick PID", ":", "N/A");
+        } else
+            cli_out("\t%-17s %s   %d", "Brick PID", ":", pid);
+
+        ret = snprintf(key, sizeof(key), "%s.brick%d.data", key_prefix, i);
+        if (ret < 0) {
+            goto out;
         }
-        cli_out (INDENT_MAIN_HEAD "%d", "Snaps Taken", ":", avail);
 
-        ret = dict_get_int32 (dict, "snaps-available", &count);
+        ret = dict_get_str(dict, key, &buffer);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not fetch snaps-available");
-                goto out;
+            gf_log("cli", GF_LOG_INFO, "Unable to get Data Percent");
+            cli_out("\t%-17s %s   %s", "Data Percentage", ":", "N/A");
+        } else
+            cli_out("\t%-17s %s   %s", "Data Percentage", ":", buffer);
+
+        ret = snprintf(key, sizeof(key), "%s.brick%d.lvsize", key_prefix, i);
+        if (ret < 0) {
+            goto out;
         }
-        cli_out (INDENT_MAIN_HEAD "%d", "Snaps Available", ":", count);
+        ret = dict_get_str(dict, key, &buffer);
+        if (ret) {
+            gf_log("cli", GF_LOG_INFO, "Unable to get LV Size");
+            cli_out("\t%-17s %s   %s", "LV Size", ":", "N/A");
+        } else
+            cli_out("\t%-17s %s   %s", "LV Size", ":", buffer);
+    }
 
-        for (i = 1 ; i <= avail ; i++) {
-                snprintf (key, sizeof (key), "snap%d", i);
-                ret = cli_get_each_snap_info (dict, key, _gf_false);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Unable to print snap details");
-                        goto out;
-                }
+    ret = 0;
+out:
+    return ret;
+}
 
-                ret = snprintf (key, sizeof (key), "snap%d.vol1", i);
-                if (ret < 0) {
-                        goto out;
-                }
-                ret = cli_get_each_volinfo_in_snap (dict, key, _gf_false);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Could not get volume "
-                                "related information");
-                        goto out;
-                }
+static int
+cli_get_single_snap_status(dict_t *dict, char *keyprefix)
+{
+    int ret = -1;
+    char key[64] = ""; /* keyprefix is ""status.snap0" */
+    int i = 0;
+    int volcount = 0;
+    char *get_buffer = NULL;
+
+    GF_ASSERT(dict);
+    GF_ASSERT(keyprefix);
+
+    ret = snprintf(key, sizeof(key), "%s.snapname", keyprefix);
+    if (ret < 0) {
+        goto out;
+    }
+
+    ret = dict_get_str(dict, key, &get_buffer);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to get snapname");
+        goto out;
+    }
+    cli_out("\nSnap Name : %s", get_buffer);
+
+    ret = snprintf(key, sizeof(key), "%s.uuid", keyprefix);
+    if (ret < 0) {
+        goto out;
+    }
+
+    ret = dict_get_str(dict, key, &get_buffer);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to get snap UUID");
+        goto out;
+    }
+    cli_out("Snap UUID : %s", get_buffer);
+
+    ret = snprintf(key, sizeof(key), "%s.volcount", keyprefix);
+    if (ret < 0) {
+        goto out;
+    }
+
+    ret = dict_get_int32(dict, key, &volcount);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to get volume count");
+        goto out;
+    }
+
+    for (i = 0; i < volcount; i++) {
+        ret = snprintf(key, sizeof(key), "%s.vol%d", keyprefix, i);
+        if (ret < 0) {
+            goto out;
+        }
 
-                cli_out (" ");
+        ret = cli_get_snap_volume_status(dict, key);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, "Could not get snap volume status");
+            goto out;
         }
+    }
 out:
-        return ret;
+    return ret;
 }
 
-int
-cli_snapshot_list (dict_t *dict) {
-        int     snapcount       =       0;
-        char    key[PATH_MAX]   =       "";
-        int     ret             =       -1;
-        int     i               =       0;
-        char    *get_buffer     =       NULL;
+static int32_t
+cli_populate_req_dict_for_delete(dict_t *snap_dict, dict_t *dict, size_t index)
+{
+    int32_t ret = -1;
+    char key[PATH_MAX] = "";
+    char *buffer = NULL;
+
+    GF_ASSERT(snap_dict);
+    GF_ASSERT(dict);
+
+    ret = dict_set_int32_sizen(snap_dict, "sub-cmd", GF_SNAP_DELETE_TYPE_ITER);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Could not save command type in snap dictionary");
+        goto out;
+    }
+
+    ret = snprintf(key, sizeof(key), "snapname%zu", index);
+    if (ret < 0) {
+        goto out;
+    }
+
+    ret = dict_get_str(dict, key, &buffer);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get snapname");
+        goto out;
+    }
+
+    ret = dict_set_dynstr_with_alloc(snap_dict, "snapname", buffer);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to save snapname");
+        goto out;
+    }
+
+    ret = dict_set_int32_sizen(snap_dict, "type", GF_SNAP_OPTION_TYPE_DELETE);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to save command type");
+        goto out;
+    }
+
+    ret = dict_set_dynstr_with_alloc(snap_dict, "cmd-str", "snapshot delete");
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not save command string as delete");
+        goto out;
+    }
+out:
+    return ret;
+}
 
-        GF_ASSERT (dict);
+static int
+cli_populate_req_dict_for_status(dict_t *snap_dict, dict_t *dict, int index)
+{
+    int ret = -1;
+    char key[PATH_MAX] = "";
+    char *buffer = NULL;
+
+    GF_ASSERT(snap_dict);
+    GF_ASSERT(dict);
+
+    ret = dict_set_uint32(snap_dict, "sub-cmd", GF_SNAP_STATUS_TYPE_ITER);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not save command type in snap dict");
+        goto out;
+    }
+
+    ret = snprintf(key, sizeof(key), "status.snap%d.snapname", index);
+    if (ret < 0) {
+        goto out;
+    }
+
+    ret = dict_get_strn(dict, key, ret, &buffer);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not get snapname");
+        goto out;
+    }
+
+    ret = dict_set_str_sizen(snap_dict, "snapname", buffer);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not save snapname in snap dict");
+        goto out;
+    }
+
+    ret = dict_set_int32_sizen(snap_dict, "type", GF_SNAP_OPTION_TYPE_STATUS);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not save command type");
+        goto out;
+    }
+
+    ret = dict_set_dynstr_with_alloc(snap_dict, "cmd-str", "snapshot status");
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not save command string as status");
+        goto out;
+    }
+
+    ret = dict_set_int32_sizen(snap_dict, "hold_vol_locks", _gf_false);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Setting volume lock flag failed");
+        goto out;
+    }
 
-        ret = dict_get_int32 (dict, "snapcount", &snapcount);
+out:
+    return ret;
+}
+
+static int
+cli_snapshot_status(dict_t *dict, gf_cli_rsp *rsp, call_frame_t *frame)
+{
+    int ret = -1;
+    int status_cmd = -1;
+    cli_local_t *local = NULL;
+
+    GF_ASSERT(dict);
+    GF_ASSERT(rsp);
+    GF_ASSERT(frame);
+
+    local = ((call_frame_t *)frame)->local;
+    if (!local) {
+        gf_log("cli", GF_LOG_ERROR, "frame->local is NULL");
+        goto out;
+    }
+
+    if (rsp->op_ret) {
+        if (rsp->op_errstr) {
+            ret = dict_set_dynstr_with_alloc(local->dict, "op_err_str",
+                                             rsp->op_errstr);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Failed to set op_errstr in local dictionary");
+                goto out;
+            }
+        }
+        ret = rsp->op_ret;
+        goto out;
+    }
+
+    ret = dict_get_int32_sizen(dict, "sub-cmd", &status_cmd);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not fetch status type");
+        goto out;
+    }
+
+    if ((status_cmd != GF_SNAP_STATUS_TYPE_SNAP) &&
+        (status_cmd != GF_SNAP_STATUS_TYPE_ITER)) {
+        dict_copy(dict, local->dict);
+        goto out;
+    }
+
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_snapshot_status_single_snap(local, dict, "status.snap0");
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not fetch snap count");
-                goto out;
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to create xml output for snapshot status");
         }
-
-        if (snapcount == 0) {
-                cli_out ("No snapshots present");
+    } else {
+        ret = cli_get_single_snap_status(dict, "status.snap0");
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, "Could not fetch status of snap");
         }
+    }
 
-        for (i = 1 ; i <= snapcount ; i++) {
-                ret = snprintf (key, sizeof (key), "snapname%d", i);
-                if (ret < 0) {
-                        goto out;
-                }
-
-                ret = dict_get_str (dict, key, &get_buffer);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Could not get %s ", key);
-                        goto out;
-                } else {
-                        cli_out ("%s", get_buffer);
-                }
-        }
 out:
-        return ret;
+    return ret;
 }
 
-int
-cli_get_snap_volume_status (dict_t *dict, char *key_prefix)
+static int
+gf_cli_generate_snapshot_event(gf_cli_rsp *rsp, dict_t *dict, int32_t type,
+                               char *snap_name, char *volname, char *snap_uuid,
+                               char *clone_name)
 {
-        int             ret             =       -1;
-        char            key[PATH_MAX]   =       "";
-        char            *buffer         =       NULL;
-        int             brickcount      =       0;
-        int             i               =       0;
-        int             pid             =       0;
-
-        GF_ASSERT (dict);
-        GF_ASSERT (key_prefix);
-
-        ret = snprintf (key, sizeof (key), "%s.brickcount", key_prefix);
-        if (ret < 0) {
+    int ret = -1;
+    int config_command = 0;
+    int32_t delete_cmd = -1;
+    uint64_t hard_limit = 0;
+    uint64_t soft_limit = 0;
+    char *auto_delete = NULL;
+    char *snap_activate = NULL;
+    char msg[PATH_MAX] = {
+        0,
+    };
+    char option[512] = {
+        0,
+    };
+
+    GF_VALIDATE_OR_GOTO("cli", dict, out);
+    GF_VALIDATE_OR_GOTO("cli", rsp, out);
+
+    switch (type) {
+        case GF_SNAP_OPTION_TYPE_CREATE:
+            if (!snap_name) {
+                gf_log("cli", GF_LOG_ERROR, "Failed to get snap name");
                 goto out;
-        }
-        ret = dict_get_int32 (dict, key, &brickcount);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to fetch brickcount");
+            }
+
+            if (!volname) {
+                gf_log("cli", GF_LOG_ERROR, "Failed to get volume name");
                 goto out;
-        }
+            }
 
-        for (i = 0 ; i < brickcount ; i++) {
-                ret = snprintf (key, sizeof (key), "%s.brick%d.path",
-                                key_prefix, i);
-                if (ret < 0) {
-                        goto out;
-                }
+            if (rsp->op_ret != 0) {
+                gf_event(EVENT_SNAPSHOT_CREATE_FAILED,
+                         "snapshot_name=%s;volume_name=%s;error=%s", snap_name,
+                         volname,
+                         rsp->op_errstr ? rsp->op_errstr
+                                        : "Please check log file for details");
+                ret = 0;
+                break;
+            }
 
-                ret = dict_get_str (dict, key, &buffer);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_INFO,
-                                "Unable to get Brick Path");
-                        continue;
-                }
-                cli_out ("\n\t%-17s %s   %s", "Brick Path", ":", buffer);
+            if (!snap_uuid) {
+                gf_log("cli", GF_LOG_ERROR, "Failed to get snap uuid");
+                goto out;
+            }
 
-                ret = snprintf (key, sizeof (key), "%s.brick%d.vgname",
-                                key_prefix, i);
-                if (ret < 0) {
-                        goto out;
-                }
+            gf_event(EVENT_SNAPSHOT_CREATED,
+                     "snapshot_name=%s;volume_name=%s;snapshot_uuid=%s",
+                     snap_name, volname, snap_uuid);
 
-                ret = dict_get_str (dict, key, &buffer);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_INFO,
-                                "Unable to get Volume Group");
-                        cli_out ("\t%-17s %s   %s", "Volume Group", ":",
-                                 "N/A");
-                } else
-                        cli_out ("\t%-17s %s   %s", "Volume Group", ":",
-                                 buffer);
-
-                ret = snprintf (key, sizeof (key), "%s.brick%d.status",
-                                key_prefix, i);
-                if (ret < 0) {
-                        goto out;
-                }
+            ret = 0;
+            break;
 
-                ret = dict_get_str (dict, key, &buffer);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_INFO,
-                                "Unable to get Brick Running");
-                        cli_out ("\t%-17s %s   %s", "Brick Running", ":",
-                                 "N/A");
-                } else
-                        cli_out ("\t%-17s %s   %s", "Brick Running", ":",
-                                 buffer);
-
-                ret = snprintf (key, sizeof (key), "%s.brick%d.pid",
-                                key_prefix, i);
-                if (ret < 0) {
-                        goto out;
-                }
+        case GF_SNAP_OPTION_TYPE_ACTIVATE:
+            if (!snap_name) {
+                gf_log("cli", GF_LOG_ERROR, "Failed to get snap name");
+                goto out;
+            }
 
-                ret = dict_get_int32 (dict, key, &pid);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_INFO,
-                                "Unable to get pid");
-                        cli_out ("\t%-17s %s   %s", "Brick PID", ":", "N/A");
-                } else
-                        cli_out ("\t%-17s %s   %d", "Brick PID", ":", pid);
-
-                ret = snprintf (key, sizeof (key), "%s.brick%d.data",
-                                key_prefix, i);
-                if (ret < 0) {
-                        goto out;
-                }
+            if (rsp->op_ret != 0) {
+                gf_event(EVENT_SNAPSHOT_ACTIVATE_FAILED,
+                         "snapshot_name=%s;error=%s", snap_name,
+                         rsp->op_errstr ? rsp->op_errstr
+                                        : "Please check log file for details");
+                ret = 0;
+                break;
+            }
 
-                ret = dict_get_str (dict, key, &buffer);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_INFO,
-                                "Unable to get Data Percent");
-                        cli_out ("\t%-17s %s   %s", "Data Percentage", ":",
-                                 "N/A");
-                } else
-                        cli_out ("\t%-17s %s   %s", "Data Percentage", ":",
-                                 buffer);
-
-                ret = snprintf (key, sizeof (key), "%s.brick%d.lvsize",
-                                key_prefix, i);
-                if (ret < 0) {
-                        goto out;
-                }
-                ret = dict_get_str (dict, key, &buffer);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_INFO, "Unable to get LV Size");
-                        cli_out ("\t%-17s %s   %s", "LV Size", ":", "N/A");
-                } else
-                        cli_out ("\t%-17s %s   %s", "LV Size", ":", buffer);
+            if (!snap_uuid) {
+                gf_log("cli", GF_LOG_ERROR, "Failed to get snap uuid");
+                goto out;
+            }
 
-        }
+            gf_event(EVENT_SNAPSHOT_ACTIVATED,
+                     "snapshot_name=%s;snapshot_uuid=%s", snap_name, snap_uuid);
 
-        ret = 0;
-out:
-        return ret;
-}
+            ret = 0;
+            break;
 
+        case GF_SNAP_OPTION_TYPE_DEACTIVATE:
+            if (!snap_name) {
+                gf_log("cli", GF_LOG_ERROR, "Failed to get snap name");
+                goto out;
+            }
 
+            if (rsp->op_ret != 0) {
+                gf_event(EVENT_SNAPSHOT_DEACTIVATE_FAILED,
+                         "snapshot_name=%s;error=%s", snap_name,
+                         rsp->op_errstr ? rsp->op_errstr
+                                        : "Please check log file for details");
+                ret = 0;
+                break;
+            }
 
-int
-cli_get_single_snap_status (dict_t *dict, char *keyprefix)
-{
-        int             ret             =       -1;
-        char            key[PATH_MAX]   =       "";
-        int             i               =       0;
-        int             volcount        =       0;
-        char            *get_buffer     =       NULL;
+            if (!snap_uuid) {
+                gf_log("cli", GF_LOG_ERROR, "Failed to get snap uuid");
+                goto out;
+            }
 
-        GF_ASSERT (dict);
-        GF_ASSERT (keyprefix);
+            gf_event(EVENT_SNAPSHOT_DEACTIVATED,
+                     "snapshot_name=%s;snapshot_uuid=%s", snap_name, snap_uuid);
 
-        ret = snprintf (key, sizeof (key), "%s.snapname", keyprefix);
-        if (ret < 0) {
-                goto out;
-        }
+            ret = 0;
+            break;
 
-        ret = dict_get_str (dict, key, &get_buffer);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to get snapname");
+        case GF_SNAP_OPTION_TYPE_RESTORE:
+            if (!snap_name) {
+                gf_log("cli", GF_LOG_ERROR, "Failed to get snap name");
                 goto out;
-        }
-        cli_out ("\nSnap Name : %s", get_buffer);
+            }
 
-        ret = snprintf (key, sizeof (key), "%s.uuid", keyprefix);
-        if (ret < 0) {
+            if (rsp->op_ret != 0) {
+                gf_event(EVENT_SNAPSHOT_RESTORE_FAILED,
+                         "snapshot_name=%s;error=%s", snap_name,
+                         rsp->op_errstr ? rsp->op_errstr
+                                        : "Please check log file for details");
+                ret = 0;
+                break;
+            }
+
+            if (!snap_uuid) {
+                ret = -1;
+                gf_log("cli", GF_LOG_ERROR, "Failed to get snap uuid");
                 goto out;
-        }
+            }
 
-        ret = dict_get_str (dict, key, &get_buffer);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to get snap UUID");
+            if (!volname) {
+                ret = -1;
+                gf_log("cli", GF_LOG_ERROR, "Failed to get volname");
                 goto out;
-        }
-        cli_out ("Snap UUID : %s", get_buffer);
+            }
 
-        ret = snprintf (key, sizeof (key), "%s.volcount", keyprefix);
-        if (ret < 0) {
+            gf_event(EVENT_SNAPSHOT_RESTORED,
+                     "snapshot_name=%s;snapshot_uuid=%s;volume_name=%s",
+                     snap_name, snap_uuid, volname);
+
+            ret = 0;
+            break;
+
+        case GF_SNAP_OPTION_TYPE_DELETE:
+            ret = dict_get_int32_sizen(dict, "sub-cmd", &delete_cmd);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR, "Could not get sub-cmd");
                 goto out;
-        }
+            }
 
-        ret = dict_get_int32 (dict, key, &volcount);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to get volume count");
+            /*
+             * Need not generate any event (success or failure) for delete *
+             * all, as it will trigger individual delete for all snapshots *
+             */
+            if (delete_cmd == GF_SNAP_DELETE_TYPE_ALL) {
+                ret = 0;
+                break;
+            }
+
+            if (!snap_name) {
+                gf_log("cli", GF_LOG_ERROR, "Failed to get snap name");
                 goto out;
-        }
+            }
 
-        for (i = 0 ; i < volcount ; i++) {
-                ret = snprintf (key, sizeof (key), "%s.vol%d", keyprefix, i);
-                if (ret < 0) {
-                        goto out;
-                }
+            if (rsp->op_ret != 0) {
+                gf_event(EVENT_SNAPSHOT_DELETE_FAILED,
+                         "snapshot_name=%s;error=%s", snap_name,
+                         rsp->op_errstr ? rsp->op_errstr
+                                        : "Please check log file for details");
+                ret = 0;
+                break;
+            }
 
-                ret = cli_get_snap_volume_status (dict, key);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Could not get snap volume status");
-                        goto out;
-                }
-        }
-out:
-        return ret;
-}
+            if (!snap_uuid) {
+                ret = -1;
+                gf_log("cli", GF_LOG_ERROR, "Failed to get snap uuid");
+                goto out;
+            }
 
-int32_t
-cli_populate_req_dict_for_delete (dict_t *snap_dict, dict_t *dict, size_t index)
-{
-        int32_t         ret             = -1;
-        char            key[PATH_MAX]   = "";
-        char            *buffer         = NULL;
-        int             type            = 0;
-        int             snapcount       = 0;
+            gf_event(EVENT_SNAPSHOT_DELETED,
+                     "snapshot_name=%s;snapshot_uuid=%s", snap_name, snap_uuid);
 
-        GF_ASSERT (snap_dict);
-        GF_ASSERT (dict);
+            ret = 0;
+            break;
 
-        ret = dict_set_int32 (snap_dict, "delete-cmd",
-                              GF_SNAP_DELETE_TYPE_SNAP);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not save command "
-                        "type in snap dictionary");
+        case GF_SNAP_OPTION_TYPE_CLONE:
+            if (!clone_name) {
+                gf_log("cli", GF_LOG_ERROR, "Failed to get clone name");
                 goto out;
-        }
+            }
 
-        ret = snprintf (key, sizeof (key), "snapname%lu", index);
-        if (ret < 0) {
+            if (!snap_name) {
+                gf_log("cli", GF_LOG_ERROR, "Failed to get snapname name");
                 goto out;
-        }
+            }
 
-        ret = dict_get_str (dict, key, &buffer);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get snapname");
-                goto out;
-        }
+            if (rsp->op_ret != 0) {
+                gf_event(EVENT_SNAPSHOT_CLONE_FAILED,
+                         "snapshot_name=%s;clone_name=%s;error=%s", snap_name,
+                         clone_name,
+                         rsp->op_errstr ? rsp->op_errstr
+                                        : "Please check log file for details");
+                ret = 0;
+                break;
+            }
 
-        ret = dict_set_dynstr_with_alloc (snap_dict, "snapname", buffer);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to save snapname");
+            if (!snap_uuid) {
+                ret = -1;
+                gf_log("cli", GF_LOG_ERROR, "Failed to get snap uuid");
                 goto out;
-        }
+            }
 
-        ret = dict_set_int32 (snap_dict, "type", GF_SNAP_OPTION_TYPE_DELETE);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to save command type");
-                goto out;
-        }
+            gf_event(EVENT_SNAPSHOT_CLONED,
+                     "snapshot_name=%s;clone_name=%s;clone_uuid=%s", snap_name,
+                     clone_name, snap_uuid);
 
-        ret = dict_set_dynstr_with_alloc (snap_dict, "cmd-str",
-                                       "snapshot delete");
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR,
-                        "Could not save command string as delete");
-                goto out;
-        }
-out:
-        return ret;
-}
+            ret = 0;
+            break;
 
-int
-cli_populate_req_dict_for_status (dict_t *snap_dict, dict_t *dict, int index) {
-        int             ret             =       -1;
-        char            key[PATH_MAX]   =       "";
-        char            *buffer         =       NULL;
-        int             type            =       0;
-        int             snapcount       =       0;
-
-        GF_ASSERT (snap_dict);
-        GF_ASSERT (dict);
-
-        ret = dict_set_uint32 (snap_dict, "status-cmd",
-                               GF_SNAP_STATUS_TYPE_SNAP);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not save command "
-                        "type in snap dict");
+        case GF_SNAP_OPTION_TYPE_CONFIG:
+            if (rsp->op_ret != 0) {
+                gf_event(EVENT_SNAPSHOT_CONFIG_UPDATE_FAILED, "error=%s",
+                         rsp->op_errstr ? rsp->op_errstr
+                                        : "Please check log file for details");
+                ret = 0;
+                break;
+            }
+
+            ret = dict_get_int32_sizen(dict, "config-command", &config_command);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR, "Could not fetch config type");
                 goto out;
-        }
+            }
+
+            if (config_command == GF_SNAP_CONFIG_DISPLAY) {
+                ret = 0;
+                break;
+            }
 
-        ret = snprintf (key, sizeof (key), "status.snap%d.snapname", index);
-         if (ret < 0) {
-                 goto out;
-         }
+            /* These are optional parameters therefore ignore the error */
+            ret = dict_get_uint64(dict, "snap-max-hard-limit", &hard_limit);
+            ret = dict_get_uint64(dict, "snap-max-soft-limit", &soft_limit);
+            ret = dict_get_str_sizen(dict, "auto-delete", &auto_delete);
+            ret = dict_get_str_sizen(dict, "snap-activate-on-create",
+                                     &snap_activate);
 
-        ret = dict_get_str (dict, key, &buffer);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not get snapname");
+            if (!hard_limit && !soft_limit && !auto_delete && !snap_activate) {
+                ret = -1;
+                gf_log("cli", GF_LOG_ERROR,
+                       "At least one option from "
+                       "snap-max-hard-limit, snap-max-soft-limit, "
+                       "auto-delete and snap-activate-on-create "
+                       "should be set");
                 goto out;
-        }
+            }
 
-        ret = dict_set_str (snap_dict, "snapname", buffer);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not save snapname "
-                        "in snap dict");
-                goto out;
+            if (hard_limit || soft_limit) {
+                snprintf(option, sizeof(option), "%s=%" PRIu64,
+                         hard_limit ? "hard_limit" : "soft_limit",
+                         hard_limit ? hard_limit : soft_limit);
+            } else if (auto_delete || snap_activate) {
+                snprintf(option, sizeof(option), "%s=%s",
+                         auto_delete ? "auto-delete" : "snap-activate",
+                         auto_delete ? auto_delete : snap_activate);
+            }
 
-        }
+            volname = NULL;
+            ret = dict_get_str_sizen(dict, "volname", &volname);
 
-        ret = dict_set_int32 (snap_dict, "type", GF_SNAP_OPTION_TYPE_STATUS);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR,
-                        "Could not save command type");
-                goto out;
-        }
+            snprintf(msg, sizeof(msg), "config_type=%s;%s",
+                     volname ? "volume_config" : "system_config", option);
 
-        ret = dict_set_dynstr_with_alloc (snap_dict, "cmd-str",
-                                       "snapshot status");
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR,
-                        "Could not save command string as status");
-                goto out;
-        }
+            gf_event(EVENT_SNAPSHOT_CONFIG_UPDATED, "%s", msg);
 
-        ret = dict_set_int32 (snap_dict, "hold_vol_locks", _gf_false);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR,
-                        "Setting volume lock flag failed");
-                goto out;
-        }
+            ret = 0;
+            break;
+
+        default:
+            gf_log("cli", GF_LOG_WARNING,
+                   "Cannot generate event for unknown type.");
+            ret = 0;
+            goto out;
+    }
 
 out:
-        return ret;
+    return ret;
 }
 
-int
-cli_snapshot_status (dict_t *dict, gf_cli_rsp *rsp,
-                             call_frame_t *frame)
+/*
+ * Fetch necessary data from dict at one place instead of *
+ * repeating the same code again and again.               *
+ */
+static int
+gf_cli_snapshot_get_data_from_dict(dict_t *dict, char **snap_name,
+                                   char **volname, char **snap_uuid,
+                                   int8_t *soft_limit_flag, char **clone_name)
 {
-        char            key[PATH_MAX]   =       "";
-        int             ret             =       -1;
-        int             status_cmd      =       -1;
-        cli_local_t     *local          =       NULL;
+    int ret = -1;
 
-        GF_ASSERT (dict);
-        GF_ASSERT (rsp);
-        GF_ASSERT (frame);
+    GF_VALIDATE_OR_GOTO("cli", dict, out);
 
-        local = ((call_frame_t *) frame) -> local;
-        if (!local) {
-                gf_log ("cli", GF_LOG_ERROR, "frame->local is NULL");
-                goto out;
-        }
-
-        if (rsp->op_ret) {
-                if (rsp->op_errstr) {
-                        ret = dict_set_dynstr_with_alloc (local->dict,
-                                                          "op_err_str",
-                                                          rsp->op_errstr);
-                        if (ret) {
-                                gf_log ("cli", GF_LOG_ERROR, "Failed to set "
-                                        "op_errstr in local dictionary");
-                                goto out;
-                        }
-                }
-                ret = rsp->op_ret;
-                goto out;
+    if (snap_name) {
+        ret = dict_get_str_sizen(dict, "snapname", snap_name);
+        if (ret) {
+            gf_log("cli", GF_LOG_DEBUG, "failed to get snapname from dict");
         }
+    }
 
-        ret = dict_get_int32 (dict, "status-cmd", &status_cmd);
+    if (volname) {
+        ret = dict_get_str_sizen(dict, "volname1", volname);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not fetch status type");
-                goto out;
+            gf_log("cli", GF_LOG_DEBUG, "failed to get volname1 from dict");
         }
+    }
 
-        if (status_cmd != GF_SNAP_STATUS_TYPE_SNAP) {
-                dict_copy (dict, local->dict);
-                goto out;
+    if (snap_uuid) {
+        ret = dict_get_str_sizen(dict, "snapuuid", snap_uuid);
+        if (ret) {
+            gf_log("cli", GF_LOG_DEBUG, "failed to get snapuuid from dict");
         }
+    }
 
-
-        ret = snprintf (key, sizeof (key), "status.snap0");
-        if (ret < 0) {
-                goto out;
+    if (soft_limit_flag) {
+        ret = dict_get_int8(dict, "soft-limit-reach", soft_limit_flag);
+        if (ret) {
+            gf_log("cli", GF_LOG_DEBUG,
+                   "failed to get soft-limit-reach from dict");
         }
+    }
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_snapshot_status_single_snap (local, dict, key);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to create "
-                                "xml output for snapshot status");
-                        goto out;
-                }
-        } else {
-                ret = cli_get_single_snap_status (dict, key);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Could not fetch "
-                                "status of snap");
-                        goto out;
-                }
+    if (clone_name) {
+        ret = dict_get_str_sizen(dict, "clonename", clone_name);
+        if (ret) {
+            gf_log("cli", GF_LOG_DEBUG, "failed to get clonename from dict");
         }
+    }
 
-        ret = 0;
+    ret = 0;
 out:
-        return ret;
+    return ret;
 }
 
-int
-gf_cli_snapshot_cbk (struct rpc_req *req, struct iovec *iov,
-                     int count, void *myframe)
-{
-        int                   ret                      = -1;
-        gf_cli_rsp            rsp                      = {0, };
-        dict_t               *dict                     = NULL;
-        char                 *snap_name                = NULL;
-        int32_t               type                     =  0;
-        call_frame_t         *frame                    = NULL;
-        gf_boolean_t         snap_driven               = _gf_false;
-        int8_t               soft_limit_flag           = -1;
-        char                 *volname                  = NULL;
-
-        if (req->rpc_status == -1) {
-                ret = -1;
-                goto out;
+static int
+gf_cli_snapshot_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                    void *myframe)
+{
+    int ret = -1;
+    gf_cli_rsp rsp = {
+        0,
+    };
+    dict_t *dict = NULL;
+    char *snap_name = NULL;
+    char *clone_name = NULL;
+    int32_t type = 0;
+    call_frame_t *frame = NULL;
+    gf_boolean_t snap_driven = _gf_false;
+    int8_t soft_limit_flag = -1;
+    char *volname = NULL;
+    char *snap_uuid = NULL;
+
+    GF_ASSERT(myframe);
+
+    if (req->rpc_status == -1) {
+        goto out;
+    }
+
+    frame = myframe;
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(frame->this->name, GF_LOG_ERROR, XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    dict = dict_new();
+
+    if (!dict) {
+        ret = -1;
+        goto out;
+    }
+
+    ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &dict);
+
+    if (ret)
+        goto out;
+
+    ret = dict_get_int32_sizen(dict, "type", &type);
+    if (ret) {
+        gf_log(frame->this->name, GF_LOG_ERROR, "failed to get type");
+        goto out;
+    }
+
+    ret = gf_cli_snapshot_get_data_from_dict(
+        dict, &snap_name, &volname, &snap_uuid, &soft_limit_flag, &clone_name);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to fetch data from dict.");
+        goto out;
+    }
+
+#if (USE_EVENTS)
+    ret = gf_cli_generate_snapshot_event(&rsp, dict, type, snap_name, volname,
+                                         snap_uuid, clone_name);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to generate snapshot event");
+        goto out;
+    }
+#endif
+
+    /* Snapshot status and delete command is handled separately */
+    if (global_state->mode & GLUSTER_MODE_XML &&
+        GF_SNAP_OPTION_TYPE_STATUS != type &&
+        GF_SNAP_OPTION_TYPE_DELETE != type) {
+        ret = cli_xml_output_snapshot(type, dict, rsp.op_ret, rsp.op_errno,
+                                      rsp.op_errstr);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
         }
 
-        frame = myframe;
+        goto out;
+    }
 
-        ret = xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (frame->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
+    switch (type) {
+        case GF_SNAP_OPTION_TYPE_CREATE:
+            if (rsp.op_ret) {
+                cli_err("snapshot create: failed: %s",
+                        rsp.op_errstr ? rsp.op_errstr
+                                      : "Please check log file for details");
+                ret = rsp.op_ret;
                 goto out;
-        }
+            }
 
-        dict = dict_new ();
-
-        if (!dict) {
+            if (!snap_name) {
                 ret = -1;
+                gf_log("cli", GF_LOG_ERROR, "Failed to get snap name");
                 goto out;
-        }
+            }
 
-        ret = dict_unserialize (rsp.dict.dict_val, rsp.dict.dict_len, &dict);
-
-        if (ret)
+            if (!volname) {
+                ret = -1;
+                gf_log("cli", GF_LOG_ERROR, "Failed to get volume name");
+                goto out;
+            }
+
+            cli_out("snapshot create: success: Snap %s created successfully",
+                    snap_name);
+
+            if (soft_limit_flag == 1) {
+                cli_out(
+                    "Warning: Soft-limit of volume (%s) is "
+                    "reached. Snapshot creation is not possible "
+                    "once hard-limit is reached.",
+                    volname);
+            }
+            ret = 0;
+            break;
+
+        case GF_SNAP_OPTION_TYPE_CLONE:
+            if (rsp.op_ret) {
+                cli_err("snapshot clone: failed: %s",
+                        rsp.op_errstr ? rsp.op_errstr
+                                      : "Please check log file for details");
+                ret = rsp.op_ret;
                 goto out;
+            }
 
-        ret = dict_get_int32 (dict, "type", &type);
-        if (ret) {
-                gf_log (frame->this->name, GF_LOG_ERROR, "failed to get type");
+            if (!clone_name) {
+                gf_log("cli", GF_LOG_ERROR, "Failed to get clone name");
                 goto out;
-        }
-
-        /* Snapshot status command is handled separately */
-        if (global_state->mode & GLUSTER_MODE_XML &&
-            GF_SNAP_OPTION_TYPE_STATUS != type) {
-                ret = cli_xml_output_snapshot (type, dict, rsp.op_ret,
-                                               rsp.op_errno, rsp.op_errstr);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Error outputting to xml");
-                }
+            }
 
+            if (!snap_name) {
+                gf_log("cli", GF_LOG_ERROR, "Failed to get snapname name");
                 goto out;
-        }
-
-        switch (type) {
-        case GF_SNAP_OPTION_TYPE_CREATE:
-                if (rsp.op_ret) {
-                        cli_err("snapshot create: failed: %s",
-                                 rsp.op_errstr ? rsp.op_errstr :
-                                 "Please check log file for details");
-                                 ret = rsp.op_ret;
-                                 goto out;
-                }
-
-                ret = dict_get_str (dict, "snapname", &snap_name);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Failed to get snap name");
-                        goto out;
-                }
+            }
 
-                /* TODO : Instead of using volname1 directly use
-                 * volname$i in loop once snapshot of multiple
-                 * volumes are supported
-                 */
-                ret = dict_get_str (dict, "volname1", &volname);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Failed to get volume name");
-                        goto out;
-                }
+            cli_out("snapshot clone: success: Clone %s created successfully",
+                    clone_name);
 
-                cli_out ("snapshot create: success: Snap %s created "
-                                        "successfully", snap_name);
-
-                ret = dict_get_int8 (dict, "soft-limit-reach",
-                                    &soft_limit_flag);
-                if (soft_limit_flag == 1) {
-                        cli_out ("Warning: Soft-limit of volume (%s) is "
-                                "reached. Snapshot creation is not possible "
-                                "once hard-limit is reached.", volname);
-                }
-                ret = 0;
-                break;
+            ret = 0;
+            break;
 
         case GF_SNAP_OPTION_TYPE_RESTORE:
-                /* TODO: Check if rsp.op_ret needs to be checked here. Or is
-                 * it ok to check this in the start of the function where we
-                 * get rsp.*/
-                if (rsp.op_ret) {
-                        cli_err("snapshot restore: failed: %s",
-                                 rsp.op_errstr ? rsp.op_errstr :
-                                 "Please check log file for details");
-                                 ret = rsp.op_ret;
-                                 goto out;
-                }
+            if (rsp.op_ret) {
+                cli_err("snapshot restore: failed: %s",
+                        rsp.op_errstr ? rsp.op_errstr
+                                      : "Please check log file for details");
+                ret = rsp.op_ret;
+                goto out;
+            }
 
-                ret = dict_get_str (dict, "snapname", &snap_name);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Failed to get snap name");
-                        goto out;
-                }
+            if (!snap_name) {
+                gf_log("cli", GF_LOG_ERROR, "Failed to get snap name");
+                goto out;
+            }
 
-                cli_out ("Snapshot restore: %s: Snap restored "
-                                 "successfully", snap_name);
+            cli_out("Snapshot restore: %s: Snap restored successfully",
+                    snap_name);
 
-                ret = 0;
-                break;
+            ret = 0;
+            break;
         case GF_SNAP_OPTION_TYPE_ACTIVATE:
-                /* TODO: Check if rsp.op_ret needs to be checked here. Or is
-                 * it ok to check this in the start of the function where we
-                 * get rsp.*/
-                if (rsp.op_ret) {
-                        cli_err("snapshot activate: failed: %s",
-                                 rsp.op_errstr ? rsp.op_errstr :
-                                 "Please check log file for details");
-                                 ret = rsp.op_ret;
-                                 goto out;
-                }
+            if (rsp.op_ret) {
+                cli_err("snapshot activate: failed: %s",
+                        rsp.op_errstr ? rsp.op_errstr
+                                      : "Please check log file for details");
+                ret = rsp.op_ret;
+                goto out;
+            }
 
-                ret = dict_get_str (dict, "snapname", &snap_name);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Failed to get snap name");
-                        goto out;
-                }
+            if (!snap_name) {
+                gf_log("cli", GF_LOG_ERROR, "Failed to get snap name");
+                goto out;
+            }
 
-                cli_out ("Snapshot activate: %s: Snap activated "
-                                 "successfully", snap_name);
+            cli_out("Snapshot activate: %s: Snap activated successfully",
+                    snap_name);
 
-                ret = 0;
-                break;
+            ret = 0;
+            break;
 
         case GF_SNAP_OPTION_TYPE_DEACTIVATE:
-                /* TODO: Check if rsp.op_ret needs to be checked here. Or is
-                 * it ok to check this in the start of the function where we
-                 * get rsp.*/
-                if (rsp.op_ret) {
-                        cli_err("snapshot deactivate: failed: %s",
-                                 rsp.op_errstr ? rsp.op_errstr :
-                                 "Please check log file for details");
-                                 ret = rsp.op_ret;
-                                 goto out;
-                }
+            if (rsp.op_ret) {
+                cli_err("snapshot deactivate: failed: %s",
+                        rsp.op_errstr ? rsp.op_errstr
+                                      : "Please check log file for details");
+                ret = rsp.op_ret;
+                goto out;
+            }
 
-                ret = dict_get_str (dict, "snapname", &snap_name);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Failed to get snap name");
-                        goto out;
-                }
+            if (!snap_name) {
+                gf_log("cli", GF_LOG_ERROR, "Failed to get snap name");
+                goto out;
+            }
 
-                cli_out ("Snapshot deactivate: %s: Snap deactivated "
-                                 "successfully", snap_name);
+            cli_out("Snapshot deactivate: %s: Snap deactivated successfully",
+                    snap_name);
 
-                ret = 0;
-                break;
+            ret = 0;
+            break;
         case GF_SNAP_OPTION_TYPE_INFO:
-                if (rsp.op_ret) {
-                        cli_err ("Snapshot info : failed: %s",
-                                  rsp.op_errstr ? rsp.op_errstr :
-                                  "Please check log file for details");
-                        ret = rsp.op_ret;
-                        goto out;
-                }
+            if (rsp.op_ret) {
+                cli_err("Snapshot info : failed: %s",
+                        rsp.op_errstr ? rsp.op_errstr
+                                      : "Please check log file for details");
+                ret = rsp.op_ret;
+                goto out;
+            }
 
-                snap_driven = dict_get_str_boolean (dict, "snap-driven",
-                                                    _gf_false);
-                if (snap_driven == _gf_true) {
-                        ret = cli_call_snapshot_info  (dict, snap_driven);
-                        if (ret) {
-                                gf_log ("cli", GF_LOG_ERROR,
-                                        "Snapshot info failed");
-                                goto out;
-                        }
-                } else if (snap_driven == _gf_false) {
-                        ret = cli_get_snaps_in_volume (dict);
-                        if (ret) {
-                                gf_log ("cli", GF_LOG_ERROR,
-                                        "Snapshot info failed");
-                                goto out;
-                        }
+            snap_driven = dict_get_str_boolean(dict, "snap-driven", _gf_false);
+            if (snap_driven == _gf_true) {
+                ret = cli_call_snapshot_info(dict, snap_driven);
+                if (ret) {
+                    gf_log("cli", GF_LOG_ERROR, "Snapshot info failed");
+                    goto out;
                 }
-                break;
-
-        case GF_SNAP_OPTION_TYPE_CONFIG:
-                ret = cli_snapshot_config_display (dict, &rsp);
+            } else if (snap_driven == _gf_false) {
+                ret = cli_get_snaps_in_volume(dict);
                 if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to display "
-                                "snapshot config output.");
-                        goto out;
+                    gf_log("cli", GF_LOG_ERROR, "Snapshot info failed");
+                    goto out;
                 }
-                break;
+            }
+            break;
+
+        case GF_SNAP_OPTION_TYPE_CONFIG:
+            ret = cli_snapshot_config_display(dict, &rsp);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Failed to display snapshot config output.");
+                goto out;
+            }
+            break;
 
         case GF_SNAP_OPTION_TYPE_LIST:
-                if (rsp.op_ret) {
-                        cli_err ("Snapshot list : failed: %s",
-                                 rsp.op_errstr ? rsp.op_errstr :
-                                "Please check log file for details");
-                        ret = rsp.op_ret;
-                        goto out;
-                }
+            if (rsp.op_ret) {
+                cli_err("Snapshot list : failed: %s",
+                        rsp.op_errstr ? rsp.op_errstr
+                                      : "Please check log file for details");
+                ret = rsp.op_ret;
+                goto out;
+            }
 
-                ret = cli_snapshot_list (dict);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to display "
-                                "snapshot list");
-                        goto out;
-                }
-                break;
+            ret = cli_snapshot_list(dict);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR, "Failed to display snapshot list");
+                goto out;
+            }
+            break;
 
         case GF_SNAP_OPTION_TYPE_DELETE:
-                ret = cli_snapshot_remove_reply (&rsp, dict, frame);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Failed to delete snap");
-                        goto out;
-                }
-                break;
+            ret = cli_snapshot_remove_reply(&rsp, dict, frame);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR, "Failed to delete snap");
+                goto out;
+            }
+            break;
 
         case GF_SNAP_OPTION_TYPE_STATUS:
-                ret = cli_snapshot_status (dict, &rsp, frame);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to display "
-                                "snapshot status output.");
-                        goto out;
-                }
-                break;
+            ret = cli_snapshot_status(dict, &rsp, frame);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Failed to display snapshot status output.");
+                goto out;
+            }
+            break;
 
         default:
-                cli_err ("Unknown command executed");
-                ret = -1;
-                goto out;
-        }
+            cli_err("Unknown command executed");
+            ret = -1;
+            goto out;
+    }
 out:
-        if (dict)
-                dict_unref (dict);
-        cli_cmd_broadcast_response (ret);
+    if (dict)
+        dict_unref(dict);
+    cli_cmd_broadcast_response(ret);
 
-        free (rsp.dict.dict_val);
-        free (rsp.op_errstr);
+    free(rsp.dict.dict_val);
+    free(rsp.op_errstr);
 
-        return ret;
+    return ret;
 }
 
 int32_t
-gf_cli_snapshot_for_delete (call_frame_t *frame, xlator_t *this,
-                            void *data)
-{
-        gf_cli_req      req                     = {{0,}};
-        dict_t          *options                = NULL;
-        int32_t         ret                     = -1;
-        int32_t         cmd                     = -1;
-        cli_local_t     *local                  = NULL;
-        dict_t          *snap_dict              = NULL;
-        int32_t         snapcount               = 0;
-        int             i                       = 0;
-        char            question[PATH_MAX]      = "";
-        char            *volname                = NULL;
-        gf_answer_t     answer                  = GF_ANSWER_NO;
-
-        GF_VALIDATE_OR_GOTO ("cli", frame,  out);
-        GF_VALIDATE_OR_GOTO ("cli", frame->local, out);
-        GF_VALIDATE_OR_GOTO ("cli", this, out);
-        GF_VALIDATE_OR_GOTO ("cli", data, out);
-
-        local = frame->local;
-
-        options = data;
-
-        ret = dict_get_int32 (local->dict, "delete-cmd", &cmd);
+gf_cli_snapshot_for_delete(call_frame_t *frame, xlator_t *this, void *data)
+{
+    gf_cli_req req = {{
+        0,
+    }};
+    int32_t ret = -1;
+    int32_t cmd = -1;
+    cli_local_t *local = NULL;
+    dict_t *snap_dict = NULL;
+    int32_t snapcount = 0;
+    int i = 0;
+    char question[PATH_MAX] = "";
+    char *volname = NULL;
+    gf_answer_t answer = GF_ANSWER_NO;
+
+    GF_VALIDATE_OR_GOTO("cli", frame, out);
+    GF_VALIDATE_OR_GOTO("cli", frame->local, out);
+    GF_VALIDATE_OR_GOTO("cli", this, out);
+    GF_VALIDATE_OR_GOTO("cli", data, out);
+
+    local = frame->local;
+
+    ret = dict_get_int32_sizen(local->dict, "sub-cmd", &cmd);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get sub-cmd");
+        goto out;
+    }
+
+    /* No need multiple RPCs for individual snapshot delete*/
+    if (cmd == GF_SNAP_DELETE_TYPE_SNAP) {
+        ret = 0;
+        goto out;
+    }
+
+    ret = dict_get_int32_sizen(local->dict, "snapcount", &snapcount);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not get snapcount");
+        goto out;
+    }
+
+    if (global_state->mode & GLUSTER_MODE_XML) {
+#ifdef HAVE_LIB_XML
+        ret = xmlTextWriterWriteFormatElement(
+            local->writer, (xmlChar *)"snapCount", "%d", snapcount);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get "
-                        "delete-cmd");
-                goto out;
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to write xml element \"snapCount\"");
+            goto out;
+        }
+#endif /* HAVE_LIB_XML */
+    } else if (snapcount == 0) {
+        cli_out("No snapshots present");
+        goto out;
+    }
+
+    if (cmd == GF_SNAP_DELETE_TYPE_ALL) {
+        snprintf(question, sizeof(question),
+                 "System contains %d snapshot(s).\nDo you still "
+                 "want to continue and delete them? ",
+                 snapcount);
+    } else {
+        ret = dict_get_str_sizen(local->dict, "volname", &volname);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to fetch volname from local dictionary");
+            goto out;
         }
 
-        /* No need multiple RPCs for individual snapshot delete*/
-        if (cmd == GF_SNAP_DELETE_TYPE_SNAP) {
-                ret = 0;
-                goto out;
-        }
+        snprintf(question, sizeof(question),
+                 "Volume (%s) contains %d snapshot(s).\nDo you still want to "
+                 "continue and delete them? ",
+                 volname, snapcount);
+    }
 
-        ret = dict_get_int32 (local->dict, "snapcount",
-                              &snapcount);
+    answer = cli_cmd_get_confirmation(global_state, question);
+    if (GF_ANSWER_NO == answer) {
+        ret = 0;
+        gf_log("cli", GF_LOG_DEBUG,
+               "User cancelled snapshot delete operation for snap delete");
+        goto out;
+    }
+
+    for (i = 1; i <= snapcount; i++) {
+        ret = -1;
+
+        snap_dict = dict_new();
+        if (!snap_dict)
+            goto out;
+
+        ret = cli_populate_req_dict_for_delete(snap_dict, local->dict, i);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not get "
-                        "snapcount");
-                goto out;
+            gf_log("cli", GF_LOG_ERROR,
+                   "Could not populate snap request dictionary");
+            goto out;
         }
 
-        if (snapcount == 0) {
-                cli_out ("No snapshots present");
-                goto out;
+        ret = cli_to_glusterd(&req, frame, gf_cli_snapshot_cbk,
+                              (xdrproc_t)xdr_gf_cli_req, snap_dict,
+                              GLUSTER_CLI_SNAP, this, cli_rpc_prog, NULL);
+        if (ret) {
+            /* Fail the operation if deleting one of the
+             * snapshots is failed
+             */
+            gf_log("cli", GF_LOG_ERROR,
+                   "cli_to_glusterd for snapshot delete failed");
+            goto out;
         }
+        dict_unref(snap_dict);
+        snap_dict = NULL;
+    }
 
-        if (cmd == GF_SNAP_DELETE_TYPE_ALL) {
-                snprintf (question, sizeof (question), "System contains %d "
-                          "snapshot(s).\nDo you still "
-                          "want to continue and delete them? ",
-                           snapcount);
-        } else {
-                ret = dict_get_str (local->dict, "volname", &volname);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to fetch "
-                                "volname from local dictionary");
-                        goto out;
-                }
+out:
+    if (snap_dict)
+        dict_unref(snap_dict);
+    GF_FREE(req.dict.dict_val);
 
-                snprintf (question, sizeof (question), "Volume (%s) contains "
-                          "%d snapshot(s).\nDo you still want to "
-                           "continue and delete them? ", volname,
-                           snapcount);
-        }
+    return ret;
+}
 
-        answer = cli_cmd_get_confirmation (global_state, question);
-        if (GF_ANSWER_NO == answer) {
-                ret = 0;
-                gf_log ("cli", GF_LOG_DEBUG, "User cancelled "
-                        "snapshot delete operation for snap delete");
-                goto out;
-        }
+static int32_t
+gf_cli_snapshot_for_status(call_frame_t *frame, xlator_t *this, void *data)
+{
+    gf_cli_req req = {{
+        0,
+    }};
+    int ret = -1;
+    int32_t cmd = -1;
+    cli_local_t *local = NULL;
+    dict_t *snap_dict = NULL;
+    int snapcount = 0;
+    int i = 0;
+
+    GF_VALIDATE_OR_GOTO("cli", frame, out);
+    GF_VALIDATE_OR_GOTO("cli", frame->local, out);
+    GF_VALIDATE_OR_GOTO("cli", this, out);
+    GF_VALIDATE_OR_GOTO("cli", data, out);
+
+    local = frame->local;
+
+    ret = dict_get_int32_sizen(local->dict, "sub-cmd", &cmd);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get sub-cmd");
+        goto out;
+    }
+
+    /* Snapshot status of single snap (i.e. GF_SNAP_STATUS_TYPE_SNAP)
+     * is already handled. Therefore we can return from here.
+     * If want to get status of all snaps in the system or volume then
+     * we should get them one by one.*/
+    if ((cmd == GF_SNAP_STATUS_TYPE_SNAP) ||
+        (cmd == GF_SNAP_STATUS_TYPE_ITER)) {
+        ret = 0;
+        goto out;
+    }
 
-        for (i = 1 ; i <= snapcount ; i++) {
-                ret = -1;
+    ret = dict_get_int32_sizen(local->dict, "status.snapcount", &snapcount);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not get snapcount");
+        goto out;
+    }
 
-                snap_dict = dict_new();
-                if (!snap_dict)
-                        goto out;
+    if (snapcount == 0 && !(global_state->mode & GLUSTER_MODE_XML)) {
+        cli_out("No snapshots present");
+    }
 
-                ret = cli_populate_req_dict_for_delete (snap_dict,
-                                                       local->dict, i);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Could not "
-                                "populate snap request dictionary");
-                        goto out;
-                }
+    for (i = 0; i < snapcount; i++) {
+        ret = -1;
 
-                ret = cli_to_glusterd (&req, frame,
-                                gf_cli_snapshot_cbk,
-                                (xdrproc_t) xdr_gf_cli_req, snap_dict,
-                                GLUSTER_CLI_SNAP, this, cli_rpc_prog,
-                                NULL);
-                if (ret) {
-                        /* Fail the operation if deleting one of the
-                         * snapshots is failed
-                         */
-                        gf_log ("cli", GF_LOG_ERROR, "cli_to_glusterd "
-                                "for snapshot delete failed");
-                        goto out;
-                }
-                dict_unref (snap_dict);
-                snap_dict = NULL;
+        snap_dict = dict_new();
+        if (!snap_dict)
+            goto out;
+
+        ret = cli_populate_req_dict_for_status(snap_dict, local->dict, i);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Could not populate snap request dictionary");
+            goto out;
+        }
+
+        ret = cli_to_glusterd(&req, frame, gf_cli_snapshot_cbk,
+                              (xdrproc_t)xdr_gf_cli_req, snap_dict,
+                              GLUSTER_CLI_SNAP, this, cli_rpc_prog, NULL);
+
+        /* Ignore the return value and error for snapshot
+         * status of type "ALL" or "VOL"
+         *
+         * Scenario : There might be case where status command
+         * and delete command might be issued at the same time.
+         * In that case when status tried to fetch detail of
+         * snap which has been deleted by concurrent command,
+         * then it will show snapshot not present. Which will
+         * not be appropriate.
+         */
+        if (ret && (cmd != GF_SNAP_STATUS_TYPE_ALL &&
+                    cmd != GF_SNAP_STATUS_TYPE_VOL)) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "cli_to_glusterd for snapshot status failed");
+            goto out;
         }
+        dict_unref(snap_dict);
+        snap_dict = NULL;
+    }
 
+    ret = 0;
 out:
-        if (snap_dict)
-                dict_unref (snap_dict);
+    if (snap_dict)
+        dict_unref(snap_dict);
+    GF_FREE(req.dict.dict_val);
 
-        return ret;
+    return ret;
 }
 
-int32_t
-gf_cli_snapshot_for_status (call_frame_t *frame, xlator_t *this,
-                            void *data)
+static int32_t
+gf_cli_snapshot(call_frame_t *frame, xlator_t *this, void *data)
 {
+    gf_cli_req req = {{
+        0,
+    }};
+    dict_t *options = NULL;
+    int ret = -1;
+    int tmp_ret = -1;
+    cli_local_t *local = NULL;
+    char *err_str = NULL;
+    int type = -1;
 
-        gf_cli_req        req           = {{0,}};
-        dict_t           *options       = NULL;
-        int               ret           = -1;
-        int32_t           cmd           = -1;
-        cli_local_t      *local         = NULL;
-        dict_t           *snap_dict     = NULL;
-        int              snapcount      =  0;
-        int              i              =  0;
+    if (!frame || !frame->local || !this || !data)
+        goto out;
 
-        GF_VALIDATE_OR_GOTO ("cli", frame, out);
-        GF_VALIDATE_OR_GOTO ("cli", frame->local, out);
-        GF_VALIDATE_OR_GOTO ("cli", this, out);
-        GF_VALIDATE_OR_GOTO ("cli", data, out);
+    local = frame->local;
 
-        local = frame->local;
-        options = data;
+    options = data;
 
-        ret = dict_get_int32 (local->dict, "status-cmd", &cmd);
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_snapshot_begin_composite_op(local);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get status-cmd");
-                goto out;
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to begin snapshot xml composite op");
+            goto out;
         }
+    }
 
-        /* Snapshot status of single snap (i.e. GF_SNAP_STATUS_TYPE_SNAP)
-         * is already handled. Therefore we can return from here.
-         * If want to get status of all snaps in the system or volume then
-         * we should get them one by one.*/
-        if (cmd == GF_SNAP_STATUS_TYPE_SNAP) {
-                ret = 0;
-                goto out;
+    ret = cli_to_glusterd(&req, frame, gf_cli_snapshot_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, options, GLUSTER_CLI_SNAP,
+                          this, cli_rpc_prog, NULL);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "cli_to_glusterd for snapshot failed");
+        goto xmlend;
+    }
+
+    ret = dict_get_int32_sizen(local->dict, "type", &type);
+
+    if (GF_SNAP_OPTION_TYPE_STATUS == type) {
+        ret = gf_cli_snapshot_for_status(frame, this, data);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "cli to glusterd for snapshot status command failed");
         }
 
-        ret = dict_get_int32 (local->dict, "status.snapcount", &snapcount);
+        goto xmlend;
+    }
+
+    if (GF_SNAP_OPTION_TYPE_DELETE == type) {
+        ret = gf_cli_snapshot_for_delete(frame, this, data);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not get snapcount");
-                goto out;
+            gf_log("cli", GF_LOG_ERROR,
+                   "cli to glusterd for snapshot delete command failed");
         }
 
-        if (snapcount == 0) {
-                cli_out ("No snapshots present");
+        goto xmlend;
+    }
+
+    ret = 0;
+
+xmlend:
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_snapshot_end_composite_op(local);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to end snapshot xml composite op");
+            goto out;
+        }
+    }
+out:
+    if (ret && local && GF_SNAP_OPTION_TYPE_STATUS == type) {
+        tmp_ret = dict_get_str_sizen(local->dict, "op_err_str", &err_str);
+        if (tmp_ret || !err_str) {
+            cli_err("Snapshot Status : failed: %s",
+                    "Please check log file for details");
+        } else {
+            cli_err("Snapshot Status : failed: %s", err_str);
+            dict_del_sizen(local->dict, "op_err_str");
         }
+    }
 
-        for (i = 0 ; i < snapcount; i++) {
-                ret = -1;
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
 
-                snap_dict = dict_new();
-                if (!snap_dict)
-                        goto out;
+    GF_FREE(req.dict.dict_val);
 
-                ret = cli_populate_req_dict_for_status (snap_dict,
-                                local->dict, i);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Could not "
-                                        "populate snap request dictionary");
-                        goto out;
-                }
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        /* XML mode handles its own error */
+        ret = 0;
+    }
+    return ret;
+}
 
-                ret = cli_to_glusterd (&req, frame,
-                                gf_cli_snapshot_cbk,
-                                (xdrproc_t) xdr_gf_cli_req, snap_dict,
-                                GLUSTER_CLI_SNAP, this, cli_rpc_prog,
-                                NULL);
-
-                /* Ignore the return value and error for snapshot
-                 * status of type "ALL" or "VOL"
-                 *
-                 * Scenario : There might be case where status command
-                 * and delete command might be issued at the same time.
-                 * In that case when status tried to fetch detail of
-                 * snap which has been deleted by concurrent command,
-                 * then it will show snapshot not present. Which will
-                 * not be appropriate.
-                 */
-                dict_unref (snap_dict);
-                snap_dict = NULL;
+static int32_t
+gf_cli_barrier_volume_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                          void *myframe)
+{
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status)
+        goto out;
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
+    gf_log("cli", GF_LOG_DEBUG, "Received response to barrier");
+
+    if (rsp.op_ret) {
+        if (rsp.op_errstr && (strlen(rsp.op_errstr) > 1)) {
+            cli_err("volume barrier: command unsuccessful : %s", rsp.op_errstr);
+        } else {
+            cli_err("volume barrier: command unsuccessful");
         }
+    } else {
+        cli_out("volume barrier: command successful");
+    }
+    ret = rsp.op_ret;
 
-        ret = 0;
 out:
-        if (snap_dict)
-                dict_unref (snap_dict);
-
-        return ret;
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
+    return ret;
 }
 
-int32_t
-gf_cli_snapshot (call_frame_t *frame, xlator_t *this,
-                 void *data)
-{
-        gf_cli_req        req           = {{0,}};
-        dict_t           *options       = NULL;
-        int               ret           = -1;
-        int               tmp_ret       = -1;
-        cli_local_t      *local         = NULL;
-        char             *err_str       = NULL;
-        int              type           =  -1;
-
-        if (!frame || !this || !data)
-                goto out;
+static int
+gf_cli_barrier_volume(call_frame_t *frame, xlator_t *this, void *data)
+{
+    gf_cli_req req = {{
+        0,
+    }};
+    dict_t *options = NULL;
+    int ret = -1;
 
-        if (!frame->local)
-                goto out;
+    options = data;
 
-        local = frame->local;
+    ret = cli_to_glusterd(&req, frame, gf_cli_barrier_volume_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, options,
+                          GLUSTER_CLI_BARRIER_VOLUME, this, cli_rpc_prog, NULL);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
 
-        options = data;
+    GF_FREE(req.dict.dict_val);
+    return ret;
+}
 
-        ret = dict_get_int32 (local->dict, "type", &type);
+static int32_t
+gf_cli_get_vol_opt_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                       void *myframe)
+{
+    gf_cli_rsp rsp = {
+        0,
+    };
+    int ret = -1;
+    dict_t *dict = NULL;
+    char *key = NULL;
+    char *value = NULL;
+    char msg[1024] = {
+        0,
+    };
+    int i = 0;
+    char dict_key[50] = {
+        0,
+    };
+
+    GF_ASSERT(myframe);
+
+    if (-1 == req->rpc_status)
+        goto out;
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
+    gf_log("cli", GF_LOG_DEBUG, "Received response to get volume option");
+
+    if (rsp.op_ret) {
+        if (strcmp(rsp.op_errstr, ""))
+            snprintf(msg, sizeof(msg), "volume get option: failed: %s",
+                     rsp.op_errstr);
+        else
+            snprintf(msg, sizeof(msg), "volume get option: failed");
 
-        if (GF_SNAP_OPTION_TYPE_STATUS == type &&
-            global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_snap_status_begin (local, 0, 0, NULL);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Error creating xml "
-                                "output");
-                        goto out;
-                }
+        if (global_state->mode & GLUSTER_MODE_XML) {
+            ret = cli_xml_output_str("volGetopts", msg, rsp.op_ret,
+                                     rsp.op_errno, rsp.op_errstr);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+            }
+        } else {
+            cli_err("%s", msg);
         }
-
-
-        ret = cli_to_glusterd (&req, frame, gf_cli_snapshot_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, options,
-                               GLUSTER_CLI_SNAP, this, cli_rpc_prog,
-                               NULL);
+        ret = rsp.op_ret;
+        goto out_nolog;
+    }
+    dict = dict_new();
+
+    if (!dict) {
+        ret = -1;
+        goto out;
+    }
+
+    ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &dict);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, DICT_UNSERIALIZE_FAIL);
+        goto out;
+    }
+
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_vol_getopts(dict, rsp.op_ret, rsp.op_errno,
+                                         rsp.op_errstr);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "cli_to_glusterd for "
-                        "snapshot failed");
-                goto out;
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+            ret = 0;
+        }
+        goto out;
+    }
+
+    ret = dict_get_str_sizen(dict, "warning", &value);
+    if (!ret) {
+        cli_out("%s", value);
+    }
+
+    ret = dict_get_int32_sizen(dict, "count", &count);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Failed to retrieve count from the dictionary");
+        goto out;
+    }
+
+    if (count <= 0) {
+        gf_log("cli", GF_LOG_ERROR, "Value of count :%d is invalid", count);
+        ret = -1;
+        goto out;
+    }
+
+    cli_out("%-40s%-40s", "Option", "Value");
+    cli_out("%-40s%-40s", "------", "-----");
+    for (i = 1; i <= count; i++) {
+        ret = snprintf(dict_key, sizeof dict_key, "key%d", i);
+        ret = dict_get_strn(dict, dict_key, ret, &key);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to retrieve %s from the dictionary", dict_key);
+            goto out;
         }
-
-        if (GF_SNAP_OPTION_TYPE_STATUS == type) {
-                ret = gf_cli_snapshot_for_status (frame, this, data);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "cli to glusterd "
-                                "for snapshot status command failed");
-                        goto out;
-                }
-
-                if (global_state->mode & GLUSTER_MODE_XML) {
-                        ret = cli_xml_output_snap_status_end (local);
-                        if (ret) {
-                                gf_log ("cli", GF_LOG_ERROR, "Error creating "
-                                        "xml output");
-                                goto out;
-                        }
-                }
+        ret = snprintf(dict_key, sizeof dict_key, "value%d", i);
+        ret = dict_get_strn(dict, dict_key, ret, &value);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to retrieve key value for %s from the dictionary",
+                   dict_key);
+            goto out;
         }
+        cli_out("%-40s%-40s", key, value);
+    }
 
-        if (GF_SNAP_OPTION_TYPE_DELETE == type) {
-                ret = gf_cli_snapshot_for_delete (frame, this, data);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "cli to glusterd "
-                                "for snapshot delete command failed");
-                        goto out;
-                }
-        }
+out:
+    if (ret) {
+        cli_out(
+            "volume get option failed. Check the cli/glusterd log "
+            "file for more details");
+    }
 
-        ret = 0;
+out_nolog:
+    if (dict)
+        dict_unref(dict);
+    cli_cmd_broadcast_response(ret);
+    gf_free_xdr_cli_rsp(rsp);
+    return ret;
+}
 
-out:
-        if (ret && local && GF_SNAP_OPTION_TYPE_STATUS == type) {
-                tmp_ret = dict_get_str (local->dict, "op_err_str", &err_str);
-                if (err_str) {
-                        cli_err ("Snapshot Status : failed: %s", err_str);
-                        dict_del (local->dict, "op_err_str");
-                } else {
-                        cli_err ("Snapshot Status : failed: %s", "Please "
-                                "check log file for details");
-                }
-        }
+static int
+gf_cli_get_vol_opt(call_frame_t *frame, xlator_t *this, void *data)
+{
+    gf_cli_req req = {{
+        0,
+    }};
+    dict_t *options = NULL;
+    int ret = -1;
+
+    options = data;
 
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    ret = cli_to_glusterd(&req, frame, gf_cli_get_vol_opt_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, options,
+                          GLUSTER_CLI_GET_VOL_OPT, this, cli_rpc_prog, NULL);
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
 
-        GF_FREE (req.dict.dict_val);
-        return ret;
+    GF_FREE(req.dict.dict_val);
+    return ret;
 }
 
-int32_t
-gf_cli_barrier_volume_cbk (struct rpc_req *req, struct iovec *iov,
-                                  int count, void *myframe)
+static int
+add_cli_cmd_timeout_to_dict(dict_t *dict)
 {
-        gf_cli_rsp                      rsp = {0,};
-        int                             ret = -1;
-        dict_t                          *dict = NULL;
-
-        if (-1 == req->rpc_status)
-                goto out;
-        ret = xdr_to_generic (*iov, &rsp,
-                              (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
-        gf_log ("cli", GF_LOG_DEBUG, "Received response to barrier");
+    int ret = 0;
 
-        if (rsp.op_ret) {
-                if (rsp.op_errstr && (strlen (rsp.op_errstr) > 1)) {
-                        cli_err ("volume barrier: command unsuccessful : %s",
-                                 rsp.op_errstr);
-                } else {
-                        cli_err ("volume barrier: command unsuccessful");
-                }
-        } else {
-                cli_out ("volume barrier: command successful");
+    if (cli_default_conn_timeout > 120) {
+        ret = dict_set_uint32(dict, "timeout", cli_default_conn_timeout);
+        if (ret) {
+            gf_log("cli", GF_LOG_INFO, "Failed to save timeout to dict");
         }
-        ret = rsp.op_ret;
+    }
+    return ret;
+}
 
+static int
+cli_to_glusterd(gf_cli_req *req, call_frame_t *frame, fop_cbk_fn_t cbkfn,
+                xdrproc_t xdrproc, dict_t *dict, int procnum, xlator_t *this,
+                rpc_clnt_prog_t *prog, struct iobref *iobref)
+{
+    int ret = 0;
+    size_t len = 0;
+    char *cmd = NULL;
+    int i = 0;
+    const char **words = NULL;
+    cli_local_t *local = NULL;
+
+    if (!this || !frame || !frame->local || !dict) {
+        ret = -1;
+        goto out;
+    }
+
+    local = frame->local;
+
+    if (!local->words) {
+        ret = -1;
+        goto out;
+    }
+
+    words = local->words;
+
+    while (words[i])
+        len += strlen(words[i++]) + 1;
+
+    cmd = GF_MALLOC(len + 1, gf_common_mt_char);
+    if (!cmd) {
+        ret = -1;
+        goto out;
+    }
+    cmd[0] = '\0';
+
+    for (i = 0; words[i]; i++) {
+        strncat(cmd, words[i], len - 1);
+        if (words[i + 1] != NULL)
+            strncat(cmd, " ", len - 1);
+    }
+
+    ret = dict_set_dynstr_sizen(dict, "cmd-str", cmd);
+    if (ret)
+        goto out;
+
+    ret = add_cli_cmd_timeout_to_dict(dict);
+
+    ret = dict_allocate_and_serialize(dict, &(req->dict).dict_val,
+                                      &(req->dict).dict_len);
+
+    if (ret < 0) {
+        gf_log(this->name, GF_LOG_DEBUG,
+               "failed to get serialized length of dict");
+        goto out;
+    }
+
+    ret = cli_cmd_submit(NULL, req, frame, prog, procnum, iobref, this, cbkfn,
+                         (xdrproc_t)xdrproc);
 out:
-        if (dict)
-                dict_unref (dict);
-        free (rsp.op_errstr);
-        free (rsp.dict.dict_val);
-        cli_cmd_broadcast_response (ret);
-        return ret;
+    return ret;
 }
 
-int
-gf_cli_barrier_volume (call_frame_t *frame, xlator_t *this, void *data)
+static int
+gf_cli_print_bitrot_scrub_status(dict_t *dict)
 {
-        gf_cli_req                      req = {{0,}};
-        dict_t                          *options = NULL;
-        int                             ret = -1;
+    int i = 1;
+    int j = 0;
+    int ret = -1;
+    int count = 0;
+    char key[64] = {
+        0,
+    };
+    char *volname = NULL;
+    char *node_name = NULL;
+    char *scrub_freq = NULL;
+    char *state_scrub = NULL;
+    char *scrub_impact = NULL;
+    char *bad_file_str = NULL;
+    char *scrub_log_file = NULL;
+    char *bitrot_log_file = NULL;
+    uint64_t scrub_files = 0;
+    uint64_t unsigned_files = 0;
+    uint64_t scrub_time = 0;
+    uint64_t days = 0;
+    uint64_t hours = 0;
+    uint64_t minutes = 0;
+    uint64_t seconds = 0;
+    char *last_scrub = NULL;
+    uint64_t error_count = 0;
+    int8_t scrub_running = 0;
+    char *scrub_state_op = NULL;
+
+    ret = dict_get_int32_sizen(dict, "count", &count);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "failed to get count value from dictionary");
+        goto out;
+    }
+
+    ret = dict_get_str_sizen(dict, "volname", &volname);
+    if (ret)
+        gf_log("cli", GF_LOG_TRACE, "failed to get volume name");
+
+    ret = dict_get_str_sizen(dict, "features.scrub", &state_scrub);
+    if (ret)
+        gf_log("cli", GF_LOG_TRACE, "failed to get scrub state value");
+
+    ret = dict_get_str_sizen(dict, "features.scrub-throttle", &scrub_impact);
+    if (ret)
+        gf_log("cli", GF_LOG_TRACE, "failed to get scrub impact value");
+
+    ret = dict_get_str_sizen(dict, "features.scrub-freq", &scrub_freq);
+    if (ret)
+        gf_log("cli", GF_LOG_TRACE, "failed to get scrub -freq value");
+
+    ret = dict_get_str_sizen(dict, "bitrot_log_file", &bitrot_log_file);
+    if (ret)
+        gf_log("cli", GF_LOG_TRACE, "failed to get bitrot log file location");
+
+    ret = dict_get_str_sizen(dict, "scrub_log_file", &scrub_log_file);
+    if (ret)
+        gf_log("cli", GF_LOG_TRACE, "failed to get scrubber log file location");
+
+    for (i = 1; i <= count; i++) {
+        snprintf(key, sizeof(key), "scrub-running-%d", i);
+        ret = dict_get_int8(dict, key, &scrub_running);
+        if (ret)
+            gf_log("cli", GF_LOG_TRACE, "failed to get scrubbed files");
+        if (scrub_running)
+            break;
+    }
 
-        if (!frame || !this || !data)
-                goto out;
+    if (scrub_running)
+        gf_asprintf(&scrub_state_op, "%s (In Progress)", state_scrub);
+    else
+        gf_asprintf(&scrub_state_op, "%s (Idle)", state_scrub);
 
-        options = data;
+    cli_out("\n%s: %s\n", "Volume name ", volname);
 
-        ret = cli_to_glusterd (&req, frame, gf_cli_barrier_volume_cbk,
-                               (xdrproc_t) xdr_gf_cli_req, options,
-                               GLUSTER_CLI_BARRIER_VOLUME, this, cli_rpc_prog,
-                               NULL);
-out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    cli_out("%s: %s\n", "State of scrub", scrub_state_op);
 
-        GF_FREE (req.dict.dict_val);
-        return ret;
-}
+    cli_out("%s: %s\n", "Scrub impact", scrub_impact);
 
-int32_t
-gf_cli_get_vol_opt_cbk (struct rpc_req *req, struct iovec *iov, int count,
-                        void *myframe)
-{
-        gf_cli_rsp   rsp  = {0,};
-        int          ret  = -1;
-        dict_t      *dict = NULL;
-        char        *key  = NULL;
-        char        *value = NULL;
-        char         msg[1024] = {0,};
-        int          i     = 0;
-        char         dict_key[50] = {0,};
+    cli_out("%s: %s\n", "Scrub frequency", scrub_freq);
 
-        if (-1 == req->rpc_status)
-                goto out;
-        ret = xdr_to_generic (*iov, &rsp,
-                              (xdrproc_t)xdr_gf_cli_rsp);
-        if (ret < 0) {
-                gf_log (((call_frame_t *) myframe)->this->name, GF_LOG_ERROR,
-                        "Failed to decode xdr response");
-                goto out;
-        }
-        gf_log ("cli", GF_LOG_DEBUG, "Received response to get volume option");
+    cli_out("%s: %s\n", "Bitrot error log location", bitrot_log_file);
 
-        if (rsp.op_ret) {
-                if (strcmp (rsp.op_errstr, ""))
-                        snprintf (msg, sizeof (msg), "volume get option: "
-                                  "failed: %s", rsp.op_errstr);
-                else
-                        snprintf (msg, sizeof (msg), "volume get option: "
-                                  "failed");
-
-                if (global_state->mode & GLUSTER_MODE_XML) {
-                        ret = cli_xml_output_str ("volGetopts", msg, rsp.op_ret,
-                                          rsp.op_errno, rsp.op_errstr);
-                        if (ret) {
-                                gf_log ("cli", GF_LOG_ERROR,
-                                        "Error outputting to xml");
-                        }
-                } else {
-                        cli_err ("%s", msg);
-                }
-                ret = 0; /* setting ret to 0 ensures we do not end up displaying
-                            double error messages */
-                goto out;
-        }
-        dict = dict_new ();
+    cli_out("%s: %s\n", "Scrubber error log location", scrub_log_file);
 
-        if (!dict) {
-                ret = -1;
-                goto out;
-        }
+    for (i = 1; i <= count; i++) {
+        /* Reset the variables to prevent carryover of values */
+        node_name = NULL;
+        last_scrub = NULL;
+        scrub_time = 0;
+        error_count = 0;
+        scrub_files = 0;
+        unsigned_files = 0;
 
-        ret = dict_unserialize (rsp.dict.dict_val, rsp.dict.dict_len, &dict);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR,
-                        "Failed rsp_dict unserialization");
-                goto out;
-        }
+        snprintf(key, sizeof(key), "node-name-%d", i);
+        ret = dict_get_str(dict, key, &node_name);
+        if (ret)
+            gf_log("cli", GF_LOG_TRACE, "failed to get node-name");
 
-        if (global_state->mode & GLUSTER_MODE_XML) {
-                ret = cli_xml_output_vol_getopts (dict, rsp.op_ret,
-                                                  rsp.op_errno,
-                                                  rsp.op_errstr);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "xml output generation "
-                                "failed");
-                        ret = 0;
-                }
-                goto out;
-        }
+        snprintf(key, sizeof(key), "scrubbed-files-%d", i);
+        ret = dict_get_uint64(dict, key, &scrub_files);
+        if (ret)
+            gf_log("cli", GF_LOG_TRACE, "failed to get scrubbed files");
 
-        ret = dict_get_int32 (dict, "count", &count);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to retrieve count "
-                        "from the dictionary");
-                goto out;
-        }
-        if (count <= 0) {
-                gf_log ("cli", GF_LOG_ERROR, "Value of count :%d is "
-                        "invalid", count);
-                ret = -1;
-                goto out;
-        }
+        snprintf(key, sizeof(key), "unsigned-files-%d", i);
+        ret = dict_get_uint64(dict, key, &unsigned_files);
+        if (ret)
+            gf_log("cli", GF_LOG_TRACE, "failed to get unsigned files");
 
-        cli_out ("%-40s%-40s", "Option", "Value");
-        cli_out ("%-40s%-40s", "------", "-----");
-        for (i=1; i<=count; i++) {
-                sprintf (dict_key, "key%d", i);
-                ret = dict_get_str (dict, dict_key, &key);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to"
-                                " retrieve %s from the "
-                                "dictionary", dict_key);
-                        goto out;
-                }
-                sprintf (dict_key, "value%d", i);
-                ret = dict_get_str (dict, dict_key, &value);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to "
-                                "retrieve key value for %s from"
-                                "the dictionary", dict_key);
-                        goto out;
-                }
-                cli_out ("%-40s%-40s", key, value);
-        }
+        snprintf(key, sizeof(key), "scrub-duration-%d", i);
+        ret = dict_get_uint64(dict, key, &scrub_time);
+        if (ret)
+            gf_log("cli", GF_LOG_TRACE, "failed to get last scrub duration");
 
-out:
-        if (ret) {
-                cli_out ("volume get option failed. Check the cli/glusterd log "
-                         "file for more details");
-        }
-        if (dict)
-                dict_unref (dict);
-        free (rsp.op_errstr);
-        free (rsp.dict.dict_val);
-        cli_cmd_broadcast_response (ret);
-        return ret;
-}
+        snprintf(key, sizeof(key), "last-scrub-time-%d", i);
+        ret = dict_get_str(dict, key, &last_scrub);
+        if (ret)
+            gf_log("cli", GF_LOG_TRACE, "failed to get last scrub time");
+        snprintf(key, sizeof(key), "error-count-%d", i);
+        ret = dict_get_uint64(dict, key, &error_count);
+        if (ret)
+            gf_log("cli", GF_LOG_TRACE, "failed to get error count");
 
-int
-gf_cli_get_vol_opt (call_frame_t *frame, xlator_t *this, void *data)
-{
-        gf_cli_req  req     = {{0,}};
-        dict_t     *options = NULL;
-        int         ret     = -1;
+        cli_out("\n%s\n",
+                "=========================================================");
 
-        if (!frame || !this || !data)
-                goto out;
+        cli_out("%s: %s\n", "Node", node_name);
 
-        options = data;
+        cli_out("%s: %" PRIu64 "\n", "Number of Scrubbed files", scrub_files);
 
-        ret = cli_to_glusterd (&req, frame, gf_cli_get_vol_opt_cbk,
-                               (xdrproc_t)xdr_gf_cli_req, options,
-                               GLUSTER_CLI_GET_VOL_OPT, this, cli_rpc_prog,
-                               NULL);
-out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
+        cli_out("%s: %" PRIu64 "\n", "Number of Skipped files", unsigned_files);
+
+        if ((!last_scrub) || !strcmp(last_scrub, ""))
+            cli_out("%s: %s\n", "Last completed scrub time",
+                    "Scrubber pending to complete.");
+        else
+            cli_out("%s: %s\n", "Last completed scrub time", last_scrub);
+
+        /* Printing last scrub duration time in human readable form*/
+        seconds = scrub_time % 60;
+        minutes = (scrub_time / 60) % 60;
+        hours = (scrub_time / 3600) % 24;
+        days = scrub_time / 86400;
+        cli_out("%s: %" PRIu64 ":%" PRIu64 ":%" PRIu64 ":%" PRIu64 "\n",
+                "Duration of last scrub (D:M:H:M:S)", days, hours, minutes,
+                seconds);
+
+        cli_out("%s: %" PRIu64 "\n", "Error count", error_count);
+
+        if (error_count) {
+            cli_out("%s:\n", "Corrupted object's [GFID]");
+            /* Printing list of bad file's (Corrupted object's)*/
+            for (j = 0; j < error_count; j++) {
+                snprintf(key, sizeof(key), "quarantine-%d-%d", j, i);
+                ret = dict_get_str(dict, key, &bad_file_str);
+                if (!ret) {
+                    cli_out("%s\n", bad_file_str);
+                }
+            }
+        }
+    }
+    cli_out("%s\n",
+            "=========================================================");
 
-        GF_FREE (req.dict.dict_val);
-        return ret;
+out:
+    GF_FREE(scrub_state_op);
+    return 0;
 }
 
-int
-cli_to_glusterd (gf_cli_req *req, call_frame_t *frame,
-                 fop_cbk_fn_t cbkfn, xdrproc_t xdrproc, dict_t *dict,
-                 int procnum, xlator_t *this, rpc_clnt_prog_t *prog,
-                 struct iobref *iobref)
-{
-        int                ret = 0;
-        size_t             len = 0;
-        char               *cmd = NULL;
-        int                i = 0;
-        const char         **words = NULL;
-        cli_local_t        *local = NULL;
-
-        if (!this || !frame || !dict) {
-                ret = -1;
-                goto out;
-        }
+static int
+gf_cli_bitrot_cbk(struct rpc_req *req, struct iovec *iov, int count,
+                  void *myframe)
+{
+    int ret = -1;
+    int type = 0;
+    gf_cli_rsp rsp = {
+        0,
+    };
+    dict_t *dict = NULL;
+    char *scrub_cmd = NULL;
+    char *volname = NULL;
+    char *cmd_str = NULL;
+    char *cmd_op = NULL;
+
+    GF_ASSERT(myframe);
+
+    if (req->rpc_status == -1) {
+        goto out;
+    }
+
+    ret = xdr_to_generic(*iov, &rsp, (xdrproc_t)xdr_gf_cli_rsp);
+    if (ret < 0) {
+        gf_log(((call_frame_t *)myframe)->this->name, GF_LOG_ERROR,
+               XDR_DECODE_FAIL);
+        goto out;
+    }
+
+    if (rsp.op_ret) {
+        ret = -1;
+        if (global_state->mode & GLUSTER_MODE_XML)
+            goto xml_output;
 
-        if (!frame->local) {
-                ret = -1;
-                goto out;
-        }
+        if (strcmp(rsp.op_errstr, ""))
+            cli_err("Bitrot command failed : %s", rsp.op_errstr);
+        else
+            cli_err("Bitrot command : failed");
 
-        local = frame->local;
+        goto out;
+    }
 
-        if (!local->words) {
-                ret = -1;
-                goto out;
-        }
-
-        words = local->words;
+    if (rsp.dict.dict_len) {
+        /* Unserialize the dictionary */
+        dict = dict_new();
 
-        while (words[i])
-                len += strlen (words[i++]) + 1;
+        if (!dict) {
+            ret = -1;
+            goto out;
+        }
 
-        cmd = GF_CALLOC (1, len, gf_common_mt_char);
+        ret = dict_unserialize(rsp.dict.dict_val, rsp.dict.dict_len, &dict);
 
-        if (!cmd) {
-                ret = -1;
-                goto out;
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, DICT_UNSERIALIZE_FAIL);
+            goto out;
         }
+    }
 
-        for (i = 0; words[i]; i++) {
-                strncat (cmd, words[i], strlen (words[i]));
-                if (words[i+1] != NULL)
-                        strncat (cmd, " ", strlen (" "));
-        }
+    gf_log("cli", GF_LOG_DEBUG, "Received resp to bit rot command");
 
-        cmd [len - 1] = '\0';
+    ret = dict_get_int32_sizen(dict, "type", &type);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get command type");
+        goto out;
+    }
+
+    if ((type == GF_BITROT_CMD_SCRUB_STATUS) &&
+        !(global_state->mode & GLUSTER_MODE_XML)) {
+        ret = gf_cli_print_bitrot_scrub_status(dict);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, "Failed to print bitrot scrub status");
+        }
+        goto out;
+    }
+
+    /* Ignoring the error, as using dict val for cli output only */
+    ret = dict_get_str_sizen(dict, "volname", &volname);
+    if (ret)
+        gf_log("cli", GF_LOG_TRACE, "failed to get volume name");
+
+    ret = dict_get_str_sizen(dict, "scrub-value", &scrub_cmd);
+    if (ret)
+        gf_log("cli", GF_LOG_TRACE, "Failed to get scrub command");
+
+    ret = dict_get_str_sizen(dict, "cmd-str", &cmd_str);
+    if (ret)
+        gf_log("cli", GF_LOG_TRACE, "failed to get command string");
+
+    if (cmd_str)
+        cmd_op = strrchr(cmd_str, ' ') + 1;
+
+    switch (type) {
+        case GF_BITROT_OPTION_TYPE_ENABLE:
+            cli_out("volume bitrot: success bitrot enabled for volume %s",
+                    volname);
+            ret = 0;
+            goto out;
+        case GF_BITROT_OPTION_TYPE_DISABLE:
+            cli_out("volume bitrot: success bitrot disabled for volume %s",
+                    volname);
+            ret = 0;
+            goto out;
+        case GF_BITROT_CMD_SCRUB_ONDEMAND:
+            cli_out("volume bitrot: scrubber started ondemand for volume %s",
+                    volname);
+            ret = 0;
+            goto out;
+        case GF_BITROT_OPTION_TYPE_SCRUB:
+            if (!strncmp("pause", scrub_cmd, sizeof("pause")))
+                cli_out("volume bitrot: scrubber paused for volume %s",
+                        volname);
+            if (!strncmp("resume", scrub_cmd, sizeof("resume")))
+                cli_out("volume bitrot: scrubber resumed for volume %s",
+                        volname);
+            ret = 0;
+            goto out;
+        case GF_BITROT_OPTION_TYPE_SCRUB_FREQ:
+            cli_out(
+                "volume bitrot: scrub-frequency is set to %s "
+                "successfully for volume %s",
+                cmd_op, volname);
+            ret = 0;
+            goto out;
+        case GF_BITROT_OPTION_TYPE_SCRUB_THROTTLE:
+            cli_out(
+                "volume bitrot: scrub-throttle is set to %s "
+                "successfully for volume %s",
+                cmd_op, volname);
+            ret = 0;
+            goto out;
+    }
 
-        ret = dict_set_dynstr (dict, "cmd-str", cmd);
+xml_output:
+    if (global_state->mode & GLUSTER_MODE_XML) {
+        ret = cli_xml_output_vol_profile(dict, rsp.op_ret, rsp.op_errno,
+                                         rsp.op_errstr);
         if (ret)
-                goto out;
+            gf_log("cli", GF_LOG_ERROR, XML_ERROR);
+        goto out;
+    }
 
-        ret = dict_allocate_and_serialize (dict, &(req->dict).dict_val,
-                                           &(req->dict).dict_len);
+    if (!rsp.op_ret)
+        cli_out("volume bitrot: success");
 
-        if (ret < 0) {
-                gf_log (this->name, GF_LOG_DEBUG,
-                        "failed to get serialized length of dict");
-                goto out;
-        }
+    ret = rsp.op_ret;
 
-        ret = cli_cmd_submit (NULL, req, frame, prog, procnum, iobref, this,
-                              cbkfn, (xdrproc_t) xdrproc);
+out:
+    if (dict)
+        dict_unref(dict);
+
+    gf_free_xdr_cli_rsp(rsp);
+    cli_cmd_broadcast_response(ret);
+    return ret;
+}
+
+static int32_t
+gf_cli_bitrot(call_frame_t *frame, xlator_t *this, void *data)
+{
+    gf_cli_req req = {{
+        0,
+    }};
+    dict_t *options = NULL;
+    int ret = -1;
+
+    options = data;
+
+    ret = cli_to_glusterd(&req, frame, gf_cli_bitrot_cbk,
+                          (xdrproc_t)xdr_gf_cli_req, options,
+                          GLUSTER_CLI_BITROT, this, cli_rpc_prog, NULL);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "cli_to_glusterd for bitrot failed");
+        goto out;
+    }
 
 out:
-        return ret;
-
-}
-
-struct rpc_clnt_procedure gluster_cli_actors[GLUSTER_CLI_MAXVALUE] = {
-        [GLUSTER_CLI_NULL]             = {"NULL", NULL },
-        [GLUSTER_CLI_PROBE]            = {"PROBE_QUERY", gf_cli_probe},
-        [GLUSTER_CLI_DEPROBE]          = {"DEPROBE_QUERY", gf_cli_deprobe},
-        [GLUSTER_CLI_LIST_FRIENDS]     = {"LIST_FRIENDS", gf_cli_list_friends},
-        [GLUSTER_CLI_UUID_RESET]       = {"UUID_RESET", gf_cli3_1_uuid_reset},
-        [GLUSTER_CLI_UUID_GET]       = {"UUID_GET", gf_cli3_1_uuid_get},
-        [GLUSTER_CLI_CREATE_VOLUME]    = {"CREATE_VOLUME", gf_cli_create_volume},
-        [GLUSTER_CLI_DELETE_VOLUME]    = {"DELETE_VOLUME", gf_cli_delete_volume},
-        [GLUSTER_CLI_START_VOLUME]     = {"START_VOLUME", gf_cli_start_volume},
-        [GLUSTER_CLI_STOP_VOLUME]      = {"STOP_VOLUME", gf_cli_stop_volume},
-        [GLUSTER_CLI_RENAME_VOLUME]    = {"RENAME_VOLUME", gf_cli_rename_volume},
-        [GLUSTER_CLI_DEFRAG_VOLUME]    = {"DEFRAG_VOLUME", gf_cli_defrag_volume},
-        [GLUSTER_CLI_GET_VOLUME]       = {"GET_VOLUME", gf_cli_get_volume},
-        [GLUSTER_CLI_GET_NEXT_VOLUME]  = {"GET_NEXT_VOLUME", gf_cli_get_next_volume},
-        [GLUSTER_CLI_SET_VOLUME]       = {"SET_VOLUME", gf_cli_set_volume},
-        [GLUSTER_CLI_ADD_BRICK]        = {"ADD_BRICK", gf_cli_add_brick},
-        [GLUSTER_CLI_REMOVE_BRICK]     = {"REMOVE_BRICK", gf_cli_remove_brick},
-        [GLUSTER_CLI_REPLACE_BRICK]    = {"REPLACE_BRICK", gf_cli_replace_brick},
-        [GLUSTER_CLI_LOG_ROTATE]       = {"LOG ROTATE", gf_cli_log_rotate},
-        [GLUSTER_CLI_GETSPEC]          = {"GETSPEC", gf_cli_getspec},
-        [GLUSTER_CLI_PMAP_PORTBYBRICK] = {"PMAP PORTBYBRICK", gf_cli_pmap_b2p},
-        [GLUSTER_CLI_SYNC_VOLUME]      = {"SYNC_VOLUME", gf_cli_sync_volume},
-        [GLUSTER_CLI_RESET_VOLUME]     = {"RESET_VOLUME", gf_cli_reset_volume},
-        [GLUSTER_CLI_FSM_LOG]          = {"FSM_LOG", gf_cli_fsm_log},
-        [GLUSTER_CLI_GSYNC_SET]        = {"GSYNC_SET", gf_cli_gsync_set},
-        [GLUSTER_CLI_PROFILE_VOLUME]   = {"PROFILE_VOLUME", gf_cli_profile_volume},
-        [GLUSTER_CLI_QUOTA]            = {"QUOTA", gf_cli_quota},
-        [GLUSTER_CLI_TOP_VOLUME]       = {"TOP_VOLUME", gf_cli_top_volume},
-        [GLUSTER_CLI_GETWD]            = {"GETWD", gf_cli_getwd},
-        [GLUSTER_CLI_STATUS_VOLUME]    = {"STATUS_VOLUME", gf_cli_status_volume},
-        [GLUSTER_CLI_STATUS_ALL]       = {"STATUS_ALL", gf_cli_status_volume_all},
-        [GLUSTER_CLI_MOUNT]            = {"MOUNT", gf_cli_mount},
-        [GLUSTER_CLI_UMOUNT]           = {"UMOUNT", gf_cli_umount},
-        [GLUSTER_CLI_HEAL_VOLUME]      = {"HEAL_VOLUME", gf_cli_heal_volume},
-        [GLUSTER_CLI_STATEDUMP_VOLUME] = {"STATEDUMP_VOLUME", gf_cli_statedump_volume},
-        [GLUSTER_CLI_LIST_VOLUME]      = {"LIST_VOLUME", gf_cli_list_volume},
-        [GLUSTER_CLI_CLRLOCKS_VOLUME]  = {"CLEARLOCKS_VOLUME", gf_cli_clearlocks_volume},
-        [GLUSTER_CLI_COPY_FILE]        = {"COPY_FILE", gf_cli_copy_file},
-        [GLUSTER_CLI_SYS_EXEC]         = {"SYS_EXEC", gf_cli_sys_exec},
-        [GLUSTER_CLI_SNAP]             = {"SNAP", gf_cli_snapshot},
-        [GLUSTER_CLI_BARRIER_VOLUME]   = {"BARRIER VOLUME", gf_cli_barrier_volume},
-        [GLUSTER_CLI_GET_VOL_OPT]      = {"GET_VOL_OPT", gf_cli_get_vol_opt},
+    gf_log("cli", GF_LOG_DEBUG, RETURNING, ret);
+
+    GF_FREE(req.dict.dict_val);
+
+    return ret;
+}
+
+static struct rpc_clnt_procedure gluster_cli_actors[GLUSTER_CLI_MAXVALUE] = {
+    [GLUSTER_CLI_NULL] = {"NULL", NULL},
+    [GLUSTER_CLI_PROBE] = {"PROBE_QUERY", gf_cli_probe},
+    [GLUSTER_CLI_DEPROBE] = {"DEPROBE_QUERY", gf_cli_deprobe},
+    [GLUSTER_CLI_LIST_FRIENDS] = {"LIST_FRIENDS", gf_cli_list_friends},
+    [GLUSTER_CLI_UUID_RESET] = {"UUID_RESET", gf_cli3_1_uuid_reset},
+    [GLUSTER_CLI_UUID_GET] = {"UUID_GET", gf_cli3_1_uuid_get},
+    [GLUSTER_CLI_CREATE_VOLUME] = {"CREATE_VOLUME", gf_cli_create_volume},
+    [GLUSTER_CLI_DELETE_VOLUME] = {"DELETE_VOLUME", gf_cli_delete_volume},
+    [GLUSTER_CLI_START_VOLUME] = {"START_VOLUME", gf_cli_start_volume},
+    [GLUSTER_CLI_STOP_VOLUME] = {"STOP_VOLUME", gf_cli_stop_volume},
+    [GLUSTER_CLI_RENAME_VOLUME] = {"RENAME_VOLUME", gf_cli_rename_volume},
+    [GLUSTER_CLI_DEFRAG_VOLUME] = {"DEFRAG_VOLUME", gf_cli_defrag_volume},
+    [GLUSTER_CLI_GET_VOLUME] = {"GET_VOLUME", gf_cli_get_volume},
+    [GLUSTER_CLI_GET_NEXT_VOLUME] = {"GET_NEXT_VOLUME", gf_cli_get_next_volume},
+    [GLUSTER_CLI_SET_VOLUME] = {"SET_VOLUME", gf_cli_set_volume},
+    [GLUSTER_CLI_ADD_BRICK] = {"ADD_BRICK", gf_cli_add_brick},
+    [GLUSTER_CLI_REMOVE_BRICK] = {"REMOVE_BRICK", gf_cli_remove_brick},
+    [GLUSTER_CLI_REPLACE_BRICK] = {"REPLACE_BRICK", gf_cli_replace_brick},
+    [GLUSTER_CLI_LOG_ROTATE] = {"LOG ROTATE", gf_cli_log_rotate},
+    [GLUSTER_CLI_GETSPEC] = {"GETSPEC", gf_cli_getspec},
+    [GLUSTER_CLI_PMAP_PORTBYBRICK] = {"PMAP PORTBYBRICK", gf_cli_pmap_b2p},
+    [GLUSTER_CLI_SYNC_VOLUME] = {"SYNC_VOLUME", gf_cli_sync_volume},
+    [GLUSTER_CLI_RESET_VOLUME] = {"RESET_VOLUME", gf_cli_reset_volume},
+    [GLUSTER_CLI_FSM_LOG] = {"FSM_LOG", gf_cli_fsm_log},
+    [GLUSTER_CLI_GSYNC_SET] = {"GSYNC_SET", gf_cli_gsync_set},
+    [GLUSTER_CLI_PROFILE_VOLUME] = {"PROFILE_VOLUME", gf_cli_profile_volume},
+    [GLUSTER_CLI_QUOTA] = {"QUOTA", gf_cli_quota},
+    [GLUSTER_CLI_TOP_VOLUME] = {"TOP_VOLUME", gf_cli_top_volume},
+    [GLUSTER_CLI_GETWD] = {"GETWD", gf_cli_getwd},
+    [GLUSTER_CLI_STATUS_VOLUME] = {"STATUS_VOLUME", gf_cli_status_volume},
+    [GLUSTER_CLI_STATUS_ALL] = {"STATUS_ALL", gf_cli_status_volume_all},
+    [GLUSTER_CLI_MOUNT] = {"MOUNT", gf_cli_mount},
+    [GLUSTER_CLI_UMOUNT] = {"UMOUNT", gf_cli_umount},
+    [GLUSTER_CLI_HEAL_VOLUME] = {"HEAL_VOLUME", gf_cli_heal_volume},
+    [GLUSTER_CLI_STATEDUMP_VOLUME] = {"STATEDUMP_VOLUME",
+                                      gf_cli_statedump_volume},
+    [GLUSTER_CLI_LIST_VOLUME] = {"LIST_VOLUME", gf_cli_list_volume},
+    [GLUSTER_CLI_CLRLOCKS_VOLUME] = {"CLEARLOCKS_VOLUME",
+                                     gf_cli_clearlocks_volume},
+    [GLUSTER_CLI_COPY_FILE] = {"COPY_FILE", gf_cli_copy_file},
+    [GLUSTER_CLI_SYS_EXEC] = {"SYS_EXEC", gf_cli_sys_exec},
+    [GLUSTER_CLI_SNAP] = {"SNAP", gf_cli_snapshot},
+    [GLUSTER_CLI_BARRIER_VOLUME] = {"BARRIER VOLUME", gf_cli_barrier_volume},
+    [GLUSTER_CLI_GET_VOL_OPT] = {"GET_VOL_OPT", gf_cli_get_vol_opt},
+    [GLUSTER_CLI_BITROT] = {"BITROT", gf_cli_bitrot},
+    [GLUSTER_CLI_GET_STATE] = {"GET_STATE", gf_cli_get_state},
+    [GLUSTER_CLI_RESET_BRICK] = {"RESET_BRICK", gf_cli_reset_brick},
+    [GLUSTER_CLI_GANESHA] = {"GANESHA", gf_cli_ganesha},
 };
 
 struct rpc_clnt_program cli_prog = {
-        .progname  = "Gluster CLI",
-        .prognum   = GLUSTER_CLI_PROGRAM,
-        .progver   = GLUSTER_CLI_VERSION,
-        .numproc   = GLUSTER_CLI_MAXVALUE,
-        .proctable = gluster_cli_actors,
+    .progname = "Gluster CLI",
+    .prognum = GLUSTER_CLI_PROGRAM,
+    .progver = GLUSTER_CLI_VERSION,
+    .numproc = GLUSTER_CLI_MAXVALUE,
+    .proctable = gluster_cli_actors,
 };
 
-struct rpc_clnt_procedure cli_quotad_procs[GF_AGGREGATOR_MAXVALUE] = {
-        [GF_AGGREGATOR_NULL]     = {"NULL", NULL},
-        [GF_AGGREGATOR_LOOKUP]   = {"LOOKUP", NULL},
-        [GF_AGGREGATOR_GETLIMIT]   = {"GETLIMIT", cli_quotad_getlimit},
+static struct rpc_clnt_procedure cli_quotad_procs[GF_AGGREGATOR_MAXVALUE] = {
+    [GF_AGGREGATOR_NULL] = {"NULL", NULL},
+    [GF_AGGREGATOR_LOOKUP] = {"LOOKUP", NULL},
+    [GF_AGGREGATOR_GETLIMIT] = {"GETLIMIT", cli_quotad_getlimit},
 };
 
 struct rpc_clnt_program cli_quotad_clnt = {
-        .progname  = "CLI Quotad client",
-        .prognum   = GLUSTER_AGGREGATOR_PROGRAM,
-        .progver   = GLUSTER_AGGREGATOR_VERSION,
-        .numproc   = GF_AGGREGATOR_MAXVALUE,
-        .proctable = cli_quotad_procs,
+    .progname = "CLI Quotad client",
+    .prognum = GLUSTER_AGGREGATOR_PROGRAM,
+    .progver = GLUSTER_AGGREGATOR_VERSION,
+    .numproc = GF_AGGREGATOR_MAXVALUE,
+    .proctable = cli_quotad_procs,
 };
diff --git a/cli/src/cli-xml-output.c b/cli/src/cli-xml-output.c
index aacce140946..069de75801c 100644
--- a/cli/src/cli-xml-output.c
+++ b/cli/src/cli-xml-output.c
@@ -10,15 +10,12 @@
 #include <stdlib.h>
 #include "cli.h"
 #include "cli1-xdr.h"
-#include "run.h"
-#include "compat.h"
-#include "syscall.h"
+#include <glusterfs/run.h>
+#include <glusterfs/compat.h>
+#include <glusterfs/syscall.h>
+#include <glusterfs/upcall-utils.h>
 
-
-enum gf_task_types {
-    GF_TASK_TYPE_REBALANCE,
-    GF_TASK_TYPE_REMOVE_BRICK
-};
+enum gf_task_types { GF_TASK_TYPE_REBALANCE, GF_TASK_TYPE_REMOVE_BRICK };
 
 /*
  * IMPORTANT NOTE:
@@ -42,4154 +39,3796 @@ enum gf_task_types {
  *      }
  *      #endif
  *
- *  Following the above formate ensures that all xml related code is compliled
+ *  Following the above format ensures that all xml related code is compiled
  *  only when libxml2 is present, and also keeps the rest of the codebase free
  *  of #if (HAVE_LIB_XML)
  */
 
-
 #if (HAVE_LIB_XML)
 
 #include <libxml/encoding.h>
 #include <libxml/xmlwriter.h>
 
-#define XML_RET_CHECK_AND_GOTO(ret, label)      do {            \
-                if (ret < 0) {                                  \
-                        ret = -1;                               \
-                        goto label;                             \
-                }                                               \
-                else                                            \
-                        ret = 0;                                \
-        }while (0)                                              \
+#define XML_RET_CHECK_AND_GOTO(ret, label)                                     \
+    do {                                                                       \
+        if (ret < 0) {                                                         \
+            ret = -1;                                                          \
+            goto label;                                                        \
+        } else                                                                 \
+            ret = 0;                                                           \
+    } while (0)
 
 int
-cli_begin_xml_output (xmlTextWriterPtr *writer, xmlDocPtr *doc)
+cli_begin_xml_output(xmlTextWriterPtr *writer, xmlDocPtr *doc)
 {
-        int             ret = -1;
+    int ret = -1;
 
-        *writer = xmlNewTextWriterDoc (doc, 0);
-        if (writer == NULL) {
-                ret = -1;
-                goto out;
-        }
+    *writer = xmlNewTextWriterDoc(doc, 0);
+    if (*writer == NULL) {
+        ret = -1;
+        goto out;
+    }
 
-        ret = xmlTextWriterStartDocument (*writer, "1.0", "UTF-8", "yes");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = xmlTextWriterStartDocument(*writer, "1.0", "UTF-8", "yes");
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        /* <cliOutput> */
-        ret = xmlTextWriterStartElement (*writer, (xmlChar *)"cliOutput");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* <cliOutput> */
+    ret = xmlTextWriterStartElement(*writer, (xmlChar *)"cliOutput");
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 
 int
-cli_end_xml_output (xmlTextWriterPtr writer, xmlDocPtr doc)
+cli_end_xml_output(xmlTextWriterPtr writer, xmlDocPtr doc)
 {
-        int             ret = -1;
+    int ret = -1;
 
-        /* </cliOutput> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </cliOutput> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = xmlTextWriterEndDocument (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = xmlTextWriterEndDocument(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
+    /* Dump xml document to stdout and pretty format it */
+    xmlSaveFormatFileEnc("-", doc, "UTF-8", 1);
 
-        /* Dump xml document to stdout and pretty format it */
-        xmlSaveFormatFileEnc ("-", doc, "UTF-8", 1);
-
-        xmlFreeTextWriter (writer);
-        xmlFreeDoc (doc);
+    xmlFreeTextWriter(writer);
+    xmlFreeDoc(doc);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 
 int
-cli_xml_output_common (xmlTextWriterPtr writer, int op_ret, int op_errno,
-                       char *op_errstr)
+cli_xml_output_common(xmlTextWriterPtr writer, int op_ret, int op_errno,
+                      char *op_errstr)
 {
-        int             ret = -1;
+    int ret = -1;
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"opRet", "%d",
+                                          op_ret);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"opRet",
-                                               "%d", op_ret);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"opErrno", "%d",
+                                          op_errno);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"opErrno",
-                                               "%d", op_errno);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    if (op_errstr)
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"opErrstr",
+                                              "%s", op_errstr);
+    else
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"opErrstr",
+                                              "%s", "");
 
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"opErrstr",
-                                                "%s", op_errstr);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 #endif
 
 int
-cli_xml_output_str (char *op, char *str, int op_ret, int op_errno,
-                    char *op_errstr)
+cli_xml_output_str(char *op, char *str, int op_ret, int op_errno,
+                   char *op_errstr)
 {
 #if (HAVE_LIB_XML)
-        int                     ret = -1;
-        xmlTextWriterPtr        writer = NULL;
-        xmlDocPtr               doc = NULL;
+    int ret = -1;
+    xmlTextWriterPtr writer = NULL;
+    xmlDocPtr doc = NULL;
 
-        ret = cli_begin_xml_output (&writer, &doc);
-        if (ret)
-                goto out;
+    ret = cli_begin_xml_output(&writer, &doc);
+    if (ret)
+        goto out;
 
-        ret = cli_xml_output_common (writer, op_ret, op_errno, op_errstr);
-        if (ret)
-                goto out;
+    ret = cli_xml_output_common(writer, op_ret, op_errno, op_errstr);
+    if (ret)
+        goto out;
 
-        if (op) {
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"cliOp",
-                                                       "%s", op);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+    if (op) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"cliOp", "%s",
+                                              op);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        if (str) {
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"output",
-                                                       "%s", str);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+    if (str) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"output", "%s",
+                                              str);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        ret = cli_end_xml_output (writer, doc);
+    ret = cli_end_xml_output(writer, doc);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif
 }
 
 #if (HAVE_LIB_XML)
 int
-cli_xml_output_data_pair (dict_t *this, char *key, data_t *value,
-                          void *data)
+cli_xml_output_data_pair(dict_t *this, char *key, data_t *value, void *data)
 {
-        int                     ret = -1;
-        xmlTextWriterPtr        *writer = NULL;
+    int ret = -1;
+    xmlTextWriterPtr *writer = NULL;
 
-        writer = (xmlTextWriterPtr *)data;
+    writer = (xmlTextWriterPtr *)data;
 
-        ret = xmlTextWriterWriteFormatElement (*writer, (xmlChar *)key,
-                                               "%s", value->data);
+    ret = xmlTextWriterWriteFormatElement(*writer, (xmlChar *)key, "%s",
+                                          value->data);
 
-        return ret;
+    XML_RET_CHECK_AND_GOTO(ret, out);
+out:
+    return ret;
 }
 #endif
 
 int
-cli_xml_output_dict ( char *op, dict_t *dict, int op_ret, int op_errno,
-                      char *op_errstr)
+cli_xml_output_dict(char *op, dict_t *dict, int op_ret, int op_errno,
+                    char *op_errstr)
 {
 #if (HAVE_LIB_XML)
-        int                     ret = -1;
-        xmlTextWriterPtr        writer = NULL;
-        xmlDocPtr               doc = NULL;
+    int ret = -1;
+    xmlTextWriterPtr writer = NULL;
+    xmlDocPtr doc = NULL;
 
-        ret = cli_begin_xml_output (&writer, &doc);
-        if (ret)
-                goto out;
+    ret = cli_begin_xml_output(&writer, &doc);
+    if (ret)
+        goto out;
 
-        ret = cli_xml_output_common (writer, op_ret, op_errno, op_errstr);
-        if (ret)
-                goto out;
+    ret = cli_xml_output_common(writer, op_ret, op_errno, op_errstr);
+    if (ret)
+        goto out;
 
-        /* <"op"> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)op);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* <"op"> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)op);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        if (dict)
-                dict_foreach (dict, cli_xml_output_data_pair, &writer);
+    if (dict)
+        dict_foreach(dict, cli_xml_output_data_pair, &writer);
 
-        /* </"op"> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </"op"> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = cli_end_xml_output (writer, doc);
+    ret = cli_end_xml_output(writer, doc);
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif
 }
 
 #if (HAVE_LIB_XML)
 int
-cli_xml_output_vol_status_common (xmlTextWriterPtr writer, dict_t *dict,
-                                  int   brick_index, int *online,
-                                  gf_boolean_t *node_present)
+cli_xml_output_vol_status_common(xmlTextWriterPtr writer, dict_t *dict,
+                                 int brick_index, int *online,
+                                 gf_boolean_t *node_present)
 {
-        int             ret = -1;
-        char            *hostname = NULL;
-        char            *path = NULL;
-        char            *uuid = NULL;
-        int             port = 0;
-        int             status = 0;
-        int             pid = 0;
-        char            key[1024] = {0,};
-
-        snprintf (key, sizeof (key), "brick%d.hostname", brick_index);
-        ret = dict_get_str (dict, key, &hostname);
-        if (ret) {
-                *node_present = _gf_false;
-                goto out;
+    int ret = -1;
+    char *hostname = NULL;
+    char *path = NULL;
+    char *uuid = NULL;
+    int port = 0;
+    int rdma_port = 0;
+    int status = 0;
+    int pid = 0;
+    char key[1024] = {
+        0,
+    };
+
+    snprintf(key, sizeof(key), "brick%d.hostname", brick_index);
+    ret = dict_get_str(dict, key, &hostname);
+    if (ret) {
+        *node_present = _gf_false;
+        goto out;
+    }
+    *node_present = _gf_true;
+
+    /* <node>
+     * will be closed in the calling function cli_xml_output_vol_status()*/
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"node");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"hostname", "%s",
+                                          hostname);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "brick%d.path", brick_index);
+    ret = dict_get_str(dict, key, &path);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"path", "%s",
+                                          path);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "brick%d.peerid", brick_index);
+    ret = dict_get_str(dict, key, &uuid);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"peerid", "%s",
+                                          uuid);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "brick%d.status", brick_index);
+    ret = dict_get_int32(dict, key, &status);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"status", "%d",
+                                          status);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+    *online = status;
+
+    snprintf(key, sizeof(key), "brick%d.port", brick_index);
+    ret = dict_get_int32(dict, key, &port);
+    if (ret)
+        goto out;
+
+    snprintf(key, sizeof(key), "brick%d.rdma_port", brick_index);
+    ret = dict_get_int32(dict, key, &rdma_port);
+
+    /* If the process is either offline or doesn't provide a port (shd)
+     * port = "N/A"
+     * else print the port number of the process.
+     */
+
+    /*
+     * Tag 'port' can be removed once console management is started
+     * to support new tag ports.
+     */
+
+    if (*online == 1 && port != 0)
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"port", "%d",
+                                              port);
+    else
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"port", "%s",
+                                              "N/A");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"ports");
+    if (*online == 1 && (port != 0 || rdma_port != 0)) {
+        if (port) {
+            ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"tcp",
+                                                  "%d", port);
+        } else {
+            ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"tcp",
+                                                  "%s", "N/A");
         }
-        *node_present = _gf_true;
-
-        /* <node>
-         * will be closed in the calling function cli_xml_output_vol_status()*/
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"node");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"hostname",
-                                               "%s", hostname);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.path", brick_index);
-        ret = dict_get_str (dict, key, &path);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"path",
-                                               "%s", path);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.peerid", brick_index);
-        ret = dict_get_str (dict, key, &uuid);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"peerid",
-                                               "%s", uuid);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.status", brick_index);
-        ret = dict_get_int32 (dict, key, &status);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"status",
-                                               "%d", status);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-        *online = status;
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.port", brick_index);
-        ret = dict_get_int32 (dict, key, &port);
-        if (ret)
-                goto out;
-
-        /* If the process is either offline or doesn't provide a port (shd)
-         * port = "N/A"
-         * else print the port number of the process.
-         */
-
-        if (*online == 1 && port != 0)
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"port",
-                                                       "%d", port);
-        else
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"port",
-                                                       "%s", "N/A");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.pid", brick_index);
-        ret = dict_get_int32 (dict, key, &pid);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"pid",
-                                               "%d", pid);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
-}
-
-int
-cli_xml_output_vol_status_detail (xmlTextWriterPtr writer, dict_t *dict,
-                                  int brick_index)
-{
-        int             ret = -1;
-        uint64_t        size_total = 0;
-        uint64_t        size_free = 0;
-        char            *device = NULL;
-        uint64_t        block_size = 0;
-        char            *mnt_options = NULL;
-        char            *fs_name = NULL;
-        char            *inode_size = NULL;
-        uint64_t        inodes_total = 0;
-        uint64_t        inodes_free = 0;
-        char            key[1024] = {0,};
-
-        snprintf (key, sizeof (key), "brick%d.total", brick_index);
-        ret = dict_get_uint64 (dict, key, &size_total);
-        if (!ret)
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"sizeTotal",
-                                                       "%"PRIu64, size_total);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.free", brick_index);
-        ret = dict_get_uint64 (dict, key, &size_free);
-        if (!ret)
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"sizeFree",
-                                                       "%"PRIu64, size_free);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.device", brick_index);
-        ret = dict_get_str (dict, key, &device);
-        if (!ret)
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"device",
-                                                       "%s", device);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.block_size", brick_index);
-        ret = dict_get_uint64 (dict, key, &block_size);
-        if (!ret)
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"blockSize",
-                                                       "%"PRIu64, block_size);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.mnt_options", brick_index);
-        ret = dict_get_str (dict, key, &mnt_options);
-        if (!ret)
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"mntOptions",
-                                                       "%s", mnt_options);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.fs_name", brick_index);
-        ret = dict_get_str (dict, key, &fs_name);
-        if (!ret)
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"fsName",
-                                                       "%s", fs_name);
-
-        /* inode details are only available for ext 2/3/4 & xfs */
-        if (!fs_name || !IS_EXT_FS(fs_name) || strcmp (fs_name, "xfs")) {
-                        ret = 0;
-                        goto out;
+        XML_RET_CHECK_AND_GOTO(ret, out);
+        if (rdma_port) {
+            ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"rdma",
+                                                  "%d", rdma_port);
+        } else {
+            ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"rdma",
+                                                  "%s", "N/A");
         }
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    } else {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"tcp", "%s",
+                                              "N/A");
+        XML_RET_CHECK_AND_GOTO(ret, out);
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"rdma", "%s",
+                                              "N/A");
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
+
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.inode_size", brick_index);
-        ret = dict_get_str (dict, key, &inode_size);
-        if (!ret)
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"inodeSize",
-                                                       "%s", fs_name);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.total_inodes", brick_index);
-        ret = dict_get_uint64 (dict, key, &inodes_total);
-        if (!ret)
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"inodesTotal",
-                                                       "%"PRIu64, inodes_total);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.free_inodes", brick_index);
-        ret = dict_get_uint64 (dict, key, &inodes_free);
-        if (!ret)
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"inodesFree",
-                                                       "%"PRIu64, inodes_free);
+    snprintf(key, sizeof(key), "brick%d.pid", brick_index);
+    ret = dict_get_int32(dict, key, &pid);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"pid", "%d", pid);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 
 int
-cli_xml_output_vol_status_mempool (xmlTextWriterPtr writer, dict_t *dict,
-                                   char *prefix)
+cli_xml_output_vol_status_detail(xmlTextWriterPtr writer, dict_t *dict,
+                                 int brick_index)
 {
-        int             ret = -1;
-        int             mempool_count = 0;
-        char            *name = NULL;
-        int             hotcount = 0;
-        int             coldcount = 0;
-        uint64_t        paddedsizeof = 0;
-        uint64_t        alloccount = 0;
-        int             maxalloc = 0;
-        char            key[1024] = {0,};
-        int             i = 0;
-
-        /* <mempool> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"mempool");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        snprintf (key, sizeof (key), "%s.mempool-count", prefix);
-        ret = dict_get_int32 (dict, key, &mempool_count);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"count",
-                                               "%d", mempool_count);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        for (i = 0; i < mempool_count; i++) {
-                /* <pool> */
-                ret = xmlTextWriterStartElement (writer, (xmlChar *)"pool");
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.pool%d.name", prefix, i);
-                ret = dict_get_str (dict, key, &name);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"name",
-                                                       "%s", name);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.pool%d.hotcount", prefix, i);
-                ret = dict_get_int32 (dict, key, &hotcount);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"hotCount",
-                                                       "%d", hotcount);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.pool%d.coldcount", prefix, i);
-                ret = dict_get_int32 (dict, key, &coldcount);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"coldCount",
-                                                       "%d", coldcount);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.pool%d.paddedsizeof",
-                          prefix, i);
-                ret = dict_get_uint64 (dict, key, &paddedsizeof);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement
-                        (writer, (xmlChar *)"padddedSizeOf", "%"PRIu64,
-                         paddedsizeof);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.pool%d.alloccount", prefix, i);
-                ret = dict_get_uint64 (dict, key, &alloccount);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"allocCount",
-                                                       "%"PRIu64, alloccount);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.pool%d.max_alloc", prefix, i);
-                ret = dict_get_int32 (dict, key, &maxalloc);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"maxAlloc",
-                                                       "%d", maxalloc);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.pool%d.pool-misses", prefix, i);
-                ret = dict_get_uint64 (dict, key, &alloccount);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"poolMisses",
-                                                       "%"PRIu64, alloccount);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.pool%d.max-stdalloc", prefix, i);
-                ret = dict_get_int32 (dict, key, &maxalloc);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"maxStdAlloc",
-                                                       "%d", maxalloc);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-
-                /* </pool> */
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
-
-        /* </mempool> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    int ret = -1;
+    uint64_t size_total = 0;
+    uint64_t size_free = 0;
+    char *device = NULL;
+    uint64_t block_size = 0;
+    char *mnt_options = NULL;
+    char *fs_name = NULL;
+    char *inode_size = NULL;
+    uint64_t inodes_total = 0;
+    uint64_t inodes_free = 0;
+    char key[1024] = {
+        0,
+    };
+
+    snprintf(key, sizeof(key), "brick%d.total", brick_index);
+    ret = dict_get_uint64(dict, key, &size_total);
+    if (!ret) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"sizeTotal",
+                                              "%" PRIu64, size_total);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
+    snprintf(key, sizeof(key), "brick%d.free", brick_index);
+    ret = dict_get_uint64(dict, key, &size_free);
+    if (!ret) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"sizeFree",
+                                              "%" PRIu64, size_free);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
+    snprintf(key, sizeof(key), "brick%d.device", brick_index);
+    ret = dict_get_str(dict, key, &device);
+    if (!ret) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"device", "%s",
+                                              device);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
+    snprintf(key, sizeof(key), "brick%d.block_size", brick_index);
+    ret = dict_get_uint64(dict, key, &block_size);
+    if (!ret) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"blockSize",
+                                              "%" PRIu64, block_size);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
+    snprintf(key, sizeof(key), "brick%d.mnt_options", brick_index);
+    ret = dict_get_str(dict, key, &mnt_options);
+    if (!ret) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"mntOptions",
+                                              "%s", mnt_options);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
+    snprintf(key, sizeof(key), "brick%d.fs_name", brick_index);
+    ret = dict_get_str(dict, key, &fs_name);
+    if (!ret) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"fsName", "%s",
+                                              fs_name);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
+    snprintf(key, sizeof(key), "brick%d.inode_size", brick_index);
+    ret = dict_get_str(dict, key, &inode_size);
+    if (!ret) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"inodeSize",
+                                              "%s", fs_name);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
+    snprintf(key, sizeof(key), "brick%d.total_inodes", brick_index);
+    ret = dict_get_uint64(dict, key, &inodes_total);
+    if (!ret) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"inodesTotal",
+                                              "%" PRIu64, inodes_total);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
+    snprintf(key, sizeof(key), "brick%d.free_inodes", brick_index);
+    ret = dict_get_uint64(dict, key, &inodes_free);
+    if (!ret) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"inodesFree",
+                                              "%" PRIu64, inodes_free);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    } else {
+        ret = 0;
+    }
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 
 int
-cli_xml_output_vol_status_mem (xmlTextWriterPtr writer, dict_t *dict,
-                               int brick_index)
+cli_xml_output_vol_status_mempool(xmlTextWriterPtr writer, dict_t *dict,
+                                  char *prefix)
 {
-        int             ret = -1;
-        int             arena = 0;
-        int             ordblks = 0;
-        int             smblks = 0;
-        int             hblks = 0;
-        int             hblkhd = 0;
-        int             usmblks = 0;
-        int             fsmblks = 0;
-        int             uordblks = 0;
-        int             fordblks = 0;
-        int             keepcost = 0;
-        char            key[1024] = {0,};
-
-        /* <memStatus> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"memStatus");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        /* <mallinfo> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"mallinfo");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        snprintf (key, sizeof (key), "brick%d.mallinfo.arena", brick_index);
-        ret = dict_get_int32 (dict, key, &arena);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"arena",
-                                               "%d", arena);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.mallinfo.ordblks", brick_index);
-        ret = dict_get_int32 (dict, key, &ordblks);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"ordblks",
-                                               "%d", ordblks);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    int ret = -1;
+    int mempool_count = 0;
+    char *name = NULL;
+    int hotcount = 0;
+    int coldcount = 0;
+    uint64_t paddedsizeof = 0;
+    uint64_t alloccount = 0;
+    int maxalloc = 0;
+    char key[1024] = {
+        0,
+    };
+    int i = 0;
+
+    /* <mempool> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"mempool");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%s.mempool-count", prefix);
+    ret = dict_get_int32(dict, key, &mempool_count);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"count", "%d",
+                                          mempool_count);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    for (i = 0; i < mempool_count; i++) {
+        /* <pool> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"pool");
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.mallinfo.smblks", brick_index);
-        ret = dict_get_int32 (dict, key, &smblks);
+        snprintf(key, sizeof(key), "%s.pool%d.name", prefix, i);
+        ret = dict_get_str(dict, key, &name);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"smblks",
-                                               "%d", smblks);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"name", "%s",
+                                              name);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.mallinfo.hblks", brick_index);
-        ret = dict_get_int32 (dict, key, &hblks);
+        snprintf(key, sizeof(key), "%s.pool%d.hotcount", prefix, i);
+        ret = dict_get_int32(dict, key, &hotcount);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"hblks",
-                                               "%d", hblks);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"hotCount",
+                                              "%d", hotcount);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.mallinfo.hblkhd", brick_index);
-        ret = dict_get_int32 (dict, key, &hblkhd);
+        snprintf(key, sizeof(key), "%s.pool%d.coldcount", prefix, i);
+        ret = dict_get_int32(dict, key, &coldcount);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"hblkhd",
-                                               "%d", hblkhd);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"coldCount",
+                                              "%d", coldcount);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.mallinfo.usmblks", brick_index);
-        ret = dict_get_int32 (dict, key, &usmblks);
+        snprintf(key, sizeof(key), "%s.pool%d.paddedsizeof", prefix, i);
+        ret = dict_get_uint64(dict, key, &paddedsizeof);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"usmblks",
-                                               "%d", usmblks);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(
+            writer, (xmlChar *)"padddedSizeOf", "%" PRIu64, paddedsizeof);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.mallinfo.fsmblks", brick_index);
-        ret = dict_get_int32 (dict, key, &fsmblks);
+        snprintf(key, sizeof(key), "%s.pool%d.alloccount", prefix, i);
+        ret = dict_get_uint64(dict, key, &alloccount);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"fsmblks",
-                                               "%d", fsmblks);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"allocCount",
+                                              "%" PRIu64, alloccount);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.mallinfo.uordblks", brick_index);
-        ret = dict_get_int32 (dict, key, &uordblks);
+        snprintf(key, sizeof(key), "%s.pool%d.max_alloc", prefix, i);
+        ret = dict_get_int32(dict, key, &maxalloc);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"uordblks",
-                                               "%d", uordblks);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"maxAlloc",
+                                              "%d", maxalloc);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.mallinfo.fordblks", brick_index);
-        ret = dict_get_int32 (dict, key, &fordblks);
+        snprintf(key, sizeof(key), "%s.pool%d.pool-misses", prefix, i);
+        ret = dict_get_uint64(dict, key, &alloccount);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"fordblks",
-                                               "%d", fordblks);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"poolMisses",
+                                              "%" PRIu64, alloccount);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d.mallinfo.keepcost", brick_index);
-        ret = dict_get_int32 (dict, key, &keepcost);
+        snprintf(key, sizeof(key), "%s.pool%d.max-stdalloc", prefix, i);
+        ret = dict_get_int32(dict, key, &maxalloc);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"keepcost",
-                                               "%d", keepcost);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        /* </mallinfo> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"maxStdAlloc",
+                                              "%d", maxalloc);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "brick%d", brick_index);
-        ret = cli_xml_output_vol_status_mempool (writer, dict, key);
-        if (ret)
-                goto out;
+        /* </pool> */
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        /* </memStatus> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </mempool> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 
 int
-cli_xml_output_vol_status_clients (xmlTextWriterPtr writer, dict_t *dict,
-                                   int brick_index)
+cli_xml_output_vol_status_mem(xmlTextWriterPtr writer, dict_t *dict,
+                              int brick_index)
 {
-        int             ret = -1;
-        int             client_count = 0;
-        char            *hostname = NULL;
-        uint64_t        bytes_read = 0;
-        uint64_t        bytes_write = 0;
-        char            key[1024] = {0,};
-        int             i = 0;
-
-        /* <clientsStatus> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"clientsStatus");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        snprintf (key, sizeof (key), "brick%d.clientcount", brick_index);
-        ret = dict_get_int32 (dict, key, &client_count);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer,
-                                               (xmlChar *)"clientCount",
-                                               "%d", client_count);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        for (i = 0; i < client_count; i++) {
-                /* <client> */
-                ret = xmlTextWriterStartElement (writer, (xmlChar *)"client");
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.client%d.hostname",
-                          brick_index, i);
-                ret = dict_get_str (dict, key, &hostname);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"hostname",
-                                                       "%s", hostname);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.client%d.bytesread",
-                          brick_index, i);
-                ret = dict_get_uint64 (dict, key, &bytes_read);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"bytesRead",
-                                                       "%"PRIu64, bytes_read);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.client%d.byteswrite",
-                          brick_index, i);
-                ret = dict_get_uint64 (dict, key, &bytes_write);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"bytesWrite",
-                                                       "%"PRIu64, bytes_write);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                /* </client> */
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+    int ret = -1;
+    int arena = 0;
+    int ordblks = 0;
+    int smblks = 0;
+    int hblks = 0;
+    int hblkhd = 0;
+    int usmblks = 0;
+    int fsmblks = 0;
+    int uordblks = 0;
+    int fordblks = 0;
+    int keepcost = 0;
+    char key[1024] = {
+        0,
+    };
+
+    /* <memStatus> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"memStatus");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* <mallinfo> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"mallinfo");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "brick%d.mallinfo.arena", brick_index);
+    ret = dict_get_int32(dict, key, &arena);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"arena", "%d",
+                                          arena);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "brick%d.mallinfo.ordblks", brick_index);
+    ret = dict_get_int32(dict, key, &ordblks);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"ordblks", "%d",
+                                          ordblks);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "brick%d.mallinfo.smblks", brick_index);
+    ret = dict_get_int32(dict, key, &smblks);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"smblks", "%d",
+                                          smblks);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "brick%d.mallinfo.hblks", brick_index);
+    ret = dict_get_int32(dict, key, &hblks);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"hblks", "%d",
+                                          hblks);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "brick%d.mallinfo.hblkhd", brick_index);
+    ret = dict_get_int32(dict, key, &hblkhd);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"hblkhd", "%d",
+                                          hblkhd);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "brick%d.mallinfo.usmblks", brick_index);
+    ret = dict_get_int32(dict, key, &usmblks);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"usmblks", "%d",
+                                          usmblks);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "brick%d.mallinfo.fsmblks", brick_index);
+    ret = dict_get_int32(dict, key, &fsmblks);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"fsmblks", "%d",
+                                          fsmblks);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "brick%d.mallinfo.uordblks", brick_index);
+    ret = dict_get_int32(dict, key, &uordblks);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"uordblks", "%d",
+                                          uordblks);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "brick%d.mallinfo.fordblks", brick_index);
+    ret = dict_get_int32(dict, key, &fordblks);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"fordblks", "%d",
+                                          fordblks);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "brick%d.mallinfo.keepcost", brick_index);
+    ret = dict_get_int32(dict, key, &keepcost);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"keepcost", "%d",
+                                          keepcost);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* </mallinfo> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "brick%d", brick_index);
+    ret = cli_xml_output_vol_status_mempool(writer, dict, key);
+    if (ret)
+        goto out;
+
+    /* </memStatus> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        /* </clientsStatus> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 
 int
-cli_xml_output_vol_status_inode_entry (xmlTextWriterPtr writer, dict_t *dict,
-                                       char *prefix)
+cli_xml_output_vol_status_clients(xmlTextWriterPtr writer, dict_t *dict,
+                                  int brick_index)
 {
-        int             ret = -1;
-        char            *gfid = NULL;
-        uint64_t        nlookup = 0;
-        uint32_t        ref = 0;
-        int             ia_type = 0;
-        char            key[1024] = {0,};
-
-        /* <inode> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"inode");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        snprintf (key, sizeof (key), "%s.gfid", prefix);
-        ret = dict_get_str (dict, key, &gfid);
+    int ret = -1;
+    int client_count = 0;
+    char *hostname = NULL;
+    uint64_t bytes_read = 0;
+    uint64_t bytes_write = 0;
+    uint32_t opversion = 0;
+    char key[1024] = {
+        0,
+    };
+    int i = 0;
+
+    /* <clientsStatus> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"clientsStatus");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "brick%d.clientcount", brick_index);
+    ret = dict_get_int32(dict, key, &client_count);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"clientCount",
+                                          "%d", client_count);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    for (i = 0; i < client_count; i++) {
+        /* <client> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"client");
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        snprintf(key, sizeof(key), "brick%d.client%d.hostname", brick_index, i);
+        ret = dict_get_str(dict, key, &hostname);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"gfid",
-                                               "%s", gfid);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"hostname",
+                                              "%s", hostname);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        memset (key,0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.nlookup", prefix);
-        ret = dict_get_uint64 (dict, key, &nlookup);
+        snprintf(key, sizeof(key), "brick%d.client%d.bytesread", brick_index,
+                 i);
+        ret = dict_get_uint64(dict, key, &bytes_read);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"nLookup",
-                                               "%"PRIu64, nlookup);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"bytesRead",
+                                              "%" PRIu64, bytes_read);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        memset (key,0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.ref", prefix);
-        ret = dict_get_uint32 (dict, key, &ref);
+        snprintf(key, sizeof(key), "brick%d.client%d.byteswrite", brick_index,
+                 i);
+        ret = dict_get_uint64(dict, key, &bytes_write);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"ref",
-                                               "%"PRIu32, ref);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"bytesWrite",
+                                              "%" PRIu64, bytes_write);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        memset (key,0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.ia_type", prefix);
-        ret = dict_get_int32 (dict, key, &ia_type);
+        snprintf(key, sizeof(key), "brick%d.client%d.opversion", brick_index,
+                 i);
+        ret = dict_get_uint32(dict, key, &opversion);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"iaType",
-                                               "%d", ia_type);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"opVersion",
+                                              "%" PRIu32, opversion);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        /* </inode> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+        /* </client> */
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
+    /* </clientsStatus> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 
 int
-cli_xml_output_vol_status_itable (xmlTextWriterPtr writer, dict_t *dict,
-                                  char *prefix)
+cli_xml_output_vol_status_inode_entry(xmlTextWriterPtr writer, dict_t *dict,
+                                      char *prefix)
 {
-        int             ret = -1;
-        uint32_t        active_size = 0;
-        uint32_t        lru_size = 0;
-        uint32_t        purge_size = 0;
-        char            key[1024] = {0,};
-        int             i = 0;
-
-        snprintf (key, sizeof (key), "%s.active_size", prefix);
-        ret = dict_get_uint32 (dict, key, &active_size);
-        if (ret)
+    int ret = -1;
+    char *gfid = NULL;
+    uint64_t nlookup = 0;
+    uint32_t ref = 0;
+    int ia_type = 0;
+    char key[1024] = {
+        0,
+    };
+
+    /* <inode> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"inode");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%s.gfid", prefix);
+    ret = dict_get_str(dict, key, &gfid);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"gfid", "%s",
+                                          gfid);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%s.nlookup", prefix);
+    ret = dict_get_uint64(dict, key, &nlookup);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"nLookup",
+                                          "%" PRIu64, nlookup);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%s.ref", prefix);
+    ret = dict_get_uint32(dict, key, &ref);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"ref", "%" PRIu32,
+                                          ref);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%s.ia_type", prefix);
+    ret = dict_get_int32(dict, key, &ia_type);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"iaType", "%d",
+                                          ia_type);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* </inode> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+out:
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
+}
+
+int
+cli_xml_output_vol_status_itable(xmlTextWriterPtr writer, dict_t *dict,
+                                 char *prefix)
+{
+    int ret = -1;
+    uint32_t active_size = 0;
+    uint32_t lru_size = 0;
+    uint32_t purge_size = 0;
+    char key[1024] = {
+        0,
+    };
+    int i = 0;
+
+    snprintf(key, sizeof(key), "%s.active_size", prefix);
+    ret = dict_get_uint32(dict, key, &active_size);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"activeSize",
+                                          "%" PRIu32, active_size);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+    if (active_size != 0) {
+        /* <active> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"active");
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        for (i = 0; i < active_size; i++) {
+            snprintf(key, sizeof(key), "%s.active%d", prefix, i);
+            ret = cli_xml_output_vol_status_inode_entry(writer, dict, key);
+            if (ret)
                 goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"activeSize",
-                                               "%"PRIu32, active_size);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-        if (active_size != 0) {
-                /* <active> */
-                ret = xmlTextWriterStartElement (writer, (xmlChar *)"active");
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                for (i = 0; i < active_size; i++) {
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key), "%s.active%d", prefix, i);
-                        ret = cli_xml_output_vol_status_inode_entry
-                                (writer, dict, key);
-                        if (ret)
-                                goto out;
-                }
-                /* </active> */
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
         }
+        /* </active> */
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
+
+    snprintf(key, sizeof(key), "%s.lru_size", prefix);
+    ret = dict_get_uint32(dict, key, &lru_size);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"lruSize",
+                                          "%" PRIu32, lru_size);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+    if (lru_size != 0) {
+        /* <lru> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"lru");
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.lru_size", prefix);
-        ret = dict_get_uint32 (dict, key, &lru_size);
-        if (ret)
+        for (i = 0; i < lru_size; i++) {
+            snprintf(key, sizeof(key), "%s.lru%d", prefix, i);
+            ret = cli_xml_output_vol_status_inode_entry(writer, dict, key);
+            if (ret)
                 goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"lruSize",
-                                               "%"PRIu32, lru_size);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-        if (lru_size != 0) {
-                /* <lru> */
-                ret = xmlTextWriterStartElement (writer, (xmlChar *)"lru");
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                for (i = 0; i < lru_size; i++) {
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key), "%s.lru%d", prefix, i);
-                        ret = cli_xml_output_vol_status_inode_entry
-                                (writer, dict, key);
-                        if (ret)
-                                goto out;
-                }
-                /* </lru> */
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
         }
+        /* </lru> */
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
+
+    snprintf(key, sizeof(key), "%s.purge_size", prefix);
+    ret = dict_get_uint32(dict, key, &purge_size);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"purgeSize",
+                                          "%" PRIu32, purge_size);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+    if (purge_size != 0) {
+        /* <purge> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"purge");
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.purge_size", prefix);
-        ret = dict_get_uint32 (dict, key, &purge_size);
-        if (ret)
+        for (i = 0; i < purge_size; i++) {
+            snprintf(key, sizeof(key), "%s.purge%d", prefix, i);
+            ret = cli_xml_output_vol_status_inode_entry(writer, dict, key);
+            if (ret)
                 goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"purgeSize",
-                                               "%"PRIu32, purge_size);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-        if (purge_size != 0) {
-                /* <purge> */
-                ret = xmlTextWriterStartElement (writer, (xmlChar *)"purge");
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                for (i = 0; i < purge_size; i++) {
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key), "%s.purge%d", prefix, i);
-                        ret = cli_xml_output_vol_status_inode_entry
-                                (writer, dict, key);
-                        if (ret)
-                                goto out;
-                }
-                /* </purge> */
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
         }
+        /* </purge> */
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 
 int
-cli_xml_output_vol_status_inode (xmlTextWriterPtr writer, dict_t *dict,
-                                 int brick_index)
+cli_xml_output_vol_status_inode(xmlTextWriterPtr writer, dict_t *dict,
+                                int brick_index)
 {
-        int             ret = -1;
-        int             conn_count = 0;
-        char            key[1024] = {0,};
-        int             i = 0;
-
-        /* <inodeStatus> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"inodeStatus");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    int ret = -1;
+    int conn_count = 0;
+    char key[1024] = {
+        0,
+    };
+    int i = 0;
+
+    /* <inodeStatus> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"inodeStatus");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "brick%d.conncount", brick_index);
+    ret = dict_get_int32(dict, key, &conn_count);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"connections",
+                                          "%d", conn_count);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    for (i = 0; i < conn_count; i++) {
+        /* <connection> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"connection");
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        snprintf (key, sizeof (key), "brick%d.conncount", brick_index);
-        ret = dict_get_int32 (dict, key, &conn_count);
+        snprintf(key, sizeof(key), "brick%d.conn%d.itable", brick_index, i);
+        ret = cli_xml_output_vol_status_itable(writer, dict, key);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"connections",
-                                               "%d", conn_count);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        for (i = 0; i < conn_count; i++) {
-                /* <connection> */
-                ret = xmlTextWriterStartElement (writer,
-                                                 (xmlChar *)"connection");
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.conn%d.itable",
-                          brick_index, i);
-                ret = cli_xml_output_vol_status_itable (writer, dict, key);
-                if (ret)
-                        goto out;
+            goto out;
 
-                /* </connection> */
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+        /* </connection> */
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        /* </inodeStatus> */
-        ret= xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </inodeStatus> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 
 int
-cli_xml_output_vol_status_fdtable (xmlTextWriterPtr writer, dict_t *dict,
-                                   char *prefix)
+cli_xml_output_vol_status_fdtable(xmlTextWriterPtr writer, dict_t *dict,
+                                  char *prefix)
 {
-        int             ret = -1;
-        int             refcount = 0;
-        uint32_t        maxfds = 0;
-        int             firstfree = 0;
-        int             openfds = 0;
-        int             fd_pid = 0;
-        int             fd_refcount = 0;
-        int             fd_flags = 0;
-        char            key[1024] = {0,};
-        int             i = 0;
-
-        /* <fdTable> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"fdTable");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        snprintf (key, sizeof (key), "%s.refcount", prefix);
-        ret = dict_get_int32 (dict, key, &refcount);
+    int ret = -1;
+    int refcount = 0;
+    uint32_t maxfds = 0;
+    int firstfree = 0;
+    int openfds = 0;
+    int fd_pid = 0;
+    int fd_refcount = 0;
+    int fd_flags = 0;
+    char key[1024] = {
+        0,
+    };
+    int i = 0;
+
+    /* <fdTable> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"fdTable");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%s.refcount", prefix);
+    ret = dict_get_int32(dict, key, &refcount);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"refCount", "%d",
+                                          refcount);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%s.maxfds", prefix);
+    ret = dict_get_uint32(dict, key, &maxfds);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"maxFds",
+                                          "%" PRIu32, maxfds);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%s.firstfree", prefix);
+    ret = dict_get_int32(dict, key, &firstfree);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"firstFree", "%d",
+                                          firstfree);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%s.openfds", prefix);
+    ret = dict_get_int32(dict, key, &openfds);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"openFds", "%d",
+                                          openfds);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    for (i = 0; i < maxfds; i++) {
+        snprintf(key, sizeof(key), "%s.fdentry%d.pid", prefix, i);
+        ret = dict_get_int32(dict, key, &fd_pid);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"refCount",
-                                               "%d", refcount);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+            continue;
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.maxfds", prefix);
-        ret = dict_get_uint32 (dict, key, &maxfds);
+        snprintf(key, sizeof(key), "%s.fdentry%d.refcount", prefix, i);
+        ret = dict_get_int32(dict, key, &fd_refcount);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"maxFds",
-                                               "%"PRIu32, maxfds);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+            continue;
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.firstfree", prefix);
-        ret = dict_get_int32 (dict, key, &firstfree);
+        snprintf(key, sizeof(key), "%s.fdentry%d.flags", prefix, i);
+        ret = dict_get_int32(dict, key, &fd_flags);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"firstFree",
-                                               "%d", firstfree);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+            continue;
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.openfds", prefix);
-        ret = dict_get_int32 (dict, key, &openfds);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"openFds",
-                                               "%d", openfds);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        for (i = 0; i < maxfds; i++) {
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.fdentry%d.pid", prefix, i);
-                ret = dict_get_int32 (dict, key, &fd_pid);
-                if (ret)
-                        continue;
+        /* <fd> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"fd");
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.fdentry%d.refcount",
-                          prefix, i);
-                ret = dict_get_int32 (dict, key, &fd_refcount);
-                if (ret)
-                        continue;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"entry", "%d",
+                                              i + 1);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.fdentry%d.flags", prefix, i);
-                ret = dict_get_int32 (dict, key, &fd_flags);
-                if (ret)
-                        continue;
-
-                /* <fd> */
-                ret = xmlTextWriterStartElement (writer, (xmlChar *)"fd");
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"entry",
-                                                       "%d", i+1);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"pid",
-                                                       "%d", fd_pid);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"refCount",
-                                                       "%d", fd_refcount);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"flags",
-                                                       "%d", fd_flags);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                /* </fd> */
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"pid", "%d",
+                                              fd_pid);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"refCount",
+                                              "%d", fd_refcount);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"flags", "%d",
+                                              fd_flags);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        /* </fd> */
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        /* </fdTable> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </fdTable> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 
 int
-cli_xml_output_vol_status_fd (xmlTextWriterPtr writer, dict_t *dict,
-                              int brick_index)
+cli_xml_output_vol_status_fd(xmlTextWriterPtr writer, dict_t *dict,
+                             int brick_index)
 {
-        int             ret = -1;
-        int             conn_count = 0;
-        char            key[1024] = {0,};
-        int             i = 0;
-
-        /* <fdStatus> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"fdStatus");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    int ret = -1;
+    int conn_count = 0;
+    char key[1024] = {
+        0,
+    };
+    int i = 0;
+
+    /* <fdStatus> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"fdStatus");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "brick%d.conncount", brick_index);
+    ret = dict_get_int32(dict, key, &conn_count);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"connections",
+                                          "%d", conn_count);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    for (i = 0; i < conn_count; i++) {
+        /* <connection> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"connection");
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        snprintf (key, sizeof (key), "brick%d.conncount", brick_index);
-        ret = dict_get_int32 (dict, key, &conn_count);
+        snprintf(key, sizeof(key), "brick%d.conn%d.fdtable", brick_index, i);
+        ret = cli_xml_output_vol_status_fdtable(writer, dict, key);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"connections",
-                                               "%d", conn_count);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        for (i = 0; i < conn_count; i++) {
-                /* <connection> */
-                ret = xmlTextWriterStartElement (writer,
-                                                 (xmlChar *)"connection");
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.conn%d.fdtable",
-                          brick_index, i);
-                ret = cli_xml_output_vol_status_fdtable (writer, dict, key);
-                if (ret)
-                        goto out;
+            goto out;
 
-                /* </connection> */
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+        /* </connection> */
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        /* </fdStatus> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </fdStatus> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 
 int
-cli_xml_output_vol_status_callframe (xmlTextWriterPtr writer, dict_t *dict,
-                                     char *prefix)
+cli_xml_output_vol_status_callframe(xmlTextWriterPtr writer, dict_t *dict,
+                                    char *prefix)
 {
-        int             ret = -1;
-        int             ref_count = 0;
-        char            *translator = NULL;
-        int             complete = 0;
-        char            *parent = NULL;
-        char            *wind_from = NULL;
-        char            *wind_to = NULL;
-        char            *unwind_from = NULL;
-        char            *unwind_to = NULL;
-        char            key[1024] = {0,};
-
-        /* <callFrame> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"callFrame");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        snprintf (key, sizeof (key), "%s.refcount", prefix);
-        ret = dict_get_int32 (dict, key, &ref_count);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"refCount",
-                                               "%d", ref_count);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.translator", prefix);
-        ret = dict_get_str (dict, key, &translator);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"translator",
-                                               "%s", translator);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.complete", prefix);
-        ret = dict_get_int32 (dict, key, &complete);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"complete",
-                                               "%d", complete);
-        XML_RET_CHECK_AND_GOTO (ret ,out);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.parent", prefix);
-        ret = dict_get_str (dict, key, &parent);
-        if (!ret) {
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"parent",
-                                                       "%s", parent);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+    int ret = -1;
+    int ref_count = 0;
+    char *translator = NULL;
+    int complete = 0;
+    char *parent = NULL;
+    char *wind_from = NULL;
+    char *wind_to = NULL;
+    char *unwind_from = NULL;
+    char *unwind_to = NULL;
+    char key[1024] = {
+        0,
+    };
+
+    /* <callFrame> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"callFrame");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%s.refcount", prefix);
+    ret = dict_get_int32(dict, key, &ref_count);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"refCount", "%d",
+                                          ref_count);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%s.translator", prefix);
+    ret = dict_get_str(dict, key, &translator);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"translator", "%s",
+                                          translator);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%s.complete", prefix);
+    ret = dict_get_int32(dict, key, &complete);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"complete", "%d",
+                                          complete);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%s.parent", prefix);
+    ret = dict_get_str(dict, key, &parent);
+    if (!ret) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"parent", "%s",
+                                              parent);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.windfrom", prefix);
-        ret = dict_get_str (dict, key, &wind_from);
-        if (!ret) {
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"windFrom",
-                                                       "%s", wind_from);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+    snprintf(key, sizeof(key), "%s.windfrom", prefix);
+    ret = dict_get_str(dict, key, &wind_from);
+    if (!ret) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"windFrom",
+                                              "%s", wind_from);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.windto", prefix);
-        ret = dict_get_str (dict, key, &wind_to);
-        if (!ret) {
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"windTo",
-                                                       "%s", wind_to);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+    snprintf(key, sizeof(key), "%s.windto", prefix);
+    ret = dict_get_str(dict, key, &wind_to);
+    if (!ret) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"windTo", "%s",
+                                              wind_to);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.unwindfrom", prefix);
-        ret = dict_get_str (dict, key, &unwind_from);
-        if (!ret) {
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"unwindFrom",
-                                                       "%s", unwind_from);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+    snprintf(key, sizeof(key), "%s.unwindfrom", prefix);
+    ret = dict_get_str(dict, key, &unwind_from);
+    if (!ret) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"unwindFrom",
+                                              "%s", unwind_from);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.unwindto", prefix);
-        ret = dict_get_str (dict, key, &unwind_to);
-        if (!ret) {
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"unwindTo",
-                                                       "%s", unwind_to);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+    snprintf(key, sizeof(key), "%s.unwindto", prefix);
+    ret = dict_get_str(dict, key, &unwind_to);
+    if (!ret) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"unwindTo",
+                                              "%s", unwind_to);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        /* </callFrame> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </callFrame> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 
 int
-cli_xml_output_vol_status_callstack (xmlTextWriterPtr writer, dict_t *dict,
-                                     char *prefix)
+cli_xml_output_vol_status_callstack(xmlTextWriterPtr writer, dict_t *dict,
+                                    char *prefix)
 {
-        int             ret = -1;
-        int             uid = 0;
-        int             gid = 0;
-        int             pid = 0;
-        uint64_t        unique = 0;
-        int             frame_count = 0;
-        char            key[1024] = {0,};
-        int             i = 0;
-
-        /* <callStack> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"callStack");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        snprintf (key, sizeof (key), "%s.uid", prefix);
-        ret = dict_get_int32 (dict, key, &uid);
+    int ret = -1;
+    int uid = 0;
+    int gid = 0;
+    int pid = 0;
+    uint64_t unique = 0;
+    int frame_count = 0;
+    char key[1024] = {
+        0,
+    };
+    int i = 0;
+
+    /* <callStack> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"callStack");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%s.uid", prefix);
+    ret = dict_get_int32(dict, key, &uid);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"uid", "%d", uid);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%s.gid", prefix);
+    ret = dict_get_int32(dict, key, &gid);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"gid", "%d", gid);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%s.pid", prefix);
+    ret = dict_get_int32(dict, key, &pid);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"pid", "%d", pid);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%s.unique", prefix);
+    ret = dict_get_uint64(dict, key, &unique);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"unique",
+                                          "%" PRIu64, unique);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%s.count", prefix);
+    ret = dict_get_int32(dict, key, &frame_count);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"frameCount", "%d",
+                                          frame_count);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    for (i = 0; i < frame_count; i++) {
+        snprintf(key, sizeof(key), "%s.frame%d", prefix, i);
+        ret = cli_xml_output_vol_status_callframe(writer, dict, key);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"uid",
-                                               "%d", uid);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.gid", prefix);
-        ret = dict_get_int32 (dict, key, &gid);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"gid",
-                                               "%d", gid);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+            goto out;
+    }
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.pid", prefix);
-        ret = dict_get_int32 (dict, key, &pid);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"pid",
-                                               "%d", pid);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.unique", prefix);
-        ret = dict_get_uint64 (dict, key, &unique);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"unique",
-                                               "%"PRIu64, unique);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.count", prefix);
-        ret = dict_get_int32 (dict, key, &frame_count);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"frameCount",
-                                               "%d", frame_count);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        for (i = 0; i < frame_count; i++) {
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.frame%d", prefix, i);
-                ret = cli_xml_output_vol_status_callframe (writer, dict,
-                                                           key);
-                if (ret)
-                        goto out;
-        }
-
-        /* </callStack> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </callStack> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 
 int
-cli_xml_output_vol_status_callpool (xmlTextWriterPtr writer, dict_t *dict,
-                                    int brick_index)
+cli_xml_output_vol_status_callpool(xmlTextWriterPtr writer, dict_t *dict,
+                                   int brick_index)
 {
-        int             ret = -1;
-        int             call_count = 0;
-        char            key[1024] = {0,};
-        int             i = 0;
-
-        /* <callpoolStatus> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"callpoolStatus");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        snprintf (key, sizeof (key), "brick%d.callpool.count", brick_index);
-        ret = dict_get_int32 (dict, key, &call_count);
+    int ret = -1;
+    int call_count = 0;
+    char key[1024] = {
+        0,
+    };
+    int i = 0;
+
+    /* <callpoolStatus> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"callpoolStatus");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "brick%d.callpool.count", brick_index);
+    ret = dict_get_int32(dict, key, &call_count);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"count", "%d",
+                                          call_count);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    for (i = 0; i < call_count; i++) {
+        snprintf(key, sizeof(key), "brick%d.callpool.stack%d", brick_index, i);
+        ret = cli_xml_output_vol_status_callstack(writer, dict, key);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"count",
-                                               "%d", call_count);
-
-        for (i = 0; i < call_count; i++) {
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "brick%d.callpool.stack%d",
-                          brick_index, i);
-                ret = cli_xml_output_vol_status_callstack (writer, dict,
-                                                           key);
-                if (ret)
-                        goto out;
-        }
+            goto out;
+    }
 
-        /* </callpoolStatus> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </callpoolStatus> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 #endif
 
 int
-cli_xml_output_vol_status_begin (cli_local_t *local, int op_ret, int op_errno,
-                                 char *op_errstr)
+cli_xml_output_vol_status_begin(cli_local_t *local, int op_ret, int op_errno,
+                                char *op_errstr)
 {
 #if (HAVE_LIB_XML)
-        int                     ret = -1;
+    int ret = -1;
 
-        ret = cli_begin_xml_output (&(local->writer), &(local->doc));
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = cli_begin_xml_output(&(local->writer), &(local->doc));
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = cli_xml_output_common (local->writer, op_ret, op_errno,
-                                     op_errstr);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = cli_xml_output_common(local->writer, op_ret, op_errno, op_errstr);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        /* <volStatus> */
-        ret = xmlTextWriterStartElement (local->writer,
-                                         (xmlChar *) "volStatus");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* <volStatus> */
+    ret = xmlTextWriterStartElement(local->writer, (xmlChar *)"volStatus");
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        /* <volumes> */
-        ret = xmlTextWriterStartElement (local->writer, (xmlChar *)"volumes");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* <volumes> */
+    ret = xmlTextWriterStartElement(local->writer, (xmlChar *)"volumes");
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif
 }
 
 int
-cli_xml_output_vol_status_end (cli_local_t *local)
+cli_xml_output_vol_status_end(cli_local_t *local)
 {
 #if (HAVE_LIB_XML)
-        int     ret = -1;
+    int ret = -1;
 
-        /* </volumes> */
-        ret = xmlTextWriterEndElement (local->writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </volumes> */
+    ret = xmlTextWriterEndElement(local->writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        /* </volStatus> */
-        ret = xmlTextWriterEndElement (local->writer);
-        XML_RET_CHECK_AND_GOTO(ret, out);
+    /* </volStatus> */
+    ret = xmlTextWriterEndElement(local->writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = cli_end_xml_output (local->writer, local->doc);
+    ret = cli_end_xml_output(local->writer, local->doc);
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif
 }
 
 #if (HAVE_LIB_XML)
 int
-cli_xml_output_remove_brick_task_params (xmlTextWriterPtr writer, dict_t *dict,
-                                         char *prefix)
+cli_xml_output_remove_brick_task_params(xmlTextWriterPtr writer, dict_t *dict,
+                                        char *prefix)
 {
-        int             ret = -1;
-        char            key[1024] = {0,};
-        int             count = 0;
-        int             i = 0;
-        char            *brick = NULL;
-
-        /* <params> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"params");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        snprintf (key, sizeof (key), "%s.count", prefix);
-        ret = dict_get_int32 (dict, key, &count);
+    int ret = -1;
+    char key[1024] = {
+        0,
+    };
+    int count = 0;
+    int i = 0;
+    char *brick = NULL;
+
+    /* <params> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"params");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%s.count", prefix);
+    ret = dict_get_int32(dict, key, &count);
+    if (ret)
+        goto out;
+
+    for (i = 1; i <= count; i++) {
+        snprintf(key, sizeof(key), "%s.brick%d", prefix, i);
+        ret = dict_get_str(dict, key, &brick);
         if (ret)
-                goto out;
-
-        for (i = 1; i <= count; i++) {
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.brick%d", prefix, i);
-                ret = dict_get_str (dict, key, &brick);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"brick",
-                                                       "%s", brick);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-                brick = NULL;
-        }
-
-        /* </param> */
-        ret = xmlTextWriterEndElement (writer);
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"brick", "%s",
+                                              brick);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+        brick = NULL;
+    }
 
+    /* </param> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 
 int
-cli_xml_output_replace_brick_task_params (xmlTextWriterPtr writer, dict_t *dict,
-                                          char *prefix)
+cli_xml_output_vol_status_tasks(cli_local_t *local, dict_t *dict)
 {
+    int ret = -1;
+    char *task_type = NULL;
+    char *task_id_str = NULL;
+    int status = 0;
+    int tasks = 0;
+    char key[1024] = {
+        0,
+    };
+    int i = 0;
+
+    /* <tasks> */
+    ret = xmlTextWriterStartElement(local->writer, (xmlChar *)"tasks");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_int32(dict, "tasks", &tasks);
+    if (ret)
+        goto out;
+
+    for (i = 0; i < tasks; i++) {
+        /* <task> */
+        ret = xmlTextWriterStartElement(local->writer, (xmlChar *)"task");
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        int             ret = -1;
-        char            key[1024] = {0,};
-        char            *brick = NULL;
-
-        /* <params> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"params");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        snprintf (key, sizeof (key), "%s.src-brick", prefix);
-        ret = dict_get_str (dict, key, &brick);
+        snprintf(key, sizeof(key), "task%d.type", i);
+        ret = dict_get_str(dict, key, &task_type);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"srcBrick",
-                                               "%s", brick);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(local->writer, (xmlChar *)"type",
+                                              "%s", task_type);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%s.dst-brick", prefix);
-        ret = dict_get_str (dict, key, &brick);
+        snprintf(key, sizeof(key), "task%d.id", i);
+        ret = dict_get_str(dict, key, &task_id_str);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"dstBrick",
-                                               "%s", brick);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-
-        /* </param> */
-        ret = xmlTextWriterEndElement (writer);
-
-out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
-}
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(local->writer, (xmlChar *)"id",
+                                              "%s", task_id_str);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-int
-cli_xml_output_vol_status_tasks (cli_local_t *local, dict_t *dict) {
-        int                     ret = -1;
-        char                    *task_type = NULL;
-        char                    *task_id_str = NULL;
-        int                     status = 0;
-        int                     tasks = 0;
-        char                    key[1024] = {0,};
-        int                     i = 0;
-
-        /* <tasks> */
-        ret = xmlTextWriterStartElement (local->writer, (xmlChar *)"tasks");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = dict_get_int32 (dict, "tasks", &tasks);
+        snprintf(key, sizeof(key), "task%d.status", i);
+        ret = dict_get_int32(dict, key, &status);
         if (ret)
-                goto out;
-
-        for (i = 0; i < tasks; i++) {
-                /* <task> */
-                ret = xmlTextWriterStartElement (local->writer,
-                                                 (xmlChar *)"task");
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "task%d.type", i);
-                ret = dict_get_str (dict, key, &task_type);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (local->writer,
-                                                       (xmlChar *)"type",
-                                                       "%s", task_type);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "task%d.id", i);
-                ret = dict_get_str (dict, key, &task_id_str);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (local->writer,
-                                                       (xmlChar *)"id",
-                                                       "%s", task_id_str);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "task%d.status", i);
-                ret = dict_get_int32 (dict, key, &status);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (local->writer,
-                                                       (xmlChar *)"status",
-                                                       "%d", status);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                if (!strcmp (task_type, "Replace brick")) {
-                    if (status) {
-                        status = GF_DEFRAG_STATUS_COMPLETE;
-                    } else {
-                        status = GF_DEFRAG_STATUS_STARTED;
-                    }
-                }
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(
+            local->writer, (xmlChar *)"status", "%d", status);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                ret = xmlTextWriterWriteFormatElement (local->writer,
-                                                       (xmlChar *)"statusStr",
-                                                       "%s",
-                                             cli_vol_task_status_str[status]);
-
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "task%d", i);
-                if (!strcmp (task_type, "Replace brick")) {
-                        ret = cli_xml_output_replace_brick_task_params
-                                (local->writer, dict, key);
-                        if (ret)
-                                goto out;
-                } else if (!strcmp (task_type, "Remove brick")) {
-                        ret = cli_xml_output_remove_brick_task_params
-                                (local->writer, dict, key);
-                        if (ret)
-                                goto out;
-                }
+        ret = xmlTextWriterWriteFormatElement(local->writer,
+                                              (xmlChar *)"statusStr", "%s",
+                                              cli_vol_task_status_str[status]);
 
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                /* </task> */
-                ret = xmlTextWriterEndElement (local->writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        snprintf(key, sizeof(key), "task%d", i);
+        if (!strcmp(task_type, "Remove brick")) {
+            ret = cli_xml_output_remove_brick_task_params(local->writer, dict,
+                                                          key);
+            if (ret)
+                goto out;
         }
 
-        /* </tasks> */
-        ret = xmlTextWriterEndElement (local->writer);
+        /* </task> */
+        ret = xmlTextWriterEndElement(local->writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
+    /* </tasks> */
+    ret = xmlTextWriterEndElement(local->writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 
 #endif
 
 int
-cli_xml_output_vol_status_tasks_detail (cli_local_t *local, dict_t *dict)
+cli_xml_output_vol_status_tasks_detail(cli_local_t *local, dict_t *dict)
 {
 #if (HAVE_LIB_XML)
-        int    ret     = -1;
-        char  *volname = NULL;
+    int ret = -1;
+    char *volname = NULL;
 
-        /*<volume>*/
-        ret = xmlTextWriterStartElement (local->writer, (xmlChar *)"volume");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /*<volume>*/
+    ret = xmlTextWriterStartElement(local->writer, (xmlChar *)"volume");
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = dict_get_str (dict, "volname", &volname);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (local->writer,
-                                               (xmlChar *)"volName", "%s",
-                                               volname);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = dict_get_str(dict, "volname", &volname);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(local->writer, (xmlChar *)"volName",
+                                          "%s", volname);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = cli_xml_output_vol_status_tasks (local, dict);
-        if (ret)
-                goto out;
+    ret = cli_xml_output_vol_status_tasks(local, dict);
+    if (ret)
+        goto out;
 
-        /* </volume> */
-        ret = xmlTextWriterEndElement (local->writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </volume> */
+    ret = xmlTextWriterEndElement(local->writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
 out:
-        return ret;
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif
 }
 
 int
-cli_xml_output_vol_status (cli_local_t *local, dict_t *dict)
+cli_xml_output_vol_status(cli_local_t *local, dict_t *dict)
 {
 #if (HAVE_LIB_XML)
-        int                     ret = -1;
-        char                    *volname = NULL;
-        int                     brick_count = 0;
-        int                     brick_index_max = -1;
-        int                     other_count = 0;
-        int                     index_max = 0;
-        uint32_t                cmd = GF_CLI_STATUS_NONE;
-        int                     online = 0;
-        gf_boolean_t            node_present = _gf_true;
-        int                     i;
-
-
-        /* <volume> */
-        ret = xmlTextWriterStartElement (local->writer, (xmlChar *)"volume");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = dict_get_str (dict, "volname", &volname);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (local->writer,
-                                               (xmlChar *)"volName", "%s",
-                                               volname);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = dict_get_int32 (dict, "count", &brick_count);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (local->writer,
-                                               (xmlChar *)"nodeCount", "%d",
-                                               brick_count);
-        if (ret)
-                goto out;
-
-        ret = dict_get_uint32 (dict, "cmd", &cmd);
-        if (ret)
-                goto out;
-
-        ret = dict_get_int32 (dict, "brick-index-max", &brick_index_max);
-        if (ret)
-                goto out;
-        ret = dict_get_int32 (dict, "other-count", &other_count);
-        if (ret)
+    int ret = -1;
+    char *volname = NULL;
+    int brick_count = 0;
+    int brick_index_max = -1;
+    int other_count = 0;
+    int index_max = 0;
+    uint32_t cmd = GF_CLI_STATUS_NONE;
+    int online = 0;
+    gf_boolean_t node_present = _gf_true;
+    int i;
+    int type = -1;
+
+    /* <volume> */
+    ret = xmlTextWriterStartElement(local->writer, (xmlChar *)"volume");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_str(dict, "volname", &volname);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(local->writer, (xmlChar *)"volName",
+                                          "%s", volname);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_int32(dict, "count", &brick_count);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(local->writer, (xmlChar *)"nodeCount",
+                                          "%d", brick_count);
+
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_uint32(dict, "cmd", &cmd);
+    if (ret)
+        goto out;
+
+    ret = dict_get_int32(dict, "brick-index-max", &brick_index_max);
+    if (ret)
+        goto out;
+    ret = dict_get_int32(dict, "other-count", &other_count);
+    if (ret)
+        goto out;
+
+    index_max = brick_index_max + other_count;
+
+    ret = dict_get_int32(dict, "type", &type);
+    if (ret)
+        goto out;
+
+    for (i = 0; i <= index_max; i++) {
+        ret = cli_xml_output_vol_status_common(local->writer, dict, i, &online,
+                                               &node_present);
+        if (ret) {
+            if (node_present)
                 goto out;
+            else
+                continue;
+        }
 
-        index_max = brick_index_max + other_count;
+        switch (cmd & GF_CLI_STATUS_MASK) {
+            case GF_CLI_STATUS_DETAIL:
+                ret = cli_xml_output_vol_status_detail(local->writer, dict, i);
+                if (ret)
+                    goto out;
+                break;
 
-        for (i = 0; i <= index_max; i++) {
-                ret = cli_xml_output_vol_status_common (local->writer, dict, i,
-                                                        &online, &node_present);
-                if (ret) {
-                        if (node_present)
-                                goto out;
-                        else
-                                continue;
+            case GF_CLI_STATUS_MEM:
+                if (online) {
+                    ret = cli_xml_output_vol_status_mem(local->writer, dict, i);
+                    if (ret)
+                        goto out;
                 }
+                break;
 
-                switch (cmd & GF_CLI_STATUS_MASK) {
-                case GF_CLI_STATUS_DETAIL:
-                        ret = cli_xml_output_vol_status_detail (local->writer,
-                                                                dict, i);
-                        if (ret)
-                                goto out;
-                        break;
-
-                case GF_CLI_STATUS_MEM:
-                        if (online) {
-                                ret = cli_xml_output_vol_status_mem
-                                        (local->writer, dict, i);
-                                if (ret)
-                                        goto out;
-                        }
-                        break;
-
-                case GF_CLI_STATUS_CLIENTS:
-                        if (online) {
-                                ret = cli_xml_output_vol_status_clients
-                                        (local->writer, dict, i);
-                                if (ret)
-                                        goto out;
-                        }
-                        break;
-
-                case GF_CLI_STATUS_INODE:
-                        if (online) {
-                                ret = cli_xml_output_vol_status_inode
-                                        (local->writer, dict, i);
-                                if (ret)
-                                        goto out;
-                        }
-                        break;
-
-                case GF_CLI_STATUS_FD:
-                        if (online) {
-                                ret = cli_xml_output_vol_status_fd
-                                        (local->writer, dict, i);
-                                if (ret)
-                                        goto out;
-                        }
-                        break;
-
-                case GF_CLI_STATUS_CALLPOOL:
-                        if (online) {
-                                ret = cli_xml_output_vol_status_callpool
-                                        (local->writer, dict, i);
-                                if (ret)
-                                        goto out;
-                        }
-                        break;
-                default:
-                        break;
+            case GF_CLI_STATUS_CLIENTS:
+                if (online) {
+                    ret = cli_xml_output_vol_status_clients(local->writer, dict,
+                                                            i);
+                    if (ret)
+                        goto out;
+                }
+                break;
 
+            case GF_CLI_STATUS_INODE:
+                if (online) {
+                    ret = cli_xml_output_vol_status_inode(local->writer, dict,
+                                                          i);
+                    if (ret)
+                        goto out;
                 }
+                break;
 
-                /* </node>  was opened in cli_xml_output_vol_status_common()*/
-                ret = xmlTextWriterEndElement (local->writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+            case GF_CLI_STATUS_FD:
+                if (online) {
+                    ret = cli_xml_output_vol_status_fd(local->writer, dict, i);
+                    if (ret)
+                        goto out;
+                }
+                break;
 
-        /* Tasks are only present when a normal volume status call is done on a
-         * single volume or on all volumes
-         */
-        if (((cmd & GF_CLI_STATUS_MASK) == GF_CLI_STATUS_NONE) &&
-            (cmd & (GF_CLI_STATUS_VOL|GF_CLI_STATUS_ALL))) {
-                ret = cli_xml_output_vol_status_tasks (local, dict);
-                if (ret)
+            case GF_CLI_STATUS_CALLPOOL:
+                if (online) {
+                    ret = cli_xml_output_vol_status_callpool(local->writer,
+                                                             dict, i);
+                    if (ret)
                         goto out;
+                }
+                break;
+            default:
+                break;
         }
 
-        /* </volume> */
-        ret = xmlTextWriterEndElement (local->writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+        /* </node>  was opened in cli_xml_output_vol_status_common()*/
+        ret = xmlTextWriterEndElement(local->writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
+
+    /* Tasks are only present when a normal volume status call is done on a
+     * single volume or on all volumes
+     */
+    if (((cmd & GF_CLI_STATUS_MASK) == GF_CLI_STATUS_NONE) &&
+        (cmd & (GF_CLI_STATUS_VOL | GF_CLI_STATUS_ALL))) {
+        ret = cli_xml_output_vol_status_tasks(local, dict);
+        if (ret)
+            goto out;
+    }
+
+    /* </volume> */
+    ret = xmlTextWriterEndElement(local->writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif
 }
 
 #if (HAVE_LIB_XML)
 int
-cli_xml_output_vol_top_rw_perf (xmlTextWriterPtr writer, dict_t *dict,
-                                int brick_index, int member_index)
+cli_xml_output_vol_top_rw_perf(xmlTextWriterPtr writer, dict_t *dict,
+                               int brick_index, int member_index)
 {
-        int        ret = -1;
-        char      *filename = NULL;
-        uint64_t   throughput = 0;
-        long int   time_sec = 0;
-        long int   time_usec = 0;
-        char       timestr[256] = {0,};
-        char       key[1024] = {0,};
-
-        /* <file> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"file");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        snprintf (key, sizeof (key), "%d-filename-%d", brick_index,
-                  member_index);
-        ret = dict_get_str (dict, key, &filename);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"filename",
-                                               "%s", filename);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%d-value-%d", brick_index, member_index);
-        ret = dict_get_uint64 (dict, key, &throughput);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"count",
-                                               "%"PRIu64, throughput);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%d-time-sec-%d", brick_index,
-                  member_index);
-        ret = dict_get_int32 (dict, key, (int32_t *)&time_sec);
-        if (ret)
-                goto out;
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%d-time-usec-%d", brick_index,
-                  member_index);
-        ret = dict_get_int32 (dict, key, (int32_t *)&time_usec);
-        if (ret)
-                goto out;
-
-        gf_time_fmt (timestr, sizeof timestr, time_sec, gf_timefmt_FT);
-        snprintf (timestr + strlen (timestr),
-                  sizeof timestr - strlen (timestr),
-                  ".%"GF_PRI_SUSECONDS, time_usec);
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"time",
-                                               "%s", timestr);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        /* </file> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    int ret = -1;
+    char *filename = NULL;
+    uint64_t throughput = 0;
+    struct timeval tv = {
+        0,
+    };
+    char timestr[GF_TIMESTR_SIZE] = {
+        0,
+    };
+    char key[1024] = {
+        0,
+    };
+
+    /* <file> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"file");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%d-filename-%d", brick_index, member_index);
+    ret = dict_get_str(dict, key, &filename);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"filename", "%s",
+                                          filename);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%d-value-%d", brick_index, member_index);
+    ret = dict_get_uint64(dict, key, &throughput);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"count",
+                                          "%" PRIu64, throughput);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%d-time-sec-%d", brick_index, member_index);
+    ret = dict_get_int32(dict, key, (int32_t *)&tv.tv_sec);
+    if (ret)
+        goto out;
+
+    snprintf(key, sizeof(key), "%d-time-usec-%d", brick_index, member_index);
+    ret = dict_get_int32(dict, key, (int32_t *)&tv.tv_usec);
+    if (ret)
+        goto out;
+
+    gf_time_fmt_tv(timestr, sizeof timestr, &tv, gf_timefmt_FT);
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"time", "%s",
+                                          timestr);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* </file> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 
 int
-cli_xml_output_vol_top_other (xmlTextWriterPtr writer, dict_t *dict,
-                                int brick_index, int member_index)
+cli_xml_output_vol_top_other(xmlTextWriterPtr writer, dict_t *dict,
+                             int brick_index, int member_index)
 {
-        int             ret = -1;
-        char            *filename = NULL;
-        uint64_t        count = 0;
-        char            key[1024] = {0,};
-
-        /* <file> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"file");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        snprintf (key, sizeof (key), "%d-filename-%d", brick_index,
-                  member_index);
-        ret = dict_get_str (dict, key, &filename);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"filename",
-                                               "%s", filename);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%d-value-%d", brick_index, member_index);
-        ret = dict_get_uint64 (dict, key, &count);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"count",
-                                               "%"PRIu64, count);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        /* </file> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    int ret = -1;
+    char *filename = NULL;
+    uint64_t count = 0;
+    char key[1024] = {
+        0,
+    };
+
+    /* <file> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"file");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%d-filename-%d", brick_index, member_index);
+    ret = dict_get_str(dict, key, &filename);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"filename", "%s",
+                                          filename);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%d-value-%d", brick_index, member_index);
+    ret = dict_get_uint64(dict, key, &count);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"count",
+                                          "%" PRIu64, count);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* </file> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 #endif
 
 int
-cli_xml_output_vol_top (dict_t *dict, int op_ret, int op_errno,
-                        char *op_errstr)
+cli_xml_output_vol_top(dict_t *dict, int op_ret, int op_errno, char *op_errstr)
 {
 #if (HAVE_LIB_XML)
-        int                     ret = -1;
-        xmlTextWriterPtr        writer = NULL;
-        xmlDocPtr               doc = NULL;
-        int                     brick_count = 0;
-        int                     top_op = GF_CLI_TOP_NONE;
-        char                    *brick_name = NULL;
-        int                     members = 0;
-        uint64_t                current_open = 0;
-        uint64_t                max_open = 0;
-        char                    *max_open_time = NULL;
-        double                  throughput = 0.0;
-        double                  time_taken = 0.0;
-        char                    key[1024] = {0,};
-        int                     i = 0;
-        int                     j = 0;
-
-        ret = cli_begin_xml_output (&writer, &doc);
+    int ret = -1;
+    xmlTextWriterPtr writer = NULL;
+    xmlDocPtr doc = NULL;
+    int brick_count = 0;
+    int top_op = GF_CLI_TOP_NONE;
+    char *brick_name = NULL;
+    int members = 0;
+    uint64_t current_open = 0;
+    uint64_t max_open = 0;
+    char *max_open_time = NULL;
+    double throughput = 0.0;
+    double time_taken = 0.0;
+    char key[1024] = {
+        0,
+    };
+    int i = 0;
+    int j = 0;
+
+    ret = cli_begin_xml_output(&writer, &doc);
+    if (ret)
+        goto out;
+
+    ret = cli_xml_output_common(writer, op_ret, op_errno, op_errstr);
+    if (ret)
+        goto out;
+
+    /* <volTop> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"volTop");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_int32(dict, "count", &brick_count);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"brickCount", "%d",
+                                          brick_count);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_int32(dict, "1-top-op", &top_op);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"topOp", "%d",
+                                          top_op);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    while (i < brick_count) {
+        i++;
+
+        /* <brick> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"brick");
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        snprintf(key, sizeof(key), "%d-brick", i);
+        ret = dict_get_str(dict, key, &brick_name);
         if (ret)
-                goto out;
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"name", "%s",
+                                              brick_name);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = cli_xml_output_common (writer, op_ret, op_errno, op_errstr);
+        snprintf(key, sizeof(key), "%d-members", i);
+        ret = dict_get_int32(dict, key, &members);
         if (ret)
-                goto out;
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"members",
+                                              "%d", members);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        /* <volTop> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"volTop");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+        switch (top_op) {
+            case GF_CLI_TOP_OPEN:
+                snprintf(key, sizeof(key), "%d-current-open", i);
+                ret = dict_get_uint64(dict, key, &current_open);
+                if (ret)
+                    goto out;
+                ret = xmlTextWriterWriteFormatElement(
+                    writer, (xmlChar *)"currentOpen", "%" PRIu64, current_open);
+                XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = dict_get_int32 (dict, "count", &brick_count);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"brickCount",
-                                               "%d", brick_count);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+                snprintf(key, sizeof(key), "%d-max-open", i);
+                ret = dict_get_uint64(dict, key, &max_open);
+                if (ret)
+                    goto out;
+                ret = xmlTextWriterWriteFormatElement(
+                    writer, (xmlChar *)"maxOpen", "%" PRIu64, max_open);
+                XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = dict_get_int32 (dict, "1-top-op", &top_op);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"topOp",
-                                               "%d", top_op);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+                snprintf(key, sizeof(key), "%d-max-openfd-time", i);
+                ret = dict_get_str(dict, key, &max_open_time);
+                if (ret)
+                    goto out;
+                ret = xmlTextWriterWriteFormatElement(
+                    writer, (xmlChar *)"maxOpenTime", "%s", max_open_time);
+                XML_RET_CHECK_AND_GOTO(ret, out);
 
-        while (i < brick_count) {
-                i++;
+            case GF_CLI_TOP_READ:
+            case GF_CLI_TOP_WRITE:
+            case GF_CLI_TOP_OPENDIR:
+            case GF_CLI_TOP_READDIR:
 
-                /* <brick> */
-                ret = xmlTextWriterStartElement (writer, (xmlChar *)"brick");
-                XML_RET_CHECK_AND_GOTO (ret, out);
+                break;
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%d-brick", i);
-                ret = dict_get_str (dict, key, &brick_name);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"name",
-                                                       "%s", brick_name);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key , sizeof (key), "%d-members", i);
-                ret = dict_get_int32 (dict, key, &members);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"members",
-                                                       "%d", members);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                switch (top_op) {
-                case GF_CLI_TOP_OPEN:
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key), "%d-current-open", i);
-                        ret = dict_get_uint64 (dict, key, &current_open);
-                        if (ret)
-                                goto out;
-                        ret = xmlTextWriterWriteFormatElement
-                                (writer, (xmlChar *)"currentOpen", "%"PRIu64,
-                                 current_open);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key), "%d-max-open", i);
-                        ret = dict_get_uint64 (dict, key, &max_open);
-                        if (ret)
-                                goto out;
-                        ret = xmlTextWriterWriteFormatElement
-                                (writer, (xmlChar *)"maxOpen", "%"PRIu64,
-                                 max_open);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key), "%d-max-openfd-time", i);
-                        ret = dict_get_str (dict, key, &max_open_time);
-                        if (ret)
-                                goto out;
-                        ret = xmlTextWriterWriteFormatElement
-                                (writer, (xmlChar *)"maxOpenTime", "%s",
-                                 max_open_time);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-
-                case GF_CLI_TOP_READ:
-                case GF_CLI_TOP_WRITE:
-                case GF_CLI_TOP_OPENDIR:
-                case GF_CLI_TOP_READDIR:
-
-                        break;
-
-                case GF_CLI_TOP_READ_PERF:
-                case GF_CLI_TOP_WRITE_PERF:
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key), "%d-throughput", i);
-                        ret = dict_get_double (dict, key, &throughput);
-                        if (!ret) {
-                                memset (key, 0, sizeof (key));
-                                snprintf (key, sizeof (key), "%d-time", i);
-                                ret = dict_get_double (dict, key, &time_taken);
-                        }
-
-                        if (!ret) {
-                                ret = xmlTextWriterWriteFormatElement
-                                        (writer, (xmlChar *)"throughput",
-                                         "%f", throughput);
-                                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                                ret = xmlTextWriterWriteFormatElement
-                                        (writer, (xmlChar *)"timeTaken",
-                                         "%f", time_taken);
-                        }
-
-                        break;
-
-                default:
-                        ret = -1;
-                        goto out;
+            case GF_CLI_TOP_READ_PERF:
+            case GF_CLI_TOP_WRITE_PERF:
+                snprintf(key, sizeof(key), "%d-throughput", i);
+                ret = dict_get_double(dict, key, &throughput);
+                if (!ret) {
+                    snprintf(key, sizeof(key), "%d-time", i);
+                    ret = dict_get_double(dict, key, &time_taken);
                 }
 
-                for (j = 1; j <= members; j++) {
-                        if (top_op == GF_CLI_TOP_READ_PERF ||
-                            top_op == GF_CLI_TOP_WRITE_PERF) {
-                                ret = cli_xml_output_vol_top_rw_perf
-                                        (writer, dict, i, j);
-                        } else {
-                                ret = cli_xml_output_vol_top_other
-                                        (writer, dict, i, j);
-                        }
-                        if (ret)
-                                goto out;
+                if (!ret) {
+                    ret = xmlTextWriterWriteFormatElement(
+                        writer, (xmlChar *)"throughput", "%f", throughput);
+                    XML_RET_CHECK_AND_GOTO(ret, out);
+
+                    ret = xmlTextWriterWriteFormatElement(
+                        writer, (xmlChar *)"timeTaken", "%f", time_taken);
+                    XML_RET_CHECK_AND_GOTO(ret, out);
                 }
 
+                break;
 
-                /* </brick> */
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+            default:
+                ret = -1;
+                goto out;
+        }
+
+        for (j = 1; j <= members; j++) {
+            if (top_op == GF_CLI_TOP_READ_PERF ||
+                top_op == GF_CLI_TOP_WRITE_PERF) {
+                ret = cli_xml_output_vol_top_rw_perf(writer, dict, i, j);
+            } else {
+                ret = cli_xml_output_vol_top_other(writer, dict, i, j);
+            }
+            if (ret)
+                goto out;
         }
 
-        /* </volTop> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-        ret = cli_end_xml_output (writer, doc);
+        /* </brick> */
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
+
+    /* </volTop> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+    ret = cli_end_xml_output(writer, doc);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif
 }
 
 #if (HAVE_LIB_XML)
 int
-cli_xml_output_vol_profile_stats (xmlTextWriterPtr writer, dict_t *dict,
-                                  int brick_index, int interval)
+cli_xml_output_vol_profile_stats(xmlTextWriterPtr writer, dict_t *dict,
+                                 int brick_index, int interval)
 {
-        int                     ret = -1;
-        uint64_t                read_count = 0;
-        uint64_t                write_count = 0;
-        uint64_t                hits = 0;
-        double                  avg_latency = 0.0;
-        double                  max_latency = 0.0;
-        double                  min_latency = 0.0;
-        uint64_t                duration = 0;
-        uint64_t                total_read = 0;
-        uint64_t                total_write = 0;
-        char                    key[1024] = {0};
-        int                     i = 0;
-
-        /* <cumulativeStats> || <intervalStats> */
-        if (interval == -1)
-                ret = xmlTextWriterStartElement (writer,
-                                                 (xmlChar *)"cumulativeStats");
-        else
-                ret = xmlTextWriterStartElement (writer,
-                                                 (xmlChar *)"intervalStats");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        /* <blockStats> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"blockStats");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        for (i = 0; i < 32; i++) {
-                /* <block> */
-                ret = xmlTextWriterStartElement (writer, (xmlChar *)"block");
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                ret = xmlTextWriterWriteFormatElement
-                        (writer, (xmlChar *)"size", "%"PRIu32, (1 << i));
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%d-%d-read-%d", brick_index,
-                          interval, (1 << i));
-                ret = dict_get_uint64 (dict, key, &read_count);
-                if (ret)
-                        read_count = 0;
-                ret = xmlTextWriterWriteFormatElement
-                        (writer, (xmlChar *)"reads", "%"PRIu64, read_count);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%d-%d-write-%d", brick_index,
-                          interval, (1 << i));
-                ret = dict_get_uint64 (dict, key, &write_count);
-                if (ret)
-                        write_count = 0;
-                ret = xmlTextWriterWriteFormatElement
-                        (writer, (xmlChar *)"writes", "%"PRIu64, write_count);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                /* </block> */
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+    int ret = -1;
+    uint64_t read_count = 0;
+    uint64_t write_count = 0;
+    uint64_t hits = 0;
+    double avg_latency = 0.0;
+    double max_latency = 0.0;
+    double min_latency = 0.0;
+    uint64_t duration = 0;
+    uint64_t total_read = 0;
+    uint64_t total_write = 0;
+    char key[1024] = {0};
+    int i = 0;
+
+    /* <cumulativeStats> || <intervalStats> */
+    if (interval == -1)
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"cumulativeStats");
+    else
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"intervalStats");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* <blockStats> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"blockStats");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    for (i = 0; i < 32; i++) {
+        /* <block> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"block");
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        /* </blockStats> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"size",
+                                              "%" PRIu32, (1U << i));
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        /* <fopStats> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"fopStats");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+        snprintf(key, sizeof(key), "%d-%d-read-%" PRIu32, brick_index, interval,
+                 (1U << i));
+        ret = dict_get_uint64(dict, key, &read_count);
+        if (ret)
+            read_count = 0;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"reads",
+                                              "%" PRIu64, read_count);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        for (i = 0; i < GF_FOP_MAXVALUE; i++) {
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%d-%d-%d-hits", brick_index,
-                          interval, i);
-                ret = dict_get_uint64 (dict, key, &hits);
-                if (ret)
-                        goto cont;
+        snprintf(key, sizeof(key), "%d-%d-write-%" PRIu32, brick_index,
+                 interval, (1U << i));
+        ret = dict_get_uint64(dict, key, &write_count);
+        if (ret)
+            write_count = 0;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"writes",
+                                              "%" PRIu64, write_count);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%d-%d-%d-avglatency", brick_index,
-                          interval, i);
-                ret = dict_get_double (dict, key, &avg_latency);
-                if (ret)
-                        goto cont;
+        /* </block> */
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%d-%d-%d-minlatency", brick_index,
-                          interval, i);
-                ret = dict_get_double (dict, key, &min_latency);
-                if (ret)
-                        goto cont;
+    /* </blockStats> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%d-%d-%d-maxlatency", brick_index,
-                          interval, i);
-                ret = dict_get_double (dict, key, &max_latency);
-                if (ret)
-                        goto cont;
+    /* <fopStats> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"fopStats");
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-                /* <fop> */
-                ret = xmlTextWriterStartElement (writer, (xmlChar *)"fop");
-                XML_RET_CHECK_AND_GOTO (ret, out);
+    for (i = 0; i < GF_FOP_MAXVALUE; i++) {
+        snprintf(key, sizeof(key), "%d-%d-%d-hits", brick_index, interval, i);
+        ret = dict_get_uint64(dict, key, &hits);
+        if (ret)
+            goto cont;
 
-                ret = xmlTextWriterWriteFormatElement
-                        (writer, (xmlChar *)"name","%s", gf_fop_list[i]);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        snprintf(key, sizeof(key), "%d-%d-%d-avglatency", brick_index, interval,
+                 i);
+        ret = dict_get_double(dict, key, &avg_latency);
+        if (ret)
+            goto cont;
 
-                ret = xmlTextWriterWriteFormatElement
-                        (writer, (xmlChar *)"hits", "%"PRIu64, hits);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        snprintf(key, sizeof(key), "%d-%d-%d-minlatency", brick_index, interval,
+                 i);
+        ret = dict_get_double(dict, key, &min_latency);
+        if (ret)
+            goto cont;
 
-                ret = xmlTextWriterWriteFormatElement
-                        (writer, (xmlChar *)"avgLatency", "%f", avg_latency);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        snprintf(key, sizeof(key), "%d-%d-%d-maxlatency", brick_index, interval,
+                 i);
+        ret = dict_get_double(dict, key, &max_latency);
+        if (ret)
+            goto cont;
 
-                ret = xmlTextWriterWriteFormatElement
-                        (writer, (xmlChar *)"minLatency", "%f", min_latency);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        /* <fop> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"fop");
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                ret = xmlTextWriterWriteFormatElement
-                        (writer, (xmlChar *)"maxLatency", "%f", max_latency);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"name", "%s",
+                                              gf_fop_list[i]);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                /* </fop> */
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"hits",
+                                              "%" PRIu64, hits);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-cont:
-                hits = 0;
-                avg_latency = 0.0;
-                min_latency = 0.0;
-                max_latency = 0.0;
-        }
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"avgLatency",
+                                              "%f", avg_latency);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        /* </fopStats> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"minLatency",
+                                              "%f", min_latency);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%d-%d-duration", brick_index, interval);
-        ret = dict_get_uint64 (dict, key, &duration);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"duration",
-                                               "%"PRIu64, duration);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"maxLatency",
+                                              "%f", max_latency);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%d-%d-total-read", brick_index, interval);
-        ret = dict_get_uint64 (dict, key, &total_read);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"totalRead",
-                                               "%"PRIu64, total_read);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+        /* </fop> */
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        memset (key, 0, sizeof (key));
-        snprintf (key, sizeof (key), "%d-%d-total-write", brick_index, interval);
-        ret = dict_get_uint64 (dict, key, &total_write);
+    cont:
+        hits = 0;
+        avg_latency = 0.0;
+        min_latency = 0.0;
+        max_latency = 0.0;
+    }
+
+    for (i = 0; i < GF_UPCALL_FLAGS_MAXVALUE; i++) {
+        hits = 0;
+        avg_latency = 0.0;
+        min_latency = 0.0;
+        max_latency = 0.0;
+
+        snprintf(key, sizeof(key), "%d-%d-%d-upcall-hits", brick_index,
+                 interval, i);
+        ret = dict_get_uint64(dict, key, &hits);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"totalWrite",
-                                               "%"PRIu64, total_write);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+            continue;
 
-        /* </cumulativeStats> || </intervalStats> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+        /* <fop> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"fop");
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"name", "%s",
+                                              gf_fop_list[i]);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"hits",
+                                              "%" PRIu64, hits);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"avgLatency",
+                                              "%f", avg_latency);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"minLatency",
+                                              "%f", min_latency);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"maxLatency",
+                                              "%f", max_latency);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        /* </fop> */
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
+
+    /* </fopStats> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%d-%d-duration", brick_index, interval);
+    ret = dict_get_uint64(dict, key, &duration);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"duration",
+                                          "%" PRIu64, duration);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%d-%d-total-read", brick_index, interval);
+    ret = dict_get_uint64(dict, key, &total_read);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"totalRead",
+                                          "%" PRIu64, total_read);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%d-%d-total-write", brick_index, interval);
+    ret = dict_get_uint64(dict, key, &total_write);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"totalWrite",
+                                          "%" PRIu64, total_write);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* </cumulativeStats> || </intervalStats> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 #endif
 
 int
-cli_xml_output_vol_profile (dict_t *dict, int op_ret, int op_errno,
-                            char *op_errstr)
+cli_xml_output_vol_profile(dict_t *dict, int op_ret, int op_errno,
+                           char *op_errstr)
 {
 #if (HAVE_LIB_XML)
-        int                     ret = -1;
-        xmlTextWriterPtr        writer = NULL;
-        xmlDocPtr               doc = NULL;
-        char                    *volname = NULL;
-        int                     op = GF_CLI_STATS_NONE;
-        int                     info_op = GF_CLI_INFO_NONE;
-        int                     brick_count = 0;
-        char                    *brick_name = NULL;
-        int                     interval = 0;
-        char                    key[1024] = {0,};
-        int                     i = 0;
-        int                     stats_cleared = 0;
-
-        ret = cli_begin_xml_output (&writer, &doc);
-        if (ret)
-                goto out;
-
-        ret = cli_xml_output_common (writer, op_ret, op_errno, op_errstr);
-        if (ret)
-                goto out;
-
-        /* <volProfile> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"volProfile");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = dict_get_str (dict, "volname", &volname);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"volname",
-                                               "%s", volname);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = dict_get_int32 (dict, "op", &op);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"profileOp",
-                                               "%d", op);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        if (GF_CLI_STATS_INFO != op)
-                goto cont;
+    int ret = -1;
+    xmlTextWriterPtr writer = NULL;
+    xmlDocPtr doc = NULL;
+    char *volname = NULL;
+    int op = GF_CLI_STATS_NONE;
+    int info_op = GF_CLI_INFO_NONE;
+    int brick_count = 0;
+    char *brick_name = NULL;
+    int interval = 0;
+    char key[1024] = {
+        0,
+    };
+    int i = 0;
+    int stats_cleared = 0;
+
+    ret = cli_begin_xml_output(&writer, &doc);
+    if (ret)
+        goto out;
+
+    ret = cli_xml_output_common(writer, op_ret, op_errno, op_errstr);
+    if (ret)
+        goto out;
+
+    /* <volProfile> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"volProfile");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_str(dict, "volname", &volname);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"volname", "%s",
+                                          volname);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_int32(dict, "op", &op);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"profileOp", "%d",
+                                          op);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    if (GF_CLI_STATS_INFO != op)
+        goto cont;
+
+    ret = dict_get_int32(dict, "count", &brick_count);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"brickCount", "%d",
+                                          brick_count);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_int32(dict, "info-op", &info_op);
+    if (ret)
+        goto out;
+
+    while (i < brick_count) {
+        i++;
+
+        /* <brick> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"brick");
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = dict_get_int32 (dict, "count", &brick_count);
+        snprintf(key, sizeof(key), "%d-brick", i);
+        ret = dict_get_str(dict, key, &brick_name);
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"brickCount",
-                                               "%d", brick_count);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"brickName",
+                                              "%s", brick_name);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = dict_get_int32 (dict, "info-op", &info_op);
-        if (ret)
+        if (GF_CLI_INFO_CLEAR == info_op) {
+            snprintf(key, sizeof(key), "%d-stats-cleared", i);
+            ret = dict_get_int32(dict, key, &stats_cleared);
+            if (ret)
                 goto out;
 
-        while (i < brick_count) {
-                i++;
-
-                /* <brick> */
-                ret = xmlTextWriterStartElement (writer, (xmlChar *)"brick");
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                snprintf (key, sizeof (key), "%d-brick", i);
-                ret = dict_get_str (dict, key, &brick_name);
+            ret = xmlTextWriterWriteFormatElement(
+                writer, (xmlChar *)"clearStats", "%s",
+                stats_cleared ? "Cleared stats." : "Failed to clear stats.");
+            XML_RET_CHECK_AND_GOTO(ret, out);
+        } else {
+            snprintf(key, sizeof(key), "%d-cumulative", i);
+            ret = dict_get_int32(dict, key, &interval);
+            if (ret == 0) {
+                ret = cli_xml_output_vol_profile_stats(writer, dict, i,
+                                                       interval);
                 if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement
-                        (writer, (xmlChar *)"brickName", "%s", brick_name);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                if (GF_CLI_INFO_CLEAR == info_op) {
-                        snprintf (key, sizeof (key), "%d-stats-cleared", i);
-                        ret = dict_get_int32 (dict, key, &stats_cleared);
-                        if (ret)
-                                goto out;
-
-                        ret = xmlTextWriterWriteFormatElement
-                                (writer, (xmlChar *)"clearStats", "%s",
-                                stats_cleared ? "Cleared stats." :
-                                                 "Failed to clear stats.");
-                        if (ret)
-                                goto out;
-                } else {
-                        snprintf (key, sizeof (key), "%d-cumulative", i);
-                        ret = dict_get_int32 (dict, key, &interval);
-                        if (ret == 0) {
-                                ret = cli_xml_output_vol_profile_stats
-                                        (writer, dict, i, interval);
-                                if (ret)
-                                        goto out;
-                        }
-
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key), "%d-interval", i);
-                        ret = dict_get_int32 (dict, key, &interval);
-                        if (ret == 0) {
-                                ret = cli_xml_output_vol_profile_stats
-                                        (writer, dict, i, interval);
-                                if (ret)
-                                        goto out;
-                        }
-                }
-
-                /* </brick> */
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+                    goto out;
+            }
+
+            snprintf(key, sizeof(key), "%d-interval", i);
+            ret = dict_get_int32(dict, key, &interval);
+            if (ret == 0) {
+                ret = cli_xml_output_vol_profile_stats(writer, dict, i,
+                                                       interval);
+                if (ret)
+                    goto out;
+            }
         }
 
+        /* </brick> */
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
+
 cont:
-        /* </volProfile> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </volProfile> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = cli_end_xml_output (writer, doc);
+    ret = cli_end_xml_output(writer, doc);
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif
 }
 
 int
-cli_xml_output_vol_list (dict_t *dict, int op_ret, int op_errno,
-                         char *op_errstr)
+cli_xml_output_vol_list(dict_t *dict, int op_ret, int op_errno, char *op_errstr)
 {
 #if (HAVE_LIB_XML)
-        int                     ret = -1;
-        xmlTextWriterPtr        writer = NULL;
-        xmlDocPtr               doc = NULL;
-        int                     count = 0;
-        char                    *volname = NULL;
-        char                    key[1024] = {0,};
-        int                     i = 0;
-
-        ret = cli_begin_xml_output (&writer, &doc);
+    int ret = -1;
+    xmlTextWriterPtr writer = NULL;
+    xmlDocPtr doc = NULL;
+    int count = 0;
+    char *volname = NULL;
+    char key[1024] = {
+        0,
+    };
+    int i = 0;
+
+    ret = cli_begin_xml_output(&writer, &doc);
+    if (ret)
+        goto out;
+
+    ret = cli_xml_output_common(writer, op_ret, op_errno, op_errstr);
+    if (ret)
+        goto out;
+
+    /* <volList> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"volList");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_int32(dict, "count", &count);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"count", "%d",
+                                          count);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    for (i = 0; i < count; i++) {
+        snprintf(key, sizeof(key), "volume%d", i);
+        ret = dict_get_str(dict, key, &volname);
         if (ret)
-                goto out;
-
-        ret = cli_xml_output_common (writer, op_ret, op_errno, op_errstr);
-        if (ret)
-                goto out;
-
-        /* <volList> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"volList");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = dict_get_int32 (dict, "count", &count);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"count",
-                                               "%d", count);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        for (i = 0; i < count; i++) {
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "volume%d", i);
-                ret = dict_get_str (dict, key, &volname);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"volume",
-                                                       "%s", volname);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"volume", "%s",
+                                              volname);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        /* </volList> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </volList> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = cli_end_xml_output (writer, doc);
+    ret = cli_end_xml_output(writer, doc);
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif
 }
 
 #if (HAVE_LIB_XML)
 int
-cli_xml_output_vol_info_option (xmlTextWriterPtr writer, char *substr,
-                                char *optstr, char *valstr)
+cli_xml_output_vol_info_option(xmlTextWriterPtr writer, char *substr,
+                               char *optstr, char *valstr)
 {
-        int             ret = -1;
-        char            *ptr1 = NULL;
-        char            *ptr2 = NULL;
-
-        ptr1 = substr;
-        ptr2 = optstr;
-
-        while (ptr1) {
-                if (*ptr1 != *ptr2)
-                        break;
-                ptr1++;
-                ptr2++;
-                if (!ptr1)
-                        goto out;
-                if (!ptr2)
-                        goto out;
-        }
-        if (*ptr2 == '\0')
-                goto out;
-
-        /* <option> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"option");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"name",
-                                               "%s", ptr2);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"value",
-                                               "%s", valstr);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        /* </option> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    int ret = -1;
+    char *ptr1 = NULL;
+    char *ptr2 = NULL;
+
+    ptr1 = substr;
+    ptr2 = optstr;
+
+    while (ptr1) {
+        if (*ptr1 != *ptr2)
+            break;
+        ptr1++;
+        ptr2++;
+        if (!*ptr1)
+            break;
+        if (!*ptr2)
+            break;
+    }
+    if (*ptr2 == '\0')
+        goto out;
+
+    /* <option> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"option");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"name", "%s",
+                                          ptr2);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"value", "%s",
+                                          valstr);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* </option> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 
 struct tmp_xml_option_logger {
-        char *key;
-        xmlTextWriterPtr writer;
+    char *key;
+    xmlTextWriterPtr writer;
 };
 
 static int
-_output_vol_info_option (dict_t *d, char *k, data_t *v,
-                         void *data)
+_output_vol_info_option(dict_t *d, char *k, data_t *v, void *data)
 {
-        int                           ret   = 0;
-        char                         *ptr   = NULL;
-        struct tmp_xml_option_logger *tmp   = NULL;
+    int ret = 0;
+    char *ptr = NULL;
+    struct tmp_xml_option_logger *tmp = NULL;
 
-        tmp = data;
+    tmp = data;
 
-        ptr = strstr (k, "option.");
-        if (!ptr)
-                goto out;
+    ptr = strstr(k, "option.");
+    if (!ptr)
+        goto out;
 
-        if (!v) {
-                ret = -1;
-                goto out;
-        }
-        ret = cli_xml_output_vol_info_option (tmp->writer, tmp->key, k,
-                                              v->data);
+    if (!v) {
+        ret = -1;
+        goto out;
+    }
+    ret = cli_xml_output_vol_info_option(tmp->writer, tmp->key, k, v->data);
 
 out:
-        return ret;
+    return ret;
 }
 
 int
-cli_xml_output_vol_info_options (xmlTextWriterPtr writer, dict_t *dict,
-                                 char *prefix)
+cli_xml_output_vol_info_options(xmlTextWriterPtr writer, dict_t *dict,
+                                char *prefix)
 {
-        int             ret = -1;
-        int             opt_count = 0;
-        char            key[1024] = {0,};
-        struct tmp_xml_option_logger tmp = {0,};
-
-        snprintf (key, sizeof (key), "%s.opt_count", prefix);
-        ret = dict_get_int32 (dict, key, &opt_count);
-        if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"optCount",
-                                               "%d", opt_count);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        /* <options> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"options");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-        snprintf (key, sizeof (key), "%s.option.", prefix);
-
-        tmp.key = key;
-        tmp.writer = writer;
-        ret = dict_foreach (dict, _output_vol_info_option, &tmp);
-        if (ret)
-                goto out;
-
-        /* </options> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    int ret = -1;
+    int opt_count = 0;
+    char key[1024] = {
+        0,
+    };
+    struct tmp_xml_option_logger tmp = {
+        0,
+    };
+
+    snprintf(key, sizeof(key), "%s.opt_count", prefix);
+    ret = dict_get_int32(dict, key, &opt_count);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"optCount", "%d",
+                                          opt_count);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* <options> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"options");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+    snprintf(key, sizeof(key), "%s.option.", prefix);
+
+    tmp.key = key;
+    tmp.writer = writer;
+    ret = dict_foreach(dict, _output_vol_info_option, &tmp);
+    if (ret)
+        goto out;
+
+    /* </options> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 #endif
 
 int
-cli_xml_output_vol_info (cli_local_t *local, dict_t *dict)
+cli_xml_output_vol_info(cli_local_t *local, dict_t *dict)
 {
 #if (HAVE_LIB_XML)
-        int                     ret = 0;
-        int                     count = 0;
-        char                    *volname = NULL;
-        char                    *volume_id = NULL;
-        char                    *uuid = NULL;
-        int                     type = 0;
-        int                     status = 0;
-        int                     brick_count = 0;
-        int                     dist_count = 0;
-        int                     stripe_count = 0;
-        int                     replica_count = 0;
-        int                     disperse_count = 0;
-        int                     redundancy_count = 0;
-        int                     transport = 0;
-        char                    *brick = NULL;
-        char                    key[1024] = {0,};
-        int                     i = 0;
-        int                     j = 1;
-        char                    *caps = NULL;
-        int                     k __attribute__((unused)) = 0;
-
-        ret = dict_get_int32 (dict, "count", &count);
+    int ret = 0;
+    int count = 0;
+    char *volname = NULL;
+    char *volume_id = NULL;
+    char *uuid = NULL;
+    int type = 0;
+    int status = 0;
+    int brick_count = 0;
+    int dist_count = 0;
+    int stripe_count = 0;
+    int replica_count = 0;
+    int arbiter_count = 0;
+    int snap_count = 0;
+    int isArbiter = 0;
+    int disperse_count = 0;
+    int redundancy_count = 0;
+    int transport = 0;
+    char *brick = NULL;
+    char key[1024] = {
+        0,
+    };
+    int i = 0;
+    int j = 1;
+    char *caps __attribute__((unused)) = NULL;
+    int k __attribute__((unused)) = 0;
+
+    ret = dict_get_int32(dict, "count", &count);
+    if (ret)
+        goto out;
+
+    for (i = 0; i < count; i++) {
+        /* <volume> */
+        ret = xmlTextWriterStartElement(local->writer, (xmlChar *)"volume");
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        snprintf(key, sizeof(key), "volume%d.name", i);
+        ret = dict_get_str(dict, key, &volname);
         if (ret)
-                goto out;
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(local->writer, (xmlChar *)"name",
+                                              "%s", volname);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        for (i = 0; i < count; i++) {
-                /* <volume> */
-                ret = xmlTextWriterStartElement (local->writer,
-                                                 (xmlChar *)"volume");
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        snprintf(key, sizeof(key), "volume%d.volume_id", i);
+        ret = dict_get_str(dict, key, &volume_id);
+        if (ret)
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(local->writer, (xmlChar *)"id",
+                                              "%s", volume_id);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "volume%d.name", i);
-                ret = dict_get_str (dict, key, &volname);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (local->writer,
-                                                       (xmlChar *)"name",
-                                                       "%s", volname);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "volume%d.volume_id", i);
-                ret = dict_get_str (dict, key, &volume_id);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (local->writer,
-                                                       (xmlChar *)"id",
-                                                       "%s", volume_id);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "volume%d.status", i);
-                ret = dict_get_int32 (dict, key, &status);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (local->writer,
-                                                       (xmlChar *)"status",
-                                                       "%d", status);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                ret =xmlTextWriterWriteFormatElement
-                        (local->writer, (xmlChar *)"statusStr", "%s",
-                         cli_vol_status_str[status]);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "volume%d.brick_count", i);
-                ret = dict_get_int32 (dict, key, &brick_count);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (local->writer,
-                                                       (xmlChar *)"brickCount",
-                                                       "%d", brick_count);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "volume%d.dist_count", i);
-                ret = dict_get_int32 (dict, key, &dist_count);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (local->writer,
-                                                       (xmlChar *)"distCount",
-                                                       "%d", dist_count);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "volume%d.stripe_count", i);
-                ret = dict_get_int32 (dict, key, &stripe_count);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (local->writer,
-                                                       (xmlChar *)"stripeCount",
-                                                       "%d", stripe_count);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "volume%d.replica_count", i);
-                ret = dict_get_int32 (dict, key, &replica_count);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (local->writer,
-                                                       (xmlChar *)"replicaCount",
-                                                       "%d", replica_count);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "volume%d.disperse_count", i);
-                ret = dict_get_int32 (dict, key, &disperse_count);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (local->writer,
-                                                       (xmlChar *)"disperseCount",
-                                                       "%d", disperse_count);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "volume%d.redundancy_count", i);
-                ret = dict_get_int32 (dict, key, &redundancy_count);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (local->writer,
-                                                       (xmlChar *)"redundancyCount",
-                                                       "%d", redundancy_count);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "volume%d.type", i);
-                ret = dict_get_int32 (dict, key, &type);
-                if (ret)
-                        goto out;
-                /* For Distributed-(stripe,replicate,stipe-replicate,disperse)
-                   types
-                 */
-                if ((type > 0) && (dist_count < brick_count))
-                        type += 4;
-                ret = xmlTextWriterWriteFormatElement (local->writer,
-                                                       (xmlChar *)"type",
-                                                       "%d", type);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                ret = xmlTextWriterWriteFormatElement (local->writer,
-                                                       (xmlChar *)"typeStr",
-                                                       "%s",
-                                                       cli_vol_type_str[type]);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "volume%d.transport", i);
-                ret = dict_get_int32 (dict, key, &transport);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (local->writer,
-                                                       (xmlChar *)"transport",
-                                                       "%d", transport);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-#ifdef HAVE_BD_XLATOR
-                /* <xlators> */
-                ret = xmlTextWriterStartElement (local->writer,
-                                                 (xmlChar *)"xlators");
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                for (k = 0; ; k++) {
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key),"volume%d.xlator%d", i, k);
-                        ret = dict_get_str (dict, key, &caps);
-                        if (ret)
-                                break;
-
-                        /* <xlator> */
-                        ret = xmlTextWriterStartElement (local->writer,
-                                                         (xmlChar *)"xlator");
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-
-                        ret = xmlTextWriterWriteFormatElement
-                                (local->writer, (xmlChar *)"name", "%s", caps);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-
-                        /* <capabilities> */
-                        ret = xmlTextWriterStartElement (local->writer,
-                                                         (xmlChar *)
-                                                         "capabilities");
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-
-                        j = 0;
-                        for (j = 0; ;j++) {
-                                memset (key, 0, sizeof (key));
-                                snprintf (key, sizeof (key),
-                                          "volume%d.xlator%d.caps%d", i, k, j);
-                                ret = dict_get_str (dict, key, &caps);
-                                if (ret)
-                                        break;
-                                ret = xmlTextWriterWriteFormatElement
-                                        (local->writer, (xmlChar *)"capability",
-                                         "%s", caps);
-                                XML_RET_CHECK_AND_GOTO (ret, out);
-                        }
-                        /* </capabilities> */
-                        ret = xmlTextWriterEndElement (local->writer);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-                        /* </xlator> */
-                        ret = xmlTextWriterEndElement (local->writer);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-                }
-                ret = xmlTextWriterFullEndElement (local->writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-                /* </xlators> */
-#else
-                caps = 0; /* Avoid compiler warnings when BD not enabled */
-#endif
-                j = 1;
-
-                /* <bricks> */
-                ret = xmlTextWriterStartElement (local->writer,
-                                                 (xmlChar *)"bricks");
-                XML_RET_CHECK_AND_GOTO (ret, out);
-                while (j <= brick_count) {
-                        ret = xmlTextWriterStartElement
-                                (local->writer, (xmlChar *)"brick");
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key), "volume%d.brick%d.uuid",
-                                  i, j);
-                        ret = dict_get_str (dict, key, &uuid);
-                        if (ret)
-                                goto out;
-                        ret = xmlTextWriterWriteFormatAttribute
-                                (local->writer, (xmlChar *)"uuid", "%s",
-                                 uuid);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key), "volume%d.brick%d", i, j);
-                        ret = dict_get_str (dict, key, &brick);
-                        if (ret)
-                                goto out;
-                        ret = xmlTextWriterWriteFormatString
-                                (local->writer, "%s", brick);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-
-                        ret = xmlTextWriterWriteFormatElement
-                                (local->writer, (xmlChar *)"name", "%s",
-                                 brick);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-
-                        ret = xmlTextWriterWriteFormatElement
-                                (local->writer, (xmlChar *)"hostUuid", "%s",
-                                 uuid);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-
-                        /* </brick> */
-                        ret = xmlTextWriterEndElement (local->writer);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-
-                        j++;
-                }
-                /* </bricks> */
-                ret = xmlTextWriterEndElement (local->writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "volume%d", i);
-                ret = cli_xml_output_vol_info_options (local->writer, dict,
-                                                       key);
-                if (ret)
-                        goto out;
+        snprintf(key, sizeof(key), "volume%d.status", i);
+        ret = dict_get_int32(dict, key, &status);
+        if (ret)
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(
+            local->writer, (xmlChar *)"status", "%d", status);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                /* </volume> */
-                ret = xmlTextWriterEndElement (local->writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+        ret = xmlTextWriterWriteFormatElement(local->writer,
+                                              (xmlChar *)"statusStr", "%s",
+                                              cli_vol_status_str[status]);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        if (volname) {
-                GF_FREE (local->get_vol.volname);
-                local->get_vol.volname = gf_strdup (volname);
-                local->vol_count += count;
-        }
-out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
-#else
-        return 0;
-#endif
-}
+        snprintf(key, sizeof(key), "volume%d.snap_count", i);
+        ret = dict_get_int32(dict, key, &snap_count);
+        if (ret)
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(
+            local->writer, (xmlChar *)"snapshotCount", "%d", snap_count);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-int
-cli_xml_output_vol_info_begin (cli_local_t *local, int op_ret, int op_errno,
-                               char *op_errstr)
-{
-#if (HAVE_LIB_XML)
-        int                     ret = -1;
+        snprintf(key, sizeof(key), "volume%d.brick_count", i);
+        ret = dict_get_int32(dict, key, &brick_count);
+        if (ret)
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(
+            local->writer, (xmlChar *)"brickCount", "%d", brick_count);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        GF_ASSERT (local);
+        snprintf(key, sizeof(key), "volume%d.dist_count", i);
+        ret = dict_get_int32(dict, key, &dist_count);
+        if (ret)
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(local->writer,
+                                              (xmlChar *)"distCount", "%d",
+                                              (brick_count / dist_count));
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = cli_begin_xml_output (&(local->writer), &(local->doc));
+        snprintf(key, sizeof(key), "volume%d.stripe_count", i);
+        ret = dict_get_int32(dict, key, &stripe_count);
         if (ret)
-                goto out;
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(
+            local->writer, (xmlChar *)"stripeCount", "%d", stripe_count);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        snprintf(key, sizeof(key), "volume%d.replica_count", i);
+        ret = dict_get_int32(dict, key, &replica_count);
+        if (ret)
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(
+            local->writer, (xmlChar *)"replicaCount", "%d", replica_count);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = cli_xml_output_common (local->writer, op_ret, op_errno,
-                                     op_errstr);
+        snprintf(key, sizeof(key), "volume%d.arbiter_count", i);
+        ret = dict_get_int32(dict, key, &arbiter_count);
         if (ret)
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(
+            local->writer, (xmlChar *)"arbiterCount", "%d", arbiter_count);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        snprintf(key, sizeof(key), "volume%d.disperse_count", i);
+        ret = dict_get_int32(dict, key, &disperse_count);
+        if (ret)
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(
+            local->writer, (xmlChar *)"disperseCount", "%d", disperse_count);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        snprintf(key, sizeof(key), "volume%d.redundancy_count", i);
+        ret = dict_get_int32(dict, key, &redundancy_count);
+        if (ret)
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(local->writer,
+                                              (xmlChar *)"redundancyCount",
+                                              "%d", redundancy_count);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        snprintf(key, sizeof(key), "volume%d.type", i);
+        ret = dict_get_int32(dict, key, &type);
+        if (ret)
+            goto out;
+        /* For Distributed-(stripe,replicate,stipe-replicate,disperse)
+           types
+         */
+        type = get_vol_type(type, dist_count, brick_count);
+
+        ret = xmlTextWriterWriteFormatElement(local->writer, (xmlChar *)"type",
+                                              "%d", type);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        ret = xmlTextWriterWriteFormatElement(
+            local->writer, (xmlChar *)"typeStr", "%s", vol_type_str[type]);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        snprintf(key, sizeof(key), "volume%d.transport", i);
+        ret = dict_get_int32(dict, key, &transport);
+        if (ret)
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(
+            local->writer, (xmlChar *)"transport", "%d", transport);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        j = 1;
+
+        /* <bricks> */
+        ret = xmlTextWriterStartElement(local->writer, (xmlChar *)"bricks");
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        while (j <= brick_count) {
+            ret = xmlTextWriterStartElement(local->writer, (xmlChar *)"brick");
+            XML_RET_CHECK_AND_GOTO(ret, out);
+
+            snprintf(key, sizeof(key), "volume%d.brick%d.uuid", i, j);
+            ret = dict_get_str(dict, key, &uuid);
+            if (ret)
+                goto out;
+            ret = xmlTextWriterWriteFormatAttribute(
+                local->writer, (xmlChar *)"uuid", "%s", uuid);
+            XML_RET_CHECK_AND_GOTO(ret, out);
+
+            snprintf(key, sizeof(key), "volume%d.brick%d", i, j);
+            ret = dict_get_str(dict, key, &brick);
+            if (ret)
                 goto out;
+            ret = xmlTextWriterWriteFormatString(local->writer, "%s", brick);
+            XML_RET_CHECK_AND_GOTO(ret, out);
+
+            ret = xmlTextWriterWriteFormatElement(
+                local->writer, (xmlChar *)"name", "%s", brick);
+            XML_RET_CHECK_AND_GOTO(ret, out);
+
+            ret = xmlTextWriterWriteFormatElement(
+                local->writer, (xmlChar *)"hostUuid", "%s", uuid);
+            XML_RET_CHECK_AND_GOTO(ret, out);
 
-        /* <volInfo> */
-        ret = xmlTextWriterStartElement (local->writer, (xmlChar *)"volInfo");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+            snprintf(key, sizeof(key), "volume%d.brick%d.isArbiter", i, j);
+            if (dict_get(dict, key))
+                isArbiter = 1;
+            else
+                isArbiter = 0;
+            ret = xmlTextWriterWriteFormatElement(
+                local->writer, (xmlChar *)"isArbiter", "%d", isArbiter);
+            XML_RET_CHECK_AND_GOTO(ret, out);
 
-        /* <volumes> */
-        ret = xmlTextWriterStartElement (local->writer, (xmlChar *)"volumes");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+            /* </brick> */
+            ret = xmlTextWriterEndElement(local->writer);
+            XML_RET_CHECK_AND_GOTO(ret, out);
+
+            j++;
+        }
+        /* </bricks> */
+        ret = xmlTextWriterEndElement(local->writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        snprintf(key, sizeof(key), "volume%d", i);
+        ret = cli_xml_output_vol_info_options(local->writer, dict, key);
+        if (ret)
+            goto out;
 
-        /* Init vol count */
-        local->vol_count = 0;
+        /* </volume> */
+        ret = xmlTextWriterEndElement(local->writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
+    if (volname) {
+        GF_FREE(local->get_vol.volname);
+        local->get_vol.volname = gf_strdup(volname);
+        local->vol_count += count;
+    }
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif
 }
 
 int
-cli_xml_output_vol_info_end (cli_local_t *local)
+cli_xml_output_vol_info_begin(cli_local_t *local, int op_ret, int op_errno,
+                              char *op_errstr)
 {
 #if (HAVE_LIB_XML)
-        int             ret = -1;
+    int ret = -1;
+
+    GF_ASSERT(local);
 
-        GF_ASSERT (local);
+    ret = cli_begin_xml_output(&(local->writer), &(local->doc));
+    if (ret)
+        goto out;
 
-        ret = xmlTextWriterWriteFormatElement (local->writer,
-                                               (xmlChar *)"count",
-                                               "%d", local->vol_count);
+    ret = cli_xml_output_common(local->writer, op_ret, op_errno, op_errstr);
+    if (ret)
+        goto out;
 
-        /* </volumes> */
-        ret = xmlTextWriterEndElement (local->writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* <volInfo> */
+    ret = xmlTextWriterStartElement(local->writer, (xmlChar *)"volInfo");
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        /* </volInfo> */
-        ret = xmlTextWriterEndElement (local->writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* <volumes> */
+    ret = xmlTextWriterStartElement(local->writer, (xmlChar *)"volumes");
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = cli_end_xml_output (local->writer, local->doc);
+    /* Init vol count */
+    local->vol_count = 0;
 
 out:
-        gf_log ("cli", GF_LOG_ERROR, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif
 }
 
 int
-cli_xml_output_vol_quota_limit_list (char *volname, char *limit_list,
-                                     int op_ret, int op_errno,
-                                     char *op_errstr)
+cli_xml_output_vol_info_end(cli_local_t *local)
 {
 #if (HAVE_LIB_XML)
-        int                     ret = -1;
-        xmlTextWriterPtr        writer = NULL;
-        xmlDocPtr               doc = NULL;
-        int64_t                 size = 0;
-        int64_t                 limit_value = 0;
-        int                     i = 0;
-        int                     j = 0;
-        int                     k = 0;
-        int                     len = 0;
-        char                    *size_str = NULL;
-        char                    path[PATH_MAX] = {0,};
-        char                    ret_str[1024] = {0,};
-        char                    value[1024] = {0,};
-        char                    mountdir[] = "/tmp/mountXXXXXX";
-        char                    abspath[PATH_MAX] = {0,};
-        runner_t                runner = {0,};
-
-        GF_ASSERT (volname);
-        GF_ASSERT (limit_list);
-
-        ret = cli_begin_xml_output (&writer, &doc);
-        if (ret)
-                goto out;
+    int ret = -1;
 
-        ret = cli_xml_output_common (writer, op_ret, op_errno, op_errstr);
-        if (ret)
-                goto out;
-
-        /* <volQuota> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"volQuota");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    GF_ASSERT(local);
 
-        if (!limit_list)
-                goto cont;
+    ret = xmlTextWriterWriteFormatElement(local->writer, (xmlChar *)"count",
+                                          "%d", local->vol_count);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+    /* </volumes> */
+    ret = xmlTextWriterEndElement(local->writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        len = strlen (limit_list);
-        if (len == 0)
-                goto cont;
+    /* </volInfo> */
+    ret = xmlTextWriterEndElement(local->writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        if (mkdtemp (mountdir) == NULL) {
-                gf_log ("cli", GF_LOG_ERROR, "failed to create a temporary"
-                        " mount directory");
-                ret = -1;
-                goto out;
-        }
+    ret = cli_end_xml_output(local->writer, local->doc);
 
-        ret = runcmd (SBIN_DIR"/glusterfs", "-s", "localhost",
-                      "--volfile-id", volname, "-l",
-                      DEFAULT_LOG_FILE_DIRECTORY"/quota-list-xml.log",
-                      mountdir, NULL);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR,
-                        "failed to mount glusterfs client");
-                ret = -1;
-                rmdir (mountdir);
-                goto cont;
-        }
-
-        while (i < len) {
-                j = 0;
-                k = 0;
-                size = 0;
-
-                while (limit_list[i] != ':')
-                        path[k++] = limit_list[i++];
-                path[k] = '\0';
-
-                i++;
+out:
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
+#else
+    return 0;
+#endif
+}
 
-                while (limit_list[i] != ',' && limit_list[i] != '\0')
-                        value[j++] = limit_list[i++];
-                i++;
+int
+cli_xml_output_vol_quota_limit_list_end(cli_local_t *local)
+{
+#if (HAVE_LIB_XML)
+    int ret = -1;
 
-                snprintf (abspath, sizeof (abspath), "%s/%s", mountdir, path);
-                ret = sys_lgetxattr (abspath, "trusted.limit.list",
-                                     (void *)ret_str, 4096);
-                if (ret >= 0) {
-                        sscanf (ret_str, "%"SCNd64",%"SCNd64, &size,
-                                &limit_value);
-                        size_str = gf_uint64_2human_readable ((uint64_t)size);
-                }
+    ret = xmlTextWriterEndElement(local->writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-                /* <quota> */
-                ret = xmlTextWriterStartElement (writer, (xmlChar *)"quota");
-                XML_RET_CHECK_AND_GOTO (ret, unmount);
-
-                ret = xmlTextWriterWriteFormatElement
-                        (writer, (xmlChar *)"path", "%s", path);
-                XML_RET_CHECK_AND_GOTO (ret, unmount);
-
-                ret = xmlTextWriterWriteFormatElement
-                        (writer, (xmlChar *)"limit", "%s", value);
-                XML_RET_CHECK_AND_GOTO (ret, unmount);
-
-                if (size_str) {
-                        ret = xmlTextWriterWriteFormatElement
-                                (writer, (xmlChar *)"size", "%s", size_str);
-                        XML_RET_CHECK_AND_GOTO (ret, unmount);
-                        GF_FREE (size_str);
-                } else {
-                        ret = xmlTextWriterWriteFormatElement
-                                (writer, (xmlChar *)"size", "%"PRId64, size);
-                        XML_RET_CHECK_AND_GOTO (ret, unmount);
-                }
+    ret = cli_end_xml_output(local->writer, local->doc);
 
-                /* </quota> */
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, unmount);
+out:
+    return ret;
+#else
+    return 0;
+#endif
+}
 
-        }
+int
+cli_xml_output_vol_quota_limit_list_begin(cli_local_t *local, int op_ret,
+                                          int op_errno, char *op_errstr)
+{
+#if (HAVE_LIB_XML)
+    int ret = -1;
 
-unmount:
-        ret = gf_umount_lazy ("cli", mountdir, 1);
-        if (ret)
-                gf_log ("cli", GF_LOG_WARNING, "error unmounting %s: %s",
-                        mountdir, strerror (errno));
+    ret = cli_begin_xml_output(&(local->writer), &(local->doc));
+    if (ret)
+        goto out;
 
-cont:
-        /* </volQuota> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = cli_xml_output_common(local->writer, op_ret, op_errno, op_errstr);
+    if (ret)
+        goto out;
 
-        ret = cli_end_xml_output (writer, doc);
+    /* <volQuota> */
+    ret = xmlTextWriterStartElement(local->writer, (xmlChar *)"volQuota");
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
 out:
-        GF_FREE (size_str);
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif
 }
 
 #if (HAVE_LIB_XML)
 static int
-cli_xml_output_peer_hostnames (xmlTextWriterPtr writer, dict_t *dict,
-                               const char *prefix, int count)
+cli_xml_output_peer_hostnames(xmlTextWriterPtr writer, dict_t *dict,
+                              const char *prefix, int count)
 {
-        int   ret       = -1;
-        int   i         = 0;
-        char *hostname  = NULL;
-        char  key[1024] = {0,};
-
-        /* <hostnames> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"hostnames");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        for (i = 0; i < count; i++) {
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "%s.hostname%d", prefix, i);
-                ret = dict_get_str (dict, key, &hostname);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement
-                        (writer, (xmlChar *)"hostname", "%s", hostname);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-                hostname = NULL;
-        }
-
-        /* </hostnames> */
-        ret = xmlTextWriterEndElement (writer);
+    int ret = -1;
+    int i = 0;
+    char *hostname = NULL;
+    char key[1024] = {
+        0,
+    };
+
+    /* <hostnames> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"hostnames");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    for (i = 0; i < count; i++) {
+        ret = snprintf(key, sizeof(key), "%s.hostname%d", prefix, i);
+        if (ret < 0)
+            goto out;
+        ret = dict_get_str(dict, key, &hostname);
+        if (ret)
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"hostname",
+                                              "%s", hostname);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+        hostname = NULL;
+    }
 
+    /* </hostnames> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 #endif
 
 int
-cli_xml_output_peer_status (dict_t *dict, int op_ret, int op_errno,
-                            char *op_errstr)
+cli_xml_output_peer_status(dict_t *dict, int op_ret, int op_errno,
+                           char *op_errstr)
 {
 #if (HAVE_LIB_XML)
-        int                     ret = -1;
-        xmlTextWriterPtr        writer = NULL;
-        xmlDocPtr               doc = NULL;
-        int                     count = 0;
-        char                    *uuid = NULL;
-        char                    *hostname = NULL;
-        int                     connected = 0;
-        int                     state_id = 0;
-        char                    *state_str = NULL;
-        int                     hostname_count = 0;
-        int                     i = 1;
-        char                    key[1024] = {0,};
-
-        ret = cli_begin_xml_output (&writer, &doc);
-        if (ret)
-                goto out;
+    int ret = -1;
+    xmlTextWriterPtr writer = NULL;
+    xmlDocPtr doc = NULL;
+    int count = 0;
+    char *uuid = NULL;
+    char *hostname = NULL;
+    int connected = 0;
+    int state_id = 0;
+    char *state_str = NULL;
+    int hostname_count = 0;
+    int i = 1;
+    char key[1024] = {
+        0,
+    };
+
+    ret = cli_begin_xml_output(&writer, &doc);
+    if (ret)
+        goto out;
+
+    ret = cli_xml_output_common(writer, op_ret, op_errno, op_errstr);
+    if (ret)
+        goto out;
+
+    /* <peerStatus> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"peerStatus");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    if (!dict)
+        goto cont;
+
+    ret = dict_get_int32(dict, "count", &count);
+    if (ret)
+        goto out;
+
+    while (i <= count) {
+        /* <peer> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"peer");
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = cli_xml_output_common (writer, op_ret, op_errno, op_errstr);
+        snprintf(key, sizeof(key), "friend%d.uuid", i);
+        ret = dict_get_str(dict, key, &uuid);
         if (ret)
-                goto out;
-
-        /* <peerStatus> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"peerStatus");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+            goto out;
 
-        if (!dict)
-                goto cont;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"uuid", "%s",
+                                              uuid);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = dict_get_int32 (dict, "count", &count);
+        snprintf(key, sizeof(key), "friend%d.hostname", i);
+        ret = dict_get_str(dict, key, &hostname);
         if (ret)
-                goto out;
-
-        while (i <= count) {
-                /* <peer> */
-                ret = xmlTextWriterStartElement (writer, (xmlChar *)"peer");
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "friend%d.uuid", i);
-                ret = dict_get_str (dict, key, &uuid);
-                if (ret)
-                        goto out;
+            goto out;
 
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"uuid",
-                                                       "%s", uuid);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "friend%d.hostname", i);
-                ret = dict_get_str (dict, key, &hostname);
-                if (ret)
-                        goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"hostname",
+                                              "%s", hostname);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"hostname",
-                                                       "%s", hostname);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "friend%d.hostname_count", i);
-                ret = dict_get_int32 (dict, key, &hostname_count);
-                if ((ret == 0) && (hostname_count > 0)) {
-                        memset (key, 0, sizeof (key));
-                        snprintf (key, sizeof (key), "friend%d", i);
-                        ret = cli_xml_output_peer_hostnames (writer, dict, key,
-                                                             hostname_count);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-                }
+        snprintf(key, sizeof(key), "friend%d.hostname_count", i);
+        ret = dict_get_int32(dict, key, &hostname_count);
+        if ((ret == 0) && (hostname_count > 0)) {
+            snprintf(key, sizeof(key), "friend%d", i);
+            ret = cli_xml_output_peer_hostnames(writer, dict, key,
+                                                hostname_count);
+            XML_RET_CHECK_AND_GOTO(ret, out);
+        }
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "friend%d.connected", i);
-                ret = dict_get_int32 (dict, key, &connected);
-                if (ret)
-                        goto out;
+        snprintf(key, sizeof(key), "friend%d.connected", i);
+        ret = dict_get_int32(dict, key, &connected);
+        if (ret)
+            goto out;
 
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"connected",
-                                                       "%d", connected);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"connected",
+                                              "%d", connected);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "friend%d.stateId", i);
-                ret = dict_get_int32 (dict, key, &state_id);
-                if (!ret) {
-                        /* ignore */
+        snprintf(key, sizeof(key), "friend%d.stateId", i);
+        ret = dict_get_int32(dict, key, &state_id);
+        if (!ret) {
+            /* ignore */
 
-                        ret = xmlTextWriterWriteFormatElement (writer,
-                                           (xmlChar *)"state", "%d", state_id);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-                }
+            ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"state",
+                                                  "%d", state_id);
+            XML_RET_CHECK_AND_GOTO(ret, out);
+        }
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "friend%d.state", i);
-                ret = dict_get_str (dict, key, &state_str);
-                if (!ret) {
-                        /* ignore */
+        snprintf(key, sizeof(key), "friend%d.state", i);
+        ret = dict_get_str(dict, key, &state_str);
+        if (!ret) {
+            /* ignore */
 
-                        ret = xmlTextWriterWriteFormatElement (writer,
-                                       (xmlChar *)"stateStr", "%s", state_str);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-                }
+            ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"stateStr",
+                                                  "%s", state_str);
+            XML_RET_CHECK_AND_GOTO(ret, out);
+        }
 
-                /* </peer> */
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        /* </peer> */
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                i++;
-        }
+        i++;
+    }
 
 cont:
-        /* </peerStatus> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </peerStatus> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = cli_end_xml_output (writer, doc);
+    ret = cli_end_xml_output(writer, doc);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif
 }
 
 #if (HAVE_LIB_XML)
 /* Used for rebalance stop/status, remove-brick status */
 int
-cli_xml_output_vol_rebalance_status (xmlTextWriterPtr writer, dict_t *dict,
-                                     enum gf_task_types task_type)
+cli_xml_output_vol_rebalance_status(xmlTextWriterPtr writer, dict_t *dict,
+                                    enum gf_task_types task_type)
 {
-        int                     ret = -1;
-        int                     count = 0;
-        char                    *node_name = NULL;
-        char                    *node_uuid = NULL;
-        uint64_t                files = 0;
-        uint64_t                size = 0;
-        uint64_t                lookups = 0;
-        int                     status_rcd = 0;
-        uint64_t                failures = 0;
-        uint64_t                skipped = 0;
-        uint64_t                total_files = 0;
-        uint64_t                total_size = 0;
-        uint64_t                total_lookups = 0;
-        uint64_t                total_failures = 0;
-        uint64_t                total_skipped = 0;
-        char                    key[1024] = {0,};
-        int                     i = 0;
-        int                     overall_status = -1;
-        double                  elapsed = 0;
-        double                  overall_elapsed = 0;
-
-        if (!dict) {
-                ret = 0;
-                goto out;
-        }
+    int ret = -1;
+    int count = 0;
+    char *node_name = NULL;
+    char *node_uuid = NULL;
+    uint64_t files = 0;
+    uint64_t size = 0;
+    uint64_t lookups = 0;
+    int status_rcd = 0;
+    uint64_t failures = 0;
+    uint64_t skipped = 0;
+    uint64_t total_files = 0;
+    uint64_t total_size = 0;
+    uint64_t total_lookups = 0;
+    uint64_t total_failures = 0;
+    uint64_t total_skipped = 0;
+    char key[1024] = {
+        0,
+    };
+    int i = 0;
+    int overall_status = -1;
+    double elapsed = 0;
+    double overall_elapsed = 0;
+
+    if (!dict) {
+        ret = 0;
+        goto out;
+    }
+
+    ret = dict_get_int32(dict, "count", &count);
+    if (ret)
+        goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"nodeCount", "%d",
+                                          count);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    while (i < count) {
+        i++;
+        /* Getting status early, to skip nodes that don't have the
+         * rebalance process started
+         */
+        snprintf(key, sizeof(key), "status-%d", i);
+        ret = dict_get_int32(dict, key, &status_rcd);
 
-        ret = dict_get_int32 (dict, "count", &count);
+        /* If glusterd is down it fails to get the status, try
+         getting status from other nodes */
         if (ret)
-                goto out;
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"nodeCount",
-                                               "%d", count);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        while (i < count) {
-                i++;
-                /* Getting status early, to skip nodes that don't have the
-                 * rebalance process started
-                 */
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "status-%d", i);
-                ret = dict_get_int32 (dict, key, &status_rcd);
-
-                /* If glusterd is down it fails to get the status, try
-                 getting status from other nodes */
-                if (ret)
-                    continue;
-                if (GF_DEFRAG_STATUS_NOT_STARTED == status_rcd)
-                        continue;
+            continue;
+        if (GF_DEFRAG_STATUS_NOT_STARTED == status_rcd)
+            continue;
 
-                /* <node> */
-                ret = xmlTextWriterStartElement (writer, (xmlChar *)"node");
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        /* <node> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"node");
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "node-name-%d", i);
-                ret = dict_get_str (dict, key, &node_name);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"nodeName",
-                                                       "%s", node_name);
+        snprintf(key, sizeof(key), "node-name-%d", i);
+        ret = dict_get_str(dict, key, &node_name);
+        if (ret)
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"nodeName",
+                                              "%s", node_name);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "node-uuid-%d", i);
-                ret = dict_get_str (dict, key, &node_uuid);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"id",
-                                                       "%s", node_uuid);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "files-%d", i);
-                ret = dict_get_uint64 (dict, key, &files);
-                if (ret)
-                        goto out;
-                total_files += files;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"files",
-                                                       "%"PRIu64, files);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "size-%d", i);
-                ret = dict_get_uint64 (dict, key, &size);
-                if (ret)
-                        goto out;
-                total_size += size;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"size",
-                                                       "%"PRIu64,size);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "lookups-%d", i);
-                ret = dict_get_uint64 (dict, key, &lookups);
-                if (ret)
-                        goto out;
-                total_lookups += lookups;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"lookups",
-                                                       "%"PRIu64, lookups);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "failures-%d", i);
-                ret = dict_get_uint64 (dict, key, &failures);
-                if (ret)
-                        goto out;
+        snprintf(key, sizeof(key), "node-uuid-%d", i);
+        ret = dict_get_str(dict, key, &node_uuid);
+        if (ret)
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"id", "%s",
+                                              node_uuid);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                memset (key, 0, sizeof (key));
-                snprintf (key, sizeof (key), "skipped-%d", i);
+        snprintf(key, sizeof(key), "files-%d", i);
+        ret = dict_get_uint64(dict, key, &files);
+        if (ret)
+            goto out;
+        total_files += files;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"files",
+                                              "%" PRIu64, files);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                ret = dict_get_uint64 (dict, key, &skipped);
-                if (ret)
-                        goto out;
+        snprintf(key, sizeof(key), "size-%d", i);
+        ret = dict_get_uint64(dict, key, &size);
+        if (ret)
+            goto out;
+        total_size += size;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"size",
+                                              "%" PRIu64, size);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                if (task_type == GF_TASK_TYPE_REMOVE_BRICK) {
-                        failures += skipped;
-                        skipped = 0;
-                }
+        snprintf(key, sizeof(key), "lookups-%d", i);
+        ret = dict_get_uint64(dict, key, &lookups);
+        if (ret)
+            goto out;
+        total_lookups += lookups;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"lookups",
+                                              "%" PRIu64, lookups);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                total_failures += failures;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                               (xmlChar *)"failures",
-                                               "%"PRIu64, failures);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        snprintf(key, sizeof(key), "failures-%d", i);
+        ret = dict_get_uint64(dict, key, &failures);
+        if (ret)
+            goto out;
 
-                total_skipped += skipped;
+        snprintf(key, sizeof(key), "skipped-%d", i);
 
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                               (xmlChar *)"skipped",
-                                               "%"PRIu64, skipped);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        ret = dict_get_uint64(dict, key, &skipped);
+        if (ret)
+            goto out;
 
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"status",
-                                                       "%d", status_rcd);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        if (task_type == GF_TASK_TYPE_REMOVE_BRICK) {
+            failures += skipped;
+            skipped = 0;
+        }
 
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"statusStr",
-                                                       "%s",
-                                         cli_vol_task_status_str[status_rcd]);
+        total_failures += failures;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"failures",
+                                              "%" PRIu64, failures);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                memset (key, 0, 256);
-                snprintf (key, 256, "run-time-%d", i);
-                ret = dict_get_double (dict, key, &elapsed);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"runtime",
-                                                       "%.2f", elapsed);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        total_skipped += skipped;
 
-                if (elapsed > overall_elapsed) {
-                    overall_elapsed = elapsed;
-                }
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"skipped",
+                                              "%" PRIu64, skipped);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"status", "%d",
+                                              status_rcd);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        ret = xmlTextWriterWriteFormatElement(
+            writer, (xmlChar *)"statusStr", "%s",
+            cli_vol_task_status_str[status_rcd]);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        snprintf(key, sizeof(key), "run-time-%d", i);
+        ret = dict_get_double(dict, key, &elapsed);
+        if (ret)
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"runtime",
+                                              "%.2f", elapsed);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                /* Rebalance has 5 states,
-                 * NOT_STARTED, STARTED, STOPPED, COMPLETE, FAILED
-                 * The precedence used to determine the aggregate status is as
-                 * below,
-                 * STARTED > FAILED > STOPPED > COMPLETE > NOT_STARTED
-                 */
-                /* TODO: Move this to a common place utilities that both CLI and
-                 * glusterd need.
-                 * Till then if the below algorithm is changed, change it in
-                 * glusterd_volume_status_aggregate_tasks_status in
-                 * glusterd-utils.c
-                 */
-
-                if (-1 == overall_status)
-                        overall_status = status_rcd;
-                int rank[] = {
-                        [GF_DEFRAG_STATUS_STARTED] = 1,
-                        [GF_DEFRAG_STATUS_FAILED] = 2,
-                        [GF_DEFRAG_STATUS_STOPPED] = 3,
-                        [GF_DEFRAG_STATUS_COMPLETE] = 4,
-                        [GF_DEFRAG_STATUS_NOT_STARTED] = 5
-                };
-                if (rank[status_rcd] <= rank[overall_status])
-                        overall_status = status_rcd;
-
-                /* </node> */
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        if (elapsed > overall_elapsed) {
+            overall_elapsed = elapsed;
         }
 
-        /* Aggregate status */
-        /* <aggregate> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"aggregate");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+        /* Rebalance has 5 states,
+         * NOT_STARTED, STARTED, STOPPED, COMPLETE, FAILED
+         * The precedence used to determine the aggregate status is as
+         * below,
+         * STARTED > FAILED > STOPPED > COMPLETE > NOT_STARTED
+         */
+        /* TODO: Move this to a common place utilities that both CLI and
+         * glusterd need.
+         * Till then if the below algorithm is changed, change it in
+         * glusterd_volume_status_aggregate_tasks_status in
+         * glusterd-utils.c
+         */
 
-        ret = xmlTextWriterWriteFormatElement (writer,(xmlChar *)"files",
-                                               "%"PRIu64, total_files);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+        if (-1 == overall_status)
+            overall_status = status_rcd;
+        int rank[] = {[GF_DEFRAG_STATUS_STARTED] = 1,
+                      [GF_DEFRAG_STATUS_FAILED] = 2,
+                      [GF_DEFRAG_STATUS_STOPPED] = 3,
+                      [GF_DEFRAG_STATUS_COMPLETE] = 4,
+                      [GF_DEFRAG_STATUS_NOT_STARTED] = 5};
+        if (rank[status_rcd] <= rank[overall_status])
+            overall_status = status_rcd;
+
+        /* </node> */
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
+
+    /* Aggregate status */
+    /* <aggregate> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"aggregate");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"files",
+                                          "%" PRIu64, total_files);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = xmlTextWriterWriteFormatElement (writer,(xmlChar *)"size",
-                                               "%"PRIu64, total_size);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"size", "%" PRIu64,
+                                          total_size);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = xmlTextWriterWriteFormatElement (writer,(xmlChar *)"lookups",
-                                               "%"PRIu64, total_lookups);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"lookups",
+                                          "%" PRIu64, total_lookups);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = xmlTextWriterWriteFormatElement (writer,(xmlChar *)"failures",
-                                               "%"PRIu64, total_failures);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"failures",
+                                          "%" PRIu64, total_failures);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = xmlTextWriterWriteFormatElement (writer,(xmlChar *)"skipped",
-                                               "%"PRIu64, total_skipped);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"skipped",
+                                          "%" PRIu64, total_skipped);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = xmlTextWriterWriteFormatElement (writer,(xmlChar *)"status",
-                                               "%d", overall_status);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    if (overall_status == -1) {
+        overall_status = status_rcd;
+    }
 
-        ret = xmlTextWriterWriteFormatElement (writer,(xmlChar *)"statusStr",
-                                               "%s",
-                                      cli_vol_task_status_str[overall_status]);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"status", "%d",
+                                          overall_status);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = xmlTextWriterWriteFormatElement (writer,(xmlChar *)"runtime",
-                                               "%.2f", overall_elapsed);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = xmlTextWriterWriteFormatElement(
+        writer, (xmlChar *)"statusStr", "%s",
+        cli_vol_task_status_str[overall_status]);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        /* </aggregate> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"runtime", "%.2f",
+                                          overall_elapsed);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* </aggregate> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 #endif
 
 int
-cli_xml_output_vol_rebalance (gf_cli_defrag_type op, dict_t *dict, int op_ret,
-                              int op_errno, char *op_errstr)
+cli_xml_output_vol_rebalance(gf_cli_defrag_type op, dict_t *dict, int op_ret,
+                             int op_errno, char *op_errstr)
 {
 #if (HAVE_LIB_XML)
-        int                     ret = -1;
-        xmlTextWriterPtr        writer = NULL;
-        xmlDocPtr               doc = NULL;
-        char                    *task_id_str = NULL;
+    int ret = -1;
+    xmlTextWriterPtr writer = NULL;
+    xmlDocPtr doc = NULL;
+    char *task_id_str = NULL;
+
+    ret = cli_begin_xml_output(&writer, &doc);
+    if (ret)
+        goto out;
+
+    ret = cli_xml_output_common(writer, op_ret, op_errno, op_errstr);
+    if (ret)
+        goto out;
+
+    /* <volRebalance> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"volRebalance");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_str(dict, GF_REBALANCE_TID_KEY, &task_id_str);
+    if (ret == 0) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"task-id",
+                                              "%s", task_id_str);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        ret = cli_begin_xml_output (&writer, &doc);
-        if (ret)
-                goto out;
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"op", "%d", op);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = cli_xml_output_common (writer, op_ret, op_errno, op_errstr);
+    if ((GF_DEFRAG_CMD_STOP == op) || (GF_DEFRAG_CMD_STATUS == op)) {
+        ret = cli_xml_output_vol_rebalance_status(writer, dict,
+                                                  GF_TASK_TYPE_REBALANCE);
         if (ret)
-                goto out;
-
-        /* <volRebalance> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"volRebalance");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = dict_get_str (dict, GF_REBALANCE_TID_KEY, &task_id_str);
-        if (ret == 0) {
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"task-id",
-                                                       "%s", task_id_str);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
-
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"op",
-                                               "%d", op);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        if ((GF_DEFRAG_CMD_STOP == op) || (GF_DEFRAG_CMD_STATUS == op)) {
-                ret = cli_xml_output_vol_rebalance_status (writer, dict,
-                                                      GF_TASK_TYPE_REBALANCE);
-                if (ret)
-                        goto out;
-        }
+            goto out;
+    }
 
-        /* </volRebalance> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </volRebalance> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-
-        ret = cli_end_xml_output (writer, doc);
+    ret = cli_end_xml_output(writer, doc);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif
 }
 
 int
-cli_xml_output_vol_remove_brick (gf_boolean_t status_op, dict_t *dict,
-                                 int op_ret, int op_errno, char *op_errstr)
+cli_xml_output_vol_remove_brick(gf_boolean_t status_op, dict_t *dict,
+                                int op_ret, int op_errno, char *op_errstr,
+                                const char *op)
 {
 #if (HAVE_LIB_XML)
-        int                     ret = -1;
-        xmlTextWriterPtr        writer = NULL;
-        xmlDocPtr               doc = NULL;
-        char                    *task_id_str = NULL;
-
-        ret = cli_begin_xml_output (&writer, &doc);
-        if (ret)
-                goto out;
+    int ret = -1;
+    xmlTextWriterPtr writer = NULL;
+    xmlDocPtr doc = NULL;
+    char *task_id_str = NULL;
+
+    ret = cli_begin_xml_output(&writer, &doc);
+    if (ret)
+        goto out;
+
+    ret = cli_xml_output_common(writer, op_ret, op_errno, op_errstr);
+    if (ret)
+        goto out;
+
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)op);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_str(dict, GF_REMOVE_BRICK_TID_KEY, &task_id_str);
+    if (ret == 0) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"task-id",
+                                              "%s", task_id_str);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        ret = cli_xml_output_common (writer, op_ret, op_errno, op_errstr);
+    if (status_op) {
+        ret = cli_xml_output_vol_rebalance_status(writer, dict,
+                                                  GF_TASK_TYPE_REMOVE_BRICK);
         if (ret)
-                goto out;
-
-        /* <volRemoveBrick> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"volRemoveBrick");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = dict_get_str (dict, GF_REMOVE_BRICK_TID_KEY, &task_id_str);
-        if (ret == 0) {
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"task-id",
-                                                       "%s", task_id_str);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
-
-        if (status_op) {
-                ret = cli_xml_output_vol_rebalance_status (writer, dict,
-                                                   GF_TASK_TYPE_REMOVE_BRICK);
-                if (ret)
-                        goto out;
-        }
+            goto out;
+    }
 
-        /* </volRemoveBrick> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-
-        ret = cli_end_xml_output (writer, doc);
+    ret = cli_end_xml_output(writer, doc);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif
 }
 
 int
-cli_xml_output_vol_replace_brick (gf1_cli_replace_op op, dict_t *dict,
-                                  int op_ret, int op_errno, char *op_errstr)
+cli_xml_output_vol_replace_brick(dict_t *dict, int op_ret, int op_errno,
+                                 char *op_errstr)
 {
 #if (HAVE_LIB_XML)
-        int                     ret = -1;
-        int                     status = 0;
-        uint64_t                files = 0;
-        char                    *current_file = 0;
-        char                    *task_id_str = NULL;
-        xmlTextWriterPtr        writer = NULL;
-        xmlDocPtr               doc = NULL;
-
-        ret = cli_begin_xml_output (&writer, &doc);
-        if (ret)
-                goto out;
-
-        ret = cli_xml_output_common (writer, op_ret, op_errno, op_errstr);
-        if (ret)
-                goto out;
-
-        /* <volReplaceBrick> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"volReplaceBrick");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    int ret = -1;
+    xmlTextWriterPtr writer = NULL;
+    xmlDocPtr doc = NULL;
 
-        ret = dict_get_str (dict, GF_REPLACE_BRICK_TID_KEY, &task_id_str);
-        if (ret == 0) {
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"task-id",
-                                                       "%s", task_id_str);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
-
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"op",
-                                               "%d", op);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = cli_begin_xml_output(&writer, &doc);
+    if (ret)
+        goto out;
 
-        if (GF_REPLACE_OP_STATUS == op) {
-                ret = dict_get_int32 (dict, "status", &status);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"status",
-                                                       "%d", status);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = cli_xml_output_common(writer, op_ret, op_errno, op_errstr);
+    if (ret)
+        goto out;
 
-                ret = dict_get_uint64 (dict, "files", &files);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"files",
-                                                       "%"PRIu64, files);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                if (status)
-                        goto cont;
-
-                ret = dict_get_str (dict, "current_file", &current_file);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"currentFile",
-                                                       "%s", current_file);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
-cont:
-        /* </volReplaceBrick> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = cli_end_xml_output (writer, doc);
+    ret = cli_end_xml_output(writer, doc);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif
 }
 
 int
-cli_xml_output_vol_create (dict_t *dict, int op_ret, int op_errno,
-                           char *op_errstr)
+cli_xml_output_vol_create(dict_t *dict, int op_ret, int op_errno,
+                          char *op_errstr)
 {
 #if (HAVE_LIB_XML)
-        int                     ret = -1;
-        xmlTextWriterPtr        writer = NULL;
-        xmlDocPtr               doc = NULL;
-        char                    *volname = NULL;
-        char                    *volid = NULL;
+    int ret = -1;
+    xmlTextWriterPtr writer = NULL;
+    xmlDocPtr doc = NULL;
+    char *volname = NULL;
+    char *volid = NULL;
+
+    ret = cli_begin_xml_output(&writer, &doc);
+    if (ret)
+        goto out;
+
+    ret = cli_xml_output_common(writer, op_ret, op_errno, op_errstr);
+    if (ret)
+        goto out;
+
+    if (dict) {
+        /* <volCreate> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"volCreate");
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = cli_begin_xml_output (&writer, &doc);
-        if (ret)
-                goto out;
+        /* <volume> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"volume");
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = cli_xml_output_common (writer, op_ret, op_errno, op_errstr);
+        ret = dict_get_str(dict, "volname", &volname);
         if (ret)
-                goto out;
-
-        if (dict) {
-                /* <volCreate> */
-                ret = xmlTextWriterStartElement (writer,
-                                                 (xmlChar *)"volCreate");
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                /* <volume> */
-                ret = xmlTextWriterStartElement (writer, (xmlChar *)"volume");
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                ret = dict_get_str (dict, "volname", &volname);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *) "name",
-                                                       "%s", volname);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"name", "%s",
+                                              volname);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                ret = dict_get_str (dict, "volume-id", &volid);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"id",
-                                                       "%s", volid);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        ret = dict_get_str(dict, "volume-id", &volid);
+        if (ret)
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"id", "%s",
+                                              volid);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                /* </volume> */
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        /* </volume> */
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                /* </volCreate> */
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+        /* </volCreate> */
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        ret = cli_end_xml_output (writer, doc);
+    ret = cli_end_xml_output(writer, doc);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif
 }
 
 int
-cli_xml_output_generic_volume (char *op, dict_t *dict, int op_ret, int op_errno,
-                               char *op_errstr)
+cli_xml_output_generic_volume(char *op, dict_t *dict, int op_ret, int op_errno,
+                              char *op_errstr)
 {
 #if (HAVE_LIB_XML)
-        int                     ret = -1;
-        xmlTextWriterPtr        writer = NULL;
-        xmlDocPtr               doc = NULL;
-        char                    *volname = NULL;
-        char                    *volid = NULL;
+    int ret = -1;
+    xmlTextWriterPtr writer = NULL;
+    xmlDocPtr doc = NULL;
+    char *volname = NULL;
+    char *volid = NULL;
 
-        GF_ASSERT (op);
+    GF_ASSERT(op);
 
-        ret = cli_begin_xml_output (&writer, &doc);
-        if (ret)
-                goto out;
+    ret = cli_begin_xml_output(&writer, &doc);
+    if (ret)
+        goto out;
 
-        ret = cli_xml_output_common (writer, op_ret, op_errno, op_errstr);
-        if (ret)
-                goto out;
+    ret = cli_xml_output_common(writer, op_ret, op_errno, op_errstr);
+    if (ret)
+        goto out;
 
-        if (dict) {
-                /* <"op"> */
-                ret = xmlTextWriterStartElement (writer, (xmlChar *)op);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+    if (dict) {
+        /* <"op"> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)op);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                /* <volume> */
-                ret = xmlTextWriterStartElement (writer, (xmlChar *)"volume");
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        /* <volume> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"volume");
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                ret = dict_get_str (dict, "volname", &volname);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *) "name",
-                                                       "%s", volname);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        ret = dict_get_str(dict, "volname", &volname);
+        if (ret)
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"name", "%s",
+                                              volname);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                ret = dict_get_str (dict, "vol-id", &volid);
-                if (ret)
-                        goto out;
-                ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"id",
-                                                       "%s", volid);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        ret = dict_get_str(dict, "vol-id", &volid);
+        if (ret)
+            goto out;
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"id", "%s",
+                                              volid);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                /* </volume> */
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        /* </volume> */
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                /* </"op"> */
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+        /* </"op"> */
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        ret = cli_end_xml_output (writer, doc);
+    ret = cli_end_xml_output(writer, doc);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif
 }
 
 #if (HAVE_LIB_XML)
 int
-_output_gsync_config (FILE *fp, xmlTextWriterPtr writer, char *op_name)
+_output_gsync_config(FILE *fp, xmlTextWriterPtr writer, char *op_name)
 {
-        char  resbuf[256 + PATH_MAX] = {0,};
-        char *ptr                    = NULL;
-        char *v                      = NULL;
-        int   blen                   = sizeof(resbuf);
-        int   ret                    = 0;
-
-        for (;;) {
-                ptr = fgets (resbuf, blen, fp);
-                if (!ptr)
-                        break;
-
-                v = resbuf + strlen (resbuf) - 1;
-                while (isspace (*v)) {
-                        /* strip trailing space */
-                        *v-- = '\0';
-                }
-                if (v == resbuf) {
-                        /* skip empty line */
-                        continue;
-                }
+    char resbuf[256 + PATH_MAX] = {
+        0,
+    };
+    char *ptr = NULL;
+    char *v = NULL;
+    int blen = sizeof(resbuf);
+    int ret = 0;
+
+    for (;;) {
+        ptr = fgets(resbuf, blen, fp);
+        if (!ptr)
+            break;
 
-                if (op_name!= NULL){
-                        ret = xmlTextWriterWriteFormatElement (writer,
-                                                        (xmlChar *)op_name,
-                                                        "%s", resbuf);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-                        goto out;
-                }
+        v = resbuf + strlen(resbuf) - 1;
+        while (isspace(*v)) {
+            /* strip trailing space */
+            *v-- = '\0';
+        }
+        if (v == resbuf) {
+            /* skip empty line */
+            continue;
+        }
 
-                v = strchr (resbuf, ':');
-                if (!v) {
-                        ret = -1;
-                        goto out;
-                }
-                *v++ = '\0';
-                while (isspace (*v))
-                        v++;
-                v = gf_strdup (v);
-                if (!v) {
-                        ret = -1;
-                        goto out;
-                }
+        if (op_name != NULL) {
+            ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)op_name,
+                                                  "%s", resbuf);
+            XML_RET_CHECK_AND_GOTO(ret, out);
+            goto out;
+        }
 
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)resbuf,
-                                                       "%s", v);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        v = strchr(resbuf, ':');
+        if (!v) {
+            ret = -1;
+            goto out;
+        }
+        *v++ = '\0';
+        while (isspace(*v))
+            v++;
+        v = gf_strdup(v);
+        if (!v) {
+            ret = -1;
+            goto out;
         }
+
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)resbuf, "%s",
+                                              v);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 #endif
 
 #if (HAVE_LIB_XML)
 int
-get_gsync_config (runner_t *runner,
-                  int (*op_conf)(FILE *fp,
-                              xmlTextWriterPtr writer,
-                              char *op_name),
-                  xmlTextWriterPtr writer, char *op_name)
+get_gsync_config(runner_t *runner,
+                 int (*op_conf)(FILE *fp, xmlTextWriterPtr writer,
+                                char *op_name),
+                 xmlTextWriterPtr writer, char *op_name)
 {
-        int ret = 0;
+    int ret = 0;
 
-        runner_redir (runner, STDOUT_FILENO, RUN_PIPE);
-        if (runner_start (runner) != 0) {
-                gf_log ("cli", GF_LOG_ERROR, "spawning child failed");
-                return -1;
-        }
+    runner_redir(runner, STDOUT_FILENO, RUN_PIPE);
+    if (runner_start(runner) != 0) {
+        gf_log("cli", GF_LOG_ERROR, "spawning child failed");
+        return -1;
+    }
 
-        ret = op_conf (runner_chio (runner, STDOUT_FILENO), writer, op_name);
+    ret = op_conf(runner_chio(runner, STDOUT_FILENO), writer, op_name);
 
-        ret |= runner_end (runner);
-        if (ret)
-                gf_log ("cli", GF_LOG_ERROR, "reading data from child failed");
+    ret |= runner_end(runner);
+    if (ret)
+        gf_log("cli", GF_LOG_ERROR, "reading data from child failed");
 
-        return ret ? -1 : 0;
+    return ret ? -1 : 0;
 }
 #endif
 
 #if (HAVE_LIB_XML)
 int
-cli_xml_generate_gsync_config (dict_t *dict, xmlTextWriterPtr writer)
+cli_xml_generate_gsync_config(dict_t *dict, xmlTextWriterPtr writer)
 {
-        runner_t runner           = {0,};
-        char *subop               = NULL;
-        char *gwd                 = NULL;
-        char *slave               = NULL;
-        char *confpath            = NULL;
-        char *master              = NULL;
-        char *op_name             = NULL;
-        int   ret                 = -1;
-        char  conf_path[PATH_MAX] = "";
-
-        if (dict_get_str (dict, "subop", &subop) != 0) {
-                ret = -1;
-                goto out;
-        }
+    runner_t runner = {
+        0,
+    };
+    char *subop = NULL;
+    char *gwd = NULL;
+    char *slave = NULL;
+    char *confpath = NULL;
+    char *master = NULL;
+    char *op_name = NULL;
+    int ret = -1;
+    char conf_path[PATH_MAX] = "";
+
+    if (dict_get_str(dict, "subop", &subop) != 0) {
+        ret = -1;
+        goto out;
+    }
+
+    if (strcmp(subop, "get") != 0 && strcmp(subop, "get-all") != 0) {
+        ret = xmlTextWriterWriteFormatElement(
+            writer, (xmlChar *)"message", "%s",
+            GEOREP " config updated successfully");
+        XML_RET_CHECK_AND_GOTO(ret, out);
+        ret = 0;
+        goto out;
+    }
 
-        if (strcmp (subop, "get") != 0 && strcmp (subop, "get-all") != 0) {
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                 (xmlChar *)"message",
-                                 "%s",GEOREP" config updated successfully" );
-                XML_RET_CHECK_AND_GOTO (ret, out);
-                ret = 0;
-                goto out;
-        }
+    if (dict_get_str(dict, "glusterd_workdir", &gwd) != 0 ||
+        dict_get_str(dict, "slave", &slave) != 0) {
+        ret = -1;
+        goto out;
+    }
 
-        if (dict_get_str (dict, "glusterd_workdir", &gwd) != 0 ||
-            dict_get_str (dict, "slave", &slave) != 0) {
-                ret = -1;
-                goto out;
-        }
+    if (dict_get_str(dict, "master", &master) != 0)
+        master = NULL;
 
-        if (dict_get_str (dict, "master", &master) != 0)
-                master = NULL;
+    if (dict_get_str(dict, "op_name", &op_name) != 0)
+        op_name = NULL;
 
-        if (dict_get_str (dict, "op_name", &op_name) != 0)
-                op_name = NULL;
+    ret = dict_get_str(dict, "conf_path", &confpath);
+    if (!confpath) {
+        ret = snprintf(conf_path, sizeof(conf_path) - 1,
+                       "%s/" GEOREP "/gsyncd_template.conf", gwd);
+        conf_path[ret] = '\0';
+        confpath = conf_path;
+    }
 
-        ret = dict_get_str (dict, "conf_path", &confpath);
-        if (!confpath) {
-                ret = snprintf (conf_path, sizeof (conf_path) - 1,
-                                "%s/"GEOREP"/gsyncd_template.conf", gwd);
-                conf_path[ret] = '\0';
-                confpath = conf_path;
-        }
+    runinit(&runner);
+    runner_add_args(&runner, GSYNCD_PREFIX "/gsyncd", "-c", NULL);
+    runner_argprintf(&runner, "%s", confpath);
+    runner_argprintf(&runner, "--iprefix=%s", DATADIR);
 
-        runinit (&runner);
-        runner_add_args (&runner, GSYNCD_PREFIX"/gsyncd", "-c", NULL);
-        runner_argprintf (&runner, "%s", confpath);
-        runner_argprintf (&runner, "--iprefix=%s", DATADIR);
+    if (master)
+        runner_argprintf(&runner, ":%s", master);
 
-        if (master)
-                runner_argprintf (&runner, ":%s", master);
+    runner_add_arg(&runner, slave);
+    runner_argprintf(&runner, "--config-%s", subop);
 
-        runner_add_arg (&runner, slave);
-        runner_argprintf (&runner, "--config-%s", subop);
+    if (op_name)
+        runner_add_arg(&runner, op_name);
 
-        if (op_name)
-                runner_add_arg (&runner, op_name);
-
-        ret =  get_gsync_config (&runner, _output_gsync_config,
-                                 writer, op_name);
+    ret = get_gsync_config(&runner, _output_gsync_config, writer, op_name);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 #endif
 
 #if (HAVE_LIB_XML)
 int
-gf_gsync_status_t_comparator (const void *p, const void *q)
+cli_xml_output_vol_gsync_status(dict_t *dict, xmlTextWriterPtr writer)
 {
-        char *master1 = NULL;
-        char *master2 = NULL;
-
-        master1 = get_struct_variable (1, (*(gf_gsync_status_t **)p));
-        master2 = get_struct_variable (1, (*(gf_gsync_status_t **)q));
-        if (!master1 || !master2) {
-                gf_log ("cli", GF_LOG_ERROR,
-                        "struct member empty.");
-                return 0;
+    int ret = -1;
+    int i = 1;
+    int j = 0;
+    int count = 0;
+    const int number_of_fields = 20;
+    int closed = 1;
+    int session_closed = 1;
+    gf_gsync_status_t **status_values = NULL;
+    char status_value_name[PATH_MAX] = "";
+    char *tmp = NULL;
+    char *volume = NULL;
+    char *volume_next = NULL;
+    char *slave = NULL;
+    char *slave_next = NULL;
+    static const char *title_values[] = {
+        "master_node", "", "master_brick", "slave_user", "slave", "slave_node",
+        "status", "crawl_status",
+        /* last_synced */
+        "", "entry", "data", "meta", "failures",
+        /* checkpoint_time */
+        "", "checkpoint_completed",
+        /* checkpoint_completion_time */
+        "", "master_node_uuid",
+        /* last_synced_utc */
+        "last_synced",
+        /* checkpoint_time_utc */
+        "checkpoint_time",
+        /* checkpoint_completion_time_utc */
+        "checkpoint_completion_time"};
+
+    GF_ASSERT(dict);
+
+    ret = dict_get_int32(dict, "gsync-count", &count);
+    if (ret)
+        goto out;
+
+    status_values = GF_MALLOC(count * sizeof(gf_gsync_status_t *),
+                              gf_common_mt_char);
+    if (!status_values) {
+        ret = -1;
+        goto out;
+    }
+
+    for (i = 0; i < count; i++) {
+        status_values[i] = GF_CALLOC(1, sizeof(gf_gsync_status_t),
+                                     gf_common_mt_char);
+        if (!status_values[i]) {
+            ret = -1;
+            goto out;
         }
 
-        return strcmp (master1,master2);
-}
-#endif
+        snprintf(status_value_name, sizeof(status_value_name), "status_value%d",
+                 i);
 
-#if (HAVE_LIB_XML)
-int
-cli_xml_output_vol_gsync_status (dict_t *dict,
-                                 xmlTextWriterPtr writer)
-{
-        int                  ret                         = -1;
-        int                  i                           = 1;
-        int                  j                           = 0;
-        int                  count                       = 0;
-        const int            number_of_fields            = 12;
-        const int            number_of_basic_fields      = 7;
-        int                  closed                      = 1;
-        int                  session_closed              = 1;
-        gf_gsync_status_t  **status_values               = NULL;
-        gf_boolean_t         status_detail               = _gf_false;
-        char                 status_value_name[PATH_MAX] = "";
-        char                *tmp                         = NULL;
-        char                *volume                      = NULL;
-        char                *volume_next                 = NULL;
-        char                *slave                       = NULL;
-        char                *slave_next                  = NULL;
-        char                *title_values[]              = {"master_node",
-                                                            "master_node_uuid",
-                                                            "master_brick",
-                                                            "slave",
-                                                            "status",
-                                                            "checkpoint_status",
-                                                            "crawl_status",
-                                                            "files_syncd",
-                                                            "files_pending",
-                                                            "bytes_pending",
-                                                            "deletes_pending",
-                                                            "files_skipped"};
-
-        GF_ASSERT (dict);
-
-        ret = dict_get_int32 (dict, "gsync-count", &count);
-        if (ret)
-                goto out;
+        ret = dict_get_bin(dict, status_value_name,
+                           (void **)&(status_values[i]));
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, "struct member empty.");
+            goto out;
+        }
+    }
 
-        status_detail = dict_get_str_boolean (dict, "status-detail",
-                                              _gf_false);
+    qsort(status_values, count, sizeof(gf_gsync_status_t *),
+          gf_gsync_status_t_comparator);
 
-        status_values = GF_CALLOC (count, sizeof (gf_gsync_status_t *),
-                              gf_common_mt_char);
-        if (!status_values) {
+    for (i = 0; i < count; i++) {
+        if (closed) {
+            ret = xmlTextWriterStartElement(writer, (xmlChar *)"volume");
+            XML_RET_CHECK_AND_GOTO(ret, out);
+
+            tmp = get_struct_variable(1, status_values[i]);
+            if (!tmp) {
+                gf_log("cli", GF_LOG_ERROR, "struct member empty.");
                 ret = -1;
                 goto out;
-        }
+            }
 
-        for (i = 0; i < count; i++) {
-                status_values[i] = GF_CALLOC (1, sizeof (gf_gsync_status_t),
-                                         gf_common_mt_char);
-                if (!status_values[i]) {
-                        ret = -1;
-                        goto out;
-                }
+            ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"name",
+                                                  "%s", tmp);
+            XML_RET_CHECK_AND_GOTO(ret, out);
 
-                snprintf (status_value_name, sizeof (status_value_name),
-                          "status_value%d", i);
+            ret = xmlTextWriterStartElement(writer, (xmlChar *)"sessions");
+            XML_RET_CHECK_AND_GOTO(ret, out);
 
-                ret = dict_get_bin (dict, status_value_name,
-                                    (void **)&(status_values[i]));
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "struct member empty.");
-                        goto out;
-                }
+            closed = 0;
         }
 
-        qsort(status_values, count, sizeof (gf_gsync_status_t *),
-              gf_gsync_status_t_comparator);
-
-        for (i = 0; i < count; i++) {
-                if (closed) {
-                        ret = xmlTextWriterStartElement (writer,
-                                                         (xmlChar *)"volume");
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-
-                        tmp = get_struct_variable (1, status_values[i]);
-                        if (!tmp) {
-                                gf_log ("cli", GF_LOG_ERROR,
-                                        "struct member empty.");
-                                ret = -1;
-                                goto out;
-                        }
-
-                        ret = xmlTextWriterWriteFormatElement (writer,
-                                                        (xmlChar *)"name",
-                                                        "%s",tmp);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-
-                        ret = xmlTextWriterStartElement (writer,
-                                                    (xmlChar *)"sessions");
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-
-                        closed = 0;
-                }
+        if (session_closed) {
+            ret = xmlTextWriterStartElement(writer, (xmlChar *)"session");
+            XML_RET_CHECK_AND_GOTO(ret, out);
 
-                if (session_closed) {
-                        ret = xmlTextWriterStartElement (writer,
-                                                         (xmlChar *)"session");
-                        XML_RET_CHECK_AND_GOTO (ret, out);
+            session_closed = 0;
 
-                        session_closed = 0;
+            tmp = get_struct_variable(21, status_values[i]);
+            if (!tmp) {
+                gf_log("cli", GF_LOG_ERROR, "struct member empty.");
+                ret = -1;
+                goto out;
+            }
 
-                        tmp = get_struct_variable (14, status_values[i]);
-                        if (!tmp) {
-                                gf_log ("cli", GF_LOG_ERROR,
-                                        "struct member empty.");
-                                ret = -1;
-                                goto out;
-                        }
+            ret = xmlTextWriterWriteFormatElement(
+                writer, (xmlChar *)"session_slave", "%s", tmp);
+            XML_RET_CHECK_AND_GOTO(ret, out);
+        }
 
-                        ret = xmlTextWriterWriteFormatElement
-                                (writer, (xmlChar *)"session_slave", "%s", tmp);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-                }
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"pair");
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                ret = xmlTextWriterStartElement (writer, (xmlChar *)"pair");
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                for (j = 0; j < number_of_fields; j++) {
-                  // if detail option is not set and field is not under
-                  // basic fields or if field is volume then skip
-                        if(!status_detail && j >= number_of_basic_fields)
-                                continue;
-
-                        // Displaying the master_node uuid as second field
-
-                        if (j == 1)
-                                tmp = get_struct_variable (12,
-                                                           status_values[i]);
-                        else
-                                tmp = get_struct_variable (j, status_values[i]);
-                        if (!tmp) {
-                                gf_log ("cli", GF_LOG_ERROR,
-                                        "struct member empty.");
-                                ret = -1;
-                                goto out;
-                        }
-
-                        ret = xmlTextWriterWriteFormatElement (writer,
-                                                  (xmlChar *)title_values[j],
-                                                  "%s", tmp);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-                }
+        for (j = 0; j < number_of_fields; j++) {
+            /* XML ignore fields */
+            if (strcmp(title_values[j], "") == 0)
+                continue;
 
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+            tmp = get_struct_variable(j, status_values[i]);
+            if (!tmp) {
+                gf_log("cli", GF_LOG_ERROR, "struct member empty.");
+                ret = -1;
+                goto out;
+            }
 
-                if (i+1 < count) {
-                        slave = get_struct_variable (13, status_values[i]);
-                        slave_next = get_struct_variable (13,
-                                                          status_values[i+1]);
-                        volume = get_struct_variable (1, status_values[i]);
-                        volume_next = get_struct_variable (1,
-                                                           status_values[i+1]);
-                        if (!slave || !slave_next || !volume || !volume_next) {
-                                gf_log ("cli", GF_LOG_ERROR,
-                                        "struct member empty.");
-                                ret = -1;
-                                goto out;
-                        }
+            ret = xmlTextWriterWriteFormatElement(
+                writer, (xmlChar *)title_values[j], "%s", tmp);
+            XML_RET_CHECK_AND_GOTO(ret, out);
+        }
 
-                        if (strcmp (volume, volume_next)!=0) {
-                                closed = 1;
-                                session_closed = 1;
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                                ret = xmlTextWriterEndElement (writer);
-                                XML_RET_CHECK_AND_GOTO (ret, out);
+        if (i + 1 < count) {
+            slave = get_struct_variable(20, status_values[i]);
+            slave_next = get_struct_variable(20, status_values[i + 1]);
+            volume = get_struct_variable(1, status_values[i]);
+            volume_next = get_struct_variable(1, status_values[i + 1]);
+            if (!slave || !slave_next || !volume || !volume_next) {
+                gf_log("cli", GF_LOG_ERROR, "struct member empty.");
+                ret = -1;
+                goto out;
+            }
 
-                                ret = xmlTextWriterEndElement (writer);
-                                XML_RET_CHECK_AND_GOTO (ret, out);
+            if (strcmp(volume, volume_next) != 0) {
+                closed = 1;
+                session_closed = 1;
 
-                                ret = xmlTextWriterEndElement (writer);
-                                XML_RET_CHECK_AND_GOTO (ret, out);
-                        } else if (strcmp (slave, slave_next)!=0) {
+                ret = xmlTextWriterEndElement(writer);
+                XML_RET_CHECK_AND_GOTO(ret, out);
 
-                                session_closed = 1;
+                ret = xmlTextWriterEndElement(writer);
+                XML_RET_CHECK_AND_GOTO(ret, out);
 
-                                ret = xmlTextWriterEndElement (writer);
-                                XML_RET_CHECK_AND_GOTO (ret, out);
-                        }
-                } else {
+                ret = xmlTextWriterEndElement(writer);
+                XML_RET_CHECK_AND_GOTO(ret, out);
+            } else if (strcmp(slave, slave_next) != 0) {
+                session_closed = 1;
 
-                        ret = xmlTextWriterEndElement (writer);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
+                ret = xmlTextWriterEndElement(writer);
+                XML_RET_CHECK_AND_GOTO(ret, out);
+            }
+        } else {
+            ret = xmlTextWriterEndElement(writer);
+            XML_RET_CHECK_AND_GOTO(ret, out);
 
-                        ret = xmlTextWriterEndElement (writer);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
+            ret = xmlTextWriterEndElement(writer);
+            XML_RET_CHECK_AND_GOTO(ret, out);
 
-                        ret = xmlTextWriterEndElement (writer);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-                }
+            ret = xmlTextWriterEndElement(writer);
+            XML_RET_CHECK_AND_GOTO(ret, out);
         }
+    }
 out:
-        gf_log ("cli",GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    if (status_values)
+        GF_FREE(status_values);
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
 #endif
 
 int
-cli_xml_output_vol_gsync (dict_t *dict, int op_ret, int op_errno,
-                          char *op_errstr)
+cli_xml_output_vol_gsync(dict_t *dict, int op_ret, int op_errno,
+                         char *op_errstr)
 {
 #if (HAVE_LIB_XML)
-        int                     ret = -1;
-        xmlTextWriterPtr        writer = NULL;
-        xmlDocPtr               doc = NULL;
-        char                    *master = NULL;
-        char                    *slave = NULL;
-        int                     type = 0;
-
-        GF_ASSERT (dict);
-
-        ret = cli_begin_xml_output (&writer, &doc);
-        if (ret)
-                goto out;
-
-        ret = cli_xml_output_common (writer, op_ret, op_errno, op_errstr);
-        if (ret)
-                goto out;
-
-        /* <geoRep> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"geoRep");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = dict_get_int32 (dict, "type", &type);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get type");
-                goto out;
-        }
-
-        switch (type) {
+    int ret = -1;
+    xmlTextWriterPtr writer = NULL;
+    xmlDocPtr doc = NULL;
+    char *master = NULL;
+    char *slave = NULL;
+    int type = 0;
+
+    GF_ASSERT(dict);
+
+    ret = cli_begin_xml_output(&writer, &doc);
+    if (ret)
+        goto out;
+
+    ret = cli_xml_output_common(writer, op_ret, op_errno, op_errstr);
+    if (ret)
+        goto out;
+
+    /* <geoRep> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"geoRep");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_int32(dict, "type", &type);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get type");
+        goto out;
+    }
+
+    switch (type) {
         case GF_GSYNC_OPTION_TYPE_START:
         case GF_GSYNC_OPTION_TYPE_STOP:
         case GF_GSYNC_OPTION_TYPE_PAUSE:
         case GF_GSYNC_OPTION_TYPE_RESUME:
         case GF_GSYNC_OPTION_TYPE_CREATE:
         case GF_GSYNC_OPTION_TYPE_DELETE:
-                if (dict_get_str (dict, "master", &master) != 0)
-                        master = "???";
-                if (dict_get_str (dict, "slave", &slave) != 0)
-                        slave = "???";
+            if (dict_get_str(dict, "master", &master) != 0)
+                master = "???";
+            if (dict_get_str(dict, "slave", &slave) != 0)
+                slave = "???";
 
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"master",
-                                                       "%s", master);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+            ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"master",
+                                                  "%s", master);
+            XML_RET_CHECK_AND_GOTO(ret, out);
 
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"slave",
-                                                       "%s", slave);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+            ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"slave",
+                                                  "%s", slave);
+            XML_RET_CHECK_AND_GOTO(ret, out);
 
-                break;
+            break;
 
         case GF_GSYNC_OPTION_TYPE_CONFIG:
-                if (op_ret == 0) {
-                        ret = xmlTextWriterStartElement (writer, (xmlChar *)"config");
-                        XML_RET_CHECK_AND_GOTO (ret, out);
+            if (op_ret == 0) {
+                ret = xmlTextWriterStartElement(writer, (xmlChar *)"config");
+                XML_RET_CHECK_AND_GOTO(ret, out);
 
-                        ret = cli_xml_generate_gsync_config (dict, writer);
-                        if (ret)
-                                goto out;
+                ret = cli_xml_generate_gsync_config(dict, writer);
+                if (ret)
+                    goto out;
 
-                        ret = xmlTextWriterEndElement (writer);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-                }
+                ret = xmlTextWriterEndElement(writer);
+                XML_RET_CHECK_AND_GOTO(ret, out);
+            }
 
-                break;
+            break;
         case GF_GSYNC_OPTION_TYPE_STATUS:
-                ret = cli_xml_output_vol_gsync_status (dict, writer);
-                break;
+            ret = cli_xml_output_vol_gsync_status(dict, writer);
+            if (ret) {
+                gf_log("cli", GF_LOG_DEBUG, "Failed to get gsync status");
+                goto out;
+            }
+            break;
         default:
-                ret = 0;
-                break;
-        }
+            ret = 0;
+            break;
+    }
 
-        /* </geoRep> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </geoRep> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = cli_end_xml_output (writer, doc);
+    ret = cli_end_xml_output(writer, doc);
 out:
-        gf_log ("cli",GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif
 }
 
@@ -4203,114 +3842,112 @@ out:
  * @return 0 on success and -1 on failure
  */
 static int
-cli_xml_snapshot_create (xmlTextWriterPtr writer, xmlDocPtr doc, dict_t *dict)
+cli_xml_snapshot_create(xmlTextWriterPtr writer, xmlDocPtr doc, dict_t *dict)
 {
-        int     ret             = -1;
-        char   *str_value       = NULL;
-
-        GF_ASSERT (writer);
-        GF_ASSERT (doc);
-        GF_ASSERT (dict);
-
-        /* <snapCreate> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"snapCreate");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        /* <snapshot> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"snapshot");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = dict_get_str (dict, "snapname", &str_value);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get snap name");
-                goto out;
-        }
-
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *) "name",
-                                               "%s", str_value);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = dict_get_str (dict, "snapuuid", &str_value);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get snap uuid");
-                goto out;
-        }
-
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *) "uuid",
-                                               "%s", str_value);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        /* </snapshot> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        /* </snapCreate> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = 0;
+    int ret = -1;
+    char *str_value = NULL;
+
+    GF_ASSERT(writer);
+    GF_ASSERT(doc);
+    GF_ASSERT(dict);
+
+    /* <snapCreate> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"snapCreate");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* <snapshot> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"snapshot");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_str(dict, "snapname", &str_value);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get snap name");
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"name", "%s",
+                                          str_value);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_str(dict, "snapuuid", &str_value);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get snap uuid");
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"uuid", "%s",
+                                          str_value);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* </snapshot> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* </snapCreate> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = 0;
 out:
-        return ret;
+    return ret;
 }
 
-/* This function will generate snapshot delete output in xml format.
+/* This function will generate snapshot clone output in xml format.
  *
  * @param writer        xmlTextWriterPtr
  * @param doc           xmlDocPtr
- * @param dict          dict containing delete output
+ * @param dict          dict containing create output
  *
  * @return 0 on success and -1 on failure
  */
 static int
-cli_xml_snapshot_delete (xmlTextWriterPtr writer, xmlDocPtr doc, dict_t *dict)
+cli_xml_snapshot_clone(xmlTextWriterPtr writer, xmlDocPtr doc, dict_t *dict)
 {
-        int     ret             = -1;
-        char   *str_value       = NULL;
-
-        GF_ASSERT (writer);
-        GF_ASSERT (doc);
-        GF_ASSERT (dict);
-
-        /* <snapDelete> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"snapDelete");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        /* <snapshot> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"snapshot");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = dict_get_str (dict, "snapname", &str_value);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get snap name");
-                goto out;
-        }
-
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *) "name",
-                                               "%s", str_value);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = dict_get_str (dict, "snapuuid", &str_value);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get snap uuid");
-                goto out;
-        }
-
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *) "uuid",
-                                               "%s", str_value);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        /* </snapshot> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        /* </snapDelete> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = 0;
+    int ret = -1;
+    char *str_value = NULL;
+
+    GF_VALIDATE_OR_GOTO("cli", writer, out);
+    GF_VALIDATE_OR_GOTO("cli", doc, out);
+    GF_VALIDATE_OR_GOTO("cli", dict, out);
+
+    /* <CloneCreate> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"CloneCreate");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* <volume> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"volume");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_str(dict, "clonename", &str_value);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get clone name");
+        goto out;
+    }
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"name", "%s",
+                                          str_value);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_str(dict, "snapuuid", &str_value);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get clone uuid");
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"uuid", "%s",
+                                          str_value);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* </volume> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* </CloneCreate> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = 0;
 out:
-
-        return ret;
+    return ret;
 }
 
 /* This function will generate snapshot restore output in xml format.
@@ -4322,84 +3959,83 @@ out:
  * @return 0 on success and -1 on failure
  */
 static int
-cli_xml_snapshot_restore (xmlTextWriterPtr writer, xmlDocPtr doc, dict_t *dict)
+cli_xml_snapshot_restore(xmlTextWriterPtr writer, xmlDocPtr doc, dict_t *dict)
 {
-        int     ret             = -1;
-        char   *str_value       = NULL;
-
-        GF_ASSERT (writer);
-        GF_ASSERT (doc);
-        GF_ASSERT (dict);
-
-        /* <snapRestore> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"snapRestore");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        /* <volume> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"volume");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = dict_get_str (dict, "volname", &str_value);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get vol name");
-                goto out;
-        }
-
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *) "name",
-                                               "%s", str_value);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = dict_get_str (dict, "volid", &str_value);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get volume id");
-                goto out;
-        }
-
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *) "uuid",
-                                               "%s", str_value);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        /* </volume> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-
-        /* <snapshot> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"snapshot");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = dict_get_str (dict, "snapname", &str_value);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get snap name");
-                goto out;
-        }
-
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *) "name",
-                                               "%s", str_value);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = dict_get_str (dict, "snapuuid", &str_value);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get snap uuid");
-                goto out;
-        }
-
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *) "uuid",
-                                               "%s", str_value);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        /* </snapshot> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        /* </snapRestore> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = 0;
+    int ret = -1;
+    char *str_value = NULL;
+
+    GF_ASSERT(writer);
+    GF_ASSERT(doc);
+    GF_ASSERT(dict);
+
+    /* <snapRestore> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"snapRestore");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* <volume> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"volume");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_str(dict, "volname", &str_value);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get vol name");
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"name", "%s",
+                                          str_value);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_str(dict, "volid", &str_value);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get volume id");
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"uuid", "%s",
+                                          str_value);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* </volume> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* <snapshot> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"snapshot");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_str(dict, "snapname", &str_value);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get snap name");
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"name", "%s",
+                                          str_value);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_str(dict, "snapuuid", &str_value);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get snap uuid");
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"uuid", "%s",
+                                          str_value);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* </snapshot> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* </snapRestore> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = 0;
 out:
 
-        return ret;
+    return ret;
 }
 
 /* This function will generate snapshot list output in xml format.
@@ -4411,57 +4047,57 @@ out:
  * @return 0 on success and -1 on failure
  */
 static int
-cli_xml_snapshot_list (xmlTextWriterPtr writer, xmlDocPtr doc, dict_t *dict)
+cli_xml_snapshot_list(xmlTextWriterPtr writer, xmlDocPtr doc, dict_t *dict)
 {
-        int     ret             = -1;
-        int     i               = 0;
-        int     snapcount       = 0;
-        char   *str_value       = NULL;
-        char    key[PATH_MAX]   = "";
-
-        GF_ASSERT (writer);
-        GF_ASSERT (doc);
-        GF_ASSERT (dict);
-
-        /* <snapList> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"snapList");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = dict_get_int32 (dict, "snapcount", &snapcount);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get snapcount");
-                goto out;
+    int ret = -1;
+    int i = 0;
+    int snapcount = 0;
+    char *str_value = NULL;
+    char key[PATH_MAX] = "";
+
+    GF_ASSERT(writer);
+    GF_ASSERT(doc);
+    GF_ASSERT(dict);
+
+    /* <snapList> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"snapList");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_int32(dict, "snapcount", &snapcount);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get snapcount");
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"count", "%d",
+                                          snapcount);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    for (i = 1; i <= snapcount; ++i) {
+        ret = snprintf(key, sizeof(key), "snapname%d", i);
+        if (ret < 0) {
+            goto out;
         }
 
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *) "count",
-                                               "%d", snapcount);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        for (i = 1; i <= snapcount; ++i) {
-                ret = snprintf (key, sizeof (key), "snapname%d", i);
-                if (ret < 0) {
-                        goto out;
-                }
-
-                ret = dict_get_str (dict, key, &str_value);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Could not get %s ", key);
-                        goto out;
-                } else {
-                        ret = xmlTextWriterWriteFormatElement (writer,
-                                        (xmlChar *)"snapshot", "%s", str_value);
-                        XML_RET_CHECK_AND_GOTO (ret, out);
-                }
+        ret = dict_get_str(dict, key, &str_value);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, "Could not get %s ", key);
+            goto out;
+        } else {
+            ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"snapshot",
+                                                  "%s", str_value);
+            XML_RET_CHECK_AND_GOTO(ret, out);
         }
+    }
 
-        /* </snapList> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </snapList> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = 0;
+    ret = 0;
 out:
 
-        return ret;
+    return ret;
 }
 
 /* This function will generate xml output for origin volume
@@ -4475,66 +4111,66 @@ out:
  * @return 0 on success and -1 on failure
  */
 static int
-cli_xml_snapshot_info_orig_vol (xmlTextWriterPtr writer, xmlDocPtr doc,
-                                dict_t *dict, char *keyprefix)
+cli_xml_snapshot_info_orig_vol(xmlTextWriterPtr writer, xmlDocPtr doc,
+                               dict_t *dict, char *keyprefix)
 {
-        int     ret             = -1;
-        int     value           = 0;
-        char   *buffer          = NULL;
-        char    key [PATH_MAX]  = "";
+    int ret = -1;
+    int value = 0;
+    char *buffer = NULL;
+    char key[PATH_MAX] = "";
 
-        GF_ASSERT (dict);
-        GF_ASSERT (keyprefix);
-        GF_ASSERT (writer);
-        GF_ASSERT (doc);
+    GF_ASSERT(dict);
+    GF_ASSERT(keyprefix);
+    GF_ASSERT(writer);
+    GF_ASSERT(doc);
 
-        /* <originVolume> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"originVolume");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* <originVolume> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"originVolume");
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        snprintf (key, sizeof (key), "%sorigin-volname", keyprefix);
+    snprintf(key, sizeof(key), "%sorigin-volname", keyprefix);
 
-        ret = dict_get_str (dict, key, &buffer);
-        if (ret) {
-                gf_log ("cli", GF_LOG_WARNING, "Failed to get %s", key);
-                goto out;
-        }
+    ret = dict_get_str(dict, key, &buffer);
+    if (ret) {
+        gf_log("cli", GF_LOG_WARNING, "Failed to get %s", key);
+        goto out;
+    }
 
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *) "name",
-                                               "%s", buffer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"name", "%s",
+                                          buffer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        snprintf (key, sizeof (key), "%ssnapcount", keyprefix);
+    snprintf(key, sizeof(key), "%ssnapcount", keyprefix);
 
-        ret = dict_get_int32 (dict, key, &value);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get %s", key);
-                goto out;
-        }
+    ret = dict_get_int32(dict, key, &value);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get %s", key);
+        goto out;
+    }
 
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *) "snapCount",
-                                               "%d", value);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"snapCount", "%d",
+                                          value);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        snprintf (key, sizeof (key), "%ssnaps-available", keyprefix);
+    snprintf(key, sizeof(key), "%ssnaps-available", keyprefix);
 
-        ret = dict_get_int32 (dict, key, &value);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get %s", key);
-                goto out;
-        }
+    ret = dict_get_int32(dict, key, &value);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get %s", key);
+        goto out;
+    }
 
-        ret = xmlTextWriterWriteFormatElement (writer,
-                                (xmlChar *) "snapRemaining", "%d", value);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"snapRemaining",
+                                          "%d", value);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        /* </originVolume> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </originVolume> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = 0;
+    ret = 0;
 out:
-        return ret;
+    return ret;
 }
 
 /* This function will generate xml output of snapshot volume info.
@@ -4549,67 +4185,74 @@ out:
  * @return 0 on success and -1 on failure
  */
 static int
-cli_xml_snapshot_info_snap_vol (xmlTextWriterPtr writer, xmlDocPtr doc,
-                                dict_t *dict, char *keyprefix,
-                                gf_boolean_t snap_driven)
+cli_xml_snapshot_info_snap_vol(xmlTextWriterPtr writer, xmlDocPtr doc,
+                               dict_t *dict, char *keyprefix,
+                               gf_boolean_t snap_driven)
 {
-        char            key [PATH_MAX]          = "";
-        char           *buffer                  = NULL;
-        int             value                   = 0;
-        int             ret                     = -1;
-
-        GF_ASSERT (dict);
-        GF_ASSERT (keyprefix);
-        GF_ASSERT (writer);
-        GF_ASSERT (doc);
-
-        /* <snapVolume> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"snapVolume");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        snprintf (key, sizeof (key), "%s.volname", keyprefix);
-
-        ret = dict_get_str (dict, key, &buffer);
+    char key[PATH_MAX] = "";
+    char *buffer = NULL;
+    int ret = -1;
+
+    GF_ASSERT(dict);
+    GF_ASSERT(keyprefix);
+    GF_ASSERT(writer);
+    GF_ASSERT(doc);
+
+    /* <snapVolume> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"snapVolume");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = snprintf(key, sizeof(key), "%s.volname", keyprefix);
+    if (ret < 0)
+        goto out;
+
+    ret = dict_get_str(dict, key, &buffer);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get %s", key);
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"name", "%s",
+                                          buffer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = snprintf(key, sizeof(key), "%s.vol-status", keyprefix);
+    if (ret < 0)
+        goto out;
+
+    ret = dict_get_str(dict, key, &buffer);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get %s", key);
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"status", "%s",
+                                          buffer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* If the command is snap_driven then we need to show origin volume
+     * info. Else this is shown in the start of info display.*/
+    if (snap_driven) {
+        ret = snprintf(key, sizeof(key), "%s.", keyprefix);
+        if (ret < 0)
+            goto out;
+
+        ret = cli_xml_snapshot_info_orig_vol(writer, doc, dict, key);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get %s", key);
-                goto out;
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to create "
+                   "xml output for snapshot's origin volume");
+            goto out;
         }
+    }
 
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *) "name",
-                                               "%s", buffer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </snapVolume> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        snprintf (key, sizeof (key), "%s.vol-status", keyprefix);
-
-        ret = dict_get_str (dict, key, &buffer);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get %s", key);
-                goto out;
-        }
-
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *) "status",
-                                               "%s", buffer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        /* If the command is snap_driven then we need to show origin volume
-         * info. Else this is shown in the start of info display.*/
-        if (snap_driven) {
-                snprintf (key, sizeof (key), "%s.", keyprefix);
-                ret = cli_xml_snapshot_info_orig_vol (writer, doc, dict, key);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to create "
-                                "xml output for snapshot's origin volume");
-                        goto out;
-                }
-        }
-
-        /* </snapVolume> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = 0;
+    ret = 0;
 out:
-        return ret;
+    return ret;
 }
 
 /* This function will generate snapshot info of individual snapshot
@@ -4625,108 +4268,118 @@ out:
  * @return 0 on success and -1 on failure
  */
 static int
-cli_xml_snapshot_info_per_snap (xmlTextWriterPtr writer, xmlDocPtr doc,
-                                dict_t *dict, char *keyprefix,
-                                gf_boolean_t snap_driven)
+cli_xml_snapshot_info_per_snap(xmlTextWriterPtr writer, xmlDocPtr doc,
+                               dict_t *dict, char *keyprefix,
+                               gf_boolean_t snap_driven)
 {
-        char            key_buffer[PATH_MAX]    = "";
-        char           *buffer                  = NULL;
-        int             volcount                = 0;
-        int             ret                     = -1;
-        int             i                       = 0;
-
-        GF_ASSERT (dict);
-        GF_ASSERT (keyprefix);
-        GF_ASSERT (writer);
-        GF_ASSERT (doc);
-
-        /* <snapshot> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"snapshot");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        snprintf (key_buffer, sizeof (key_buffer), "%s.snapname",
-                  keyprefix);
-
-        ret = dict_get_str (dict, key_buffer, &buffer);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to fetch snapname %s ",
-                        key_buffer);
-                goto out;
-        }
-
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *) "name",
-                                               "%s", buffer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        snprintf (key_buffer, sizeof (key_buffer), "%s.snap-id", keyprefix);
-
-        ret = dict_get_str (dict, key_buffer, &buffer);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to fetch snap-id %s ",
-                                key_buffer);
-                goto out;
-        }
-
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *) "uuid",
-                                               "%s", buffer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        snprintf (key_buffer, sizeof (key_buffer), "%s.snap-desc", keyprefix);
-
-        ret = dict_get_str (dict, key_buffer, &buffer);
-        if (!ret) {
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                (xmlChar *) "description",
-                                                "%s", buffer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
-
-        snprintf (key_buffer, sizeof (key_buffer), "%s.snap-time", keyprefix);
-
-        ret = dict_get_str (dict, key_buffer, &buffer);
+    char key_buffer[PATH_MAX] = "";
+    char *buffer = NULL;
+    int volcount = 0;
+    int ret = -1;
+    int i = 0;
+
+    GF_ASSERT(dict);
+    GF_ASSERT(keyprefix);
+    GF_ASSERT(writer);
+    GF_ASSERT(doc);
+
+    /* <snapshot> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"snapshot");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = snprintf(key_buffer, sizeof(key_buffer), "%s.snapname", keyprefix);
+    if (ret < 0)
+        goto out;
+
+    ret = dict_get_str(dict, key_buffer, &buffer);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to fetch snapname %s ", key_buffer);
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"name", "%s",
+                                          buffer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = snprintf(key_buffer, sizeof(key_buffer), "%s.snap-id", keyprefix);
+    if (ret < 0)
+        goto out;
+
+    ret = dict_get_str(dict, key_buffer, &buffer);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to fetch snap-id %s ", key_buffer);
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"uuid", "%s",
+                                          buffer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = snprintf(key_buffer, sizeof(key_buffer), "%s.snap-desc", keyprefix);
+    if (ret < 0)
+        goto out;
+
+    ret = dict_get_str(dict, key_buffer, &buffer);
+    if (!ret) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"description",
+                                              "%s", buffer);
+    } else {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"description",
+                                              "%s", "");
+    }
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = snprintf(key_buffer, sizeof(key_buffer), "%s.snap-time", keyprefix);
+    if (ret < 0)
+        goto out;
+
+    ret = dict_get_str(dict, key_buffer, &buffer);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to fetch snap-time %s ", keyprefix);
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"createTime", "%s",
+                                          buffer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = snprintf(key_buffer, sizeof(key_buffer), "%s.vol-count", keyprefix);
+    if (ret < 0)
+        goto out;
+
+    ret = dict_get_int32(dict, key_buffer, &volcount);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Fail to get snap vol count");
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"volCount", "%d",
+                                          volcount);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_int32(dict, key_buffer, &volcount);
+    /* Display info of each snapshot volume */
+    for (i = 1; i <= volcount; i++) {
+        ret = snprintf(key_buffer, sizeof(key_buffer), "%s.vol%d", keyprefix,
+                       i);
+        if (ret < 0)
+            goto out;
+
+        ret = cli_xml_snapshot_info_snap_vol(writer, doc, dict, key_buffer,
+                                             snap_driven);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to fetch snap-time %s ",
-                                keyprefix);
-                goto out;
+            gf_log("cli", GF_LOG_ERROR,
+                   "Could not list "
+                   "details of volume in a snap");
+            goto out;
         }
+    }
 
-        ret = xmlTextWriterWriteFormatElement (writer,
-                                               (xmlChar *) "createTime",
-                                               "%s", buffer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        snprintf (key_buffer, sizeof (key_buffer), "%s.vol-count", keyprefix);
-        ret = dict_get_int32 (dict, key_buffer, &volcount);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Fail to get snap vol count");
-                goto out;
-        }
-
-        ret = xmlTextWriterWriteFormatElement (writer,
-                                               (xmlChar *) "volCount",
-                                               "%d", volcount);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = dict_get_int32 (dict, key_buffer, &volcount);
-        /* Display info of each snapshot volume */
-        for (i = 1 ; i <= volcount ; i++) {
-                snprintf (key_buffer, sizeof (key_buffer), "%s.vol%d",
-                          keyprefix, i);
-
-                ret = cli_xml_snapshot_info_snap_vol (writer, doc, dict,
-                                                      key_buffer, snap_driven);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Could not list "
-                                        "details of volume in a snap");
-                        goto out;
-                }
-        }
-
-        /* </snapshot> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </snapshot> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 out:
-        return ret;
+    return ret;
 }
 
 /* This function will generate snapshot info output in xml format.
@@ -4738,74 +4391,74 @@ out:
  * @return 0 on success and -1 on failure
  */
 static int
-cli_xml_snapshot_info (xmlTextWriterPtr writer, xmlDocPtr doc, dict_t *dict)
+cli_xml_snapshot_info(xmlTextWriterPtr writer, xmlDocPtr doc, dict_t *dict)
 {
-        int             ret             = -1;
-        int             i               = 0;
-        int             snapcount       = 0;
-        char            key [PATH_MAX]  = "";
-        char           *str_value       = NULL;
-        gf_boolean_t    snap_driven     = _gf_false;
-
-        GF_ASSERT (writer);
-        GF_ASSERT (doc);
-        GF_ASSERT (dict);
-
-        /* <snapInfo> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"snapInfo");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        snap_driven = dict_get_str_boolean (dict, "snap-driven", _gf_false);
-
-        /* If the approach is volume based then we should display orgin volume
-         * information first followed by per snap info*/
-        if (!snap_driven) {
-                ret = cli_xml_snapshot_info_orig_vol (writer, doc, dict, "");
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to create "
-                                "xml output for snapshot's origin volume");
-                        goto out;
-                }
-        }
-
-        ret = dict_get_int32 (dict, "snapcount", &snapcount);
+    int ret = -1;
+    int i = 0;
+    int snapcount = 0;
+    char key[PATH_MAX] = "";
+    gf_boolean_t snap_driven = _gf_false;
+
+    GF_ASSERT(writer);
+    GF_ASSERT(doc);
+    GF_ASSERT(dict);
+
+    /* <snapInfo> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"snapInfo");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snap_driven = dict_get_str_boolean(dict, "snap-driven", _gf_false);
+
+    /* If the approach is volume based then we should display origin volume
+     * information first followed by per snap info*/
+    if (!snap_driven) {
+        ret = cli_xml_snapshot_info_orig_vol(writer, doc, dict, "");
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get snapcount");
-                goto out;
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to create "
+                   "xml output for snapshot's origin volume");
+            goto out;
         }
+    }
 
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *) "count",
-                                               "%d", snapcount);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = dict_get_int32(dict, "snapcount", &snapcount);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get snapcount");
+        goto out;
+    }
 
-        /* <snapshots> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"snapshots");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"count", "%d",
+                                          snapcount);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        /* Get snapshot info of individual snapshots */
-        for (i = 1; i <= snapcount; ++i) {
-                snprintf (key, sizeof (key), "snap%d", i);
+    /* <snapshots> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"snapshots");
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-                ret = cli_xml_snapshot_info_per_snap (writer, doc, dict,
-                                                      key, snap_driven);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Could not get %s ", key);
-                        goto out;
-                }
+    /* Get snapshot info of individual snapshots */
+    for (i = 1; i <= snapcount; ++i) {
+        snprintf(key, sizeof(key), "snap%d", i);
+
+        ret = cli_xml_snapshot_info_per_snap(writer, doc, dict, key,
+                                             snap_driven);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, "Could not get %s ", key);
+            goto out;
         }
+    }
 
-        /* </snapshots> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </snapshots> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        /* </snapInfo> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </snapInfo> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = 0;
+    ret = 0;
 out:
 
-        return ret;
+    return ret;
 }
 
 /* This function will generate snapshot status of individual
@@ -4819,125 +4472,148 @@ out:
  * @return 0 on success and -1 on failure
  */
 static int
-cli_xml_snapshot_volume_status (xmlTextWriterPtr writer, xmlDocPtr doc,
-                                dict_t *dict, const char *keyprefix)
+cli_xml_snapshot_volume_status(xmlTextWriterPtr writer, xmlDocPtr doc,
+                               dict_t *dict, const char *keyprefix)
 {
-        int     ret             = -1;
-        int     brickcount      = 0;
-        int     i               = 0;
-        int     value           = 0;
-        int     pid             = 0;
-        char   *buffer          = NULL;
-        char    key[PATH_MAX]   = "";
-
-        GF_ASSERT (writer);
-        GF_ASSERT (doc);
-        GF_ASSERT (dict);
-        GF_ASSERT (keyprefix);
-
-        snprintf (key, sizeof (key), "%s.brickcount", keyprefix);
-
-        ret = dict_get_int32 (dict, key, &brickcount);
+    int ret = -1;
+    int brickcount = 0;
+    int i = 0;
+    int pid = 0;
+    char *buffer = NULL;
+    char key[PATH_MAX] = "";
+
+    GF_ASSERT(writer);
+    GF_ASSERT(doc);
+    GF_ASSERT(dict);
+    GF_ASSERT(keyprefix);
+
+    ret = snprintf(key, sizeof(key), "%s.brickcount", keyprefix);
+    if (ret < 0)
+        goto out;
+
+    ret = dict_get_int32(dict, key, &brickcount);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to fetch brickcount");
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"brickCount", "%d",
+                                          brickcount);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* Get status of every brick belonging to the snapshot volume */
+    for (i = 0; i < brickcount; i++) {
+        /* <snapInfo> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"brick");
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        ret = snprintf(key, sizeof(key), "%s.brick%d.path", keyprefix, i);
+        if (ret < 0)
+            goto out;
+
+        ret = dict_get_str(dict, key, &buffer);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to fetch brickcount");
-                goto out;
+            gf_log("cli", GF_LOG_ERROR, "Unable to get Brick Path");
+            /*
+             * If path itself is not present, then end *
+             * this brick's status and continue to the *
+             * brick                                   *
+             */
+            ret = xmlTextWriterEndElement(writer);
+            XML_RET_CHECK_AND_GOTO(ret, out);
+            continue;
         }
 
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *) "brickCount",
-                                               "%d", brickcount);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        /* Get status of every brick belonging to the snapshot volume */
-        for (i = 0 ; i < brickcount ; i++) {
-                /* <snapInfo> */
-                ret = xmlTextWriterStartElement (writer, (xmlChar *)"brick");
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"path", "%s",
+                                              buffer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                snprintf (key, sizeof (key), "%s.brick%d.path", keyprefix, i);
+        ret = snprintf(key, sizeof(key), "%s.brick%d.vgname", keyprefix, i);
+        if (ret < 0)
+            goto out;
 
-                ret = dict_get_str (dict, key, &buffer);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_INFO, "Unable to get Brick Path");
-                        goto out;
-                }
+        ret = dict_get_str(dict, key, &buffer);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, "Unable to get Volume Group");
+            ret = xmlTextWriterWriteFormatElement(
+                writer, (xmlChar *)"volumeGroup", "N/A");
+        } else
+            ret = xmlTextWriterWriteFormatElement(
+                writer, (xmlChar *)"volumeGroup", "%s", buffer);
 
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                        (xmlChar *) "path", "%s", buffer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                snprintf (key, sizeof (key), "%s.brick%d.vgname",
-                          keyprefix, i);
+        ret = snprintf(key, sizeof(key), "%s.brick%d.status", keyprefix, i);
+        if (ret < 0)
+            goto out;
 
-                ret = dict_get_str (dict, key, &buffer);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_INFO,
-                                "Unable to get Volume Group");
-                        goto out;
-                }
-
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                        (xmlChar *) "volumeGroup",
-                                        "%s", buffer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        ret = dict_get_str(dict, key, &buffer);
+        if (ret) {
+            gf_log("cli", GF_LOG_INFO, "Unable to get Brick Running");
+            ret = xmlTextWriterWriteFormatElement(
+                writer, (xmlChar *)"brick_running", "N/A");
+        } else
+            ret = xmlTextWriterWriteFormatElement(
+                writer, (xmlChar *)"brick_running", "%s", buffer);
 
-                snprintf (key, sizeof (key), "%s.brick%d.status", keyprefix, i);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                ret = dict_get_str (dict, key, &buffer);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                "Unable to get Brick Running");
-                        goto out;
-                }
+        ret = snprintf(key, sizeof(key), "%s.brick%d.pid", keyprefix, i);
+        if (ret < 0)
+            goto out;
 
-                snprintf (key, sizeof (key), "%s.brick%d.pid", keyprefix, i);
+        ret = dict_get_int32(dict, key, &pid);
+        if (ret) {
+            gf_log("cli", GF_LOG_INFO, "Unable to get pid");
+            ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"pid",
+                                                  "N/A");
+        } else
+            ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"pid",
+                                                  "%d", pid);
 
-                ret = dict_get_int32 (dict, key, &pid);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Unable to get pid");
-                        goto out;
-                }
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                        (xmlChar *) "pid", "%d", pid);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        ret = snprintf(key, sizeof(key), "%s.brick%d.data", keyprefix, i);
+        if (ret < 0)
+            goto out;
 
-                snprintf (key, sizeof (key), "%s.brick%d.data", keyprefix, i);
+        ret = dict_get_str(dict, key, &buffer);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, "Unable to get Data Percent");
+            ret = xmlTextWriterWriteFormatElement(
+                writer, (xmlChar *)"data_percentage", "N/A");
+        } else
+            ret = xmlTextWriterWriteFormatElement(
+                writer, (xmlChar *)"data_percentage", "%s", buffer);
 
-                ret = dict_get_str (dict, key, &buffer);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                        "Unable to get Data Percent");
-                        goto out;
-                }
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *) "lvUsage",
-                                                       "%s", buffer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        ret = snprintf(key, sizeof(key), "%s.brick%d.lvsize", keyprefix, i);
+        if (ret < 0)
+            goto out;
 
-                snprintf (key, sizeof (key), "%s.brick%d.lvsize",
-                          keyprefix, i);
-                ret = dict_get_str (dict, key, &buffer);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_INFO, "Unable to get LV Size");
-                        goto out;
-                }
+        ret = dict_get_str(dict, key, &buffer);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR, "Unable to get LV Size");
+            ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"lvSize",
+                                                  "N/A");
+        } else {
+            /* Truncate any newline character */
+            buffer = strtok(buffer, "\n");
 
-                /* Truncate any newline character */
-                buffer = strtok (buffer, "\n");
+            ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"lvSize",
+                                                  "%s", buffer);
+        }
 
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *) "lvSize",
-                                                       "%s", buffer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                /* </brick> */
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+        /* </brick> */
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
 out:
-        return ret;
+    return ret;
 }
 
 /* This function will generate snapshot status of individual
@@ -4950,88 +4626,85 @@ out:
  * @return 0 on success and -1 on failure
  */
 static int
-cli_xml_snapshot_status_per_snap (xmlTextWriterPtr writer, xmlDocPtr doc,
-                                  dict_t *dict, const char *keyprefix)
+cli_xml_snapshot_status_per_snap(xmlTextWriterPtr writer, xmlDocPtr doc,
+                                 dict_t *dict, const char *keyprefix)
 {
-        int     ret             = -1;
-        int     snapcount       = 0;
-        int     volcount        = 0;
-        int     i               = 0;
-        char   *buffer          = NULL;
-        char    key [PATH_MAX]  = "";
-
-        GF_ASSERT (writer);
-        GF_ASSERT (doc);
-        GF_ASSERT (dict);
-        GF_ASSERT (keyprefix);
-
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"snapshot");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        snprintf (key, sizeof (key), "%s.snapname", keyprefix);
-
-        ret = dict_get_str (dict, key, &buffer);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to get snapname");
-                goto out;
-        }
-
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *) "name",
-                                               "%s", buffer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        snprintf (key, sizeof (key), "%s.uuid", keyprefix);
-
-        ret = dict_get_str (dict, key, &buffer);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to get snap UUID");
-                goto out;
-        }
-
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *) "uuid",
-                                               "%s", buffer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    int ret = -1;
+    int volcount = 0;
+    int i = 0;
+    char *buffer = NULL;
+    char key[PATH_MAX] = "";
+
+    GF_ASSERT(writer);
+    GF_ASSERT(doc);
+    GF_ASSERT(dict);
+    GF_ASSERT(keyprefix);
+
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"snapshot");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%s.snapname", keyprefix);
+
+    ret = dict_get_str(dict, key, &buffer);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to get snapname");
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"name", "%s",
+                                          buffer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%s.uuid", keyprefix);
+
+    ret = dict_get_str(dict, key, &buffer);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to get snap UUID");
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"uuid", "%s",
+                                          buffer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    snprintf(key, sizeof(key), "%s.volcount", keyprefix);
+
+    ret = dict_get_int32(dict, key, &volcount);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Unable to get volume count");
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"volCount", "%d",
+                                          volcount);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* Get snapshot status of individual snapshot volume */
+    for (i = 0; i < volcount; i++) {
+        /* <volume> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"volume");
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        snprintf (key, sizeof (key), "%s.volcount", keyprefix);
+        snprintf(key, sizeof(key), "%s.vol%d", keyprefix, i);
 
-        ret = dict_get_int32 (dict, key, &volcount);
+        ret = cli_xml_snapshot_volume_status(writer, doc, dict, key);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Unable to get volume count");
-                goto out;
+            gf_log("cli", GF_LOG_ERROR, "Could not get snap volume status");
+            goto out;
         }
 
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *) "volCount",
-                                               "%d", volcount);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        /* Get snapshot status of individual snapshot volume */
-        for (i = 0 ; i < volcount ; i++) {
-                /* <volume> */
-                ret = xmlTextWriterStartElement (writer, (xmlChar *)"volume");
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                snprintf (key, sizeof (key), "%s.vol%d", keyprefix, i);
-
-                ret = cli_xml_snapshot_volume_status (writer, doc,
-                                                             dict, key);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR,
-                                        "Could not get snap volume status");
-                        goto out;
-                }
-
-                /* </volume> */
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+        /* </volume> */
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        /* </snapshot> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </snapshot> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = 0;
+    ret = 0;
 out:
-        return ret;
+    return ret;
 }
 
 /* This function will generate snapshot status output in xml format.
@@ -5043,64 +4716,63 @@ out:
  * @return 0 on success and -1 on failure
  */
 static int
-cli_xml_snapshot_status (xmlTextWriterPtr writer, xmlDocPtr doc, dict_t *dict)
+cli_xml_snapshot_status(xmlTextWriterPtr writer, xmlDocPtr doc, dict_t *dict)
 {
-        int     ret             = -1;
-        int     snapcount       = 0;
-        int     i               = 0;
-        int     status_cmd      = 0;
-        char   *str_value       = NULL;
-        char    key [PATH_MAX]  = "";
-
-        GF_ASSERT (writer);
-        GF_ASSERT (doc);
-        GF_ASSERT (dict);
-
-        /* <snapStatus> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"snapStatus");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = dict_get_int32 (dict, "status-cmd", &status_cmd);
+    int ret = -1;
+    int snapcount = 0;
+    int i = 0;
+    int status_cmd = 0;
+    char key[PATH_MAX] = "";
+
+    GF_ASSERT(writer);
+    GF_ASSERT(doc);
+    GF_ASSERT(dict);
+
+    /* <snapStatus> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"snapStatus");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_int32(dict, "sub-cmd", &status_cmd);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not fetch status type");
+        goto out;
+    }
+
+    if ((GF_SNAP_STATUS_TYPE_SNAP == status_cmd) ||
+        (GF_SNAP_STATUS_TYPE_ITER == status_cmd)) {
+        snapcount = 1;
+    } else {
+        ret = dict_get_int32(dict, "status.snapcount", &snapcount);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not fetch status type");
-                goto out;
+            gf_log("cli", GF_LOG_ERROR, "Could not get snapcount");
+            goto out;
         }
 
-        if (GF_SNAP_STATUS_TYPE_SNAP == status_cmd) {
-                snapcount = 1;
-        } else {
-                ret = dict_get_int32 (dict, "status.snapcount", &snapcount);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Could not get snapcount");
-                        goto out;
-                }
-
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *) "count",
-                                                       "%d", snapcount);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"count", "%d",
+                                              snapcount);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        for (i = 0 ; i < snapcount; i++) {
-                snprintf (key, sizeof (key), "status.snap%d", i);
+    for (i = 0; i < snapcount; i++) {
+        snprintf(key, sizeof(key), "status.snap%d", i);
 
-                ret = cli_xml_snapshot_status_per_snap (writer, doc,
-                                                               dict, key);
-                if (ret < 0) {
-                        gf_log ("cli", GF_LOG_ERROR, "failed to create xml "
-                                "output for snapshot status");
-                        goto out;
-                }
+        ret = cli_xml_snapshot_status_per_snap(writer, doc, dict, key);
+        if (ret < 0) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "failed to create xml "
+                   "output for snapshot status");
+            goto out;
         }
+    }
 
-        /* </snapStatus> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </snapStatus> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = 0;
+    ret = 0;
 out:
 
-        return ret;
+    return ret;
 }
 
 /* This function will generate snapshot config show output in xml format.
@@ -5112,153 +4784,149 @@ out:
  * @return 0 on success and -1 on failure
  */
 static int
-cli_xml_snapshot_config_show (xmlTextWriterPtr writer,
-                              xmlDocPtr doc, dict_t *dict)
+cli_xml_snapshot_config_show(xmlTextWriterPtr writer, xmlDocPtr doc,
+                             dict_t *dict)
 {
-        int             ret             = -1;
-        uint64_t        i               = 0;
-        uint64_t        value           = 0;
-        uint64_t        volcount        = 0;
-        char            buf[PATH_MAX]   = "";
-        char           *str_value       = NULL;
-
-        GF_ASSERT (writer);
-        GF_ASSERT (doc);
-        GF_ASSERT (dict);
-
-        /* <systemConfig> */
-        ret = xmlTextWriterStartElement (writer,
-                        (xmlChar *)"systemConfig");
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        ret = dict_get_uint64 (dict, "snap-max-hard-limit", &value);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get "
-                        "snap-max-hard-limit");
-                goto out;
-        }
-
-        ret = xmlTextWriterWriteFormatElement (writer,
-                        (xmlChar *) "hardLimit", "%"PRIu64, value);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    int ret = -1;
+    uint64_t i = 0;
+    uint64_t value = 0;
+    uint64_t volcount = 0;
+    char buf[PATH_MAX] = "";
+    char *str_value = NULL;
+
+    GF_ASSERT(writer);
+    GF_ASSERT(doc);
+    GF_ASSERT(dict);
+
+    /* <systemConfig> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"systemConfig");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_uint64(dict, "snap-max-hard-limit", &value);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Failed to get "
+               "snap-max-hard-limit");
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"hardLimit",
+                                          "%" PRIu64, value);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_uint64(dict, "snap-max-soft-limit", &value);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Failed to get "
+               "snap-max-soft-limit");
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"softLimit",
+                                          "%" PRIu64 "%%", value);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_str(dict, "auto-delete", &str_value);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not fetch auto-delete");
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"autoDelete", "%s",
+                                          str_value);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_str(dict, "snap-activate-on-create", &str_value);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Could not fetch snap-activate-on-create-delete");
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"activateOnCreate",
+                                          "%s", str_value);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* </systemConfig> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* <volumeConfig> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"volumeConfig");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_uint64(dict, "voldisplaycount", &volcount);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not fetch volcount");
+        goto out;
+    }
+
+    /* Get config of all the volumes */
+    for (i = 0; i < volcount; i++) {
+        /* <volume> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"volume");
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = dict_get_uint64 (dict, "snap-max-soft-limit", &value);
+        snprintf(buf, sizeof(buf), "volume%" PRIu64 "-volname", i);
+        ret = dict_get_str(dict, buf, &str_value);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get "
-                        "snap-max-soft-limit");
-                goto out;
+            gf_log("cli", GF_LOG_ERROR, "Could not fetch %s", buf);
+            goto out;
         }
 
-        ret = xmlTextWriterWriteFormatElement (writer,
-                        (xmlChar *) "softLimit",
-                        "%"PRIu64"%%", value);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"name", "%s",
+                                              str_value);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = dict_get_str (dict, "auto-delete", &str_value);
+        snprintf(buf, sizeof(buf), "volume%" PRIu64 "-snap-max-hard-limit", i);
+        ret = dict_get_uint64(dict, buf, &value);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not fetch auto-delete");
-                goto out;
+            gf_log("cli", GF_LOG_ERROR, "Could not fetch %s", buf);
+            goto out;
         }
 
-        ret = xmlTextWriterWriteFormatElement (writer,
-                        (xmlChar *) "autoDelete", "%s", str_value);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"hardLimit",
+                                              "%" PRIu64, value);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = dict_get_str (dict, "snap-activate-on-create", &str_value);
+        snprintf(buf, sizeof(buf), "volume%" PRIu64 "-active-hard-limit", i);
+        ret = dict_get_uint64(dict, buf, &value);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR,
-                        "Could not fetch snap-activate-on-create-delete");
-                goto out;
+            gf_log("cli", GF_LOG_ERROR,
+                   "Could not fetch"
+                   " effective snap_max_hard_limit for "
+                   "%s",
+                   str_value);
+            goto out;
         }
 
-        ret = xmlTextWriterWriteFormatElement (writer,
-                        (xmlChar *) "activateOnCreate", "%s", str_value);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        /* </systemConfig> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        /* <volumeConfig> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"volumeConfig");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+        ret = xmlTextWriterWriteFormatElement(
+            writer, (xmlChar *)"effectiveHardLimit", "%" PRIu64, value);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = dict_get_uint64 (dict, "voldisplaycount", &volcount);
+        snprintf(buf, sizeof(buf), "volume%" PRIu64 "-snap-max-soft-limit", i);
+        ret = dict_get_uint64(dict, buf, &value);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not fetch volcount");
-                goto out;
+            gf_log("cli", GF_LOG_ERROR, "Could not fetch %s", buf);
+            goto out;
         }
 
-        /* Get config of all the volumes */
-        for (i = 0; i < volcount; i++) {
-                /* <volume> */
-                ret = xmlTextWriterStartElement (writer, (xmlChar *)"volume");
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                snprintf (buf, sizeof(buf), "volume%"PRIu64"-volname", i);
-                ret = dict_get_str (dict, buf, &str_value);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Could not fetch %s", buf);
-                        goto out;
-                }
-
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                (xmlChar *) "name", "%s", str_value);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-
-                snprintf (buf, sizeof(buf),
-                                "volume%"PRIu64"-snap-max-hard-limit", i);
-                ret = dict_get_uint64 (dict, buf, &value);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Could not fetch %s", buf);
-                        goto out;
-                }
-
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                (xmlChar *) "hardLimit", "%"PRIu64, value);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                snprintf (buf, sizeof(buf),
-                                "volume%"PRIu64"-active-hard-limit", i);
-                ret = dict_get_uint64 (dict, buf, &value);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Could not fetch"
-                                " effective snap_max_hard_limit for "
-                                "%s", str_value);
-                        goto out;
-                }
-
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                (xmlChar *) "effectiveHardLimit",
-                                "%"PRIu64, value);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                snprintf (buf, sizeof(buf),
-                                "volume%"PRIu64"-snap-max-soft-limit", i);
-                ret = dict_get_uint64 (dict, buf, &value);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Could not fetch %s", buf);
-                        goto out;
-                }
-
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                        (xmlChar *) "softLimit",
-                                        "%"PRIu64, value);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                /* </volume> */
-                ret = xmlTextWriterEndElement (writer);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"softLimit",
+                                              "%" PRIu64, value);
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
         /* </volume> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        ret = 0;
+    /* </volume> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = 0;
 out:
-        return ret;
+    return ret;
 }
 
 /* This function will generate snapshot config set output in xml format.
@@ -5270,87 +4938,84 @@ out:
  * @return 0 on success and -1 on failure
  */
 static int
-cli_xml_snapshot_config_set (xmlTextWriterPtr writer, xmlDocPtr doc,
-                             dict_t *dict)
+cli_xml_snapshot_config_set(xmlTextWriterPtr writer, xmlDocPtr doc,
+                            dict_t *dict)
 {
-        int             ret             = -1;
-        uint64_t        hard_limit      = 0;
-        uint64_t        soft_limit      = 0;
-        char           *volname         = NULL;
-        char           *auto_delete     = NULL;
-        char           *snap_activate   = NULL;
-
-        GF_ASSERT (writer);
-        GF_ASSERT (doc);
-        GF_ASSERT (dict);
-
-        /* This is optional parameter therefore ignore the error */
-        ret = dict_get_uint64 (dict, "snap-max-hard-limit", &hard_limit);
-        /* This is optional parameter therefore ignore the error */
-        ret = dict_get_uint64 (dict, "snap-max-soft-limit", &soft_limit);
-        ret = dict_get_str (dict, "auto-delete", &auto_delete);
-        ret = dict_get_str (dict, "snap-activate-on-create", &snap_activate);
-
-        if (!hard_limit && !soft_limit && !auto_delete && !snap_activate) {
-                ret = -1;
-                gf_log ("cli", GF_LOG_ERROR, "At least one option from "
-                        "snap-max-hard-limit, snap-max-soft-limit, auto-delete"
-                        " and snap-activate-on-create should be set");
-                goto out;
-        }
-
-        /* Ignore the error, as volname is optional */
-        ret = dict_get_str (dict, "volname", &volname);
-
-        if (NULL == volname) {
-                /* <systemConfig> */
-                ret = xmlTextWriterStartElement (writer,
-                                                 (xmlChar *)"systemConfig");
-        } else {
-                /* <volumeConfig> */
-                ret = xmlTextWriterStartElement (writer,
-                                                 (xmlChar *)"volumeConfig");
-                XML_RET_CHECK_AND_GOTO (ret, out);
+    int ret = -1;
+    uint64_t hard_limit = 0;
+    uint64_t soft_limit = 0;
+    char *volname = NULL;
+    char *auto_delete = NULL;
+    char *snap_activate = NULL;
+
+    GF_ASSERT(writer);
+    GF_ASSERT(doc);
+    GF_ASSERT(dict);
+
+    /* This is optional parameter therefore ignore the error */
+    ret = dict_get_uint64(dict, "snap-max-hard-limit", &hard_limit);
+    /* This is optional parameter therefore ignore the error */
+    ret = dict_get_uint64(dict, "snap-max-soft-limit", &soft_limit);
+    ret = dict_get_str(dict, "auto-delete", &auto_delete);
+    ret = dict_get_str(dict, "snap-activate-on-create", &snap_activate);
+
+    if (!hard_limit && !soft_limit && !auto_delete && !snap_activate) {
+        ret = -1;
+        gf_log("cli", GF_LOG_ERROR,
+               "At least one option from "
+               "snap-max-hard-limit, snap-max-soft-limit, auto-delete"
+               " and snap-activate-on-create should be set");
+        goto out;
+    }
+
+    /* Ignore the error, as volname is optional */
+    ret = dict_get_str(dict, "volname", &volname);
+
+    if (NULL == volname) {
+        /* <systemConfig> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"systemConfig");
+    } else {
+        /* <volumeConfig> */
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"volumeConfig");
+        XML_RET_CHECK_AND_GOTO(ret, out);
 
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                (xmlChar *) "name", "%s", volname);
-        }
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"name", "%s",
+                                              volname);
+    }
 
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        if (hard_limit) {
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                (xmlChar *) "newHardLimit",
-                                                "%"PRIu64, hard_limit);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+    if (hard_limit) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"newHardLimit",
+                                              "%" PRIu64, hard_limit);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        if (soft_limit) {
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                (xmlChar *) "newSoftLimit",
-                                "%"PRIu64, soft_limit);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+    if (soft_limit) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"newSoftLimit",
+                                              "%" PRIu64, soft_limit);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        if (auto_delete) {
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                (xmlChar *) "autoDelete", "%s", auto_delete);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+    if (auto_delete) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"autoDelete",
+                                              "%s", auto_delete);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        if (snap_activate) {
-                ret = xmlTextWriterWriteFormatElement (writer,
-                           (xmlChar *) "activateOnCreate", "%s", snap_activate);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
+    if (snap_activate) {
+        ret = xmlTextWriterWriteFormatElement(
+            writer, (xmlChar *)"activateOnCreate", "%s", snap_activate);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
 
-        /* </volumeConfig> or </systemConfig> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </volumeConfig> or </systemConfig> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = 0;
+    ret = 0;
 out:
-        return ret;
+    return ret;
 }
 
 /* This function will generate snapshot config output in xml format.
@@ -5362,58 +5027,60 @@ out:
  * @return 0 on success and -1 on failure
  */
 static int
-cli_xml_snapshot_config (xmlTextWriterPtr writer, xmlDocPtr doc, dict_t *dict)
+cli_xml_snapshot_config(xmlTextWriterPtr writer, xmlDocPtr doc, dict_t *dict)
 {
-        int             ret             = -1;
-        int             config_command  = 0;
+    int ret = -1;
+    int config_command = 0;
 
-        GF_ASSERT (writer);
-        GF_ASSERT (doc);
-        GF_ASSERT (dict);
+    GF_ASSERT(writer);
+    GF_ASSERT(doc);
+    GF_ASSERT(dict);
 
-        /* <snapConfig> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"snapConfig");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* <snapConfig> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"snapConfig");
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = dict_get_int32 (dict, "config-command", &config_command);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Could not fetch config type");
-                goto out;
-        }
+    ret = dict_get_int32(dict, "config-command", &config_command);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Could not fetch config type");
+        goto out;
+    }
 
-        switch (config_command) {
+    switch (config_command) {
         case GF_SNAP_CONFIG_TYPE_SET:
-                ret = cli_xml_snapshot_config_set (writer, doc, dict);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to create xml "
-                                "output for snapshot config set command");
-                        goto out;
-                }
+            ret = cli_xml_snapshot_config_set(writer, doc, dict);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Failed to create xml "
+                       "output for snapshot config set command");
+                goto out;
+            }
 
-                break;
+            break;
         case GF_SNAP_CONFIG_DISPLAY:
-                ret = cli_xml_snapshot_config_show (writer, doc, dict);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to create xml "
-                                "output for snapshot config show command");
-                        goto out;
-                }
-                break;
+            ret = cli_xml_snapshot_config_show(writer, doc, dict);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Failed to create xml "
+                       "output for snapshot config show command");
+                goto out;
+            }
+            break;
         default:
-                gf_log ("cli", GF_LOG_ERROR, "Uknown config command :%d",
-                        config_command);
-                ret = -1;
-                goto out;
-        }
+            gf_log("cli", GF_LOG_ERROR, "Unknown config command :%d",
+                   config_command);
+            ret = -1;
+            goto out;
+    }
 
-        /* </snapConfig> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </snapConfig> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = 0;
+    ret = 0;
 out:
 
-        return ret;
+    return ret;
 }
 
 /* This function will generate snapshot activate or
@@ -5426,128 +5093,258 @@ out:
  * @return 0 on success and -1 on failure
  */
 static int
-cli_xml_snapshot_activate_deactivate (xmlTextWriterPtr writer, xmlDocPtr doc,
-                                      dict_t *dict, int cmd)
+cli_xml_snapshot_activate_deactivate(xmlTextWriterPtr writer, xmlDocPtr doc,
+                                     dict_t *dict, int cmd)
 {
-        int     ret     = -1;
-        char   *buffer  = NULL;
-        char   *tag     = NULL;
-
-        GF_ASSERT (writer);
-        GF_ASSERT (doc);
-        GF_ASSERT (dict);
-
-        if (GF_SNAP_OPTION_TYPE_ACTIVATE == cmd) {
-                tag = "snapActivate";
-        } else if (GF_SNAP_OPTION_TYPE_DEACTIVATE == cmd) {
-                tag = "snapDeactivate";
-        } else {
-                gf_log ("cli", GF_LOG_ERROR, "invalid command %d", cmd);
-                goto out;
-        }
+    int ret = -1;
+    char *buffer = NULL;
+    char *tag = NULL;
+
+    GF_ASSERT(writer);
+    GF_ASSERT(doc);
+    GF_ASSERT(dict);
+
+    if (GF_SNAP_OPTION_TYPE_ACTIVATE == cmd) {
+        tag = "snapActivate";
+    } else if (GF_SNAP_OPTION_TYPE_DEACTIVATE == cmd) {
+        tag = "snapDeactivate";
+    } else {
+        gf_log("cli", GF_LOG_ERROR, "invalid command %d", cmd);
+        goto out;
+    }
+
+    /* <snapActivate> or <snapDeactivate> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)tag);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* <snapshot> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"snapshot");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_str(dict, "snapname", &buffer);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get snap name");
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"name", "%s",
+                                          buffer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_str(dict, "snapuuid", &buffer);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get snap uuid");
+        goto out;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"uuid", "%s",
+                                          buffer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* </snapshot> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* </snapActivate> or </snapDeactivate> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = 0;
+out:
 
-        /* <snapActivate> or <snapDeactivate> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)tag);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    return ret;
+}
+#endif /* HAVE_LIB_XML */
 
-        /* <snapshot> */
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"snapshot");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+/* This function will generate snapshot delete output in xml format.
+ *
+ * @param writer        xmlTextWriterPtr
+ * @param doc           xmlDocPtr
+ * @param dict          dict containing delete output
+ * @param rsp           cli response
+ *
+ * @return 0 on success and -1 on failure
+ */
+int
+cli_xml_snapshot_delete(cli_local_t *local, dict_t *dict, gf_cli_rsp *rsp)
+{
+    int ret = -1;
+#ifdef HAVE_LIB_XML
+    char *str_value = NULL;
+    xmlTextWriterPtr writer = local->writer;
+    xmlDocPtr doc = local->doc;
+
+    GF_ASSERT(writer);
+    GF_ASSERT(doc);
+    GF_ASSERT(dict);
+
+    /* <snapshot> */
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"snapshot");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_str(dict, "snapname", &str_value);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get snap name");
+        goto xmlend;
+    }
+
+    if (!rsp->op_ret) {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"status",
+                                              "Success");
+    } else {
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"status",
+                                              "Failure");
+        XML_RET_CHECK_AND_GOTO(ret, xmlend);
+
+        ret = cli_xml_output_common(writer, rsp->op_ret, rsp->op_errno,
+                                    rsp->op_errstr);
+    }
+    XML_RET_CHECK_AND_GOTO(ret, xmlend);
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"name", "%s",
+                                          str_value);
+    XML_RET_CHECK_AND_GOTO(ret, xmlend);
+
+    ret = dict_get_str(dict, "snapuuid", &str_value);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get snap uuid");
+        goto xmlend;
+    }
+
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"uuid", "%s",
+                                          str_value);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+xmlend:
+    /* </snapshot> */
+    ret = xmlTextWriterEndElement(writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = dict_get_str (dict, "snapname", &buffer);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get snap name");
-                goto out;
-        }
+#endif /* HAVE_LIB_XML */
+    ret = 0;
+out:
 
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *) "name",
-                                               "%s", buffer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    return ret;
+}
 
-        ret = dict_get_str (dict, "snapuuid", &buffer);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to get snap uuid");
-                goto out;
-        }
+int
+cli_xml_output_snap_status_begin(cli_local_t *local, int op_ret, int op_errno,
+                                 char *op_errstr)
+{
+#if (HAVE_LIB_XML)
+    int ret = -1;
 
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *) "uuid",
-                                               "%s", buffer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    GF_ASSERT(local);
 
-        /* </snapshot> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = cli_begin_xml_output(&(local->writer), &(local->doc));
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
+    ret = cli_xml_output_common(local->writer, op_ret, op_errno, op_errstr);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        /* </snapActivate> or </snapDeactivate> */
-        ret = xmlTextWriterEndElement (writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* <snapStatus> */
+    ret = xmlTextWriterStartElement(local->writer, (xmlChar *)"snapStatus");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* <snapshots> */
+    ret = xmlTextWriterStartElement(local->writer, (xmlChar *)"snapshots");
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = 0;
 out:
+    gf_log("cli", GF_LOG_TRACE, "Returning %d", ret);
+    return ret;
+#else
+    return 0;
+#endif
+}
 
-        return ret;
+int
+cli_xml_output_snap_status_end(cli_local_t *local)
+{
+#if (HAVE_LIB_XML)
+    int ret = -1;
+
+    GF_ASSERT(local);
+
+    /* </snapshots> */
+    ret = xmlTextWriterEndElement(local->writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* </snapStatus> */
+    ret = xmlTextWriterEndElement(local->writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = cli_end_xml_output(local->writer, local->doc);
+out:
+    gf_log("cli", GF_LOG_TRACE, "Returning %d", ret);
+    return ret;
+#else
+    return 0;
+#endif
 }
-#endif /* HAVE_LIB_XML */
 
 int
-cli_xml_output_snap_status_begin (cli_local_t *local, int op_ret, int op_errno,
-                                  char *op_errstr)
+cli_xml_output_snap_delete_begin(cli_local_t *local, int op_ret, int op_errno,
+                                 char *op_errstr)
 {
 #if (HAVE_LIB_XML)
-        int                     ret = -1;
+    int ret = -1;
+    int delete_cmd = -1;
 
-        GF_ASSERT (local);
+    GF_ASSERT(local);
 
-        ret = cli_begin_xml_output (&(local->writer), &(local->doc));
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = cli_begin_xml_output(&(local->writer), &(local->doc));
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = cli_xml_output_common (local->writer, op_ret, op_errno,
-                                     op_errstr);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = dict_get_int32(local->dict, "sub-cmd", &delete_cmd);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to get sub-cmd");
+        goto out;
+    }
 
-        /* <snapStatus> */
-        ret = xmlTextWriterStartElement (local->writer,
-                                         (xmlChar *) "snapStatus");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = cli_xml_output_common(local->writer, op_ret, op_errno, op_errstr);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        /* <snapshots> */
-        ret = xmlTextWriterStartElement (local->writer, (xmlChar *)"snapshots");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* <snapStatus> */
+    ret = xmlTextWriterStartElement(local->writer, (xmlChar *)"snapDelete");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    /* <snapshots> */
+    ret = xmlTextWriterStartElement(local->writer, (xmlChar *)"snapshots");
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
 out:
-        gf_log ("cli", GF_LOG_TRACE, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_TRACE, "Returning %d", ret);
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif
 }
 
 int
-cli_xml_output_snap_status_end (cli_local_t *local)
+cli_xml_output_snap_delete_end(cli_local_t *local)
 {
 #if (HAVE_LIB_XML)
-        int     ret = -1;
+    int ret = -1;
 
-        GF_ASSERT (local);
+    GF_ASSERT(local);
 
-        /* </snapshots> */
-        ret = xmlTextWriterEndElement (local->writer);
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    /* </snapshots> */
+    ret = xmlTextWriterEndElement(local->writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        /* </snapStatus> */
-        ret = xmlTextWriterEndElement (local->writer);
-        XML_RET_CHECK_AND_GOTO(ret, out);
+    /* </snapDelete> */
+    ret = xmlTextWriterEndElement(local->writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = cli_end_xml_output (local->writer, local->doc);
+    ret = cli_end_xml_output(local->writer, local->doc);
 out:
-        gf_log ("cli", GF_LOG_TRACE, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_TRACE, "Returning %d", ret);
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif
 }
-
 /* This function will generate xml output for all the snapshot commands
  *
  * @param cmd_type      command type
@@ -5559,218 +5356,484 @@ out:
  * @return 0 on success and -1 on failure
  */
 int
-cli_xml_output_snapshot (int cmd_type, dict_t *dict, int op_ret,
-                         int op_errno, char *op_errstr)
+cli_xml_output_snapshot(int cmd_type, dict_t *dict, int op_ret, int op_errno,
+                        char *op_errstr)
 {
 #if (HAVE_LIB_XML)
-        int                     ret     = -1;
-        xmlTextWriterPtr        writer  = NULL;
-        xmlDocPtr               doc     = NULL;
-
-        GF_ASSERT (dict);
-
-        ret = cli_begin_xml_output (&writer, &doc);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to output "
-                        "xml begin block");
-                goto out;
-        }
-
-        ret = cli_xml_output_common (writer, op_ret, op_errno, op_errstr);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to output "
-                        "xml common block");
-                goto out;
-        }
-
-        /* In case of command failure just printing the error message is good
-         * enough */
-        if (0 != op_ret) {
-                goto end;
-        }
-
-        switch (cmd_type) {
+    int ret = -1;
+    xmlTextWriterPtr writer = NULL;
+    xmlDocPtr doc = NULL;
+
+    GF_ASSERT(dict);
+
+    ret = cli_begin_xml_output(&writer, &doc);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Failed to output "
+               "xml begin block");
+        goto out;
+    }
+
+    ret = cli_xml_output_common(writer, op_ret, op_errno, op_errstr);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Failed to output "
+               "xml common block");
+        goto out;
+    }
+
+    /* In case of command failure just printing the error message is good
+     * enough */
+    if (0 != op_ret) {
+        goto end;
+    }
+
+    switch (cmd_type) {
         case GF_SNAP_OPTION_TYPE_CREATE:
-                ret = cli_xml_snapshot_create (writer, doc, dict);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to create "
-                                "xml output for snapshot create command");
-                        goto out;
-                }
-                break;
-        case GF_SNAP_OPTION_TYPE_DELETE:
-                ret = cli_xml_snapshot_delete (writer, doc, dict);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to create "
-                                "xml output for snapshot delete command");
-                        goto out;
-                }
-                break;
+            ret = cli_xml_snapshot_create(writer, doc, dict);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Failed to create "
+                       "xml output for snapshot create command");
+                goto out;
+            }
+            break;
+        case GF_SNAP_OPTION_TYPE_CLONE:
+            ret = cli_xml_snapshot_clone(writer, doc, dict);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Failed to create "
+                       "xml output for snapshot clone command");
+                goto out;
+            }
+            break;
         case GF_SNAP_OPTION_TYPE_RESTORE:
-                ret = cli_xml_snapshot_restore (writer, doc, dict);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to create "
-                                "xml output for snapshot restore command");
-                        goto out;
-                }
-                break;
+            ret = cli_xml_snapshot_restore(writer, doc, dict);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Failed to create "
+                       "xml output for snapshot restore command");
+                goto out;
+            }
+            break;
         case GF_SNAP_OPTION_TYPE_LIST:
-                ret = cli_xml_snapshot_list (writer, doc, dict);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to create "
-                                "xml output for snapshot list command");
-                        goto out;
-                }
-                break;
+            ret = cli_xml_snapshot_list(writer, doc, dict);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Failed to create "
+                       "xml output for snapshot list command");
+                goto out;
+            }
+            break;
         case GF_SNAP_OPTION_TYPE_STATUS:
-                ret = cli_xml_snapshot_status (writer, doc, dict);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to create"
-                                "xml output for snapshot status command");
-                        goto out;
-                }
-                break;
+            ret = cli_xml_snapshot_status(writer, doc, dict);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Failed to create"
+                       "xml output for snapshot status command");
+                goto out;
+            }
+            break;
         case GF_SNAP_OPTION_TYPE_INFO:
-                ret = cli_xml_snapshot_info (writer, doc, dict);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to create "
-                                "xml output for snapshot info command");
-                        goto out;
-                }
-                break;
+            ret = cli_xml_snapshot_info(writer, doc, dict);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Failed to create "
+                       "xml output for snapshot info command");
+                goto out;
+            }
+            break;
         case GF_SNAP_OPTION_TYPE_ACTIVATE:
         case GF_SNAP_OPTION_TYPE_DEACTIVATE:
-                ret = cli_xml_snapshot_activate_deactivate (writer, doc,
-                                                            dict, cmd_type);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to create "
-                                "xml output for snapshot config command");
-                }
-                break;
+            ret = cli_xml_snapshot_activate_deactivate(writer, doc, dict,
+                                                       cmd_type);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Failed to create "
+                       "xml output for snapshot config command");
+            }
+            break;
         case GF_SNAP_OPTION_TYPE_CONFIG:
-                ret = cli_xml_snapshot_config (writer, doc, dict);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to create "
-                                "xml output for snapshot config command");
-                }
-                break;
+            ret = cli_xml_snapshot_config(writer, doc, dict);
+            if (ret) {
+                gf_log("cli", GF_LOG_ERROR,
+                       "Failed to create "
+                       "xml output for snapshot config command");
+            }
+            break;
         default:
-                gf_log ("cli", GF_LOG_ERROR,
-                        "Unexpected snapshot command: %d", cmd_type);
-                goto out;
-        }
+            gf_log("cli", GF_LOG_ERROR, "Unexpected snapshot command: %d",
+                   cmd_type);
+            goto out;
+    }
 
 end:
-        ret = cli_end_xml_output (writer, doc);
+    ret = cli_end_xml_output(writer, doc);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Failed to output "
+               "xml end block");
+        goto out;
+    }
+
+    ret = 0;
+out:
+    return ret;
+#else
+    return 0;
+#endif /* HAVE_LIB_XML */
+}
+
+int
+cli_xml_snapshot_begin_composite_op(cli_local_t *local)
+{
+    int ret = -1;
+#ifdef HAVE_LIB_XML
+    int cmd = -1;
+    int type = -1;
+
+    ret = dict_get_int32(local->dict, "sub-cmd", &cmd);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Failed to get "
+               "sub-cmd");
+        ret = 0;
+        goto out;
+    }
+
+    if (cmd == GF_SNAP_STATUS_TYPE_ITER || cmd == GF_SNAP_DELETE_TYPE_SNAP) {
+        ret = 0;
+        goto out;
+    }
+
+    ret = dict_get_int32(local->dict, "type", &type);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Failed to get snapshot "
+               "command type from dictionary");
+        goto out;
+    }
+
+    if (GF_SNAP_OPTION_TYPE_STATUS == type)
+        ret = cli_xml_output_snap_status_begin(local, 0, 0, NULL);
+    else if (GF_SNAP_OPTION_TYPE_DELETE == type)
+        ret = cli_xml_output_snap_delete_begin(local, 0, 0, NULL);
+
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "Error creating xml output");
+        goto out;
+    }
+
+#endif /* HAVE_LIB_XML */
+    ret = 0;
+out:
+    return ret;
+}
+
+int
+cli_xml_snapshot_end_composite_op(cli_local_t *local)
+{
+    int ret = -1;
+#ifdef HAVE_LIB_XML
+    int cmd = -1;
+    int type = -1;
+
+    ret = dict_get_int32(local->dict, "sub-cmd", &cmd);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Failed to get "
+               "sub-cmd");
+        ret = 0;
+        goto out;
+    }
+
+    if (cmd == GF_SNAP_STATUS_TYPE_ITER || cmd == GF_SNAP_DELETE_TYPE_SNAP) {
+        ret = 0;
+        goto out;
+    }
+
+    ret = dict_get_int32(local->dict, "type", &type);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Failed to get snapshot "
+               "command type from dictionary");
+        goto out;
+    }
+
+    if (GF_SNAP_OPTION_TYPE_STATUS == type)
+        ret = cli_xml_output_snap_status_end(local);
+    else if (GF_SNAP_OPTION_TYPE_DELETE == type)
+        ret = cli_xml_output_snap_delete_end(local);
+
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Error creating xml "
+               "output");
+        goto out;
+    }
+#endif /* HAVE_LIB_XML */
+    ret = 0;
+out:
+    return ret;
+}
+
+int
+cli_xml_snapshot_status_single_snap(cli_local_t *local, dict_t *dict, char *key)
+{
+#if (HAVE_LIB_XML)
+    int ret = -1;
+
+    GF_VALIDATE_OR_GOTO("cli", (local != NULL), out);
+    GF_VALIDATE_OR_GOTO("cli", (dict != NULL), out);
+    GF_VALIDATE_OR_GOTO("cli", (key != NULL), out);
+
+    ret = cli_xml_snapshot_status_per_snap(local->writer, local->doc, dict,
+                                           key);
+out:
+    return ret;
+#else
+    return 0;
+#endif /* HAVE_LIB_XML */
+}
+
+int
+cli_xml_output_vol_getopts(dict_t *dict, int op_ret, int op_errno,
+                           char *op_errstr)
+{
+#if (HAVE_LIB_XML)
+    int i = 0;
+    int ret = -1;
+    int count = 0;
+    xmlTextWriterPtr writer = NULL;
+    xmlDocPtr doc = NULL;
+    char *key = NULL;
+    char *value = NULL;
+    char dict_key[50] = {
+        0,
+    };
+
+    ret = cli_begin_xml_output(&writer, &doc);
+    if (ret)
+        goto out;
+
+    ret = cli_xml_output_common(writer, op_ret, op_errno, op_errstr);
+    if (ret)
+        goto out;
+
+    ret = xmlTextWriterStartElement(writer, (xmlChar *)"volGetopts");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = dict_get_int32(dict, "count", &count);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Failed to retrieve count "
+               "from the dictionary");
+        goto out;
+    }
+    if (count <= 0) {
+        gf_log("cli", GF_LOG_ERROR,
+               "Value of count :%d is "
+               "invalid",
+               count);
+        ret = -1;
+        goto out;
+    }
+    ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"count", "%d",
+                                          count);
+
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    for (i = 1; i <= count; i++) {
+        sprintf(dict_key, "key%d", i);
+        ret = dict_get_str(dict, dict_key, &key);
         if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to output "
-                        "xml end block");
-                goto out;
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to"
+                   " retrieve %s from the "
+                   "dictionary",
+                   dict_key);
+            goto out;
         }
+        sprintf(dict_key, "value%d", i);
+        ret = dict_get_str(dict, dict_key, &value);
+        if (ret) {
+            gf_log("cli", GF_LOG_ERROR,
+                   "Failed to "
+                   "retrieve key value for %s from"
+                   "the dictionary",
+                   dict_key);
+            goto out;
+        }
+        ret = xmlTextWriterStartElement(writer, (xmlChar *)"Opt");
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"Option", "%s",
+                                              key);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        ret = xmlTextWriterWriteFormatElement(writer, (xmlChar *)"Value", "%s",
+                                              value);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+
+        ret = xmlTextWriterEndElement(writer);
+        XML_RET_CHECK_AND_GOTO(ret, out);
+    }
+    ret = cli_end_xml_output(writer, doc);
 
-        ret = 0;
 out:
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif /* HAVE_LIB_XML */
 }
 
 int
-cli_xml_snapshot_status_single_snap (cli_local_t *local, dict_t *dict,
-                                     char *key)
+cli_quota_list_xml_error(cli_local_t *local, char *path, char *errstr)
 {
 #if (HAVE_LIB_XML)
-        int ret = -1;
+    int ret = -1;
+
+    ret = xmlTextWriterStartElement(local->writer, (xmlChar *)"limit");
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        GF_VALIDATE_OR_GOTO ("cli", (local != NULL), out);
-        GF_VALIDATE_OR_GOTO ("cli", (dict != NULL), out);
-        GF_VALIDATE_OR_GOTO ("cli", (key != NULL), out);
+    ret = xmlTextWriterWriteFormatElement(local->writer, (xmlChar *)"path",
+                                          "%s", path);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterWriteFormatElement(local->writer, (xmlChar *)"errstr",
+                                          "%s", errstr);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterEndElement(local->writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = cli_xml_snapshot_status_per_snap (local->writer, local->doc, dict,
-                                                key);
 out:
-        return ret;
+    return ret;
 #else
-        return 0;
+    return 0;
+#endif
+}
+
+int
+cli_quota_xml_output(cli_local_t *local, char *path, int64_t hl_str,
+                     char *sl_final, int64_t sl_num, int64_t used,
+                     int64_t avail, char *sl, char *hl, gf_boolean_t limit_set)
+{
+#if (HAVE_LIB_XML)
+    int ret = -1;
+
+    ret = xmlTextWriterStartElement(local->writer, (xmlChar *)"limit");
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterWriteFormatElement(local->writer, (xmlChar *)"path",
+                                          "%s", path);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterWriteFormatElement(
+        local->writer, (xmlChar *)"hard_limit", !limit_set ? "N/A" : "%" PRId64,
+        hl_str);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterWriteFormatElement(local->writer,
+                                          (xmlChar *)"soft_limit_percent",
+                                          !limit_set ? "N/A" : "%s", sl_final);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterWriteFormatElement(
+        local->writer, (xmlChar *)"soft_limit_value",
+        !limit_set ? "N/A" : "%" PRId64, sl_num);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterWriteFormatElement(
+        local->writer, (xmlChar *)"used_space", "%" PRId64, used);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterWriteFormatElement(
+        local->writer, (xmlChar *)"avail_space",
+        !limit_set ? "N/A" : "%" PRId64, avail);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterWriteFormatElement(
+        local->writer, (xmlChar *)"sl_exceeded", !limit_set ? "N/A" : "%s", sl);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterWriteFormatElement(
+        local->writer, (xmlChar *)"hl_exceeded", !limit_set ? "N/A" : "%s", hl);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterEndElement(local->writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+out:
+    return ret;
+#else
+    return 0;
 #endif /* HAVE_LIB_XML */
 }
 
 int
-cli_xml_output_vol_getopts (dict_t *dict, int op_ret, int op_errno,
-                            char *op_errstr)
+cli_quota_object_xml_output(cli_local_t *local, char *path, char *sl_str,
+                            int64_t sl_val, quota_limits_t *limits,
+                            quota_meta_t *used_space, int64_t avail, char *sl,
+                            char *hl, gf_boolean_t limit_set)
 {
 #if (HAVE_LIB_XML)
-        int                     i = 0;
-        int                     ret = -1;
-        int                     count = 0;
-        xmlTextWriterPtr        writer = NULL;
-        xmlDocPtr               doc = NULL;
-        char                    *key = NULL;
-        char                    *value = NULL;
-        char                    dict_key[50] = {0,};
-
-        ret = cli_begin_xml_output (&writer, &doc);
-        if (ret)
-                goto out;
+    int ret = -1;
 
-        ret = cli_xml_output_common (writer, op_ret, op_errno, op_errstr);
-        if (ret)
-                goto out;
+    ret = xmlTextWriterStartElement(local->writer, (xmlChar *)"limit");
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = xmlTextWriterStartElement (writer, (xmlChar *)"volGetopts");
-        XML_RET_CHECK_AND_GOTO (ret, out);
+    ret = xmlTextWriterWriteFormatElement(local->writer, (xmlChar *)"path",
+                                          "%s", path);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
-        ret = dict_get_int32 (dict, "count", &count);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "Failed to retrieve count "
-                        "from the dictionary");
-                goto out;
-        }
-        if (count <= 0) {
-                gf_log ("cli", GF_LOG_ERROR, "Value of count :%d is "
-                        "invalid", count);
-                ret = -1;
-                goto out;
-        }
-        ret = xmlTextWriterWriteFormatElement (writer, (xmlChar *)"count",
-                                               "%d", count);
-
-        XML_RET_CHECK_AND_GOTO (ret, out);
-
-        for (i=1; i<=count; i++) {
-                sprintf (dict_key, "key%d", i);
-                ret = dict_get_str (dict, dict_key, &key);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to"
-                                " retrieve %s from the "
-                                "dictionary", dict_key);
-                        goto out;
-                }
-                sprintf (dict_key, "value%d", i);
-                ret = dict_get_str (dict, dict_key, &value);
-                if (ret) {
-                        gf_log ("cli", GF_LOG_ERROR, "Failed to "
-                                "retrieve key value for %s from"
-                                "the dictionary", dict_key);
-                        goto out;
-                }
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"Option",
-                                                       "%s", key);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-
-                ret = xmlTextWriterWriteFormatElement (writer,
-                                                       (xmlChar *)"Value",
-                                                       "%s", value);
-                XML_RET_CHECK_AND_GOTO (ret, out);
-        }
-        ret = cli_end_xml_output (writer, doc);
+    ret = xmlTextWriterWriteFormatElement(
+        local->writer, (xmlChar *)"hard_limit", !limit_set ? "N/A" : "%" PRId64,
+        limits->hl);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterWriteFormatElement(local->writer,
+                                          (xmlChar *)"soft_limit_percent",
+                                          !limit_set ? "N/A" : "%s", sl_str);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterWriteFormatElement(
+        local->writer, (xmlChar *)"soft_limit_value",
+        !limit_set ? "N/A" : "%" PRIu64, sl_val);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterWriteFormatElement(local->writer,
+                                          (xmlChar *)"file_count", "%" PRId64,
+                                          used_space->file_count);
+
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterWriteFormatElement(local->writer, (xmlChar *)"dir_count",
+                                          "%" PRIu64, used_space->dir_count);
+
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterWriteFormatElement(local->writer, (xmlChar *)"available",
+                                          !limit_set ? "N/A" : "%" PRId64,
+                                          avail);
+
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterWriteFormatElement(
+        local->writer, (xmlChar *)"sl_exceeded", !limit_set ? "N/A" : "%s", sl);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterWriteFormatElement(
+        local->writer, (xmlChar *)"hl_exceeded", !limit_set ? "N/A" : "%s", hl);
+    XML_RET_CHECK_AND_GOTO(ret, out);
+
+    ret = xmlTextWriterEndElement(local->writer);
+    XML_RET_CHECK_AND_GOTO(ret, out);
 
 out:
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    return ret;
 #else
-        return 0;
+    return 0;
 #endif /* HAVE_LIB_XML */
 }
diff --git a/cli/src/cli.c b/cli/src/cli.c
index 6d9f5f6c158..a52b39c5fb8 100644
--- a/cli/src/cli.c
+++ b/cli/src/cli.c
@@ -1,5 +1,5 @@
 /*
-   Copyright (c) 2010-2012 Red Hat, Inc. <http://www.redhat.com>
+   Copyright (c) 2010-2016 Red Hat, Inc. <http://www.redhat.com>
    This file is part of GlusterFS.
 
    This file is licensed to you under your choice of the GNU Lesser
@@ -29,717 +29,884 @@
 #include <semaphore.h>
 #include <errno.h>
 
-#ifndef _CONFIG_H
-#define _CONFIG_H
-#include "config.h"
-#endif
-
 #ifdef HAVE_MALLOC_H
 #include <malloc.h>
 #endif
 
-#ifdef HAVE_MALLOC_STATS
-#ifdef DEBUG
-#include <mcheck.h>
-#endif
-#endif
-
 #include "cli.h"
 #include "cli-quotad-client.h"
 #include "cli-cmd.h"
 #include "cli-mem-types.h"
 
-#include "xlator.h"
-#include "glusterfs.h"
-#include "compat.h"
-#include "logging.h"
-#include "dict.h"
-#include "list.h"
-#include "timer.h"
-#include "stack.h"
-#include "revision.h"
-#include "common-utils.h"
-#include "event.h"
-#include "globals.h"
-#include "syscall.h"
-#include "call-stub.h"
+#include <glusterfs/xlator.h>
+#include <glusterfs/glusterfs.h>
+#include <glusterfs/compat.h>
+#include <glusterfs/logging.h>
+#include <glusterfs/dict.h>
+#include <glusterfs/list.h>
+#include <glusterfs/timer.h>
+#include <glusterfs/stack.h>
+#include <glusterfs/revision.h>
+#include <glusterfs/common-utils.h>
+#include <glusterfs/gf-event.h>
+#include <glusterfs/syscall.h>
+#include <glusterfs/call-stub.h>
 #include <fnmatch.h>
 
 #include "xdr-generic.h"
 
-extern int connected;
 /* using argp for command line parsing */
 
-const char *argp_program_version = ""                                 \
-        PACKAGE_NAME" "PACKAGE_VERSION" built on "__DATE__" "__TIME__ \
-        "\nRepository revision: " GLUSTERFS_REPOSITORY_REVISION "\n"  \
-        "Copyright (c) 2006-2011 Gluster Inc. "                       \
-        "<http://www.gluster.com>\n"                                  \
-        "GlusterFS comes with ABSOLUTELY NO WARRANTY.\n"              \
-        "You may redistribute copies of GlusterFS under the terms of "\
-        "the GNU General Public License.";
-
+const char *argp_program_version =
+    "" PACKAGE_NAME " " PACKAGE_VERSION
+    "\nRepository revision: " GLUSTERFS_REPOSITORY_REVISION
+    "\n"
+    "Copyright (c) 2006-2016 Red Hat, Inc. "
+    "<https://www.gluster.org/>\n"
+    "GlusterFS comes with ABSOLUTELY NO WARRANTY.\n"
+    "It is licensed to you under your choice of the GNU Lesser\n"
+    "General Public License, version 3 or any later version (LGPLv3\n"
+    "or later), or the GNU General Public License, version 2 (GPLv2),\n"
+    "in all cases as published by the Free Software Foundation.";
 const char *argp_program_bug_address = "<" PACKAGE_BUGREPORT ">";
 
-
-
 struct rpc_clnt *global_quotad_rpc;
+
 struct rpc_clnt *global_rpc;
 
 rpc_clnt_prog_t *cli_rpc_prog;
 
-
 extern struct rpc_clnt_program cli_prog;
 
+int cli_default_conn_timeout = 120;
+int cli_ten_minutes_timeout = 600;
+
 static int
-glusterfs_ctx_defaults_init (glusterfs_ctx_t *ctx)
+glusterfs_ctx_defaults_init(glusterfs_ctx_t *ctx)
 {
-        cmd_args_t    *cmd_args = NULL;
-        struct rlimit  lim = {0, };
-        call_pool_t   *pool = NULL;
-        int            ret         = -1;
-
-        ret = xlator_mem_acct_init (THIS, cli_mt_end);
-        if (ret != 0) {
-                return ret;
-        }
-
-        ctx->process_uuid = generate_glusterfs_ctx_id ();
-        if (!ctx->process_uuid)
-                return -1;
-
-        ctx->page_size  = 128 * GF_UNIT_KB;
-
-        ctx->iobuf_pool = iobuf_pool_new ();
-        if (!ctx->iobuf_pool)
-                return -1;
-
-        ctx->event_pool = event_pool_new (DEFAULT_EVENT_POOL_SIZE);
-        if (!ctx->event_pool)
-                return -1;
-
-        pool = GF_CALLOC (1, sizeof (call_pool_t),
-                          cli_mt_call_pool_t);
-        if (!pool)
-                return -1;
-
-        /* frame_mem_pool size 112 * 64 */
-        pool->frame_mem_pool = mem_pool_new (call_frame_t, 32);
-        if (!pool->frame_mem_pool)
-                return -1;
-
-        /* stack_mem_pool size 256 * 128 */
-        pool->stack_mem_pool = mem_pool_new (call_stack_t, 16);
-
-        if (!pool->stack_mem_pool)
-                return -1;
-
-        ctx->stub_mem_pool = mem_pool_new (call_stub_t, 16);
-        if (!ctx->stub_mem_pool)
-                return -1;
-
-        ctx->dict_pool = mem_pool_new (dict_t, 32);
-        if (!ctx->dict_pool)
-                return -1;
-
-        ctx->dict_pair_pool = mem_pool_new (data_pair_t, 512);
-        if (!ctx->dict_pair_pool)
-                return -1;
-
-        ctx->dict_data_pool = mem_pool_new (data_t, 512);
-        if (!ctx->dict_data_pool)
-                return -1;
-
-        INIT_LIST_HEAD (&pool->all_frames);
-        LOCK_INIT (&pool->lock);
-        ctx->pool = pool;
-
-        pthread_mutex_init (&(ctx->lock), NULL);
-
-        cmd_args = &ctx->cmd_args;
-
-        INIT_LIST_HEAD (&cmd_args->xlator_options);
+    cmd_args_t *cmd_args = NULL;
+    struct rlimit lim = {
+        0,
+    };
+    call_pool_t *pool = NULL;
+    int ret = -1;
+
+    if (!ctx)
+        return ret;
 
-        lim.rlim_cur = RLIM_INFINITY;
-        lim.rlim_max = RLIM_INFINITY;
-        setrlimit (RLIMIT_CORE, &lim);
+    ret = xlator_mem_acct_init(THIS, cli_mt_end);
+    if (ret != 0) {
+        gf_log("cli", GF_LOG_ERROR, "Memory accounting init failed.");
+        return ret;
+    }
+
+    /* Resetting ret to -1 to so in case of failure
+     * we can relese allocated resource.
+     */
+    ret = -1;
+
+    ctx->process_uuid = generate_glusterfs_ctx_id();
+    if (!ctx->process_uuid) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to generate uuid.");
+        goto out;
+    }
+
+    ctx->page_size = 128 * GF_UNIT_KB;
+
+    ctx->iobuf_pool = iobuf_pool_new();
+    if (!ctx->iobuf_pool) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to create iobuf pool.");
+        goto out;
+    }
+
+    ctx->event_pool = gf_event_pool_new(DEFAULT_EVENT_POOL_SIZE,
+                                        STARTING_EVENT_THREADS);
+    if (!ctx->event_pool) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to create event pool.");
+        goto out;
+    }
+
+    pool = GF_CALLOC(1, sizeof(call_pool_t), cli_mt_call_pool_t);
+    if (!pool) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to create call pool.");
+        goto out;
+    }
+
+    /* frame_mem_pool size 112 * 64 */
+    pool->frame_mem_pool = mem_pool_new(call_frame_t, 32);
+    if (!pool->frame_mem_pool) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to create frame mem pool.");
+        goto out;
+    }
+
+    /* stack_mem_pool size 256 * 128 */
+    pool->stack_mem_pool = mem_pool_new(call_stack_t, 16);
+
+    if (!pool->stack_mem_pool) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to create stack mem pool.");
+        goto out;
+    }
+
+    ctx->stub_mem_pool = mem_pool_new(call_stub_t, 16);
+    if (!ctx->stub_mem_pool) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to stub mem pool.");
+        goto out;
+    }
+
+    ctx->dict_pool = mem_pool_new(dict_t, 32);
+    if (!ctx->dict_pool) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to create dict pool.");
+        goto out;
+    }
+
+    ctx->dict_pair_pool = mem_pool_new(data_pair_t, 512);
+    if (!ctx->dict_pair_pool) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to create dict pair pool.");
+        goto out;
+    }
+
+    ctx->dict_data_pool = mem_pool_new(data_t, 512);
+    if (!ctx->dict_data_pool) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to create dict data pool.");
+        goto out;
+    }
+
+    ctx->logbuf_pool = mem_pool_new(log_buf_t, 256);
+    if (!ctx->logbuf_pool) {
+        gf_log("cli", GF_LOG_ERROR, "Failed to create logbuf pool.");
+        goto out;
+    }
+
+    INIT_LIST_HEAD(&pool->all_frames);
+    LOCK_INIT(&pool->lock);
+    ctx->pool = pool;
+
+    cmd_args = &ctx->cmd_args;
+
+    INIT_LIST_HEAD(&cmd_args->xlator_options);
+
+    lim.rlim_cur = RLIM_INFINITY;
+    lim.rlim_max = RLIM_INFINITY;
+    setrlimit(RLIMIT_CORE, &lim);
+
+    ret = 0;
 
-        return 0;
+out:
+    if (ret != 0) {
+        if (pool) {
+            mem_pool_destroy(pool->frame_mem_pool);
+            mem_pool_destroy(pool->stack_mem_pool);
+        }
+        GF_FREE(pool);
+        pool = NULL;
+        GF_FREE(ctx->process_uuid);
+        mem_pool_destroy(ctx->stub_mem_pool);
+        mem_pool_destroy(ctx->dict_pool);
+        mem_pool_destroy(ctx->dict_pair_pool);
+        mem_pool_destroy(ctx->dict_data_pool);
+        mem_pool_destroy(ctx->logbuf_pool);
+    }
+
+    return ret;
 }
 
-
 static int
-logging_init (glusterfs_ctx_t *ctx, struct cli_state *state)
+logging_init(glusterfs_ctx_t *ctx, struct cli_state *state)
 {
-        char *log_file = state->log_file ? state->log_file :
-                         DEFAULT_CLI_LOG_FILE_DIRECTORY "/cli.log";
-
-        /* passing ident as NULL means to use default ident for syslog */
-        if (gf_log_init (ctx, log_file, NULL) == -1) {
-                fprintf (stderr, "ERROR: failed to open logfile %s\n",
-                         log_file);
-                return -1;
-        }
-
-        /* CLI should not have something to DEBUG after the release,
-           hence defaulting to INFO loglevel */
-        gf_log_set_loglevel ((state->log_level == GF_LOG_NONE) ? GF_LOG_INFO :
-                             state->log_level);
-
-        return 0;
+    char *log_file = state->log_file ? state->log_file
+                                     : DEFAULT_CLI_LOG_FILE_DIRECTORY
+                         "/cli.log";
+
+    /* passing ident as NULL means to use default ident for syslog */
+    if (gf_log_init(ctx, log_file, NULL) == -1) {
+        fprintf(stderr, "ERROR: failed to open logfile %s\n", log_file);
+    }
+
+    /* CLI should not have something to DEBUG after the release,
+       hence defaulting to INFO loglevel */
+    gf_log_set_loglevel(ctx, (state->log_level == GF_LOG_NONE)
+                                 ? GF_LOG_INFO
+                                 : state->log_level);
+
+    return 0;
 }
 
 int
-cli_submit_request (struct rpc_clnt *rpc, void *req, call_frame_t *frame,
-                    rpc_clnt_prog_t *prog,
-                    int procnum, struct iobref *iobref,
-                    xlator_t *this, fop_cbk_fn_t cbkfn, xdrproc_t xdrproc)
+cli_submit_request(struct rpc_clnt *rpc, void *req, call_frame_t *frame,
+                   rpc_clnt_prog_t *prog, int procnum, struct iobref *iobref,
+                   xlator_t *this, fop_cbk_fn_t cbkfn, xdrproc_t xdrproc)
 {
-        int                     ret         = -1;
-        int                     count      = 0;
-        struct iovec            iov         = {0, };
-        struct iobuf            *iobuf = NULL;
-        char                    new_iobref = 0;
-        ssize_t                 xdr_size   = 0;
-
-        GF_ASSERT (this);
-
-        if (req) {
-                xdr_size = xdr_sizeof (xdrproc, req);
-                iobuf = iobuf_get2 (this->ctx->iobuf_pool, xdr_size);
-                if (!iobuf) {
-                        goto out;
-                };
-
-                if (!iobref) {
-                        iobref = iobref_new ();
-                        if (!iobref) {
-                                goto out;
-                        }
-
-                        new_iobref = 1;
-                }
-
-                iobref_add (iobref, iobuf);
-
-                iov.iov_base = iobuf->ptr;
-                iov.iov_len  = iobuf_size (iobuf);
-
-
-                /* Create the xdr payload */
-                ret = xdr_serialize_generic (iov, req, xdrproc);
-                if (ret == -1) {
-                        goto out;
-                }
-                iov.iov_len = ret;
-                count = 1;
+    int ret = -1;
+    int count = 0;
+    struct iovec iov = {
+        0,
+    };
+    struct iobuf *iobuf = NULL;
+    char new_iobref = 0;
+    ssize_t xdr_size = 0;
+
+    GF_ASSERT(this);
+
+    if (req) {
+        xdr_size = xdr_sizeof(xdrproc, req);
+        iobuf = iobuf_get2(this->ctx->iobuf_pool, xdr_size);
+        if (!iobuf) {
+            goto out;
+        };
+
+        if (!iobref) {
+            iobref = iobref_new();
+            if (!iobref) {
+                goto out;
+            }
+
+            new_iobref = 1;
         }
 
-        if (!rpc)
-                rpc = global_rpc;
-        /* Send the msg */
-        ret = rpc_clnt_submit (rpc, prog, procnum, cbkfn,
-                               &iov, count,
-                               NULL, 0, iobref, frame, NULL, 0, NULL, 0, NULL);
-        ret = 0;
+        iobref_add(iobref, iobuf);
+
+        iov.iov_base = iobuf->ptr;
+        iov.iov_len = iobuf_size(iobuf);
+
+        /* Create the xdr payload */
+        ret = xdr_serialize_generic(iov, req, xdrproc);
+        if (ret == -1) {
+            goto out;
+        }
+        iov.iov_len = ret;
+        count = 1;
+    }
+
+    if (!rpc)
+        rpc = global_rpc;
+    /* Send the msg */
+    ret = rpc_clnt_submit(rpc, prog, procnum, cbkfn, &iov, count, NULL, 0,
+                          iobref, frame, NULL, 0, NULL, 0, NULL);
+    ret = 0;
 
 out:
-        if (new_iobref)
-                iobref_unref (iobref);
-        if (iobuf)
-                iobuf_unref (iobuf);
-        return ret;
+    if (new_iobref)
+        iobref_unref(iobref);
+    if (iobuf)
+        iobuf_unref(iobuf);
+    return ret;
 }
 
 int
-cli_rpc_notify (struct rpc_clnt *rpc, void *mydata, rpc_clnt_event_t event,
-                void *data)
+cli_rpc_notify(struct rpc_clnt *rpc, void *mydata, rpc_clnt_event_t event,
+               void *data)
 {
-        xlator_t                *this = NULL;
-        int                     ret = 0;
-
-        this = mydata;
+    xlator_t *this = NULL;
+    int ret = 0;
 
-        switch (event) {
-        case RPC_CLNT_CONNECT:
-        {
+    this = mydata;
 
-                cli_cmd_broadcast_connected ();
-                gf_log (this->name, GF_LOG_TRACE, "got RPC_CLNT_CONNECT");
-               break;
+    switch (event) {
+        case RPC_CLNT_CONNECT: {
+            cli_cmd_broadcast_connected(_gf_true);
+            gf_log(this->name, GF_LOG_TRACE, "got RPC_CLNT_CONNECT");
+            break;
         }
 
-        case RPC_CLNT_DISCONNECT:
-        {
-                gf_log (this->name, GF_LOG_TRACE, "got RPC_CLNT_DISCONNECT");
-                connected = 0;
-                if (!global_state->prompt && global_state->await_connected) {
-                        ret = 1;
-                        cli_out ("Connection failed. Please check if gluster "
-                                  "daemon is operational.");
-                        exit (ret);
-                }
-                break;
+        case RPC_CLNT_DISCONNECT: {
+            cli_cmd_broadcast_connected(_gf_false);
+            gf_log(this->name, GF_LOG_TRACE, "got RPC_CLNT_DISCONNECT");
+            if (!global_state->prompt && global_state->await_connected) {
+                ret = 1;
+                cli_out(
+                    "Connection failed. Please check if gluster "
+                    "daemon is operational.");
+                exit(ret);
+            }
+            break;
         }
 
         default:
-                gf_log (this->name, GF_LOG_TRACE,
-                        "got some other RPC event %d", event);
-                ret = 0;
-                break;
-        }
+            gf_log(this->name, GF_LOG_TRACE, "got some other RPC event %d",
+                   event);
+            ret = 0;
+            break;
+    }
 
-        return ret;
+    return ret;
 }
 
+static gf_boolean_t
+is_valid_int(char *str)
+{
+    if (*str == '-')
+        ++str;
+
+    /* Handle empty string or just "-".*/
+    if (!*str)
+        return _gf_false;
+
+    /* Check for non-digit chars in the rest of the string */
+    while (*str) {
+        if (!isdigit(*str))
+            return _gf_false;
+        else
+            ++str;
+    }
+    return _gf_true;
+}
 
 /*
  * ret: 0: option successfully processed
  *      1: signalling end of option list
- *     -1: unknown option or other issue
+ *     -1: unknown option
+ *     -2: parsing issue (avoid unknown option error)
  */
 int
-cli_opt_parse (char *opt, struct cli_state *state)
+cli_opt_parse(char *opt, struct cli_state *state)
 {
-        char            *oarg           = NULL;
-        gf_boolean_t    secure_mgmt_tmp = 0;
-
-        if (strcmp (opt, "") == 0)
-                return 1;
-
-        if (strcmp (opt, "version") == 0) {
-                cli_out ("%s", argp_program_version);
-                exit (0);
-        }
-
-        if (strcmp (opt, "print-logdir") == 0) {
-                cli_out ("%s", DEFAULT_LOG_FILE_DIRECTORY);
-                exit (0);
-        }
-
-        if (strcmp (opt, "print-statedumpdir") == 0) {
-                cli_out ("%s", DEFAULT_VAR_RUN_DIRECTORY);
-                exit (0);
-        }
-
-        if (strcmp (opt, "xml") == 0) {
+    char *oarg = NULL;
+    gf_boolean_t secure_mgmt_tmp = 0;
+
+    if (strcmp(opt, "") == 0)
+        return 1;
+    if (strcmp(opt, "help") == 0) {
+        cli_out(
+            " peer help                - display help for peer commands\n"
+            " volume help              - display help for volume commands\n"
+            " volume bitrot help       - display help for volume"
+            " bitrot commands\n"
+            " volume quota help        - display help for volume"
+            " quota commands\n"
+            " snapshot help            - display help for snapshot commands\n"
+            " global help              - list global commands\n");
+        exit(0);
+    }
+
+    if (strcmp(opt, "version") == 0) {
+        cli_out("%s", argp_program_version);
+        exit(0);
+    }
+
+    if (strcmp(opt, "print-logdir") == 0) {
+        cli_out("%s", DEFAULT_LOG_FILE_DIRECTORY);
+        exit(0);
+    }
+
+    if (strcmp(opt, "print-statedumpdir") == 0) {
+        cli_out("%s", DEFAULT_VAR_RUN_DIRECTORY);
+        exit(0);
+    }
+
+    if (strcmp(opt, "xml") == 0) {
 #if (HAVE_LIB_XML)
-                state->mode |= GLUSTER_MODE_XML;
+        state->mode |= GLUSTER_MODE_XML;
 #else
-                cli_err ("XML output not supported. Ignoring '--xml' option");
+        cli_err("XML output not supported. Ignoring '--xml' option");
 #endif
-                return 0;
-        }
+        return 0;
+    }
 
-        if (strcmp (opt, "wignore") == 0) {
-                state->mode |= GLUSTER_MODE_WIGNORE;
-                return 0;
-        }
+    if (strcmp(opt, "nolog") == 0) {
+        state->mode |= GLUSTER_MODE_GLFSHEAL_NOLOG;
+        return 0;
+    }
 
-        oarg = strtail (opt, "mode=");
-        if (oarg) {
-                if (strcmp (oarg, "script") == 0) {
-                        state->mode |= GLUSTER_MODE_SCRIPT;
-                        return 0;
-                }
+    if (strcmp(opt, "wignore-partition") == 0) {
+        state->mode |= GLUSTER_MODE_WIGNORE_PARTITION;
+        return 0;
+    }
 
-                if (strcmp (oarg, "interactive") == 0)
-                        return 0;
+    if (strcmp(opt, "wignore") == 0) {
+        state->mode |= GLUSTER_MODE_WIGNORE;
+        return 0;
+    }
 
-                return -1;
+    oarg = strtail(opt, "mode=");
+    if (oarg) {
+        if (strcmp(oarg, "script") == 0) {
+            state->mode |= GLUSTER_MODE_SCRIPT;
+            return 0;
         }
 
-        oarg = strtail (opt, "remote-host=");
-        if (oarg) {
-                state->remote_host = oarg;
-                return 0;
-        }
+        if (strcmp(oarg, "interactive") == 0)
+            return 0;
 
-        oarg = strtail (opt, "log-file=");
-        if (oarg) {
-                state->log_file = oarg;
-                return 0;
-        }
+        return -1;
+    }
 
-        oarg = strtail (opt, "log-level=");
-        if (oarg) {
-                state->log_level = glusterd_check_log_level(oarg);
-                if (state->log_level == -1)
-                        return -1;
-                return 0;
-        }
+    oarg = strtail(opt, "remote-host=");
+    if (oarg) {
+        state->remote_host = oarg;
+        return 0;
+    }
 
-        oarg = strtail (opt, "glusterd-sock=");
-        if (oarg) {
-                state->glusterd_sock = oarg;
-                return 0;
+    oarg = strtail(opt, "inet6");
+    if (oarg) {
+        state->address_family = "inet6";
+        return 0;
+    }
+
+    oarg = strtail(opt, "log-file=");
+    if (oarg) {
+        state->log_file = oarg;
+        return 0;
+    }
+    oarg = strtail(opt, "timeout=");
+    if (oarg) {
+        if (!is_valid_int(oarg) || atoi(oarg) <= 0) {
+            cli_err("timeout value should be a positive integer");
+            return -2; /* -2 instead of -1 to avoid unknown option
+                          error */
         }
+        cli_default_conn_timeout = atoi(oarg);
+        return 0;
+    }
+
+    oarg = strtail(opt, "log-level=");
+    if (oarg) {
+        int log_level = glusterd_check_log_level(oarg);
+        if (log_level == -1)
+            return -1;
+        state->log_level = (gf_loglevel_t)log_level;
+        return 0;
+    }
 
-        oarg = strtail (opt, "secure-mgmt=");
-        if (oarg) {
-                if (gf_string2boolean(oarg,&secure_mgmt_tmp) == 0) {
-                        if (secure_mgmt_tmp) {
-                                /* See declaration for why this is an int. */
-                                state->ctx->secure_mgmt = 1;
-                        }
-                }
-                else {
-                        cli_err ("invalide secure-mgmt value (ignored)");
-                }
-                return 0;
+    oarg = strtail(opt, "glusterd-sock=");
+    if (oarg) {
+        state->glusterd_sock = oarg;
+        return 0;
+    }
+
+    oarg = strtail(opt, "secure-mgmt=");
+    if (oarg) {
+        if (gf_string2boolean(oarg, &secure_mgmt_tmp) == 0) {
+            if (secure_mgmt_tmp) {
+                /* See declaration for why this is an int. */
+                state->ctx->secure_mgmt = 1;
+            }
+        } else {
+            cli_err("invalid secure-mgmt value (ignored)");
         }
+        return 0;
+    }
 
-        return -1;
+    return -1;
 }
 
 int
-parse_cmdline (int argc, char *argv[], struct cli_state *state)
+parse_cmdline(int argc, char *argv[], struct cli_state *state)
 {
-        int         ret = 0;
-        int         i = 0;
-        int         j = 0;
-        char        *opt = NULL;
-
-        state->argc=argc-1;
-        state->argv=&argv[1];
-
-        /* Do this first so that an option can override. */
-        if (access(SECURE_ACCESS_FILE,F_OK) == 0) {
-                state->ctx->secure_mgmt = 1;
+    int ret = 0;
+    int i = 0;
+    int j = 0;
+    char *opt = NULL;
+
+    state->argc = argc - 1;
+    state->argv = &argv[1];
+
+    /* Do this first so that an option can override. */
+    if (sys_access(SECURE_ACCESS_FILE, F_OK) == 0) {
+        state->ctx->secure_mgmt = 1;
+        state->ctx->ssl_cert_depth = glusterfs_read_secure_access_file();
+    }
+
+    if (state->argc > GEO_REP_CMD_CONFIG_INDEX &&
+        strtail(state->argv[GEO_REP_CMD_INDEX], "geo") &&
+        strtail(state->argv[GEO_REP_CMD_CONFIG_INDEX], "co"))
+        goto done;
+
+    for (i = 0; i < state->argc; i++) {
+        opt = strtail(state->argv[i], "--");
+        if (!opt)
+            continue;
+        ret = cli_opt_parse(opt, state);
+        if (ret == -1) {
+            cli_out("unrecognized option --%s\n", opt);
+            usage();
+            return ret;
+        } else if (ret == -2) {
+            return ret;
         }
-
-        for (i = 0; i < state->argc; i++) {
-                opt = strtail (state->argv[i], "--");
-                if (opt) {
-                        ret = cli_opt_parse (opt, state);
-                        if (ret == -1) {
-                                cli_out ("unrecognized option --%s", opt);
-                                return ret;
-                        }
-                        for (j = i; j < state->argc - 1; j++)
-                                state->argv[j] = state->argv[j + 1];
-                        state->argc--;
-                        /* argv shifted, next check should be at i again */
-                        i--;
-                        if (ret == 1) {
-                                /* end of cli options */
-                                ret = 0;
-                                break;
-                        }
-                }
+        for (j = i; j < state->argc - 1; j++)
+            state->argv[j] = state->argv[j + 1];
+        state->argc--;
+        /* argv shifted, next check should be at i again */
+        i--;
+        if (ret == 1) {
+            /* end of cli options */
+            ret = 0;
+            break;
         }
+    }
 
-        state->argv[state->argc] = NULL;
+done:
+    state->argv[state->argc] = NULL;
 
-        return ret;
+    return ret;
 }
 
-
 int
-cli_cmd_tree_init (struct cli_cmd_tree *tree)
+cli_cmd_tree_init(struct cli_cmd_tree *tree)
 {
-        struct cli_cmd_word  *root = NULL;
-        int                   ret = 0;
+    struct cli_cmd_word *root = NULL;
+    int ret = 0;
 
-        root = &tree->root;
-        root->tree = tree;
+    root = &tree->root;
+    root->tree = tree;
 
-        return ret;
+    return ret;
 }
 
-
 int
-cli_state_init (struct cli_state *state)
+cli_state_init(struct cli_state *state)
 {
-        struct cli_cmd_tree  *tree = NULL;
-        int                   ret = 0;
-
+    struct cli_cmd_tree *tree = NULL;
+    int ret = 0;
 
-        state->log_level = -1;
+    state->log_level = GF_LOG_NONE;
 
-        tree = &state->tree;
-        tree->state = state;
+    tree = &state->tree;
+    tree->state = state;
 
-        ret = cli_cmd_tree_init (tree);
+    ret = cli_cmd_tree_init(tree);
 
-        return ret;
+    return ret;
 }
 
 int
-cli_usage_out (const char *usage)
+cli_usage_out(const char *usage)
 {
-        GF_ASSERT (usage);
-        GF_ASSERT (usage[0] != '\0');
+    GF_ASSERT(usage);
 
-        if (!usage || usage[0] == '\0')
-                return -1;
+    if (!usage || usage[0] == '\0')
+        return -1;
 
-        cli_err ("Usage: %s", usage);
-        return 0;
+    cli_err("\nUsage:\n%s\n", usage);
+    return 0;
 }
 
 int
-_cli_err (const char *fmt, ...)
+_cli_err(const char *fmt, ...)
 {
-        struct cli_state *state = NULL;
-        va_list           ap;
-        int               ret = 0;
-
-        state = global_state;
+    va_list ap;
+    int ret = 0;
+#ifdef HAVE_READLINE
+    struct cli_state *state = global_state;
+#endif
 
-        va_start (ap, fmt);
+    va_start(ap, fmt);
 
 #ifdef HAVE_READLINE
-        if (state->rl_enabled && !state->rl_processing)
-                return cli_rl_err(state, fmt, ap);
+    if (state->rl_enabled && !state->rl_processing) {
+        ret = cli_rl_err(state, fmt, ap);
+        va_end(ap);
+        return ret;
+    }
 #endif
 
-        ret = vfprintf (stderr, fmt, ap);
-        fprintf (stderr, "\n");
-        va_end (ap);
+    ret = vfprintf(stderr, fmt, ap);
+    fprintf(stderr, "\n");
+    va_end(ap);
 
-        return ret;
+    return ret;
 }
 
-
 int
-_cli_out (const char *fmt, ...)
+_cli_out(const char *fmt, ...)
 {
-        struct cli_state *state = NULL;
-        va_list           ap;
-        int               ret = 0;
-
-        state = global_state;
-
-        va_start (ap, fmt);
+    va_list ap;
+    int ret = 0;
+#ifdef HAVE_READLINE
+    struct cli_state *state = global_state;
+#endif
 
+    va_start(ap, fmt);
 #ifdef HAVE_READLINE
-        if (state->rl_enabled && !state->rl_processing)
-                return cli_rl_out(state, fmt, ap);
+    if (state->rl_enabled && !state->rl_processing) {
+        ret = cli_rl_out(state, fmt, ap);
+        va_end(ap);
+        return ret;
+    }
 #endif
 
-        ret = vprintf (fmt, ap);
-        printf ("\n");
-        va_end (ap);
+    ret = vprintf(fmt, ap);
+    printf("\n");
+    va_end(ap);
 
-        return ret;
+    return ret;
 }
 
 struct rpc_clnt *
-cli_quotad_clnt_rpc_init (void)
+cli_quotad_clnt_rpc_init(void)
 {
-        struct rpc_clnt *rpc = NULL;
-        dict_t          *rpc_opts = NULL;
-        int             ret = -1;
+    struct rpc_clnt *rpc = NULL;
+    dict_t *rpc_opts = NULL;
+    int ret = -1;
+
+    rpc_opts = dict_new();
+    if (!rpc_opts) {
+        ret = -1;
+        goto out;
+    }
+
+    ret = dict_set_str(rpc_opts, "transport.address-family", "unix");
+    if (ret)
+        goto out;
+
+    ret = dict_set_str(rpc_opts, "transport-type", "socket");
+    if (ret)
+        goto out;
+
+    ret = dict_set_str(rpc_opts, "transport.socket.connect-path",
+                       "/var/run/gluster/quotad.socket");
+    if (ret)
+        goto out;
+
+    rpc = cli_quotad_clnt_init(THIS, rpc_opts);
+    if (!rpc)
+        goto out;
+
+    global_quotad_rpc = rpc;
+out:
+    if (rpc_opts) {
+        dict_unref(rpc_opts);
+    }
+    return rpc;
+}
 
-        rpc_opts = dict_new ();
-        if (!rpc_opts) {
-                        ret = -1;
-                        goto out;
-                }
+struct rpc_clnt *
+cli_rpc_init(struct cli_state *state)
+{
+    struct rpc_clnt *rpc = NULL;
+    dict_t *options = NULL;
+    int ret = -1;
+    int port = CLI_GLUSTERD_PORT;
+    xlator_t *this = NULL;
+#ifdef IPV6_DEFAULT
+    char *addr_family = "inet6";
+#else
+    char *addr_family = "inet";
+#endif
 
-        ret = dict_set_str (rpc_opts, "transport.address-family", "unix");
+    this = THIS;
+    cli_rpc_prog = &cli_prog;
+
+    options = dict_new();
+    if (!options)
+        goto out;
+
+    /* If address family specified in CLI */
+    if (state->address_family) {
+        addr_family = state->address_family;
+    }
+
+    /* Connect to glusterd using the specified method, giving preference
+     * to a unix socket connection.  If nothing is specified, connect to
+     * the default glusterd socket.
+     */
+    if (state->glusterd_sock) {
+        gf_log("cli", GF_LOG_INFO,
+               "Connecting to glusterd using "
+               "sockfile %s",
+               state->glusterd_sock);
+        ret = rpc_transport_unix_options_build(options, state->glusterd_sock,
+                                               0);
         if (ret)
-                goto out;
-
-        ret = dict_set_str (rpc_opts, "transport-type", "socket");
+            goto out;
+    } else if (state->remote_host) {
+        gf_log("cli", GF_LOG_INFO,
+               "Connecting to remote glusterd at "
+               "%s",
+               state->remote_host);
+
+        ret = dict_set_str(options, "remote-host", state->remote_host);
         if (ret)
-                goto out;
+            goto out;
 
-        ret = dict_set_str (rpc_opts, "transport.socket.connect-path",
-                                            "/tmp/quotad.socket");
-        if (ret)
-                goto out;
+        if (state->remote_port)
+            port = state->remote_port;
 
-        rpc = cli_quotad_clnt_init (THIS, rpc_opts);
-        if (!rpc)
-                goto out;
-
-        global_quotad_rpc = rpc;
-out:
-        if (ret) {
-                if (rpc_opts)
-                        dict_destroy(rpc_opts);
-        }
-        return rpc;
-}
-
-struct rpc_clnt *
-cli_rpc_init (struct cli_state *state)
-{
-        struct rpc_clnt         *rpc = NULL;
-        dict_t                  *options = NULL;
-        int                     ret = -1;
-        int                     port = CLI_GLUSTERD_PORT;
-        xlator_t                *this = NULL;
-
-        this = THIS;
-        cli_rpc_prog = &cli_prog;
-        options = dict_new ();
-        if (!options)
-                goto out;
+        ret = dict_set_int32(options, "remote-port", port);
+        if (ret)
+            goto out;
 
-        /* Connect to glusterd using the specified method, giving preference
-         * to a unix socket connection.  If nothing is specified, connect to
-         * the default glusterd socket.
-         */
-        if (state->glusterd_sock) {
-                gf_log ("cli", GF_LOG_INFO, "Connecting to glusterd using "
-                        "sockfile %s", state->glusterd_sock);
-                ret = rpc_transport_unix_options_build (&options,
-                                                        state->glusterd_sock,
-                                                        0);
-                if (ret)
-                        goto out;
-        }
-        else if (state->remote_host) {
-                gf_log ("cli", GF_LOG_INFO, "Connecting to remote glusterd at "
-                        "%s", state->remote_host);
-                ret = dict_set_str (options, "remote-host", state->remote_host);
-                if (ret)
-                        goto out;
-
-                if (state->remote_port)
-                        port = state->remote_port;
-
-                ret = dict_set_int32 (options, "remote-port", port);
-                if (ret)
-                        goto out;
-
-                ret = dict_set_str (options, "transport.address-family",
-                                    "inet");
-                if (ret)
-                        goto out;
-        }
-        else {
-                gf_log ("cli", GF_LOG_DEBUG, "Connecting to glusterd using "
-                        "default socket");
-                ret = rpc_transport_unix_options_build
-                        (&options, DEFAULT_GLUSTERD_SOCKFILE, 0);
-                if (ret)
-                        goto out;
-        }
+        ret = dict_set_str(options, "transport.address-family", addr_family);
+        if (ret)
+            goto out;
+    } else {
+        gf_log("cli", GF_LOG_DEBUG,
+               "Connecting to glusterd using "
+               "default socket");
+        ret = rpc_transport_unix_options_build(options,
+                                               DEFAULT_GLUSTERD_SOCKFILE, 0);
+        if (ret)
+            goto out;
+    }
 
-        rpc = rpc_clnt_new (options, this->ctx, this->name, 16);
-        if (!rpc)
-                goto out;
+    rpc = rpc_clnt_new(options, this, this->name, 16);
+    if (!rpc)
+        goto out;
 
-        ret = rpc_clnt_register_notify (rpc, cli_rpc_notify, this);
-        if (ret) {
-                gf_log ("cli", GF_LOG_ERROR, "failed to register notify");
-                goto out;
-        }
+    ret = rpc_clnt_register_notify(rpc, cli_rpc_notify, this);
+    if (ret) {
+        gf_log("cli", GF_LOG_ERROR, "failed to register notify");
+        goto out;
+    }
 
-        ret = rpc_clnt_start (rpc);
+    ret = rpc_clnt_start(rpc);
 out:
-        if (ret) {
-                if (rpc)
-                        rpc_clnt_unref (rpc);
-                rpc = NULL;
-        }
-        return rpc;
+    if (options)
+        dict_unref(options);
+
+    if (ret) {
+        if (rpc)
+            rpc_clnt_unref(rpc);
+        rpc = NULL;
+    }
+    return rpc;
 }
 
 cli_local_t *
-cli_local_get ()
+cli_local_get()
 {
-        cli_local_t     *local = NULL;
+    cli_local_t *local = NULL;
 
-        local = GF_CALLOC (1, sizeof (*local), cli_mt_cli_local_t);
+    local = GF_CALLOC(1, sizeof(*local), cli_mt_cli_local_t);
+    LOCK_INIT(&local->lock);
+    INIT_LIST_HEAD(&local->dict_list);
 
-        return local;
+    return local;
 }
 
 void
-cli_local_wipe (cli_local_t *local)
+cli_local_wipe(cli_local_t *local)
 {
-        if (local) {
-                GF_FREE (local->get_vol.volname);
-                if (local->dict)
-                        dict_unref (local->dict);
-                GF_FREE (local);
-        }
-
-        return;
+    if (local) {
+        GF_FREE(local->get_vol.volname);
+        if (local->dict)
+            dict_unref(local->dict);
+        GF_FREE(local);
+    }
+
+    return;
 }
 
 struct cli_state *global_state;
 
 int
-main (int argc, char *argv[])
+main(int argc, char *argv[])
 {
-        struct cli_state   state = {0, };
-        int                ret = -1;
-        glusterfs_ctx_t   *ctx = NULL;
+    struct cli_state state = {
+        0,
+    };
+    int ret = -1;
+    glusterfs_ctx_t *ctx = NULL;
 
-        ctx = glusterfs_ctx_new ();
-        if (!ctx)
-                return ENOMEM;
+    mem_pools_init();
+
+    ctx = glusterfs_ctx_new();
+    if (!ctx)
+        return ENOMEM;
 
 #ifdef DEBUG
-        gf_mem_acct_enable_set (ctx);
+    gf_mem_acct_enable_set(ctx);
 #endif
 
-        ret = glusterfs_globals_init (ctx);
-        if (ret)
-                return ret;
+    ret = glusterfs_globals_init(ctx);
+    if (ret)
+        return ret;
 
-	THIS->ctx = ctx;
+    THIS->ctx = ctx;
 
-        ret = glusterfs_ctx_defaults_init (ctx);
-        if (ret)
-                goto out;
+    ret = glusterfs_ctx_defaults_init(ctx);
+    if (ret)
+        goto out;
 
-        ret = cli_state_init (&state);
-        if (ret)
-                goto out;
+    cli_default_conn_timeout = 120;
+    cli_ten_minutes_timeout = 600;
 
-        state.ctx = ctx;
-        global_state = &state;
+    ret = cli_state_init(&state);
+    if (ret)
+        goto out;
 
-        ret = parse_cmdline (argc, argv, &state);
-        if (ret)
-                goto out;
+    state.ctx = ctx;
+    global_state = &state;
 
-        ret = logging_init (ctx, &state);
-        if (ret)
-                goto out;
+    ret = parse_cmdline(argc, argv, &state);
+    if (ret)
+        goto out;
 
-        global_rpc = cli_rpc_init (&state);
-        if (!global_rpc)
-                goto out;
+    ret = logging_init(ctx, &state);
+    if (ret)
+        goto out;
 
-        global_quotad_rpc = cli_quotad_clnt_rpc_init ();
-        if (!global_quotad_rpc)
-                goto out;
+    gf_log("cli", GF_LOG_INFO, "Started running %s with version %s", argv[0],
+           PACKAGE_VERSION);
 
-        ret = cli_cmds_register (&state);
-        if (ret)
-                goto out;
+    global_rpc = cli_rpc_init(&state);
+    if (!global_rpc)
+        goto out;
 
-        ret = cli_cmd_cond_init ();
-        if (ret)
-                goto out;
+    global_quotad_rpc = cli_quotad_clnt_rpc_init();
+    if (!global_quotad_rpc)
+        goto out;
 
-        ret = cli_input_init (&state);
-        if (ret)
-                goto out;
+    ret = cli_cmds_register(&state);
+    if (ret)
+        goto out;
+
+    ret = cli_input_init(&state);
+    if (ret)
+        goto out;
 
-        ret = event_dispatch (ctx->event_pool);
+    ret = gf_event_dispatch(ctx->event_pool);
 
 out:
-//        glusterfs_ctx_destroy (ctx);
+    //        glusterfs_ctx_destroy (ctx);
 
-        return ret;
+    mem_pools_fini();
+
+    return ret;
 }
 
 void
-cli_print_line (int len)
+cli_print_line(int len)
 {
-        GF_ASSERT (len > 0);
+    GF_ASSERT(len > 0);
+
+    while (len--)
+        printf("-");
+
+    printf("\n");
+}
 
-        while (len--)
-                printf ("-");
+void
+print_quota_list_header(int type)
+{
+    if (type == GF_QUOTA_OPTION_TYPE_LIST) {
+        cli_out(
+            "                  Path                   Hard-limit "
+            " Soft-limit      Used  Available  Soft-limit "
+            "exceeded? Hard-limit exceeded?");
+        cli_out(
+            "-----------------------------------------------------"
+            "-----------------------------------------------------"
+            "---------------------");
+    } else {
+        cli_out(
+            "                  Path                   Hard-limit  "
+            " Soft-limit      Files       Dirs     Available  "
+            "Soft-limit exceeded? Hard-limit exceeded?");
+        cli_out(
+            "-----------------------------------------------------"
+            "-----------------------------------------------------"
+            "-------------------------------------");
+    }
+}
 
-        printf ("\n");
+void
+print_quota_list_empty(char *path, int type)
+{
+    if (type == GF_QUOTA_OPTION_TYPE_LIST)
+        cli_out("%-40s %7s %9s %10s %7s %15s %20s", path, "N/A", "N/A", "N/A",
+                "N/A", "N/A", "N/A");
+    else
+        cli_out("%-40s %9s %9s %12s %10s %10s %15s %20s", path, "N/A", "N/A",
+                "N/A", "N/A", "N/A", "N/A", "N/A");
 }
diff --git a/cli/src/cli.h b/cli/src/cli.h
index 89aedc5697d..c0d933e8f8a 100644
--- a/cli/src/cli.h
+++ b/cli/src/cli.h
@@ -10,405 +10,507 @@
 #ifndef __CLI_H__
 #define __CLI_H__
 
-#ifndef _CONFIG_H
-#define _CONFIG_H
-#include "config.h"
-#endif
-
 #include "rpc-clnt.h"
-#include "glusterfs.h"
+#include <glusterfs/glusterfs.h>
 #include "protocol-common.h"
-#include "logging.h"
+#include <glusterfs/logging.h>
+#include <glusterfs/quota-common-utils.h>
 
 #include "cli1-xdr.h"
+#include "gd-common-utils.h"
 
 #if (HAVE_LIB_XML)
 #include <libxml/encoding.h>
 #include <libxml/xmlwriter.h>
 #endif
 
-#define DEFAULT_EVENT_POOL_SIZE            16384
-#define CLI_GLUSTERD_PORT                  24007
-#define CLI_DEFAULT_CONN_TIMEOUT             120
-#define CLI_DEFAULT_CMD_TIMEOUT              120
-#define CLI_TEN_MINUTES_TIMEOUT              600 //Longer timeout for volume top
-#define DEFAULT_CLI_LOG_FILE_DIRECTORY     DATADIR "/log/glusterfs"
-#define CLI_VOL_STATUS_BRICK_LEN              55
-#define CLI_TAB_LENGTH                         8
-#define CLI_BRICK_STATUS_LINE_LEN             78
+#define DEFAULT_EVENT_POOL_SIZE 16384
+#define CLI_GLUSTERD_PORT 24007
+#define DEFAULT_CLI_LOG_FILE_DIRECTORY DATADIR "/log/glusterfs"
+#define CLI_VOL_STATUS_BRICK_LEN 43
+#define CLI_TAB_LENGTH 8
+#define CLI_BRICK_STATUS_LINE_LEN 78
+
+/* Geo-rep command positional arguments' index  */
+#define GEO_REP_CMD_INDEX 1
+#define GEO_REP_CMD_CONFIG_INDEX 4
 
 enum argp_option_keys {
-	ARGP_DEBUG_KEY = 133,
-	ARGP_PORT_KEY = 'p',
+    ARGP_DEBUG_KEY = 133,
+    ARGP_PORT_KEY = 'p',
 };
 
-#define GLUSTER_MODE_SCRIPT    (1 << 0)
+extern int cli_default_conn_timeout;
+extern int cli_ten_minutes_timeout;
+
+typedef enum {
+    COLD_BRICK_COUNT,
+    COLD_TYPE,
+    COLD_DIST_COUNT,
+    COLD_REPLICA_COUNT,
+    COLD_ARBITER_COUNT,
+    COLD_DISPERSE_COUNT,
+    COLD_REDUNDANCY_COUNT,
+    HOT_BRICK_COUNT,
+    HOT_TYPE,
+    HOT_REPLICA_COUNT,
+    MAX
+} values;
+
+#define GLUSTER_MODE_SCRIPT (1 << 0)
 #define GLUSTER_MODE_ERR_FATAL (1 << 1)
-#define GLUSTER_MODE_XML       (1 << 2)
-#define GLUSTER_MODE_WIGNORE   (1 << 3)
-
-
-#define GLUSTERD_GET_QUOTA_AUX_MOUNT_PATH(abspath, volname, path)      \
-        snprintf (abspath, sizeof (abspath)-1,                          \
-                  DEFAULT_VAR_RUN_DIRECTORY"/%s%s", volname, path);
+#define GLUSTER_MODE_XML (1 << 2)
+#define GLUSTER_MODE_WIGNORE (1 << 3)
+#define GLUSTER_MODE_WIGNORE_PARTITION (1 << 4)
+#define GLUSTER_MODE_GLFSHEAL_NOLOG (1 << 5)
 
-#define GLUSTERFS_GET_AUX_MOUNT_PIDFILE(pidfile,volname) {               \
-                snprintf (pidfile, PATH_MAX-1,                             \
-                          DEFAULT_VAR_RUN_DIRECTORY"/%s.pid", volname);  \
-        }
+#define GLUSTERD_GET_QUOTA_LIST_MOUNT_PATH(abspath, volname, path)             \
+    do {                                                                       \
+        snprintf(abspath, sizeof(abspath) - 1,                                 \
+                 DEFAULT_VAR_RUN_DIRECTORY "/%s_quota_list%s", volname, path); \
+    } while (0)
 
 struct cli_state;
 struct cli_cmd_word;
 struct cli_cmd_tree;
 struct cli_cmd;
 
-extern char *cli_vol_type_str[];
 extern char *cli_vol_status_str[];
 extern char *cli_vol_task_status_str[];
 
-typedef int (cli_cmd_cbk_t)(struct cli_state *state,
-                            struct cli_cmd_word *word,
-                            const char **words,
-                            int wordcount);
-typedef void (cli_cmd_reg_cbk_t)( struct cli_cmd *this);
+typedef int(cli_cmd_cbk_t)(struct cli_state *state, struct cli_cmd_word *word,
+                           const char **words, int wordcount);
+typedef void(cli_cmd_reg_cbk_t)(struct cli_cmd *this);
 
-typedef int (cli_cmd_match_t)(struct cli_cmd_word *word);
-typedef int (cli_cmd_filler_t)(struct cli_cmd_word *word);
+typedef int(cli_cmd_match_t)(struct cli_cmd_word *word);
+typedef int(cli_cmd_filler_t)(struct cli_cmd_word *word);
 
 struct cli_cmd_word {
-        struct cli_cmd_tree   *tree;
-        const char            *word;
-        cli_cmd_filler_t      *filler;
-        cli_cmd_match_t       *match;
-        cli_cmd_cbk_t         *cbkfn;
-        const char            *desc;
-        const char            *pattern;
-        int                    nextwords_cnt;
-        struct cli_cmd_word  **nextwords;
+    struct cli_cmd_tree *tree;
+    const char *word;
+    cli_cmd_filler_t *filler;
+    cli_cmd_match_t *match;
+    cli_cmd_cbk_t *cbkfn;
+    const char *desc;
+    const char *pattern;
+    int nextwords_cnt;
+    struct cli_cmd_word **nextwords;
 };
 
-
 struct cli_cmd_tree {
-        struct cli_state      *state;
-        struct cli_cmd_word    root;
+    struct cli_state *state;
+    struct cli_cmd_word root;
 };
 
-
 struct cli_state {
-        int                   argc;
-        char                **argv;
+    int argc;
+    char **argv;
 
-        char                  debug;
+    char debug;
 
-        /* for events dispatching */
-        glusterfs_ctx_t      *ctx;
+    /* for events dispatching */
+    glusterfs_ctx_t *ctx;
 
-        /* registry of known commands */
-        struct cli_cmd_tree   tree;
+    /* registry of known commands */
+    struct cli_cmd_tree tree;
 
-        /* the thread which "executes" the command in non-interactive mode */
-        /* also the thread which reads from stdin in non-readline mode */
-        pthread_t             input;
+    /* the thread which "executes" the command in non-interactive mode */
+    /* also the thread which reads from stdin in non-readline mode */
+    pthread_t input;
 
-        /* terminal I/O */
-        const char           *prompt;
-        int                   rl_enabled;
-        int                   rl_async;
-        int                   rl_processing;
+    /* terminal I/O */
+    const char *prompt;
+    int rl_enabled;
+    int rl_async;
+    int rl_processing;
 
-        /* autocompletion state */
-        char                **matches;
-        char                **matchesp;
+    /* autocompletion state */
+    char **matches;
+    char **matchesp;
 
-        char                 *remote_host;
-        int                   remote_port;
-        int                   mode;
-        int                   await_connected;
+    char *remote_host;
+    int remote_port;
+    int mode;
+    int await_connected;
 
-        char                 *log_file;
-        gf_loglevel_t         log_level;
+    char *log_file;
+    gf_loglevel_t log_level;
 
-        char                 *glusterd_sock;
+    char *glusterd_sock;
+    char *address_family;
 };
 
 struct cli_local {
-        struct {
-                char    *volname;
-                int     flags;
-        } get_vol;
-
-        dict_t          *dict;
-        const char      **words;
-        /* Marker for volume status all */
-        gf_boolean_t    all;
+    struct {
+        char *volname;
+        int flags;
+    } get_vol;
+
+    dict_t *dict;
+    const char **words;
+    /* Marker for volume status all */
+    gf_boolean_t all;
 #if (HAVE_LIB_XML)
-        xmlTextWriterPtr        writer;
-        xmlDocPtr               doc;
-        int                     vol_count;
+    xmlTextWriterPtr writer;
+    xmlDocPtr doc;
+    int vol_count;
 #endif
+    gf_lock_t lock;
+    struct list_head dict_list;
 };
 
 struct cli_volume_status {
-        int            port;
-        int            online;
-        uint64_t       block_size;
-        uint64_t       total_inodes;
-        uint64_t       free_inodes;
-        char          *brick;
-        char          *pid_str;
-        char          *free;
-        char          *total;
-#ifdef GF_LINUX_HOST_OS
-        char          *fs_name;
-        char          *mount_options;
-        char          *device;
-        char          *inode_size;
-#endif
+    int port;
+    int rdma_port;
+    int online;
+    uint64_t block_size;
+    uint64_t total_inodes;
+    uint64_t free_inodes;
+    char *brick;
+    char *pid_str;
+    char *free;
+    char *total;
+    char *fs_name;
+    char *mount_options;
+    char *device;
+    char *inode_size;
 };
 
 struct snap_config_opt_vals_ {
-        char           *op_name;
-        char           *question;
+    char *op_name;
+    char *question;
 };
 
 typedef struct cli_volume_status cli_volume_status_t;
 
 typedef struct cli_local cli_local_t;
 
-typedef ssize_t (*cli_serialize_t) (struct iovec outmsg, void *args);
+typedef ssize_t (*cli_serialize_t)(struct iovec outmsg, void *args);
 
 extern struct cli_state *global_state; /* use only in readline callback */
 
-typedef const char *(*cli_selector_t) (void *wcon);
+extern struct rpc_clnt *global_quotad_rpc;
 
-char *get_struct_variable (int mem_num, gf_gsync_status_t *sts_val);
+extern struct rpc_clnt *global_rpc;
 
-void *cli_getunamb (const char *tok, void **choices, cli_selector_t sel);
+extern rpc_clnt_prog_t *cli_rpc_prog;
 
-int cli_cmd_register (struct cli_cmd_tree *tree, struct cli_cmd *cmd);
-int cli_cmds_register (struct cli_state *state);
+typedef const char *(*cli_selector_t)(void *wcon);
 
-int cli_input_init (struct cli_state *state);
+char *
+get_struct_variable(int mem_num, gf_gsync_status_t *sts_val);
 
-int cli_cmd_process (struct cli_state *state, int argc, char *argv[]);
-int cli_cmd_process_line (struct cli_state *state, const char *line);
+void *
+cli_getunamb(const char *tok, void **choices, cli_selector_t sel);
 
-int cli_rl_enable (struct cli_state *state);
-int cli_rl_out (struct cli_state *state, const char *fmt, va_list ap);
-int cli_rl_err (struct cli_state *state, const char *fmt, va_list ap);
+int
+cli_cmd_register(struct cli_cmd_tree *tree, struct cli_cmd *cmd);
+int
+cli_cmds_register(struct cli_state *state);
 
-int cli_usage_out (const char *usage);
+int
+cli_input_init(struct cli_state *state);
 
-int _cli_out (const char *fmt, ...);
-int _cli_err (const char *fmt, ...);
+int
+cli_cmd_process(struct cli_state *state, int argc, char *argv[]);
+int
+cli_cmd_process_line(struct cli_state *state, const char *line);
 
-#define cli_out(fmt...) do {                       \
-                FMT_WARN (fmt);                    \
-                                                   \
-                _cli_out(fmt);                     \
-                                                   \
-        } while (0)
+int
+cli_rl_enable(struct cli_state *state);
+int
+cli_rl_out(struct cli_state *state, const char *fmt, va_list ap);
+int
+cli_rl_err(struct cli_state *state, const char *fmt, va_list ap);
 
-#define cli_err(fmt...) do {                       \
-                FMT_WARN (fmt);                    \
-                                                   \
-                _cli_err(fmt);                     \
-                                                   \
-        } while (0)
+int
+cli_usage_out(const char *usage);
 
 int
-cli_submit_request (struct rpc_clnt *rpc, void *req, call_frame_t *frame,
-                    rpc_clnt_prog_t *prog,
-                    int procnum, struct iobref *iobref,
-                    xlator_t *this, fop_cbk_fn_t cbkfn, xdrproc_t xdrproc);
+_cli_out(const char *fmt, ...);
+int
+_cli_err(const char *fmt, ...);
+
+#define cli_out(fmt...)                                                        \
+    do {                                                                       \
+        FMT_WARN(fmt);                                                         \
+                                                                               \
+        _cli_out(fmt);                                                         \
+                                                                               \
+    } while (0)
+
+#define cli_err(fmt...)                                                        \
+    do {                                                                       \
+        FMT_WARN(fmt);                                                         \
+                                                                               \
+        _cli_err(fmt);                                                         \
+                                                                               \
+    } while (0)
+
+#define usage()                                                                \
+    do {                                                                       \
+        cli_out(                                                               \
+            " Usage: gluster [options] <help> <peer>"                          \
+            " <pool> <volume>\n"                                               \
+            " Options:\n"                                                      \
+            " --help  Shows the help information\n"                            \
+            " --version  Shows the version\n"                                  \
+            " --print-logdir  Shows the log directory\n"                       \
+            " --print-statedumpdir Shows the state dump directory\n");         \
+                                                                               \
+    } while (0)
+
+int
+cli_submit_request(struct rpc_clnt *rpc, void *req, call_frame_t *frame,
+                   rpc_clnt_prog_t *prog, int procnum, struct iobref *iobref,
+                   xlator_t *this, fop_cbk_fn_t cbkfn, xdrproc_t xdrproc);
+
+int32_t
+cli_cmd_volume_create_parse(struct cli_state *state, const char **words,
+                            int wordcount, dict_t **options, char **bricks);
+
+int32_t
+cli_cmd_volume_reset_parse(const char **words, int wordcount, dict_t **opt);
+
+int32_t
+cli_cmd_gsync_set_parse(struct cli_state *state, const char **words,
+                        int wordcount, dict_t **opt, char **errstr);
+
+int32_t
+cli_cmd_quota_parse(const char **words, int wordcount, dict_t **opt);
 
 int32_t
-cli_cmd_volume_create_parse (struct cli_state *state, const char **words,
-                             int wordcount, dict_t **options);
+cli_cmd_inode_quota_parse(const char **words, int wordcount, dict_t **opt);
 
 int32_t
-cli_cmd_volume_reset_parse (const char **words, int wordcount, dict_t **opt);
+cli_cmd_bitrot_parse(const char **words, int wordcount, dict_t **opt);
 
 int32_t
-cli_cmd_gsync_set_parse (const char **words, int wordcount, dict_t **opt);
+cli_cmd_volume_set_parse(struct cli_state *state, const char **words,
+                         int wordcount, dict_t **options, char **op_errstr);
 
 int32_t
-cli_cmd_quota_parse (const char **words, int wordcount, dict_t **opt);
+cli_cmd_ganesha_parse(struct cli_state *state, const char **words,
+                      int wordcount, dict_t **options, char **op_errstr);
 
 int32_t
-cli_cmd_volume_set_parse (const char **words, int wordcount,
-                          dict_t **options, char **op_errstr);
+cli_cmd_get_state_parse(struct cli_state *state, const char **words,
+                        int wordcount, dict_t **options, char **op_errstr);
 
 int32_t
-cli_cmd_volume_add_brick_parse (const char **words, int wordcount,
-                                dict_t **options);
+cli_cmd_volume_add_brick_parse(struct cli_state *state, const char **words,
+                               int wordcount, dict_t **options, int *type);
 
 int32_t
-cli_cmd_volume_remove_brick_parse (const char **words, int wordcount,
-                                   dict_t **options, int *question);
+cli_cmd_volume_remove_brick_parse(struct cli_state *state, const char **words,
+                                  int wordcount, dict_t **options,
+                                  int *question, int *brick_count,
+                                  int32_t *command);
 
 int32_t
-cli_cmd_volume_replace_brick_parse (const char **words, int wordcount,
+cli_cmd_volume_replace_brick_parse(const char **words, int wordcount,
                                    dict_t **options);
 
 int32_t
-cli_cmd_log_rotate_parse (const char **words, int wordcount, dict_t **options);
+cli_cmd_volume_reset_brick_parse(const char **words, int wordcount,
+                                 dict_t **options);
+
+int32_t
+cli_cmd_log_rotate_parse(const char **words, int wordcount, dict_t **options);
 int32_t
-cli_cmd_log_locate_parse (const char **words, int wordcount, dict_t **options);
+cli_cmd_log_locate_parse(const char **words, int wordcount, dict_t **options);
 int32_t
-cli_cmd_log_filename_parse (const char **words, int wordcount, dict_t **options);
+cli_cmd_log_filename_parse(const char **words, int wordcount, dict_t **options);
 
 int32_t
-cli_cmd_volume_statedump_options_parse (const char **words, int wordcount,
-                                        dict_t **options);
+cli_cmd_volume_statedump_options_parse(const char **words, int wordcount,
+                                       dict_t **options);
 int32_t
-cli_cmd_volume_clrlks_opts_parse (const char **words, int wordcount,
-                                  dict_t **options);
+cli_cmd_volume_clrlks_opts_parse(const char **words, int wordcount,
+                                 dict_t **options);
 
-cli_local_t * cli_local_get ();
+cli_local_t *
+cli_local_get();
 
 void
-cli_local_wipe (cli_local_t *local);
+cli_local_wipe(cli_local_t *local);
+
+gf_boolean_t
+cli_cmd_connected();
 
 int32_t
-cli_cmd_await_connected ();
+cli_cmd_await_connected(unsigned timeout);
 
 int32_t
-cli_cmd_broadcast_connected ();
+cli_cmd_broadcast_connected(gf_boolean_t status);
 
 int
-cli_rpc_notify (struct rpc_clnt *rpc, void *mydata, rpc_clnt_event_t event,
-                void *data);
+cli_rpc_notify(struct rpc_clnt *rpc, void *mydata, rpc_clnt_event_t event,
+               void *data);
 
 int32_t
-cli_cmd_volume_profile_parse (const char **words, int wordcount,
-                              dict_t **options);
+cli_cmd_volume_profile_parse(const char **words, int wordcount,
+                             dict_t **options);
 int32_t
-cli_cmd_volume_top_parse (const char **words, int wordcount,
-                              dict_t **options);
+cli_cmd_volume_top_parse(const char **words, int wordcount, dict_t **options);
 
 int32_t
-cli_cmd_log_level_parse (const char **words, int wordcount,
-                         dict_t **options);
+cli_cmd_log_level_parse(const char **words, int wordcount, dict_t **options);
 
 int32_t
-cli_cmd_volume_status_parse (const char **words, int wordcount,
-                             dict_t **options);
+cli_cmd_volume_status_parse(const char **words, int wordcount,
+                            dict_t **options);
 
 int
-cli_cmd_volume_heal_options_parse (const char **words, int wordcount,
-                                   dict_t **options);
+cli_cmd_volume_heal_options_parse(const char **words, int wordcount,
+                                  dict_t **options);
 
 int
-cli_cmd_volume_defrag_parse (const char **words, int wordcount,
-                             dict_t **options);
+cli_cmd_volume_defrag_parse(const char **words, int wordcount,
+                            dict_t **options);
 
 int
-cli_print_brick_status (cli_volume_status_t *status);
+cli_print_brick_status(cli_volume_status_t *status);
 
 void
-cli_print_detailed_status (cli_volume_status_t *status);
+cli_print_detailed_status(cli_volume_status_t *status);
 
 int
-cli_get_detail_status (dict_t *dict, int i, cli_volume_status_t *status);
+cli_get_detail_status(dict_t *dict, int i, cli_volume_status_t *status);
 
 void
-cli_print_line (int len);
+cli_print_line(int len);
+
+int
+cli_xml_output_str(char *op, char *str, int op_ret, int op_errno,
+                   char *op_errstr);
 
 int
-cli_xml_output_str (char *op, char *str, int op_ret, int op_errno,
+cli_xml_output_dict(char *op, dict_t *dict, int op_ret, int op_errno,
                     char *op_errstr);
 
 int
-cli_xml_output_dict (char *op, dict_t *dict, int op_ret, int op_errno,
-                     char *op_errstr);
+cli_xml_output_vol_top(dict_t *dict, int op_ret, int op_errno, char *op_errstr);
 
 int
-cli_xml_output_vol_top (dict_t *dict, int op_ret, int op_errno,
-                        char *op_errstr);
+cli_xml_output_vol_profile(dict_t *dict, int op_ret, int op_errno,
+                           char *op_errstr);
 
 int
-cli_xml_output_vol_profile (dict_t *dict, int op_ret, int op_errno,
-                            char *op_errstr);
+cli_xml_output_vol_status_begin(cli_local_t *local, int op_ret, int op_errno,
+                                char *op_errstr);
 
 int
-cli_xml_output_vol_status_begin (cli_local_t *local, int op_ret, int op_errno,
-                                 char *op_errstr);
+cli_xml_output_vol_status_end(cli_local_t *local);
 
 int
-cli_xml_output_vol_status_end (cli_local_t *local);
+cli_xml_output_vol_status(cli_local_t *local, dict_t *dict);
 
 int
-cli_xml_output_vol_status (cli_local_t *local, dict_t *dict);
+cli_xml_output_vol_list(dict_t *dict, int op_ret, int op_errno,
+                        char *op_errstr);
 
 int
-cli_xml_output_vol_list (dict_t *dict, int op_ret, int op_errno,
-                         char *op_errstr);
+cli_xml_output_vol_info_begin(cli_local_t *local, int op_ret, int op_errno,
+                              char *op_errstr);
 
 int
-cli_xml_output_vol_info_begin (cli_local_t *local, int op_ret, int op_errno,
-                               char *op_errstr);
+cli_xml_output_vol_info_end(cli_local_t *local);
 
 int
-cli_xml_output_vol_info_end (cli_local_t *local);
+cli_xml_output_vol_info(cli_local_t *local, dict_t *dict);
 
 int
-cli_xml_output_vol_info (cli_local_t *local, dict_t *dict);
+cli_xml_output_vol_quota_limit_list_begin(cli_local_t *local, int op_ret,
+                                          int op_errno, char *op_errstr);
+int
+cli_xml_output_vol_quota_limit_list_end(cli_local_t *local);
 
 int
-cli_xml_output_vol_quota_limit_list (char *volname, char *limit_list,
-                                      int op_ret, int op_errno,
-                                      char *op_errstr);
+cli_quota_list_xml_error(cli_local_t *local, char *path, char *errstr);
 
 int
-cli_xml_output_peer_status (dict_t *dict, int op_ret, int op_errno,
-                            char *op_errstr);
+cli_quota_xml_output(cli_local_t *local, char *path, int64_t hl_str,
+                     char *sl_final, int64_t sl_num, int64_t used,
+                     int64_t avail, char *sl, char *hl, gf_boolean_t limit_set);
 
 int
-cli_xml_output_vol_rebalance (gf_cli_defrag_type op, dict_t *dict, int op_ret,
-                              int op_errno, char *op_errstr);
+cli_quota_object_xml_output(cli_local_t *local, char *path, char *sl_str,
+                            int64_t sl_val, quota_limits_t *limits,
+                            quota_meta_t *used_space, int64_t avail, char *sl,
+                            char *hl, gf_boolean_t limit_set);
 
 int
-cli_xml_output_vol_remove_brick (gf_boolean_t status_op, dict_t *dict,
-                                 int op_ret, int op_errno, char *op_errstr);
+cli_xml_output_peer_status(dict_t *dict, int op_ret, int op_errno,
+                           char *op_errstr);
 
 int
-cli_xml_output_vol_replace_brick (gf1_cli_replace_op op, dict_t *dict,
-                                  int op_ret, int op_errno, char *op_errstr);
+cli_xml_output_vol_rebalance(gf_cli_defrag_type op, dict_t *dict, int op_ret,
+                             int op_errno, char *op_errstr);
 
 int
-cli_xml_output_vol_create (dict_t *dict, int op_ret, int op_errno,
-                           char *op_errstr);
+cli_xml_output_vol_remove_brick(gf_boolean_t status_op, dict_t *dict,
+                                int op_ret, int op_errno, char *op_errstr,
+                                const char *op);
 
 int
-cli_xml_output_generic_volume (char *op, dict_t *dict, int op_ret, int op_errno,
-                               char *op_errstr);
+cli_xml_output_vol_replace_brick(dict_t *dict, int op_ret, int op_errno,
+                                 char *op_errstr);
 
 int
-cli_xml_output_vol_gsync (dict_t *dict, int op_ret, int op_errno,
+cli_xml_output_vol_create(dict_t *dict, int op_ret, int op_errno,
                           char *op_errstr);
+
 int
-cli_xml_output_vol_status_tasks_detail (cli_local_t *local, dict_t *dict);
+cli_xml_output_generic_volume(char *op, dict_t *dict, int op_ret, int op_errno,
+                              char *op_errstr);
 
 int
-cli_xml_output_snap_status_begin (cli_local_t *local, int op_ret, int op_errno,
-                                  char *op_errstr);
+cli_xml_output_vol_gsync(dict_t *dict, int op_ret, int op_errno,
+                         char *op_errstr);
 int
-cli_xml_output_snap_status_end (cli_local_t *local);
+cli_xml_output_vol_status_tasks_detail(cli_local_t *local, dict_t *dict);
+
 int
-cli_xml_output_snapshot (int cmd_type, dict_t *dict, int op_ret,
-                         int op_errno, char *op_errstr);
+cli_xml_snapshot_delete(cli_local_t *local, dict_t *dict, gf_cli_rsp *rsp);
+
 int
-cli_xml_snapshot_status_single_snap (cli_local_t *local, dict_t *dict,
-                                     char *key);
-char *
-is_server_debug_xlator (void *myframe);
+cli_xml_snapshot_begin_composite_op(cli_local_t *local);
+
+int
+cli_xml_snapshot_end_composite_op(cli_local_t *local);
+
+int
+cli_xml_output_snap_delete_begin(cli_local_t *local, int op_ret, int op_errno,
+                                 char *op_errstr);
+int
+cli_xml_output_snap_delete_end(cli_local_t *local);
 
+int
+cli_xml_output_snap_status_begin(cli_local_t *local, int op_ret, int op_errno,
+                                 char *op_errstr);
+int
+cli_xml_output_snap_status_end(cli_local_t *local);
+int
+cli_xml_output_snapshot(int cmd_type, dict_t *dict, int op_ret, int op_errno,
+                        char *op_errstr);
+int
+cli_xml_snapshot_status_single_snap(cli_local_t *local, dict_t *dict,
+                                    char *key);
 int32_t
-cli_cmd_snapshot_parse (const char **words, int wordcount, dict_t **options,
-                        struct cli_state *state);
+cli_cmd_snapshot_parse(const char **words, int wordcount, dict_t **options,
+                       struct cli_state *state);
+
+int
+cli_xml_output_vol_getopts(dict_t *dict, int op_ret, int op_errno,
+                           char *op_errstr);
+
+void
+print_quota_list_header(int type);
+
+void
+print_quota_list_empty(char *path, int type);
 
 int
-cli_xml_output_vol_getopts (dict_t *dict, int op_ret, int op_errno,
-                             char *op_errstr);
+gf_gsync_status_t_comparator(const void *p, const void *q);
 #endif /* __CLI_H__ */
diff --git a/cli/src/input.c b/cli/src/input.c
index 26f337c3c27..5ac1a20edb1 100644
--- a/cli/src/input.c
+++ b/cli/src/input.c
@@ -13,85 +13,81 @@
 #include <stdint.h>
 #include <pthread.h>
 
-#ifndef _CONFIG_H
-#define _CONFIG_H
-#include "config.h"
-#endif
-
 #include "cli.h"
 #include "cli-mem-types.h"
 
 #define CMDBUFSIZ 1024
 
 void *
-cli_batch (void *d)
+cli_batch(void *d)
 {
-        struct cli_state *state = NULL;
-        int               ret = 0;
+    struct cli_state *state = NULL;
+    int ret = 0;
 
-        state = d;
+    state = d;
 
-        ret = cli_cmd_process (state, state->argc, state->argv);
+    ret = cli_cmd_process(state, state->argc, state->argv);
 
-        gf_log ("", GF_LOG_INFO, "Exiting with: %d", ret);
-        exit (-ret);
+    gf_log("", GF_LOG_INFO, "Exiting with: %d", ret);
+    exit(-ret);
 
-        return NULL;
+    return NULL;
 }
 
-
 void *
-cli_input (void *d)
+cli_input(void *d)
 {
-        struct cli_state *state = NULL;
-        int               ret = 0;
-        char              cmdbuf[CMDBUFSIZ];
-        char             *cmd = NULL;
-        size_t            len = 0;
-
-        state = d;
-
-        for (;;) {
-                printf ("%s", state->prompt);
-
-                cmd = fgets (cmdbuf, CMDBUFSIZ, stdin);
-                if (!cmd)
-                        break;
-                len = strlen(cmd);
-                if (len > 0 && cmd[len - 1] == '\n') //strip trailing \n
-                        cmd[len - 1] = '\0';
-                ret = cli_cmd_process_line (state, cmd);
-                if (ret != 0 && state->mode & GLUSTER_MODE_ERR_FATAL)
-                        break;
-        }
-
-        exit (-ret);
-
-        return NULL;
+    struct cli_state *state = NULL;
+    int ret = 0;
+    char cmdbuf[CMDBUFSIZ];
+    char *cmd = NULL;
+    size_t len = 0;
+
+    state = d;
+
+    fprintf(stderr,
+            "Welcome to gluster prompt, type 'help' to see the available "
+            "commands.\n");
+    for (;;) {
+        printf("%s", state->prompt);
+
+        cmd = fgets(cmdbuf, CMDBUFSIZ, stdin);
+        if (!cmd)
+            break;
+        len = strlen(cmd);
+        if (len > 0 && cmd[len - 1] == '\n')  // strip trailing \n
+            cmd[len - 1] = '\0';
+        ret = cli_cmd_process_line(state, cmd);
+        if (ret != 0 && state->mode & GLUSTER_MODE_ERR_FATAL)
+            break;
+    }
+
+    exit(-ret);
+
+    return NULL;
 }
 
-
 int
-cli_input_init (struct cli_state *state)
+cli_input_init(struct cli_state *state)
 {
-        int  ret = 0;
+    int ret = 0;
 
-        if (state->argc) {
-                ret = pthread_create (&state->input, NULL, cli_batch, state);
-                return ret;
-        }
+    if (state->argc) {
+        ret = pthread_create(&state->input, NULL, cli_batch, state);
+        return ret;
+    }
 
-        if (isatty (STDIN_FILENO)) {
-                state->prompt = "gluster> ";
+    if (isatty(STDIN_FILENO)) {
+        state->prompt = "gluster> ";
 
-                cli_rl_enable (state);
-        } else {
-                state->prompt = "";
-                state->mode |= GLUSTER_MODE_SCRIPT | GLUSTER_MODE_ERR_FATAL;
-        }
+        cli_rl_enable(state);
+    } else {
+        state->prompt = "";
+        state->mode |= GLUSTER_MODE_SCRIPT | GLUSTER_MODE_ERR_FATAL;
+    }
 
-        if (!state->rl_enabled)
-                ret = pthread_create (&state->input, NULL, cli_input, state);
+    if (!state->rl_enabled)
+        ret = pthread_create(&state->input, NULL, cli_input, state);
 
-        return ret;
+    return ret;
 }
diff --git a/cli/src/registry.c b/cli/src/registry.c
index c4abe3be0b0..85f7686ade1 100644
--- a/cli/src/registry.c
+++ b/cli/src/registry.c
@@ -8,11 +8,6 @@
    cases as published by the Free Software Foundation.
 */
 
-#ifndef _CONFIG_H
-#define _CONFIG_H
-#include "config.h"
-#endif
-
 #include <stdio.h>
 #include <string.h>
 #include <stdlib.h>
@@ -20,20 +15,18 @@
 #include "cli.h"
 #include "cli-cmd.h"
 
-
 static int
-__is_spc (int ch)
+__is_spc(int ch)
 {
-        if (ch == ' ')
-                return 1;
-        return 0;
+    if (ch == ' ')
+        return 1;
+    return 0;
 }
 
-
 static int
-__is_div (int ch)
+__is_div(int ch)
 {
-        switch (ch) {
+    switch (ch) {
         case '(':
         case ')':
         case '<':
@@ -43,369 +36,356 @@ __is_div (int ch)
         case '{':
         case '}':
         case '|':
-                return 1;
-        }
+            return 1;
+    }
 
-        return 0;
+    return 0;
 }
 
-
 static int
-__is_word (const char *word)
+__is_word(const char *word)
 {
-        return (!__is_div (*word) && !__is_spc (*word));
+    return (!__is_div(*word) && !__is_spc(*word));
 }
 
-
 int
-counter_char (int ch)
+counter_char(int ch)
 {
-        switch (ch) {
+    switch (ch) {
         case '(':
-                return ')';
+            return ')';
         case '<':
-                return '>';
+            return '>';
         case '[':
-                return ']';
+            return ']';
         case '{':
-                return '}';
-        }
+            return '}';
+    }
 
-        return -1;
+    return -1;
 }
 
-
 const char *
-__is_template_balanced (const char *template)
+__is_template_balanced(const char *template)
 {
-        const char *trav = NULL;
-        int   ch = 0;
-
-        trav = template;
-
-        while (*trav) {
-                ch = *trav;
-
-                switch (ch) {
-                case '<':
-                case '(':
-                case '[':
-                        trav = __is_template_balanced (trav+1);
-                        if (!trav)
-                                return NULL;
-                        if (*trav != counter_char (ch))
-                                return NULL;
-                        break;
-                case '>':
-                case ')':
-                case ']':
-                        return trav;
-                }
+    const char *trav = NULL;
+    int ch = 0;
+
+    trav = template;
+
+    while (*trav) {
+        ch = *trav;
 
-                trav++;
+        switch (ch) {
+            case '<':
+            case '(':
+            case '[':
+                trav = __is_template_balanced(trav + 1);
+                if (!trav)
+                    return NULL;
+                if (*trav != counter_char(ch))
+                    return NULL;
+                break;
+            case '>':
+            case ')':
+            case ']':
+                return trav;
         }
 
-        return trav;
-}
+        trav++;
+    }
 
+    return trav;
+}
 
 int
-is_template_balanced (const char *template)
+is_template_balanced(const char *template)
 {
-        const char *trav = NULL;
+    const char *trav = NULL;
 
-        trav = __is_template_balanced (template);
-        if (!trav || *trav)
-                return -1;
+    trav = __is_template_balanced(template);
+    if (!trav || *trav)
+        return -1;
 
-        return 0;
+    return 0;
 }
 
-
 int
-cli_cmd_token_count (const char *template)
+cli_cmd_token_count(const char *template)
 {
-        int         count = 0;
-        const char *trav = NULL;
-        int         is_alnum = 0;
-
-        for (trav = template; *trav; trav++) {
-                switch (*trav) {
-                case '<':
-                case '>':
-                case '(':
-                case ')':
-                case '[':
-                case ']':
-                case '{':
-                case '}':
-                case '|':
-                        count++;
-                        /* fall through */
-                case ' ':
-                        is_alnum = 0;
-                        break;
-                default:
-                        if (!is_alnum) {
-                                is_alnum = 1;
-                                count++;
-                        }
+    int count = 0;
+    const char *trav = NULL;
+    int is_alnum = 0;
+
+    for (trav = template; *trav; trav++) {
+        switch (*trav) {
+            case '<':
+            case '>':
+            case '(':
+            case ')':
+            case '[':
+            case ']':
+            case '{':
+            case '}':
+            case '|':
+                count++;
+                /* fall through */
+            case ' ':
+                is_alnum = 0;
+                break;
+            default:
+                if (!is_alnum) {
+                    is_alnum = 1;
+                    count++;
                 }
         }
+    }
 
-        return count + 1;
+    return count + 1;
 }
 
-
 void
-cli_cmd_tokens_destroy (char **tokens)
+cli_cmd_tokens_destroy(char **tokens)
 {
-        char **tokenp = NULL;
+    char **tokenp = NULL;
 
-        if (!tokens)
-                return;
+    if (!tokens)
+        return;
 
-        tokenp = tokens;
-        while (*tokenp) {
-                free (*tokenp);
-                tokenp++;
-        }
+    tokenp = tokens;
+    while (*tokenp) {
+        free(*tokenp);
+        tokenp++;
+    }
 
-        free (tokens);
+    free(tokens);
 }
 
-
 int
-cli_cmd_tokens_fill (char **tokens, const char *template)
+cli_cmd_tokens_fill(char **tokens, const char *template)
 {
-        const char  *trav = NULL;
-        char       **tokenp = NULL;
-        char        *token = NULL;
-        int          ret = 0;
-        int          ch = 0;
+    const char *trav = NULL;
+    char **tokenp = NULL;
+    char *token = NULL;
+    int ret = 0;
+    int ch = 0;
 
-        tokenp = tokens;
+    tokenp = tokens;
 
-        for (trav = template; *trav; trav++) {
-                ch = *trav;
+    for (trav = template; *trav; trav++) {
+        ch = *trav;
 
-                if (__is_spc (ch))
-                        continue;
+        if (__is_spc(ch))
+            continue;
 
-                if (__is_div (ch)) {
-                        token = calloc (2, 1);
-                        if (!token)
-                                return -1;
-                        token[0] = ch;
+        if (__is_div(ch)) {
+            token = calloc(2, 1);
+            if (!token)
+                return -1;
+            token[0] = ch;
 
-                        *tokenp = token;
-                        tokenp++;
+            *tokenp = token;
+            tokenp++;
 
-                        continue;
-                }
+            continue;
+        }
 
-                token = strdup (trav);
-                *tokenp = token;
-                tokenp++;
+        token = strdup(trav);
+        *tokenp = token;
+        tokenp++;
 
-                for (token++; *token; token++) {
-                        if (__is_spc (*token) || __is_div (*token)) {
-                                *token = 0;
-                                break;
-                        }
-                        trav++;
-                }
+        for (token++; *token; token++) {
+            if (__is_spc(*token) || __is_div(*token)) {
+                *token = 0;
+                break;
+            }
+            trav++;
         }
+    }
 
-        return ret;
+    return ret;
 }
 
-
 char **
-cli_cmd_tokenize (const char *template)
+cli_cmd_tokenize(const char *template)
 {
-        char **tokens = NULL;
-        int    ret = 0;
-        int    count = 0;
+    char **tokens = NULL;
+    int ret = 0;
+    int count = 0;
 
-        ret = is_template_balanced (template);
-        if (ret)
-                return NULL;
+    ret = is_template_balanced(template);
+    if (ret)
+        return NULL;
 
-        count = cli_cmd_token_count (template);
-        if (count <= 0)
-                return NULL;
+    count = cli_cmd_token_count(template);
+    if (count <= 0)
+        return NULL;
 
-        tokens = calloc (count + 1, sizeof (char *));
-        if (!tokens)
-                return NULL;
+    tokens = calloc(count + 1, sizeof(char *));
+    if (!tokens)
+        return NULL;
 
-        ret = cli_cmd_tokens_fill (tokens, template);
-        if (ret)
-                goto err;
+    ret = cli_cmd_tokens_fill(tokens, template);
+    if (ret)
+        goto err;
 
-        return tokens;
+    return tokens;
 err:
-        cli_cmd_tokens_destroy (tokens);
-        return NULL;
+    cli_cmd_tokens_destroy(tokens);
+    return NULL;
 }
 
 void *
-cli_getunamb (const char *tok, void **choices, cli_selector_t sel)
+cli_getunamb(const char *tok, void **choices, cli_selector_t sel)
 {
-        void  **wcon = NULL;
-        char      *w = NULL;
-        unsigned  mn = 0;
-        void    *ret = NULL;
-
-        if (!choices || !tok || !*tok)
-                return NULL;
-
-        for (wcon = choices; *wcon; wcon++) {
-                w = strtail ((char *)sel (*wcon), tok);
-                if (!w)
-                        /* no match */
-                        continue;
-                if (!*w)
-                        /* exact match */
-                        return *wcon;
-
-                ret = *wcon;
-                mn++;
-        }
+    void **wcon = NULL;
+    char *w = NULL;
+    unsigned mn = 0;
+    void *ret = NULL;
 
-#ifdef FORCE_MATCH_EXACT
+    if (!choices || !tok || !*tok)
         return NULL;
+
+    for (wcon = choices; *wcon; wcon++) {
+        w = strtail((char *)sel(*wcon), tok);
+        if (!w)
+            /* no match */
+            continue;
+        if (!*w)
+            /* exact match */
+            return *wcon;
+
+        ret = *wcon;
+        mn++;
+    }
+
+#ifdef FORCE_MATCH_EXACT
+    return NULL;
 #else
-        return (mn == 1) ? ret : NULL;
+    return (mn == 1) ? ret : NULL;
 #endif
 }
 
 static const char *
-sel_cmd_word (void *wcon)
+sel_cmd_word(void *wcon)
 {
-        return ((struct cli_cmd_word *)wcon)->word;
+    return ((struct cli_cmd_word *)wcon)->word;
 }
 
 struct cli_cmd_word *
-cli_cmd_nextword (struct cli_cmd_word *word, const char *token)
+cli_cmd_nextword(struct cli_cmd_word *word, const char *token)
 {
-        return (struct cli_cmd_word *)cli_getunamb (token,
-                                                    (void **)word->nextwords,
-                                                    sel_cmd_word);
+    return (struct cli_cmd_word *)cli_getunamb(token, (void **)word->nextwords,
+                                               sel_cmd_word);
 }
 
-
 struct cli_cmd_word *
-cli_cmd_newword (struct cli_cmd_word *word, const char *token)
+cli_cmd_newword(struct cli_cmd_word *word, const char *token)
 {
-        struct cli_cmd_word **nextwords = NULL;
-        struct cli_cmd_word  *nextword = NULL;
+    struct cli_cmd_word **nextwords = NULL;
+    struct cli_cmd_word *nextword = NULL;
 
-        nextwords = realloc (word->nextwords,
-                             (word->nextwords_cnt + 2) * sizeof (*nextwords));
-        if (!nextwords)
-                return NULL;
+    nextwords = realloc(word->nextwords,
+                        (word->nextwords_cnt + 2) * sizeof(*nextwords));
+    if (!nextwords)
+        return NULL;
 
-        word->nextwords = nextwords;
+    word->nextwords = nextwords;
 
-        nextword = calloc (1, sizeof (*nextword));
-        if (!nextword)
-                return NULL;
+    nextword = calloc(1, sizeof(*nextword));
+    if (!nextword)
+        return NULL;
 
-        nextword->word = strdup (token);
-        if (!nextword->word) {
-                free (nextword);
-                return NULL;
-        }
+    nextword->word = strdup(token);
+    if (!nextword->word) {
+        free(nextword);
+        return NULL;
+    }
 
-        nextword->tree = word->tree;
-        nextwords[word->nextwords_cnt++] = nextword;
-        nextwords[word->nextwords_cnt] = NULL;
+    nextword->tree = word->tree;
+    nextwords[word->nextwords_cnt++] = nextword;
+    nextwords[word->nextwords_cnt] = NULL;
 
-        return nextword;
+    return nextword;
 }
 
-
 int
-cli_cmd_ingest (struct cli_cmd_tree *tree, char **tokens, cli_cmd_cbk_t *cbkfn,
-                const char *desc, const char *pattern)
+cli_cmd_ingest(struct cli_cmd_tree *tree, char **tokens, cli_cmd_cbk_t *cbkfn,
+               const char *desc, const char *pattern)
 {
-        int                    ret = 0;
-        char                 **tokenp = NULL;
-        char                  *token = NULL;
-        struct cli_cmd_word   *word = NULL;
-        struct cli_cmd_word   *next = NULL;
-
-        word = &tree->root;
+    int ret = 0;
+    char **tokenp = NULL;
+    char *token = NULL;
+    struct cli_cmd_word *word = NULL;
+    struct cli_cmd_word *next = NULL;
 
-        for (tokenp = tokens; (token = *tokenp); tokenp++) {
-                if (!__is_word (token))
-                        break;
+    word = &tree->root;
 
-                next = cli_cmd_nextword (word, token);
-                if (!next)
-                        next = cli_cmd_newword (word, token);
+    for (tokenp = tokens; (token = *tokenp); tokenp++) {
+        if (!__is_word(token))
+            break;
 
-                word = next;
-                if (!word)
-                        break;
-        }
+        next = cli_cmd_nextword(word, token);
+        if (!next)
+            next = cli_cmd_newword(word, token);
 
+        word = next;
         if (!word)
-                return -1;
+            break;
+    }
 
-        if (word->cbkfn) {
-                /* warning - command already registered */
-        }
+    if (!word)
+        return -1;
 
-        word->cbkfn = cbkfn;
-        word->desc  = desc;
-        word->pattern = pattern;
+    if (word->cbkfn) {
+        /* warning - command already registered */
+    }
 
-        /* end of static strings in command template */
+    word->cbkfn = cbkfn;
+    word->desc = desc;
+    word->pattern = pattern;
 
-        /* TODO: autocompletion beyond this point is just "nice to have" */
+    /* end of static strings in command template */
 
-        return ret;
-}
+    /* TODO: autocompletion beyond this point is just "nice to have" */
 
+    return ret;
+}
 
 int
-cli_cmd_register (struct cli_cmd_tree *tree, struct cli_cmd *cmd)
+cli_cmd_register(struct cli_cmd_tree *tree, struct cli_cmd *cmd)
 {
-        char **tokens = NULL;
-        int    ret = 0;
+    char **tokens = NULL;
+    int ret = 0;
 
-        GF_ASSERT (cmd);
+    GF_ASSERT(cmd);
 
-        if (cmd->reg_cbk)
-                cmd->reg_cbk (cmd);
+    if (cmd->reg_cbk)
+        cmd->reg_cbk(cmd);
 
-        if (cmd->disable) {
-                ret = 0;
-                goto out;
-        }
+    if (cmd->disable) {
+        ret = 0;
+        goto out;
+    }
 
-        tokens = cli_cmd_tokenize (cmd->pattern);
-        if (!tokens) {
-                ret = -1;
-                goto out;
-        }
+    tokens = cli_cmd_tokenize(cmd->pattern);
+    if (!tokens) {
+        ret = -1;
+        goto out;
+    }
 
-        ret = cli_cmd_ingest (tree, tokens, cmd->cbk, cmd->desc, cmd->pattern);
-        if (ret) {
-                ret = -1;
-                goto out;
-        }
+    ret = cli_cmd_ingest(tree, tokens, cmd->cbk, cmd->desc, cmd->pattern);
+    if (ret) {
+        ret = -1;
+        goto out;
+    }
 
-        ret = 0;
+    ret = 0;
 
 out:
-        if (tokens)
-                cli_cmd_tokens_destroy (tokens);
+    if (tokens)
+        cli_cmd_tokens_destroy(tokens);
 
-        gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
-        return ret;
+    gf_log("cli", GF_LOG_DEBUG, "Returning %d", ret);
+    return ret;
 }
-
diff --git a/configure.ac b/configure.ac
index 54364cda71f..e2d6fd66cec 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1,4 +1,4 @@
-dnl  Copyright (c) 2006-2012 Red Hat, Inc. <http://www.redhat.com>
+dnl  Copyright (c) 2006-2016 Red Hat, Inc. <http://www.redhat.com>
 dnl  This file is part of GlusterFS.
 dnl
 dnl  This file is licensed to you under your choice of the GNU Lesser
@@ -13,7 +13,7 @@ AC_INIT([glusterfs],
 AC_SUBST([PACKAGE_RELEASE],
          [m4_esyscmd([build-aux/pkg-version --release])])
 
-AM_INIT_AUTOMAKE
+AM_INIT_AUTOMAKE([tar-pax foreign])
 
 # Removes warnings when using automake 1.14 around (...but option 'subdir-objects' is disabled )
 #but libglusterfs fails to build with contrib (Then are not set up that way?)
@@ -21,26 +21,21 @@ AM_INIT_AUTOMAKE
 
 m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES(yes)])
 
-if make --help 2>&1 | grep -q no-print-directory; then
-   AM_MAKEFLAGS="$AM_MAKEFLAGS --no-print-directory";
-fi
-
-if make --help 2>&1 | grep -q quiet; then
-   AM_MAKEFLAGS="$AM_MAKEFLAGS --quiet"
-fi
-
-if libtool --help 2>&1 | grep -q quiet; then
-   AM_LIBTOOLFLAGS="--quiet";
-fi
-
-AC_CONFIG_HEADERS([config.h])
+AC_CONFIG_HEADERS([config.h site.h])
 
 AC_CONFIG_FILES([Makefile
                 libglusterfs/Makefile
                 libglusterfs/src/Makefile
+                libglusterd/Makefile
+                libglusterd/src/Makefile
                 geo-replication/src/peer_gsec_create
-                geo-replication/src/peer_add_secret_pub
-                geo-replication/syncdaemon/configinterface.py
+                geo-replication/src/peer_mountbroker
+                geo-replication/src/peer_mountbroker.py
+                geo-replication/src/peer_georep-sshkey.py
+                extras/peer_add_secret_pub
+                geo-replication/syncdaemon/conf.py
+                geo-replication/gsyncd.conf
+                extras/snap_scheduler/conf.py
                 glusterfsd/Makefile
                 glusterfsd/src/Makefile
                 rpc/Makefile
@@ -49,8 +44,6 @@ AC_CONFIG_FILES([Makefile
                 rpc/rpc-transport/Makefile
                 rpc/rpc-transport/socket/Makefile
                 rpc/rpc-transport/socket/src/Makefile
-                rpc/rpc-transport/rdma/Makefile
-                rpc/rpc-transport/rdma/src/Makefile
                 rpc/xdr/Makefile
                 rpc/xdr/src/Makefile
                 xlators/Makefile
@@ -65,13 +58,9 @@ AC_CONFIG_FILES([Makefile
                 xlators/storage/Makefile
                 xlators/storage/posix/Makefile
                 xlators/storage/posix/src/Makefile
-                xlators/storage/bd/Makefile
-                xlators/storage/bd/src/Makefile
                 xlators/cluster/Makefile
                 xlators/cluster/afr/Makefile
                 xlators/cluster/afr/src/Makefile
-                xlators/cluster/stripe/Makefile
-                xlators/cluster/stripe/src/Makefile
                 xlators/cluster/dht/Makefile
                 xlators/cluster/dht/src/Makefile
                 xlators/cluster/ec/Makefile
@@ -87,19 +76,23 @@ AC_CONFIG_FILES([Makefile
                 xlators/performance/io-threads/src/Makefile
                 xlators/performance/io-cache/Makefile
                 xlators/performance/io-cache/src/Makefile
-                xlators/performance/symlink-cache/Makefile
-                xlators/performance/symlink-cache/src/Makefile
                 xlators/performance/quick-read/Makefile
                 xlators/performance/quick-read/src/Makefile
                 xlators/performance/open-behind/Makefile
                 xlators/performance/open-behind/src/Makefile
                 xlators/performance/md-cache/Makefile
                 xlators/performance/md-cache/src/Makefile
+                xlators/performance/nl-cache/Makefile
+                xlators/performance/nl-cache/src/Makefile
                 xlators/debug/Makefile
+                xlators/debug/sink/Makefile
+                xlators/debug/sink/src/Makefile
                 xlators/debug/trace/Makefile
                 xlators/debug/trace/src/Makefile
                 xlators/debug/error-gen/Makefile
                 xlators/debug/error-gen/src/Makefile
+                xlators/debug/delay-gen/Makefile
+                xlators/debug/delay-gen/src/Makefile
                 xlators/debug/io-stats/Makefile
                 xlators/debug/io-stats/src/Makefile
                 xlators/protocol/Makefile
@@ -113,51 +106,69 @@ AC_CONFIG_FILES([Makefile
                 xlators/protocol/server/Makefile
                 xlators/protocol/server/src/Makefile
                 xlators/features/Makefile
+                xlators/features/arbiter/Makefile
+                xlators/features/arbiter/src/Makefile
+                xlators/features/thin-arbiter/Makefile
+                xlators/features/thin-arbiter/src/Makefile
                 xlators/features/changelog/Makefile
                 xlators/features/changelog/src/Makefile
                 xlators/features/changelog/lib/Makefile
                 xlators/features/changelog/lib/src/Makefile
-                xlators/features/glupy/Makefile
-                xlators/features/glupy/examples/Makefile
-                xlators/features/glupy/src/Makefile
-                xlators/features/glupy/src/setup.py
-                xlators/features/glupy/src/__init__.py
                 xlators/features/locks/Makefile
                 xlators/features/locks/src/Makefile
                 xlators/features/quota/Makefile
                 xlators/features/quota/src/Makefile
                 xlators/features/marker/Makefile
                 xlators/features/marker/src/Makefile
+                xlators/features/selinux/Makefile
+                xlators/features/selinux/src/Makefile
+                xlators/features/sdfs/Makefile
+                xlators/features/sdfs/src/Makefile
                 xlators/features/read-only/Makefile
                 xlators/features/read-only/src/Makefile
                 xlators/features/compress/Makefile
                 xlators/features/compress/src/Makefile
-                xlators/features/mac-compat/Makefile
-                xlators/features/mac-compat/src/Makefile
+                xlators/features/namespace/Makefile
+                xlators/features/namespace/src/Makefile
                 xlators/features/quiesce/Makefile
                 xlators/features/quiesce/src/Makefile
                 xlators/features/barrier/Makefile
                 xlators/features/barrier/src/Makefile
                 xlators/features/index/Makefile
                 xlators/features/index/src/Makefile
-                xlators/features/protect/Makefile
-                xlators/features/protect/src/Makefile
                 xlators/features/gfid-access/Makefile
                 xlators/features/gfid-access/src/Makefile
+                xlators/features/trash/Makefile
+                xlators/features/trash/src/Makefile
                 xlators/features/snapview-server/Makefile
                 xlators/features/snapview-server/src/Makefile
                 xlators/features/snapview-client/Makefile
                 xlators/features/snapview-client/src/Makefile
+                xlators/features/upcall/Makefile
+                xlators/features/upcall/src/Makefile
+                xlators/features/shard/Makefile
+                xlators/features/shard/src/Makefile
+                xlators/features/bit-rot/Makefile
+                xlators/features/bit-rot/src/Makefile
+                xlators/features/bit-rot/src/stub/Makefile
+                xlators/features/bit-rot/src/bitd/Makefile
+                xlators/features/leases/Makefile
+                xlators/features/leases/src/Makefile
+                xlators/features/cloudsync/Makefile
+                xlators/features/cloudsync/src/Makefile
+                xlators/features/utime/Makefile
+                xlators/features/utime/src/Makefile
+                xlators/features/cloudsync/src/cloudsync-plugins/Makefile
+                xlators/features/cloudsync/src/cloudsync-plugins/src/Makefile
+                xlators/features/cloudsync/src/cloudsync-plugins/src/cloudsyncs3/Makefile
+                xlators/features/cloudsync/src/cloudsync-plugins/src/cloudsyncs3/src/Makefile
+                xlators/features/cloudsync/src/cloudsync-plugins/src/cvlt/Makefile
+                xlators/features/cloudsync/src/cloudsync-plugins/src/cvlt/src/Makefile
+                xlators/features/metadisp/Makefile
+                xlators/features/metadisp/src/Makefile
                 xlators/playground/Makefile
                 xlators/playground/template/Makefile
                 xlators/playground/template/src/Makefile
-                xlators/encryption/Makefile
-                xlators/encryption/rot-13/Makefile
-                xlators/encryption/rot-13/src/Makefile
-                xlators/encryption/crypt/Makefile
-                xlators/encryption/crypt/src/Makefile
-                xlators/features/qemu-block/Makefile
-                xlators/features/qemu-block/src/Makefile
                 xlators/system/Makefile
                 xlators/system/posix-acl/Makefile
                 xlators/system/posix-acl/src/Makefile
@@ -172,14 +183,26 @@ AC_CONFIG_FILES([Makefile
                 doc/Makefile
                 extras/Makefile
                 extras/glusterd.vol
+                extras/cliutils/Makefile
                 extras/init.d/Makefile
                 extras/init.d/glusterd.plist
                 extras/init.d/glusterd-Debian
                 extras/init.d/glusterd-Redhat
                 extras/init.d/glusterd-FreeBSD
                 extras/init.d/glusterd-SuSE
+                extras/init.d/glustereventsd-Debian
+                extras/init.d/glustereventsd-Redhat
+                extras/init.d/glustereventsd-FreeBSD
+                extras/ganesha/Makefile
+                extras/ganesha/config/Makefile
+                extras/ganesha/scripts/Makefile
+                extras/ganesha/ocf/Makefile
                 extras/systemd/Makefile
                 extras/systemd/glusterd.service
+                extras/systemd/glustereventsd.service
+                extras/systemd/glusterfssharedstorage.service
+                extras/systemd/gluster-ta-volume.service
+                extras/run-gluster.tmpfiles
                 extras/benchmarking/Makefile
                 extras/hook-scripts/Makefile
                 extras/ocf/Makefile
@@ -187,9 +210,15 @@ AC_CONFIG_FILES([Makefile
                 extras/ocf/volume
                 extras/LinuxRPM/Makefile
                 extras/geo-rep/Makefile
+                extras/geo-rep/schedule_georep.py
+                extras/firewalld/Makefile
                 extras/hook-scripts/add-brick/Makefile
                 extras/hook-scripts/add-brick/pre/Makefile
                 extras/hook-scripts/add-brick/post/Makefile
+                extras/hook-scripts/create/Makefile
+                extras/hook-scripts/create/post/Makefile
+                extras/hook-scripts/delete/Makefile
+                extras/hook-scripts/delete/pre/Makefile
                 extras/hook-scripts/start/Makefile
                 extras/hook-scripts/start/post/Makefile
                 extras/hook-scripts/set/Makefile
@@ -199,21 +228,33 @@ AC_CONFIG_FILES([Makefile
                 extras/hook-scripts/reset/Makefile
                 extras/hook-scripts/reset/post/Makefile
                 extras/hook-scripts/reset/pre/Makefile
+                extras/python/Makefile
+                extras/snap_scheduler/Makefile
+                events/Makefile
+                events/src/Makefile
+                events/src/eventsapiconf.py
+                events/tools/Makefile
                 contrib/fuse-util/Makefile
                 contrib/umountd/Makefile
-                contrib/uuid/uuid_types.h
                 glusterfs-api.pc
                 libgfchangelog.pc
                 api/Makefile
                 api/src/Makefile
-                api/src/gfapi.map
                 api/examples/Makefile
                 geo-replication/Makefile
                 geo-replication/src/Makefile
                 geo-replication/syncdaemon/Makefile
+                tools/Makefile
+                tools/gfind_missing_files/Makefile
                 heal/Makefile
                 heal/src/Makefile
-                glusterfs.spec])
+                glusterfs.spec
+                tools/glusterfind/src/tool.conf
+                tools/glusterfind/glusterfind
+                tools/glusterfind/Makefile
+                tools/glusterfind/src/Makefile
+                tools/setgfid2path/Makefile
+                tools/setgfid2path/src/Makefile])
 
 AC_CANONICAL_HOST
 
@@ -234,10 +275,124 @@ AC_ARG_ENABLE([debug],
                              [Enable debug build options.]))
 if test "x$enable_debug" = "xyes"; then
         BUILD_DEBUG=yes
-        CFLAGS="${CFLAGS} -g -O0 -DDEBUG"
+        GF_CFLAGS="${GF_CFLAGS} -g -O0 -DDEBUG"
 else
         BUILD_DEBUG=no
-        CFLAGS="${CFLAGS} -g -O2"
+fi
+
+SANITIZER=none
+
+AC_ARG_ENABLE([asan],
+              AC_HELP_STRING([--enable-asan],
+                             [Enable Address Sanitizer support]))
+if test "x$enable_asan" = "xyes"; then
+        SANITIZER=asan
+        AC_CHECK_LIB([asan], [__asan_init], ,
+                [AC_MSG_ERROR([--enable-asan requires libasan.so, exiting])])
+        GF_CFLAGS="${GF_CFLAGS} -O2 -g -fsanitize=address -fno-omit-frame-pointer"
+        GF_LDFLAGS="${GF_LDFLAGS} -lasan"
+fi
+
+AC_ARG_ENABLE([tsan],
+              AC_HELP_STRING([--enable-tsan],
+                             [Enable Thread Sanitizer support]))
+if test "x$enable_tsan" = "xyes"; then
+        if test "x$SANITIZER" != "xnone"; then
+                AC_MSG_ERROR([only one sanitizer can be enabled at once])
+        fi
+        SANITIZER=tsan
+        AC_CHECK_LIB([tsan], [__tsan_init], ,
+                [AC_MSG_ERROR([--enable-tsan requires libtsan.so, exiting])])
+        if test "x$ac_cv_lib_tsan___tsan_init" = xyes; then
+                AC_MSG_CHECKING([whether tsan API can be used])
+                saved_CFLAGS=${CFLAGS}
+                CFLAGS="${CFLAGS} -fsanitize=thread"
+                AC_COMPILE_IFELSE(
+                [AC_LANG_PROGRAM([
+                   [#include <sanitizer/tsan_interface.h>]],
+                   [[__tsan_create_fiber(0)]])],
+                   [TSAN_API=yes], [TSAN_API=no])
+                AC_MSG_RESULT([$TSAN_API])
+                if test x$TSAN_API = "xyes"; then
+                   AC_DEFINE(HAVE_TSAN_API, 1, [Define if tsan API can be used.])
+                fi
+                CFLAGS=${saved_CFLAGS}
+        fi
+        GF_CFLAGS="${GF_CFLAGS} -O2 -g -fsanitize=thread -fno-omit-frame-pointer"
+        GF_LDFLAGS="${GF_LDFLAGS} -ltsan"
+fi
+
+AC_ARG_ENABLE([ubsan],
+              AC_HELP_STRING([--enable-ubsan],
+                             [Enable Undefined Behavior Sanitizer support]))
+if test "x$enable_ubsan" = "xyes"; then
+        if test "x$SANITIZER" != "xnone"; then
+                AC_MSG_ERROR([only one sanitizer can be enabled at once])
+        fi
+        SANITIZER=ubsan
+        AC_CHECK_LIB([ubsan], [__ubsan_default_options], ,
+                [AC_MSG_ERROR([--enable-ubsan requires libubsan.so, exiting])])
+        GF_CFLAGS="${GF_CFLAGS} -O2 -g -fsanitize=undefined -fno-omit-frame-pointer"
+        GF_LDFLAGS="${GF_LDFLAGS} -lubsan"
+fi
+
+# Initialize CFLAGS before usage
+BUILD_TCMALLOC=no
+AC_ARG_ENABLE([tcmalloc],
+              AC_HELP_STRING([--enable-tcmalloc],
+                             [Enable linking with tcmalloc library.]))
+if test "x$enable_tcmalloc" = "xyes"; then
+    BUILD_TCMALLOC=yes
+    GF_CFLAGS="${GF_CFLAGS} -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free"
+    AC_CHECK_LIB([tcmalloc], [malloc], [],
+                 [AC_MSG_ERROR([when --enable-tcmalloc is used, tcmalloc library needs to be present])])
+    GF_LDFLAGS="-ltcmalloc ${GF_LDFLAGS}"
+fi
+
+
+dnl When possible, prefer libtirpc over glibc rpc.
+dnl
+dnl On newer linux with only libtirpc, use libtirpc. (Specifying
+dnl --without-libtirpc is an error.)
+dnl
+dnl on older linux with glibc rpc and WITH libtirpc, use libtirpc
+dnl by default except when configured with --without-libtirpc.
+dnl
+dnl on old linux with glibc rpc and WITHOUT libtirpc, default to
+dnl use glibc rpc.
+dnl
+AC_ARG_WITH([libtirpc],
+        [AC_HELP_STRING([--without-libtirpc], [Use legacy glibc RPC.])],
+        [with_libtirpc="no"], [with_libtirpc="yes"])
+
+dnl ipv6-default is off by default
+dnl
+dnl ipv6-default requires libtirpc. (glibc rpc does not support IPv6.)
+dnl ipv6-default can only be enabled if libtipc is enabled.
+dnl
+AC_ARG_WITH([ipv6-default],
+        AC_HELP_STRING([--with-ipv6-default], [Set IPv6 as default.]),
+        [with_ipv6_default=${with_libtirpc}], [with_ipv6_default="no"])
+
+AC_CHECK_FILE([/etc/centos-release])
+if test "x$ac_cv_file__etc_centos_release" = "xyes"; then
+        if grep "release 6" /etc/centos-release; then
+                with_ipv6_default="no"
+        fi
+fi
+
+dnl On some distributions '-ldl' isn't automatically added to LIBS
+AC_CHECK_LIB([dl], [dlopen], [LIB_DL=-ldl])
+AC_SUBST(LIB_DL)
+
+AC_ARG_ENABLE([privport_tracking],
+              AC_HELP_STRING([--disable-privport_tracking],
+                             [Disable internal tracking of privileged ports.]))
+TRACK_PRIVPORTS="yes"
+if test x"$enable_privport_tracking" = x"no"; then
+        TRACK_PRIVPORTS="no"
+        AC_DEFINE(GF_DISABLE_PRIVPORT_TRACKING, 1,
+                  [Disable internal tracking of privileged ports.])
 fi
 
 case $host_os in
@@ -247,18 +402,42 @@ case $host_os in
     fi
     # OSX version lesser than 9 has llvm/clang optimization issues which leads to various segfaults
     if test "`/usr/bin/sw_vers | grep ProductVersion: | cut -f 2 | cut -d. -f2`" -lt 9; then
-       CFLAGS="${CFLAGS} -g -O0 -DDEBUG"
+       GF_CFLAGS="${GF_CFLAGS} -g -O0 -DDEBUG"
     fi
     ;;
 esac
 
+# --enable-valgrind prevents calling dlclose(), this leaks memory
+AC_ARG_ENABLE([valgrind],
+              AC_HELP_STRING([--enable-valgrind@<:@=memcheck,drd@:>@],
+                             [Enable valgrind for resource leak (memcheck, which is
+                              the default) or thread synchronization (drd) debugging.]))
+case x$enable_valgrind in
+  xmemcheck|xyes)
+    AC_DEFINE(RUN_WITH_MEMCHECK, 1,
+              [Define if all processes should run under 'valgrind --tool=memcheck'.])
+    VALGRIND_TOOL=memcheck
+    ;;
+  xdrd)
+    AC_DEFINE(RUN_WITH_DRD, 1,
+              [Define if all processes should run under 'valgrind --tool=drd'.])
+    VALGRIND_TOOL=drd
+    ;;
+  x|xno)
+    VALGRIND_TOOL=no
+    ;;
+  *)
+    AC_MSG_ERROR([Please specify --enable-valgrind@<:@=memcheck,drd@:>@])
+    ;;
+esac
+
 AC_ARG_WITH([previous-options],
         [AS_HELP_STRING([--with-previous-options],
                         [read config.status for configure options])
         ],
         [ if test -r ./config.status && \
              args=$(grep 'ac_cs_config=' config.status | \
-                    sed -e 's/.*"\(.*\)".*/\1/'| sed -e "s/'//g") ; then
+                    sed -e 's/.*"\(.*\)".*/\1/' -e "s/'//g" -e "s/--with-previous-options//g") ; then
             echo "###"
             echo "### Rerunning as '$0 $args'"
             echo "###"
@@ -279,10 +458,11 @@ AC_ARG_WITH(mountutildir,
 AC_SUBST(mountutildir)
 
 AC_ARG_WITH(systemddir,
-            [  --with-systemddir=DIR systemd service files in DIR @<:@/usr/lib/systemd/system@:>@],
+            [  --with-systemddir=DIR systemd service files in DIR @<:@PREFIX/lib/systemd/system@:>@],
             [systemddir=$withval],
-            [systemddir='/usr/lib/systemd/system'])
+            [systemddir='${prefix}/lib/systemd/system'])
 AC_SUBST(systemddir)
+AM_CONDITIONAL([USE_SYSTEMD], test [ -d '/usr/lib/systemd/system' ])
 
 AC_ARG_WITH(initdir,
             [  --with-initdir=DIR init.d scripts in DIR @<:@/etc/init.d@:>@],
@@ -296,6 +476,13 @@ AC_ARG_WITH(launchddir,
             [launchddir='/Library/LaunchDaemons'])
 AC_SUBST(launchddir)
 
+AC_ARG_WITH(tmpfilesdir,
+            AC_HELP_STRING([--with-tmpfilesdir=DIR],
+                           [tmpfiles config in DIR, disabled by default]),
+            [tmpfilesdir=$withval],
+            [tmpfilesdir=''])
+AC_SUBST(tmpfilesdir)
+
 AC_ARG_WITH([ocf],
             [AS_HELP_STRING([--without-ocf], [build OCF-compliant cluster resource agents])],
             ,
@@ -303,6 +490,13 @@ AC_ARG_WITH([ocf],
             )
 AC_SUBST(OCF_SUBDIR)
 
+AC_ARG_WITH([server],
+            [AS_HELP_STRING([--without-server], [do not build server components])],
+            [with_server='no'],
+            [with_server='yes'],
+            )
+AM_CONDITIONAL([WITH_SERVER], [test x$with_server = xyes])
+
 # LEX needs a check
 AC_PROG_LEX
 if test  "x${LEX}" != "xflex" -a "x${FLEX}" != "xlex"; then
@@ -337,21 +531,6 @@ AC_CHECK_LIB([crypto], [MD5], , AC_MSG_ERROR([OpenSSL crypto library is required
 
 AC_CHECK_LIB([pthread], [pthread_mutex_init], , AC_MSG_ERROR([Posix threads library is required to build glusterfs]))
 
-
-PKG_CHECK_MODULES([UNITTEST], [cmockery2], [
-    UNITTEST_CPPFLAGS=`${PKG_CONFIG} --print-errors --cflags-only-I "cmockery2"`
-    UNITTEST_CFLAGS=`${PKG_CONFIG} --print-errors --cflags-only-other "cmockery2"`
-    UNITTEST_LDFLAGS=`${PKG_CONFIG} --print-errors --libs-only-L "cmockery2"`
-    UNITTEST_LIBS=`${PKG_CONFIG} --print-errors --libs-only-l "cmockery2"`
-],[
-  AC_CHECK_LIB([cmockery], [mock_assert], [
-    UNITTEST_LDFLAGS="-lcmockery -lgcov"
-    UNITTEST_CFLAGS="-Wall -Werror -DUNIT_TESTING=1"
-  ], [
-    AC_MSG_ERROR([cmockery2 library is required to build glusterfs])
-  ])
-])
-
 AC_CHECK_FUNC([dlopen], [has_dlopen=yes], AC_CHECK_LIB([dl], [dlopen], , AC_MSG_ERROR([Dynamic linking library required to build glusterfs])))
 
 AC_CHECK_LIB([readline], [rl_do_undo], [RL_UNDO="yes"], [RL_UNDO="no"])
@@ -364,6 +543,92 @@ AC_CHECK_HEADERS([sys/ioctl.h], AC_DEFINE(HAVE_IOCTL_IN_SYS_IOCTL_H, 1, [have sy
 
 AC_CHECK_HEADERS([sys/extattr.h])
 
+AC_CHECK_HEADERS([openssl/dh.h])
+
+AC_CHECK_HEADERS([openssl/ecdh.h])
+
+AC_CHECK_LIB([ssl], [SSL_CTX_get0_param], [AC_DEFINE([HAVE_SSL_CTX_GET0_PARAM], [1], [define if found OpenSSL SSL_CTX_get0_param])])
+
+dnl Math library
+AC_CHECK_LIB([m], [pow], [MATH_LIB='-lm'], [MATH_LIB=''])
+AC_SUBST(MATH_LIB)
+
+dnl depend on libuuid.so
+PKG_CHECK_MODULES([UUID], [uuid],
+        [have_uuid=yes
+         AC_DEFINE(HAVE_LIBUUID, 1, [have libuuid.so])
+         PKGCONFIG_UUID=uuid],
+        [have_uuid=no])
+AM_CONDITIONAL([HAVE_LIBUUID], [test x$have_uuid = xyes])
+
+dnl older version of libuuid (from e2fsprogs) require including uuid/uuid.h
+saved_CFLAGS=${CFLAGS}
+CFLAGS="${CFLAGS} ${UUID_CFLAGS}"
+AC_CHECK_HEADER([uuid.h], [], [AC_CHECK_HEADER([uuid/uuid.h])],
+                [[#if HAVE_UUID_H
+                  #include <uuid.h>
+                  #endif
+                ]])
+CFLAGS=${saved_CFLAGS}
+if test "x$ac_cv_header_uuid_uuid_h" = "xyes"; then
+    UUID_CFLAGS="${UUID_CFLAGS} -I$(pkg-config --variable=includedir uuid)/uuid"
+    have_uuid=yes
+fi
+
+if test "x$have_uuid" != "xyes"; then
+    case $host_os in
+        *freebsd*)
+            AC_MSG_ERROR([e2fsprogs-libuuid is required to build glusterfs])
+            ;;
+        linux*)
+            AC_MSG_ERROR([libuuid is required to build glusterfs])
+            ;;
+        *)
+            AC_MSG_ERROR([a Linux compatible libuuid is required to build glusterfs])
+            ;;
+    esac
+fi
+
+dnl libglusterfs needs uuid.h, practically everything depends on it
+GF_CFLAGS="${GF_CFLAGS} ${UUID_CFLAGS}"
+dnl PKGCONFIG_UUID is used for the dependency in *.pc.in files
+AC_SUBST(PKGCONFIG_UUID)
+
+dnl NetBSD does not support POSIX ACLs :-(
+case $host_os in
+     *netbsd* | darwin*)
+        AC_MSG_WARN([platform does not support POSIX ACLs... disabling them])
+        ACL_LIBS=''
+        USE_POSIX_ACLS='0'
+        BUILD_POSIX_ACLS='no'
+        ;;
+     *)
+        AC_CHECK_HEADERS([sys/acl.h], ,
+                         AC_MSG_ERROR([Support for POSIX ACLs is required]))
+        USE_POSIX_ACLS='1'
+        BUILD_POSIX_ACLS='yes'
+        case $host_os in
+             linux*)
+                ACL_LIBS='-lacl'
+                ;;
+             solaris*)
+                ACL_LIBS='-lsec'
+                ;;
+             *freebsd*)
+                ACL_LIBS='-lc'
+                ;;
+             darwin*)
+                ACL_LIBS='-lc'
+                ;;
+        esac
+        if test "x${ACL_LIBS}" = "x-lacl"; then
+          AC_CHECK_HEADERS([acl/libacl.h], , AC_MSG_ERROR([libacl is required for building on ${host_os}]))
+        fi
+        ;;
+esac
+AC_SUBST(ACL_LIBS)
+AC_SUBST(USE_POSIX_ACLS)
+
 # libglusterfs/checksum
 AC_CHECK_HEADERS([openssl/md5.h])
 AC_CHECK_LIB([z], [adler32], [ZLIB_LIBS="-lz"], AC_MSG_ERROR([zlib is required to build glusterfs]))
@@ -371,6 +636,8 @@ AC_SUBST(ZLIB_LIBS)
 
 AC_CHECK_HEADERS([linux/falloc.h])
 
+AC_CHECK_HEADERS([linux/oom.h], AC_DEFINE(HAVE_LINUX_OOM_H, 1, [have linux/oom.h]))
+
 dnl Mac OS X does not have spinlocks
 AC_CHECK_FUNC([pthread_spin_init], [have_spinlock=yes])
 if test "x${have_spinlock}" = "xyes"; then
@@ -400,6 +667,57 @@ if test "x${have_umount2}" = "xyes"; then
    AC_DEFINE(HAVE_UMOUNT2, 1, [define if found umount2])
 fi
 
+dnl Check Python Availability
+have_python=no
+dnl if the user has not specified a python, pick one
+if test -z "${PYTHON}"; then
+  case $host_os in
+    freebsd*)
+      if test -x /usr/local/bin/python3; then
+        PYTHON=/usr/local/bin/python3
+      else
+        PYTHON=/usr/local/bin/python2
+      fi
+      ;;
+    *)
+      if test -x /usr/bin/python3; then
+        PYTHON=/usr/bin/python3
+      else
+        PYTHON=/usr/bin/python2
+      fi
+      ;;
+  esac
+fi
+AM_PATH_PYTHON([2.6],,[:])
+if test -n "${PYTHON}"; then
+   have_python=yes
+fi
+AM_CONDITIONAL(HAVE_PYTHON, test "x$have_python" = "xyes")
+
+dnl Use pkg-config to get runtime search path missing from ${PYTHON}-config
+dnl Just do "true" on failure so that configure does not bail out
+dnl Note: python 2.6's devel pkg (e.g. in CentOS/RHEL 6) does not have
+dnl pkg-config files, so this work-around instead
+if test "x${PYTHON_VERSION}" = "x2.6"; then
+  PYTHON_CFLAGS=$(python-config --includes)
+  PYTHON_LIBS=$(python-config --libs)
+else
+  PKG_CHECK_MODULES([PYTHON], "python-${PYTHON_VERSION}",,true)
+fi
+
+PYTHON_CFLAGS=$(echo ${PYTHON_CFLAGS} | sed -e 's|-I|-isystem |')
+
+BUILD_PYTHON_SITE_PACKAGES=${pythondir}
+AC_SUBST(BUILD_PYTHON_SITE_PACKAGES)
+
+# Eval two times to expand fully. First eval replaces $exec_prefix into $prefix
+# Second eval will expand $prefix
+build_python_site_packages_temp="${pythondir}"
+eval build_python_site_packages_temp=\"${build_python_site_packages_temp}\"
+eval build_python_site_packages_temp=\"${build_python_site_packages_temp}\"
+BUILD_PYTHON_SITE_PACKAGES_EXPANDED=${build_python_site_packages_temp}
+AC_SUBST(BUILD_PYTHON_SITE_PACKAGES_EXPANDED)
+
 # FUSE section
 AC_ARG_ENABLE([fuse-client],
               AC_HELP_STRING([--disable-fuse-client],
@@ -411,65 +729,25 @@ if test "x$enable_fuse_client" != "xno"; then
    BUILD_FUSE_CLIENT="yes"
 fi
 
-AC_ARG_ENABLE([bd-xlator],
-              AC_HELP_STRING([--enable-bd-xlator], [Build BD xlator]))
-
-if test "x$enable_bd_xlator" != "xno"; then
-  AC_CHECK_LIB([lvm2app],
-               [lvm_init,lvm_lv_from_name],
-               [HAVE_BD_LIB="yes"],
-               [HAVE_BD_LIB="no"])
-
-if test "x$HAVE_BD_LIB" = "xyes"; then
-    # lvm_lv_from_name() has been made public with lvm2-2.02.79
-    AC_CHECK_DECLS(
-                  [lvm_lv_from_name],
-                  [NEED_LVM_LV_FROM_NAME_DECL="no"],
-                  [NEED_LVM_LV_FROM_NAME_DECL="yes"],
-                  [[#include <lvm2app.h>]])
-  fi
-fi
-
-if test "x$enable_bd_xlator" = "xyes" -a "x$HAVE_BD_LIB" = "xno"; then
-   echo "BD xlator requested but required lvm2 development library not found."
-   exit 1
-fi
-
-BUILD_BD_XLATOR=no
-if test "x${enable-bd-xlator}" != "xno" -a "x${HAVE_BD_LIB}" = "xyes"; then
-  BUILD_BD_XLATOR=yes
-  AC_DEFINE(HAVE_BD_XLATOR, 1, [define if lvm2app library found and bd xlator
-                            enabled])
-  if test "x$NEED_LVM_LV_FROM_NAME_DECL" = "xyes"; then
-    AC_DEFINE(NEED_LVM_LV_FROM_NAME_DECL, 1, [defined if lvm_lv_from_name()
-                  was not found in the lvm2app.h header, but can be linked])
-  fi
-fi
-
-AM_CONDITIONAL([ENABLE_BD_XLATOR], [test x$BUILD_BD_XLATOR = xyes])
+AC_SUBST(FUSE_CLIENT_SUBDIR)
 
-# start encryption/crypt section
+AC_ARG_ENABLE([fuse-notifications],
+  AS_HELP_STRING([--disable-fuse-notifications], [Disable FUSE notifications]))
 
-AC_CHECK_HEADERS([openssl/cmac.h], [have_cmac_h=yes], [have_cmac_h=no])
+AS_IF([test "x$enable_fuse_notifications" != "xno"], [
+  AC_DEFINE([HAVE_FUSE_NOTIFICATIONS], [1], [Use FUSE notifications])
+])
 
-AC_ARG_ENABLE([crypt-xlator],
-	AC_HELP_STRING([--enable-crypt-xlator], [Build crypt encryption xlator]))
+# end FUSE section
 
-if test "x$enable_crypt_xlator" = "xyes" -a "x$have_cmac_h" = "xno"; then
-   AC_MSG_ERROR([Encryption xlator requires OpenSSL with cmac.h])
-fi
 
-BUILD_CRYPT_XLATOR=no
-if test "x$enable_crypt_xlator" != "xno" -a "x$have_cmac_h" = "xyes"; then
-   BUILD_CRYPT_XLATOR=yes
-   AC_DEFINE(HAVE_CRYPT_XLATOR, 1, [enable building crypt encryption xlator])
+AC_CHECK_LIB([ssl], TLS_method, [HAVE_OPENSSL_1_1="yes"], [HAVE_OPENSSL_1_1="no"])
+if test "x$HAVE_OPENSSL_1_1" = "xyes"; then
+    AC_DEFINE([HAVE_TLS_METHOD], [1], [Using OpenSSL-1.1 TLS_method])
+else
+    AC_CHECK_LIB([ssl], TLSv1_2_method, [AC_DEFINE([HAVE_TLSV1_2_METHOD], [1], [Using OpenSSL-1.0 TLSv1_2_method])])
 fi
 
-AM_CONDITIONAL([ENABLE_CRYPT_XLATOR], [test x$BUILD_CRYPT_XLATOR = xyes])
-
-AC_SUBST(FUSE_CLIENT_SUBDIR)
-# end FUSE section
-
 
 # FUSERMOUNT section
 AC_ARG_ENABLE([fusermount],
@@ -487,33 +765,6 @@ fi
 AC_SUBST(FUSERMOUNT_SUBDIR)
 #end FUSERMOUNT section
 
-# QEMU_BLOCK section
-
-AC_ARG_ENABLE([qemu-block],
-                AC_HELP_STRING([--enable-qemu-block],
-                        [Build QEMU Block formats translator]))
-
-if test "x$enable_qemu_block" != "xno"; then
-  PKG_CHECK_MODULES([GLIB], [glib-2.0],
-                    [HAVE_GLIB_2="yes"],
-  		    [HAVE_GLIB_2="no"])
-fi
-
-if test "x$enable_qemu_block" = "xyes" -a "x$HAVE_GLIB_2" = "xno"; then
-   echo "QEMU Block formats translator requires libglib-2.0, but missing."
-   exit 1
-fi
-
-BUILD_QEMU_BLOCK=no
-if test "x${enable_qemu_block}" != "xno" -a "x${HAVE_GLIB_2}" = "xyes"; then
-  BUILD_QEMU_BLOCK=yes
-  AC_DEFINE(HAVE_QEMU_BLOCK, 1, [define if libglib-2.0 library found and QEMU
-                  	     Block translator enabled])
-fi
-
-
-# end QEMU_BLOCK section
-
 # EPOLL section
 AC_ARG_ENABLE([epoll],
               AC_HELP_STRING([--disable-epoll],
@@ -527,46 +778,6 @@ if test "x$enable_epoll" != "xno"; then
 fi
 # end EPOLL section
 
-
-# IBVERBS section
-AC_ARG_ENABLE([ibverbs],
-              AC_HELP_STRING([--disable-ibverbs],
-                             [Do not build the ibverbs transport]))
-
-if test "x$enable_ibverbs" != "xno"; then
-  AC_CHECK_LIB([ibverbs],
-               [ibv_get_device_list],
-               [HAVE_LIBIBVERBS="yes"],
-               [HAVE_LIBIBVERBS="no"])
-  AC_CHECK_LIB([rdmacm], [rdma_create_id], [HAVE_RDMACM="yes"], [HAVE_RDMACM="no"])
-fi
-
-if test "x$enable_ibverbs" = "xyes"; then
-   if test "x$HAVE_LIBIBVERBS" = "xno"; then
-      echo "ibverbs-transport requested, but libibverbs is not present."
-      exit 1
-   fi
-
-   if test "x$HAVE_RDMACM" = "xno"; then
-      echo "ibverbs-transport requested, but librdmacm is not present."
-      exit 1
-   fi
-fi
-
-BUILD_RDMA=no
-BUILD_IBVERBS=no
-if test "x$enable_ibverbs" != "xno" -a "x$HAVE_LIBIBVERBS" = "xyes" -a "x$HAVE_RDMACM" = "xyes"; then
-  IBVERBS_SUBDIR=ib-verbs
-  BUILD_IBVERBS=yes
-  RDMA_SUBDIR=rdma
-  BUILD_RDMA=yes
-fi
-
-AC_SUBST(IBVERBS_SUBDIR)
-AC_SUBST(RDMA_SUBDIR)
-# end IBVERBS section
-
-
 # SYNCDAEMON section
 AC_ARG_ENABLE([georeplication],
               AC_HELP_STRING([--disable-georeplication],
@@ -574,6 +785,9 @@ AC_ARG_ENABLE([georeplication],
 
 BUILD_SYNCDAEMON=no
 case $host_os in
+     freebsd*)
+#do nothing
+       ;;
      linux*)
 #do nothing
        ;;
@@ -586,31 +800,75 @@ case $host_os in
         ;;
 esac
 SYNCDAEMON_COMPILE=0
-if test "x$enable_georeplication" != "xno"; then
-  SYNCDAEMON_SUBDIR=geo-replication
-  SYNCDAEMON_COMPILE=1
-
-  BUILD_SYNCDAEMON="yes"
-  AM_PATH_PYTHON([2.4])
-  echo -n "checking if python is python 2.x... "
-  if echo $PYTHON_VERSION | grep ^2; then
-    :
-  else
-    echo no
-    AC_MSG_ERROR([only python 2.x is supported])
-  fi
-  echo -n "checking if python has ctypes support... "
-  if "$PYTHON" -c 'import ctypes' 2>/dev/null; then
-    echo yes
+if test "x${with_server}" = "xyes" -a "x${enable_georeplication}" != "xno"; then
+  if test "x${have_python}" = "xno" ; then
+    AC_MSG_ERROR([only python 2 and 3 are supported])
   else
-    echo no
-    AC_MSG_ERROR([python does not have ctypes support])
+    SYNCDAEMON_SUBDIR=geo-replication
+    SYNCDAEMON_COMPILE=1
+
+    BUILD_SYNCDAEMON="yes"
+    AC_MSG_CHECKING([if python has ctypes support...])
+    if "${PYTHON}" -c 'import ctypes' 2>/dev/null; then
+      AC_MSG_RESULT("yes")
+    else
+      AC_MSG_ERROR([python does not have ctypes support])
+    fi
   fi
 fi
 AC_SUBST(SYNCDAEMON_COMPILE)
 AC_SUBST(SYNCDAEMON_SUBDIR)
 # end SYNCDAEMON section
 
+# only install scripts from extras/geo-rep when enabled
+if test "x${with_server}" = "xyes" -a "x$enable_georeplication" != "xno"; then
+  GEOREP_EXTRAS_SUBDIR=geo-rep
+fi
+AC_SUBST(GEOREP_EXTRAS_SUBDIR)
+AM_CONDITIONAL(USE_GEOREP, test "x$enable_georeplication" != "xno")
+
+# METADISP section
+AC_ARG_ENABLE([metadisp],
+              AC_HELP_STRING([--enable-metadisp],
+                             [Enable the metadata dispersal xlator]))
+BUILD_METADISP=no
+if test "x${enable_metadisp}" = "xyes"; then
+  BUILD_METADISP=yes
+fi
+AM_CONDITIONAL([BUILD_METADISP], [test "x$BUILD_METADISP" = "xyes"])
+# end METADISP section
+
+# Events section
+AC_ARG_ENABLE([events],
+              AC_HELP_STRING([--disable-events],
+                             [Do not install Events components]))
+
+BUILD_EVENTS=no
+EVENTS_ENABLED=0
+EVENTS_SUBDIR=
+if test "x$enable_events" != "xno"; then
+  EVENTS_SUBDIR=events
+  EVENTS_ENABLED=1
+
+  BUILD_EVENTS="yes"
+
+  if test "x${have_python}" = "xno"; then
+     if test "x${enable_events}" = "xyes"; then
+        AC_MSG_ERROR([python 2 or 3 required. exiting.])
+     fi
+     AC_MSG_WARN([python not found, disabling events])
+     EVENTS_SUBDIR=
+     EVENTS_ENABLED=0
+     BUILD_EVENTS="no"
+  else
+    AC_DEFINE(USE_EVENTS, 1, [define if events enabled])
+  fi
+fi
+AC_SUBST(EVENTS_ENABLED)
+AC_SUBST(EVENTS_SUBDIR)
+AM_CONDITIONAL([BUILD_EVENTS], [test "x${BUILD_EVENTS}" = "xyes"])
+# end Events section
+
 # CDC xlator - check if libz is present if so enable HAVE_LIB_Z
 BUILD_CDC=yes
 PKG_CHECK_MODULES([ZLIB], [zlib >= 1.2.0],,
@@ -627,37 +885,21 @@ AC_SUBST(ZLIB_CFLAGS)
 AC_SUBST(ZLIB_LIBS)
 # end CDC xlator secion
 
-# check for systemtap/dtrace
-BUILD_SYSTEMTAP=no
-AC_MSG_CHECKING([whether to include systemtap tracing support])
-AC_ARG_ENABLE([systemtap],
-              [AS_HELP_STRING([--enable-systemtap],
-              [Enable inclusion of systemtap trace support])],
-              [ENABLE_SYSTEMTAP="${enableval}"], [ENABLE_SYSTEMTAP="def"])
+#start firewalld section
+BUILD_FIREWALLD="no"
+AC_ARG_ENABLE([firewalld],
+              AC_HELP_STRING([--enable-firewalld],
+                             [enable installation configuration for firewalld]),
+              [BUILD_FIREWALLD="${enableval}"], [BUILD_FIREWALLD="no"])
 
-AM_CONDITIONAL([ENABLE_SYSTEMTAP], [test "x${ENABLE_SYSTEMTAP}" = "xyes"])
-AC_MSG_RESULT(${ENABLE_SYSTEMTAP})
-
-if test "x${ENABLE_SYSTEMTAP}" != "xno"; then
-    AC_CHECK_PROG(DTRACE, dtrace, "yes", "no")
-    AC_CHECK_HEADER([sys/sdt.h], [SDT_H_FOUND="yes"],
-                    [SDT_H_FOUND="no"])
+if test "x${with_server}" = "xyes" -a "x${BUILD_FIREWALLD}" = "xyes"; then
+        if !(test -d /usr/lib/firewalld/services 1>/dev/null 2>&1) ; then
+                BUILD_FIREWALLD="no (firewalld not installed)"
+        fi
 fi
+AM_CONDITIONAL([USE_FIREWALLD],test ["x${BUILD_FIREWALLD}" =  "xyes"])
 
-if test "x${ENABLE_SYSTEMTAP}" = "xyes"; then
-    if test "x${DTRACE}" = "xno"; then
-        AC_MSG_ERROR([dtrace not found])
-    elif test "$x{SDT_H_FOUND}" = "xno"; then
-        AC_MSG_ERROR([systemtap support needs sys/sdt.h header])
-    fi
-fi
-
-if test "x${DTRACE}" = "xyes" -a "x${SDT_H_FOUND}" = "xyes"; then
-    AC_MSG_CHECKING([x"${DTRACE}"xy"${SDT_H_FOUND}"y])
-    AC_DEFINE([HAVE_SYSTEMTAP], [1], [Define to 1 if using  probes.])
-    BUILD_SYSTEMTAP=yes
-fi
-# end of systemtap/dtrace
+#endof firewald section
 
 # xml-output
 AC_ARG_ENABLE([xml-output],
@@ -665,17 +907,16 @@ AC_ARG_ENABLE([xml-output],
                              [Disable the xml output]))
 BUILD_XML_OUTPUT="yes"
 if test "x$enable_xml_output" != "xno"; then
-    #check if libxml is present if so enable HAVE_LIB_XML
-    m4_ifdef([AM_PATH_XML2],[AM_PATH_XML2([2.6.19])], [no_xml=yes])
+    PKG_CHECK_MODULES([XML], [libxml-2.0], [], [no_xml="yes"])
     if test "x${no_xml}" = "x"; then
         AC_DEFINE([HAVE_LIB_XML], [1], [Define to 1 if using libxml2.])
     else
-	if test "x$enable_georeplication" != "xno"; then
+        if test "x$enable_georeplication" != "xno"; then
            AC_MSG_ERROR([libxml2 devel libraries not found])
-	else
-	   AC_MSG_WARN([libxml2 devel libraries not found disabling XML support])
+        else
+           AC_MSG_WARN([libxml2 devel libraries not found disabling XML support])
            BUILD_XML_OUTPUT="no"
-	fi
+        fi
 
     fi
 else
@@ -686,27 +927,86 @@ else
 fi
 # end of xml-output
 
+dnl cloudsync section
+BUILD_CLOUDSYNC="no"
+AC_CHECK_LIB([curl], [curl_easy_setopt], [LIBCURL="-lcurl"])
+if test -n "$LIBCURL";then
+        HAVE_LIBCURL="yes"
+fi
+AC_CHECK_HEADERS([openssl/hmac.h openssl/evp.h openssl/bio.h openssl/buffer.h], [HAVE_OPENSSL="yes"])
+if test "x$HAVE_LIBCURL" = "xyes" -a "x$HAVE_OPENSSL" = "xyes";then
+   HAVE_AMAZONS3="yes"
+fi
+AM_CONDITIONAL([BUILD_AMAZONS3_PLUGIN], [test "x$HAVE_AMAZONS3" = "xyes"])
+if test "x$HAVE_AMAZONS3" = "xyes";then
+   BUILD_CLOUDSYNC="yes"
+fi
+BUILD_CVLT_PLUGIN="no"
+case $host_os in
+#enable cvlt plugin only for linux platforms
+     linux*)
+       BUILD_CVLT_PLUGIN="yes"
+       BUILD_CLOUDSYNC="yes"
+       ;;
+     *)
+       ;;
+esac
+AM_CONDITIONAL([BUILD_CVLT_PLUGIN], [test "x$BUILD_CVLT_PLUGIN" = "xyes"])
+AM_CONDITIONAL([BUILD_CLOUDSYNC], [test "x$BUILD_CLOUDSYNC" = "xyes"])
+dnl end cloudsync section
+
+dnl SELinux feature enablement
+case $host_os in
+  linux*)
+    AC_ARG_ENABLE([selinux],
+                  AC_HELP_STRING([--disable-selinux],
+                                 [Disable SELinux features]),
+                  [USE_SELINUX="${enableval}"], [USE_SELINUX="yes"])
+    ;;
+  *)
+    USE_SELINUX=no
+    ;;
+esac
+AM_CONDITIONAL(USE_SELINUX, test "x${USE_SELINUX}" = "xyes")
+dnl end of SELinux feature enablement
+
 AC_CHECK_HEADERS([execinfo.h], [have_backtrace=yes])
 if test "x${have_backtrace}" = "xyes"; then
    AC_DEFINE(HAVE_BACKTRACE, 1, [define if found backtrace])
 fi
 AC_SUBST(HAVE_BACKTRACE)
 
+dnl Old (before C11) compiler can compile (but not link) this:
+dnl
+dnl int main () {
+dnl     _Static_assert(1, "True");
+dnl     return 0;
+dnl }
+dnl
+dnl assuming that _Static_assert is an implicitly declared function. So
+dnl we're trying to link just to make sure that this is not the case.
+
+AC_MSG_CHECKING([whether $CC supports C11 _Static_assert])
+AC_TRY_LINK([], [_Static_assert(1, "True");],
+            [STATIC_ASSERT=yes], [STATIC_ASSERT=no])
+
+AC_MSG_RESULT([$STATIC_ASSERT])
+if test x$STATIC_ASSERT = "xyes"; then
+   AC_DEFINE(HAVE_STATIC_ASSERT, 1, [Define if C11 _Static_assert is supported.])
+fi
+
 if test "x${have_backtrace}" != "xyes"; then
 AC_TRY_COMPILE([#include <math.h>], [double x=0.0; x=ceil(0.0);],
-   [have_math_h=yes],
+   [],
    AC_MSG_ERROR([need math library for libexecinfo]))
-if test "x${have_math_h}" = "xyes"; then
-   LIBS="$LIBS -lm"
-fi
 fi
 
 dnl glusterfs prints memory usage to stderr by sending it SIGUSR1
-AC_CHECK_FUNC([malloc_stats], [have_malloc_stats=yes])
-if test "x${have_malloc_stats}" = "xyes"; then
-   AC_DEFINE(HAVE_MALLOC_STATS, 1, [define if found malloc_stats])
+AC_CHECK_FUNC([mallinfo], [have_mallinfo=yes])
+if test "x${have_mallinfo}" = "xyes"; then
+   AC_DEFINE(HAVE_MALLINFO, 1, [define if found mallinfo])
 fi
-AC_SUBST(HAVE_MALLOC_STATS)
+AC_SUBST(HAVE_MALLINFO)
 
 dnl Linux, Solaris, Cygwin
 AC_CHECK_MEMBERS([struct stat.st_atim.tv_nsec])
@@ -714,7 +1014,7 @@ dnl FreeBSD, NetBSD
 AC_CHECK_MEMBERS([struct stat.st_atimespec.tv_nsec])
 case $host_os in
         *netbsd*)
-        CFLAGS="${CFLAGS} -D_INCOMPLETE_XOPEN_C063 -DCONFIG_MACHINE_BSWAP_H"
+        GF_CFLAGS="${GF_CFLAGS} -D_INCOMPLETE_XOPEN_C063 -DCONFIG_MACHINE_BSWAP_H"
         ;;
 esac
 AC_CHECK_FUNC([linkat], [have_linkat=yes])
@@ -727,37 +1027,42 @@ dnl check for Monotonic clock
 AC_CHECK_LIB([rt], [clock_gettime], ,
              AC_MSG_WARN([System doesn't have monotonic clock using contrib]))
 
-dnl Add cmockery2 for unit tests
+dnl check for argp, FreeBSD has the header in /usr/local/include
 case $host_os in
-  freebsd*)
-    dnl remove --coverage on FreeBSD due to a known llvm packaging bug
-    UNITTEST_CFLAGS="${UNITTEST_CPPFLAGS} ${UNITTEST_CFLAGS} -g -DDEBUG -O0"
-    UNITTEST_LDFLAGS="${UNITTEST_LIBS} ${UNITTEST_LDFLAGS}"
+    *freebsd*)
+    CFLAGS="${CFLAGS} -isystem /usr/local/include"
+    ARGP_LDADD=-largp
     ;;
-  *)
-    UNITTEST_CFLAGS="${UNITTEST_CPPFLAGS} ${UNITTEST_CFLAGS} -g -DDEBUG -O0 --coverage"
-    UNITTEST_LDFLAGS="${UNITTEST_LIBS} ${UNITTEST_LDFLAGS}"
+    *netbsd*)
+    ARGP_LDADD=-largp
     ;;
 esac
-
-AC_SUBST(UNITTEST_CFLAGS)
-AC_SUBST(UNITTEST_LDFLAGS)
-
-dnl Check for argp
+dnl argp-standalone does not provide a pkg-config file
 AC_CHECK_HEADER([argp.h], AC_DEFINE(HAVE_ARGP, 1, [have argp]))
-AC_CONFIG_SUBDIRS(contrib/argp-standalone)
-
-BUILD_ARGP_STANDALONE=no
-if test "x${ac_cv_header_argp_h}" = "xno"; then
-   BUILD_ARGP_STANDALONE=yes
-   ARGP_STANDALONE_CPPFLAGS='-I${top_srcdir}/contrib/argp-standalone'
-   ARGP_STANDALONE_LDADD='${top_builddir}/contrib/argp-standalone/libargp.a'
-   ARGP_STANDALONE_DIR='${top_builddir}/contrib/argp-standalone'
+if test "x$ac_cv_header_argp_h" != "xyes"; then
+    AC_MSG_ERROR([argp.h not found, install libargp or argp-standalone])
 fi
-
-AC_SUBST(ARGP_STANDALONE_CPPFLAGS)
-AC_SUBST(ARGP_STANDALONE_LDADD)
-AC_SUBST(ARGP_STANDALONE_DIR)
+AC_SUBST(ARGP_LDADD)
+
+dnl Check for atomic operation support
+AC_MSG_CHECKING([for gcc __atomic builtins])
+AC_TRY_LINK([], [int v; __atomic_load_n(&v, __ATOMIC_ACQUIRE);],
+            [have_atomic_builtins=yes], [have_atomic_builtins=no])
+if test "x${have_atomic_builtins}" = "xyes"; then
+   AC_DEFINE(HAVE_ATOMIC_BUILTINS, 1, [define if __atomic_*() builtins are available])
+fi
+AC_SUBST(HAVE_ATOMIC_BUILTINS)
+AC_MSG_RESULT([$have_atomic_builtins])
+
+dnl __sync_*() will not be needed if __atomic_*() is available
+AC_MSG_CHECKING([for gcc __sync builtins])
+AC_TRY_LINK([], [__sync_synchronize();],
+            [have_sync_builtins=yes], [have_sync_builtins=no])
+if test "x${have_sync_builtins}" = "xyes"; then
+   AC_DEFINE(HAVE_SYNC_BUILTINS, 1, [define if __sync_*() builtins are available])
+fi
+AC_SUBST(HAVE_SYNC_BUILTINS)
+AC_MSG_RESULT([$have_sync_builtins])
 
 AC_CHECK_HEADER([malloc.h], AC_DEFINE(HAVE_MALLOC_H, 1, [have malloc.h]))
 
@@ -781,6 +1086,63 @@ if test "x${have_posix_fallocate}" = "xyes"; then
    AC_DEFINE(HAVE_POSIX_FALLOCATE, 1, [define if posix_fallocate exists])
 fi
 
+# On fedora-29, copy_file_range syscall and the libc API both are present.
+# Whereas, on some machines such as centos-7, RHEL-7, the API is not there.
+# Only the system call is present. So, this change is to determine whether
+# the API is present or not. If not, then check whether the system call is
+# present or not. Accordingly sys_copy_file_range function will first call
+# the API if it is there. Otherwise it will call syscall(SYS_copy_file_range).
+AC_CHECK_FUNC([copy_file_range], [have_copy_file_range=yes])
+if test "x${have_copy_file_range}" = "xyes"; then
+   AC_DEFINE(HAVE_COPY_FILE_RANGE, 1, [define if copy_file_range exists])
+else
+   OLD_CFLAGS=${CFLAGS}
+   CFLAGS="-D_GNU_SOURCE"
+   AC_CHECK_DECL([SYS_copy_file_range], , , [#include <sys/syscall.h>])
+   if test "x${ac_cv_have_decl_SYS_copy_file_range}" = "xyes"; then
+      AC_DEFINE(HAVE_COPY_FILE_RANGE_SYS, 1, [define if SYS_copy_file_range is available])
+   fi
+   CFLAGS=${OLD_CFLAGS}
+fi
+
+AC_CHECK_FUNC([syncfs], [have_syncfs=yes])
+if test "x${have_syncfs}" = "xyes"; then
+   AC_DEFINE(HAVE_SYNCFS, 1, [define if syncfs exists])
+else
+   OLD_CFLAGS=${CFLAGS}
+   CFLAGS="-D_GNU_SOURCE"
+   AC_CHECK_DECL([SYS_syncfs], , , [#include <sys/syscall.h>])
+   if test "x${ac_cv_have_decl_SYS_syncfs}" = "xyes"; then
+      AC_DEFINE(HAVE_SYNCFS_SYS, 1, [define if SYS_syncfs is available])
+   fi
+   CFLAGS=${OLD_CFLAGS}
+fi
+
+BUILD_NANOSECOND_TIMESTAMPS=no
+AC_CHECK_FUNC([utimensat], [have_utimensat=yes])
+if test "x${have_utimensat}" = "xyes"; then
+   BUILD_NANOSECOND_TIMESTAMPS=yes
+   AC_DEFINE(HAVE_UTIMENSAT, 1, [define if utimensat exists])
+fi
+
+OLD_CFLAGS=${CFLAGS}
+CFLAGS="-D_GNU_SOURCE"
+AC_CHECK_DECL([SEEK_HOLE], , , [#include <unistd.h>])
+if test "x${ac_cv_have_decl_SEEK_HOLE}" = "xyes"; then
+   AC_DEFINE(HAVE_SEEK_HOLE, 1, [define if SEEK_HOLE is available])
+fi
+CFLAGS=${OLD_CFLAGS}
+
+AC_CHECK_FUNC([accept4], [have_accept4=yes])
+if test "x${have_accept4}" = "xyes"; then
+   AC_DEFINE(HAVE_ACCEPT4, 1, [define if accept4 exists])
+fi
+
+AC_CHECK_FUNC([paccept], [have_paccept=yes])
+if test "x${have_paccept}" = "xyes"; then
+AC_DEFINE(HAVE_PACCEPT, 1, [define if paccept exists])
+fi
+
 # Check the distribution where you are compiling glusterfs on
 
 GF_DISTRIBUTION=
@@ -801,30 +1163,56 @@ fi
 AC_SUBST(GF_DISTRIBUTION)
 
 GF_HOST_OS=""
-GF_LDFLAGS="-rdynamic"
+GF_LDFLAGS="${GF_LDFLAGS} -rdynamic"
+
+dnl see --with-libtirpc option check above, libtirpc(-devel) is required for
+dnl ipv6-default
+if test "x${with_libtirpc}" = "xyes" || test "x${with_ipv6_default}" = "xyes" ; then
+    PKG_CHECK_MODULES([TIRPC], [libtirpc],
+       [with_libtirpc="yes"; GF_CFLAGS="$GF_CFLAGS $TIRPC_CFLAGS"; GF_LDFLAGS="$GF_LDFLAGS $TIRPC_LIBS";],
+       [with_libtirpc="missing"; with_ipv6_default="no"])
+fi
+
+if test "x${with_libtirpc}" = "xmissing" ; then
+    AC_CHECK_HEADERS([rpc/rpc.h],[
+        AC_MSG_WARN([
+            ---------------------------------------------------------------------------------
+            libtirpc (and/or ipv6-default) were enabled but libtirpc-devel is not installed.
+            Disabling libtirpc and ipv6-default and falling back to legacy glibc rpc headers.
+            This is a transitional warning message. Eventually it will be an error message.
+            ---------------------------------------------------------------------------------])],[
+        AC_MSG_ERROR([
+            ---------------------------------------------------------------------------------
+            libtirpc (and/or ipv6-default) were enabled but libtirpc-devel is not installed
+            and there were no legacy glibc rpc headers and library to fall back to.
+            ---------------------------------------------------------------------------------])])
+fi
+
+if test "x$with_ipv6_default" = "xyes" ; then
+   GF_CFLAGS="$GF_CFLAGS -DIPV6_DEFAULT"
+fi
 
 dnl check for gcc -Werror=format-security
-saved_GF_CFLAGS="-Wformat -Werror=format-security"
+saved_CFLAGS=$CFLAGS
+CFLAGS="-Wformat -Werror=format-security"
 AC_MSG_CHECKING([whether $CC accepts -Werror=format-security])
 AC_COMPILE_IFELSE([AC_LANG_PROGRAM()], [cc_werror_format_security=yes], [cc_werror_format_security=no])
 echo $cc_werror_format_security
-if test "x$cc_werror_format_security" = "xno"; then
-    GF_CFLAGS="$GF_CFLAGS"
-else
-    GF_CFLAGS="$saved_GF_CFLAGS $GF_CFLAGS"
+if test "x$cc_werror_format_security" = "xyes"; then
+    GF_CFLAGS="$GF_CFLAGS ${CFLAGS}"
 fi
+CFLAGS="$saved_CFLAGS"
 
 dnl check for gcc -Werror=implicit-function-declaration
-saved_GF_CFLAGS=$GF_CFLAGS
-GF_CFLAGS="-Werror=implicit-function-declaration"
+saved_CFLAGS=$CFLAGS
+CFLAGS="-Werror=implicit-function-declaration"
 AC_MSG_CHECKING([whether $CC accepts -Werror=implicit-function-declaration])
 AC_COMPILE_IFELSE([AC_LANG_PROGRAM()], [cc_werror_implicit=yes], [cc_werror_implicit=no])
 echo $cc_werror_implicit
-if test "x$cc_werror_implicit" = "xno"; then
-    GF_CFLAGS="$saved_GF_CFLAGS"
-else
-    GF_CFLAGS="$saved_GF_CFLAGS $GF_CFLAGS"
+if test "x$cc_werror_implicit" = "xyes"; then
+    GF_CFLAGS="${GF_CFLAGS} ${CFLAGS}"
 fi
+CFLAGS="$saved_CFLAGS"
 
 dnl clang is mostly GCC-compatible, but its version is much lower,
 dnl so we have to check for it.
@@ -841,7 +1229,7 @@ AC_COMPILE_IFELSE(
 AC_MSG_RESULT([$CLANG])
 
 if test "x$CLANG" = "xyes"; then
-   GF_COMPILER_FLAGS="-Wno-gnu"
+   GF_CFLAGS="${GF_CFLAGS} -Wno-gnu"
 fi
 
 if test "x$ac_cv_header_execinfo_h" = "xno"; then
@@ -855,45 +1243,63 @@ if test "x$ac_cv_header_execinfo_h" = "xno"; then
    # execinfo.c in ./contrib/libexecinfo uses __builtin_frame_address(n)
    # for providing cross platform backtrace*() functions.
    if test "x$CLANG" = "xno"; then
-      CFLAGS="$CFLAGS -fno-omit-frame-pointer"
+      GF_CFLAGS="${GF_CFLAGS} -fno-omit-frame-pointer"
    fi
 fi
 
+old_prefix=$prefix
+if test "x$prefix" = xNONE; then
+        prefix=$ac_default_prefix
+fi
+old_exec_prefix=$exec_prefix
+if test "x$exec_prefix" = xNONE; then
+        exec_prefix="$(eval echo $prefix)"
+fi
+GLUSTERFS_LIBEXECDIR="$(eval echo $libexecdir)/glusterfs"
+prefix=$old_prefix
+exec_prefix=$old_exec_prefix
+
 ### Dirty hacky stuff to make LOCALSTATEDIR work
 if test "x$prefix" = xNONE; then
    test $localstatedir = '${prefix}/var' && localstatedir=$ac_default_prefix/var
    localstatedir=/var
-   LOCALSTATEDIR=$(eval echo ${localstatedir})
-else
-   LOCALSTATEDIR=$(eval echo ${localstatedir})
 fi
+localstatedir="$(eval echo ${localstatedir})"
+LOCALSTATEDIR=$localstatedir
+
+GLUSTERFSD_MISCDIR="$(eval echo ${localstatedir})/lib/misc/glusterfsd"
+
+old_prefix=$prefix
+if test "x$prefix" = xNONE; then
+    prefix=$ac_default_prefix
+fi
+GLUSTERD_VOLFILE="$(eval echo ${sysconfdir})/glusterfs/glusterd.vol"
+prefix=$old_prefix
+
 
+GFAPI_EXTRA_LDFLAGS='-Wl,--version-script=$(top_srcdir)/api/src/gfapi.map'
 case $host_os in
      linux*)
         GF_HOST_OS="GF_LINUX_HOST_OS"
-        GF_CFLAGS="${GF_COMPILER_FLAGS} ${ARGP_STANDALONE_CPPFLAGS}"
-        GF_LDADD="${ARGP_STANDALONE_LDADD}"
         GF_FUSE_CFLAGS="-DFUSERMOUNT_DIR=\\\"\$(bindir)\\\""
         GLUSTERD_WORKDIR="${LOCALSTATEDIR}/lib/glusterd"
         ;;
      solaris*)
         GF_HOST_OS="GF_SOLARIS_HOST_OS"
-        GF_CFLAGS="${ARGP_STANDALONE_CPPFLAGS} -D_REENTRANT -D_POSIX_PTHREAD_SEMANTICS -m64"
-        GF_LDFLAGS=""
-        GF_LDADD="${ARGP_STANDALONE_LDADD}"
+        GF_CFLAGS="${GF_CFLAGS} -D_REENTRANT -D_POSIX_PTHREAD_SEMANTICS -m64"
         BUILD_FUSE_CLIENT=no
         FUSE_CLIENT_SUBDIR=""
         GLUSTERD_WORKDIR="${LOCALSTATEDIR}/lib/glusterd"
         ;;
      *netbsd*)
         GF_HOST_OS="GF_BSD_HOST_OS"
-        GF_CFLAGS="${ARGP_STANDALONE_CPPFLAGS} -D_INCOMPLETE_XOPEN_C063"
+        GF_CFLAGS="${GF_CFLAGS} -D_INCOMPLETE_XOPEN_C063"
         GF_CFLAGS="${GF_CFLAGS} -DTHREAD_UNSAFE_BASENAME"
         GF_CFLAGS="${GF_CFLAGS} -DTHREAD_UNSAFE_DIRNAME"
         GF_FUSE_CFLAGS="-DFUSERMOUNT_DIR=\\\"\$(sbindir)\\\""
-        GF_LDADD="${ARGP_STANDALONE_LDADD}"
+        GF_LDADD="${ARGP_LDADD}"
         if test "x$ac_cv_header_execinfo_h" = "xyes"; then
-           GF_LDFLAGS="-lexecinfo"
+           GF_LDFLAGS="${GF_LDFLAGS} -lexecinfo"
         fi
         GF_FUSE_LDADD="-lperfuse"
         BUILD_FUSE_CLIENT=yes
@@ -904,17 +1310,13 @@ case $host_os in
         ;;
      *freebsd*)
         GF_HOST_OS="GF_BSD_HOST_OS"
-        GF_CFLAGS="${GF_COMPILER_FLAGS} ${ARGP_STANDALONE_CPPFLAGS} -O0"
-        GF_CFLAGS="${GF_CFLAGS} -DTHREAD_UNSAFE_BASENAME"
-        GF_CFLAGS="${GF_CFLAGS} -DTHREAD_UNSAFE_DIRNAME"
-        GF_CFLAGS="${GF_CFLAGS} -D_LIBGEN_H_"
         GF_CFLAGS="${GF_CFLAGS} -DO_DSYNC=0"
         GF_CFLAGS="${GF_CFLAGS} -Dxdr_quad_t=xdr_longlong_t"
         GF_CFLAGS="${GF_CFLAGS} -Dxdr_u_quad_t=xdr_u_longlong_t"
         GF_FUSE_CFLAGS="-DFUSERMOUNT_DIR=\\\"\$(sbindir)\\\""
-        GF_LDADD="${ARGP_STANDALONE_LDADD}"
+        GF_LDADD="${ARGP_LDADD}"
         if test "x$ac_cv_header_execinfo_h" = "xyes"; then
-           GF_LDFLAGS="-lexecinfo"
+           GF_LDFLAGS="${GF_LDFLAGS} -lexecinfo"
         fi
         BUILD_FUSE_CLIENT=yes
         BUILD_FUSERMOUNT=no
@@ -924,30 +1326,47 @@ case $host_os in
      darwin*)
         GF_HOST_OS="GF_DARWIN_HOST_OS"
         LIBTOOL=glibtool
-        GF_CFLAGS="${GF_COMPILER_FLAGS} ${ARGP_STANDALONE_CPPFLAGS} "
         GF_CFLAGS="${GF_CFLAGS} -D_REENTRANT -D_XOPEN_SOURCE "
         GF_CFLAGS="${GF_CFLAGS} -D_DARWIN_USE_64_BIT_INODE "
         GF_CFLAGS="${GF_CFLAGS} -DTHREAD_UNSAFE_BASENAME"
         GF_CFLAGS="${GF_CFLAGS} -DTHREAD_UNSAFE_DIRNAME"
-        GF_LDADD="${ARGP_STANDALONE_LDADD}"
-        GF_LDFLAGS=""
+        GF_LDADD="${ARGP_LDADD}"
+        GF_LDFLAGS="${GF_LDFLAGS}"
         GF_FUSE_CFLAGS="-I\$(CONTRIBDIR)/macfuse"
         BUILD_FUSERMOUNT="no"
         FUSERMOUNT_SUBDIR=""
         GLUSTERD_WORKDIR="${LOCALSTATEDIR}/db/glusterd"
+        GFAPI_EXTRA_LDFLAGS='-Wl,-alias_list,$(top_srcdir)/api/src/gfapi.aliases'
         ;;
 esac
 
+# Default value for sbindir
+prefix_temp=$prefix
+exec_prefix_temp=$exec_prefix
+
+test "${prefix}" = "NONE" && prefix="${ac_default_prefix}"
+test "${exec_prefix}" = "NONE" && exec_prefix='${prefix}'
+sbintemp="${sbindir}"
+eval sbintemp=\"${sbintemp}\"
+eval sbintemp=\"${sbintemp}\"
+SBIN_DIR=${sbintemp}
+
+sysconfdirtemp="${sysconfdir}"
+eval sysconfdirtemp=\"${sysconfdirtemp}\"
+SYSCONF_DIR=${sysconfdirtemp}
+
+prefix=$prefix_temp
+exec_prefix=$exec_prefix_temp
+
+AC_SUBST(SBIN_DIR)
+AC_SUBST(SYSCONF_DIR)
+
 # lazy umount emulation
 UMOUNTD_SUBDIR=""
 if test "x${GF_HOST_OS}" != "xGF_LINUX_HOST_OS" ; then
         UMOUNTD_SUBDIR="contrib/umountd"
 fi
 AC_SUBST(UMOUNTD_SUBDIR)
-        
-
-# enable/disable QEMU
-AM_CONDITIONAL([ENABLE_QEMU_BLOCK], [test x$BUILD_QEMU_BLOCK = xyes])
 
 
 # enable debug section
@@ -955,6 +1374,15 @@ AC_ARG_ENABLE([debug],
               AC_HELP_STRING([--enable-debug],
                              [Enable debug build options.]))
 
+AC_ARG_ENABLE([mempool],
+              AC_HELP_STRING([--disable-mempool],
+                             [Disable the Gluster memory pooler.]))
+
+USE_MEMPOOL="yes"
+if test "x$enable_mempool" = "xno"; then
+        USE_MEMPOOL="no"
+        AC_DEFINE(GF_DISABLE_MEMPOOL, 1, [Disable the Gluster memory pooler.])
+fi
 
 # syslog section
 AC_ARG_ENABLE([syslog],
@@ -975,7 +1403,7 @@ AC_CHECK_LIB([readline -lcurses],[readline],[RLLIBS="-lreadline -lcurses"])
 AC_CHECK_LIB([readline -ltermcap],[readline],[RLLIBS="-lreadline -ltermcap"])
 AC_CHECK_LIB([readline -lncurses],[readline],[RLLIBS="-lreadline -lncurses"])
 
-if test "x$RLLIBS" != "x"; then
+if test -n "$RLLIBS"; then
    if test "x$RL_UNDO" = "xyes"; then
       AC_DEFINE(HAVE_READLINE, 1, [readline enabled CLI])
       BUILD_READLINE=yes
@@ -988,100 +1416,183 @@ fi
 BUILD_LIBAIO=no
 AC_CHECK_LIB([aio],[io_setup],[LIBAIO="-laio"])
 
-if test "x$LIBAIO" != "x"; then
+if test -n "$LIBAIO"; then
    AC_DEFINE(HAVE_LIBAIO, 1, [libaio based POSIX enabled])
    BUILD_LIBAIO=yes
 fi
 
-# glupy section
-BUILD_GLUPY=no
-have_python2=no
-have_Python_h=no
-
-AM_PATH_PYTHON()
-if echo $PYTHON_VERSION | grep ^2; then
-  have_python2=yes
+dnl gnfs section
+BUILD_GNFS="no"
+RPCBIND_SERVICE=""
+AC_ARG_ENABLE([gnfs],
+              AC_HELP_STRING([--enable-gnfs],
+                             [Enable legacy gnfs server xlator.]))
+if test "x${with_server}" = "xyes" -a "x$enable_gnfs" = "xyes"; then
+    BUILD_GNFS="yes"
+    GF_CFLAGS="$GF_CFLAGS -DBUILD_GNFS"
+    RPCBIND_SERVICE="rpcbind.service"
+fi
+AM_CONDITIONAL([BUILD_GNFS], [test x$BUILD_GNFS = xyes])
+AC_SUBST(BUILD_GNFS)
+AC_SUBST(RPCBIND_SERVICE)
+dnl end gnfs section
+
+dnl Check for userspace-rcu
+PKG_CHECK_MODULES([URCU], [liburcu-bp], [],
+  [AC_CHECK_HEADERS([urcu-bp.h],
+     [URCU_LIBS='-lurcu-bp'],
+     AC_MSG_ERROR([liburcu-bp not found]))])
+PKG_CHECK_MODULES([URCU_CDS], [liburcu-cds >= 0.8], [],
+  [PKG_CHECK_MODULES([URCU_CDS], [liburcu-cds >= 0.7],
+    [AC_DEFINE(URCU_OLD, 1, [Define if liburcu 0.6 or 0.7 is found])
+     USE_CONTRIB_URCU='yes'],
+    [AC_CHECK_HEADERS([urcu/cds.h],
+      [AC_DEFINE(URCU_OLD, 1, [Define if liburcu 0.6 or 0.7 is found])
+       URCU_CDS_LIBS='-lurcu-cds'
+       USE_CONTRIB_URCU='yes'],
+      [AC_MSG_ERROR([liburcu-cds not found])])])])
+
+BUILD_UNITTEST="no"
+AC_ARG_ENABLE([cmocka],
+              AC_HELP_STRING([--enable-cmocka],
+                             [Enable cmocka build options.]))
+if test "x$enable_cmocka" = "xyes"; then
+    BUILD_UNITTEST="yes"
+    PKG_CHECK_MODULES([UNITTEST], [cmocka >= 1.0.1], [BUILD_UNITTEST="yes"],
+        [AC_MSG_ERROR([cmocka library is required to build glusterfs])]
+    )
 fi
+AM_CONDITIONAL([UNITTEST], [test x$BUILD_UNITTEST = xyes])
 
-# Save flags before testing python
-saved_CFLAGS=$CFLAGS
-saved_CPPFLAGS=$CPPFLAGS
-saved_LDFLAGS=$LDFLAGS
-
-CFLAGS="`${PYTHON}-config --cflags`"
-CPPFLAGS=$CFLAGS
-LDFLAGS="-L`${PYTHON}-config --prefix`/$libdir `${PYTHON}-config --ldflags`"
-
-AC_CHECK_HEADERS([python$PYTHON_VERSION/Python.h],[have_Python_h=yes],[])
-AC_ARG_ENABLE([glupy],
-              AS_HELP_STRING([--enable-glupy],
-                             [build glupy]))
-case x$enable_glupy in
-   xyes)
-      if test "x$have_python2" = "xyes" -a "x$have_Python_h" = "xyes"; then
-        BUILD_GLUPY=yes
-        PYTHONDEV_CFLAGS="$CFLAGS"
-        PYTHONDEV_CPPFLAGS="$CPPFLAGS"
-        PYTHONDEV_LDFLAGS="$LDFLAGS"
-        AC_SUBST(PYTHONDEV_CFLAGS)
-        AC_SUBST(PYTHONDEV_CPPFLAGS)
-        AC_SUBST(PYTHONDEV_LDFLAGS)
-      else
-        AC_MSG_ERROR([glupy requires python-devel/python-dev package and python2.x])
-      fi
-      ;;
-   xno)
-      ;;
-   *)
-      if test "x$have_python2" = "xyes" -a "x$have_Python_h" = "xyes"; then
-        BUILD_GLUPY=yes
-        PYTHONDEV_CFLAGS="$CFLAGS"
-        PYTHONDEV_CPPFLAGS="$CPPFLAGS"
-        PYTHONDEV_LDFLAGS="$LDFLAGS"
-        AC_SUBST(PYTHONDEV_CFLAGS)
-        AC_SUBST(PYTHONDEV_CPPFLAGS)
-        AC_SUBST(PYTHONDEV_LDFLAGS)
-      else
-        AC_MSG_WARN([
-        ---------------------------------------------------------------------------------
-         cannot build glupy. python 2.x and python-devel/python-dev package are required.
-        ---------------------------------------------------------------------------------])
-      fi
-      ;;
-esac
-# Restore flags
-CFLAGS=$saved_CFLAGS
-CPPFLAGS=$saved_CPPFLAGS
-LDFLAGS=$saved_LDFLAGS
+dnl Define UNIT_TESTING only for building cmocka binaries.
+UNITTEST_CFLAGS="${UNITTEST_CFLAGS} -DUNIT_TESTING=1"
 
+dnl Add cmocka for unit tests
 case $host_os in
-     darwin*)
-        BUILD_GLUPY=no
-        ;;
+  freebsd*)
+    dnl remove --coverage on FreeBSD due to a known llvm packaging bug
+    UNITTEST_CFLAGS="${UNITTEST_CPPFLAGS} ${UNITTEST_CFLAGS} -g -DDEBUG -O0"
+    UNITTEST_LDFLAGS="${UNITTEST_LIBS} ${UNITTEST_LDFLAGS}"
+    ;;
+  *)
+    UNITTEST_CFLAGS="${UNITTEST_CPPFLAGS} ${UNITTEST_CFLAGS} -g -DDEBUG -O0 --coverage"
+    UNITTEST_LDFLAGS="${UNITTEST_LIBS} ${UNITTEST_LDFLAGS}"
+    ;;
 esac
 
-if test "x$BUILD_GLUPY" = "xyes"; then
-   BUILD_PYTHON_SITE_PACKAGES=`$PYTHON -c 'from distutils.sysconfig import get_python_lib; print(get_python_lib())'`
-   BUILD_PYTHON_INC=`$PYTHON -c "from distutils import sysconfig; print sysconfig.get_python_inc()"`
-   BUILD_PYTHON_LIB=python$PYTHON_VERSION
-   GLUPY_SUBDIR=glupy
-   GLUPY_SUBDIR_MAKEFILE=xlators/features/glupy/Makefile
-   GLUPY_SUBDIR_SRC_MAKEFILE=xlators/features/glupy/src/Makefile
-   echo "building glupy with -isystem $BUILD_PYTHON_INC -l $BUILD_PYTHON_LIB"
-   AC_SUBST(BUILD_PYTHON_SITE_PACKAGES)
-   AC_SUBST(BUILD_PYTHON_INC)
-   AC_SUBST(BUILD_PYTHON_LIB)
-   AC_SUBST(GLUPY_SUBDIR)
-   AC_SUBST(GLUPY_SUBDIR_MAKEFILE)
-   AC_SUBST(GLUPY_SUBDIR_SRC_MAKEFILE)
-fi
-# end glupy section
+AC_SUBST(UNITTEST_CFLAGS)
+AC_SUBST(UNITTEST_LDFLAGS)
 
 AC_SUBST(CFLAGS)
 # end enable debug section
 
-dnl Required if cmockery2 headers are not in standar paths
-GF_CFLAGS="${GF_CFLAGS} ${UNITTEST_CPPFLAGS}"
+# EC dynamic code generation section
+
+EC_DYNAMIC_SUPPORT="none"
+EC_DYNAMIC_ARCH="none"
+
+AC_ARG_ENABLE([ec-dynamic],
+              AC_HELP_STRING([--disable-ec-dynamic],
+                             [Disable all dynamic code generation extensions for EC module]))
+
+AC_ARG_ENABLE([ec-dynamic-intel],
+              AC_HELP_STRING([--disable-ec-dynamic-intel],
+                             [Disable all INTEL dynamic code generation extensions for EC module]))
+
+AC_ARG_ENABLE([ec-dynamic-arm],
+              AC_HELP_STRING([--disable-ec-dynamic-arm],
+                             [Disable all ARM dynamic code generation extensions for EC module]))
+
+AC_ARG_ENABLE([ec-dynamic-x64],
+              AC_HELP_STRING([--disable-ec-dynamic-x64],
+                             [Disable dynamic INTEL x64 code generation for EC module]))
+
+AC_ARG_ENABLE([ec-dynamic-sse],
+              AC_HELP_STRING([--disable-ec-dynamic-sse],
+                             [Disable dynamic INTEL SSE code generation for EC module]))
+
+AC_ARG_ENABLE([ec-dynamic-avx],
+              AC_HELP_STRING([--disable-ec-dynamic-avx],
+                             [Disable dynamic INTEL AVX code generation for EC module]))
+
+AC_ARG_ENABLE([ec-dynamic-neon],
+              AC_HELP_STRING([--disable-ec-dynamic-neon],
+                             [Disable dynamic ARM NEON code generation for EC module]))
+
+if test "x$enable_ec_dynamic" != "xno"; then
+  case $host in
+    x86_64*)
+      if test "x$enable_ec_dynamic_intel" != "xno"; then
+        if test "x$enable_ec_dynamic_x64" != "xno"; then
+          EC_DYNAMIC_SUPPORT="$EC_DYNAMIC_SUPPORT x64"
+          AC_DEFINE(USE_EC_DYNAMIC_X64, 1, [Defined if using dynamic INTEL x64 code])
+        fi
+        if test "x$enable_ec_dynamic_sse" != "xno"; then
+          EC_DYNAMIC_SUPPORT="$EC_DYNAMIC_SUPPORT sse"
+          AC_DEFINE(USE_EC_DYNAMIC_SSE, 1, [Defined if using dynamic INTEL SSE code])
+        fi
+        if test "x$enable_ec_dynamic_avx" != "xno"; then
+          EC_DYNAMIC_SUPPORT="$EC_DYNAMIC_SUPPORT avx"
+          AC_DEFINE(USE_EC_DYNAMIC_AVX, 1, [Defined if using dynamic INTEL AVX code])
+        fi
+
+        if test "x$EC_DYNAMIC_SUPPORT" != "xnone"; then
+          EC_DYNAMIC_ARCH="intel"
+        fi
+      fi
+      ;;
+    arm*)
+      if test "x$enable_ec_dynamic_arm" != "xno"; then
+        if test "x$enable_ec_dynamic_neon" != "xno"; then
+          EC_DYNAMIC_SUPPORT="$EC_DYNAMIC_SUPPORT neon"
+          AC_DEFINE(USE_EC_DYNAMIC_NEON, 1, [Defined if using dynamic ARM NEON code])
+        fi
+
+        if test "x$EC_DYNAMIC_SUPPORT" != "xnone"; then
+          EC_DYNAMIC_ARCH="arm"
+        fi
+      fi
+      ;;
+  esac
+
+  EC_DYNAMIC_SUPPORT="${EC_DYNAMIC_SUPPORT#none }"
+fi
+
+AM_CONDITIONAL([ENABLE_EC_DYNAMIC_INTEL], [test "x$EC_DYNAMIC_ARCH" = "xintel"])
+AM_CONDITIONAL([ENABLE_EC_DYNAMIC_ARM], [test "x$EC_DYNAMIC_ARCH" = "xarm"])
+
+AM_CONDITIONAL([ENABLE_EC_DYNAMIC_X64], [test "x${EC_DYNAMIC_SUPPORT##*x64*}" = "x"])
+AM_CONDITIONAL([ENABLE_EC_DYNAMIC_SSE], [test "x${EC_DYNAMIC_SUPPORT##*sse*}" = "x"])
+AM_CONDITIONAL([ENABLE_EC_DYNAMIC_AVX], [test "x${EC_DYNAMIC_SUPPORT##*avx*}" = "x"])
+AM_CONDITIONAL([ENABLE_EC_DYNAMIC_NEON], [test "x${EC_DYNAMIC_SUPPORT##*neon*}" = "x"])
+
+AC_SUBST(USE_EC_DYNAMIC_X64)
+AC_SUBST(USE_EC_DYNAMIC_SSE)
+AC_SUBST(USE_EC_DYNAMIC_AVX)
+AC_SUBST(USE_EC_DYNAMIC_NEON)
+
+# end EC dynamic code generation section
+
+dnl libglusterfs.so uses math functions
+GF_LDADD="${GF_LDADD} ${MATH_LIB}"
+
+case $host_os in
+  dnl Can't use libtool's portable "-no-undefined" as it seems to be ignored on Linux
+  linux*)
+    GF_NO_UNDEFINED='-Wl,--no-undefined'
+    ;;
+  darwin*)
+    GF_NO_UNDEFINED='-Wl,-undefined'
+    ;;
+  *)
+  dnl There's an issue on FreeBSD with reference to __progname used in some parts of code
+    GF_NO_UNDEFINED=''
+    ;;
+esac
+dnl GF_XLATOR_DEFAULT_LDFLAGS is for most xlators that expose a common set of symbols
+GF_XLATOR_DEFAULT_LDFLAGS='-avoid-version -export-symbols $(top_srcdir)/xlators/xlator.sym $(UUID_LIBS) $(GF_NO_UNDEFINED) $(TIRPC_LIBS)'
+dnl GF_XLATOR_LDFLAGS is for xlators that expose extra symbols, e.g. dht
+GF_XLATOR_LDFLAGS='-avoid-version $(UUID_LIBS) $(GF_NO_UNDEFINED) $(TIRPC_LIBS)'
 
 AC_SUBST(GF_HOST_OS)
 AC_SUBST(GF_CFLAGS)
@@ -1093,33 +1604,52 @@ AC_SUBST(RLLIBS)
 AC_SUBST(LIBAIO)
 AC_SUBST(AM_MAKEFLAGS)
 AC_SUBST(AM_LIBTOOLFLAGS)
+AC_SUBST(GF_NO_UNDEFINED)
+AC_SUBST(GF_XLATOR_DEFAULT_LDFLAGS)
+AC_SUBST(GF_XLATOR_LDFLAGS)
+AC_SUBST(GF_XLATOR_MGNT_LIBADD)
+
+case $host_os in
+     *freebsd*)
+        GF_XLATOR_MGNT_LIBADD="-lutil -lprocstat"
+        ;;
+esac
 
 CONTRIBDIR='$(top_srcdir)/contrib'
 AC_SUBST(CONTRIBDIR)
 
 GF_CPPDEFINES='-D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D$(GF_HOST_OS)'
-GF_CPPINCLUDES='-I$(top_srcdir)/libglusterfs/src -I$(CONTRIBDIR)/uuid'
-GF_CPPFLAGS="$GF_CPPDEFINES $GF_CPPINCLUDES"
+GF_CPPINCLUDES='-include $(top_builddir)/config.h -include $(top_builddir)/site.h -I$(top_srcdir)/libglusterfs/src -I$(top_builddir)/libglusterfs/src'
+if test "x${USE_CONTRIB_URCU}" = "xyes"; then
+    GF_CPPINCLUDES="${GF_CPPINCLUDES} -I\$(CONTRIBDIR)/userspace-rcu"
+fi
+GF_CPPFLAGS="$GF_CPPFLAGS $GF_CPPDEFINES $GF_CPPINCLUDES"
 AC_SUBST([GF_CPPFLAGS])
 
 AM_CONDITIONAL([GF_LINUX_HOST_OS], test "${GF_HOST_OS}" = "GF_LINUX_HOST_OS")
 AM_CONDITIONAL([GF_DARWIN_HOST_OS], test "${GF_HOST_OS}" = "GF_DARWIN_HOST_OS")
 AM_CONDITIONAL([GF_BSD_HOST_OS], test "${GF_HOST_OS}" = "GF_BSD_HOST_OS")
+if  test "${GF_HOST_OS}" = "GF_BSD_HOST_OS"; then
+    AC_DEFINE(GF_BSD_HOST_OS, 1, [This is a BSD compatible OS.])
+fi
 
 AC_SUBST(GLUSTERD_WORKDIR)
 AM_CONDITIONAL([GF_INSTALL_GLUSTERD_WORKDIR], test ! -d ${GLUSTERD_WORKDIR} && test -d ${sysconfdir}/glusterd )
+AC_SUBST(GLUSTERD_VOLFILE)
+AC_SUBST(GLUSTERFS_LIBEXECDIR)
+AC_SUBST(GLUSTERFSD_MISCDIR)
 
 dnl pkg-config versioning
 dnl
-dnl Once we released gluster-api.pc with version=4. Since then we undid the
+dnl Once we released gluster-api.pc with version=7. Since then we undid the
 dnl library versioning and replaced it with symbol-versioning. The current
 dnl libgfapi.so has version 0, but the symbols have the version from the main
 dnl package at the time they were added.
 dnl
 dnl Because other packages (like samba) use the pkg-config version, we can not
 dnl drop it, or decrease the version easily. The simplest solution is to keep
-dnl the version=4 and add sub-digits for the actual package/symbol versions.
-GFAPI_VERSION="4."${PACKAGE_VERSION}
+dnl the version=7 and add sub-digits for the actual package/symbol versions.
+GFAPI_VERSION="7."${PACKAGE_VERSION}
 LIBGFCHANGELOG_VERSION="0.0.1"
 AC_SUBST(GFAPI_VERSION)
 AC_SUBST(LIBGFCHANGELOG_VERSION)
@@ -1135,6 +1665,10 @@ AC_SUBST(LIBGFRPC_LT_VERSION)
 AC_SUBST(LIBGLUSTERFS_LT_VERSION)
 AC_SUBST(LIBGFCHANGELOG_LT_VERSION)
 AC_SUBST(GFAPI_LT_VERSION)
+AC_SUBST(GFAPI_EXTRA_LDFLAGS)
+
+GFAPI_LIBS="${ACL_LIBS}"
+AC_SUBST(GFAPI_LIBS)
 
 dnl this change necessary for run-tests.sh
 AC_CONFIG_FILES([tests/env.rc],[ln -s ${ac_abs_builddir}/env.rc ${ac_abs_srcdir}/env.rc 2>/dev/null])
@@ -1145,19 +1679,39 @@ echo
 echo "GlusterFS configure summary"
 echo "==========================="
 echo "FUSE client          : $BUILD_FUSE_CLIENT"
-echo "Infiniband verbs     : $BUILD_IBVERBS"
 echo "epoll IO multiplex   : $BUILD_EPOLL"
-echo "argp-standalone      : $BUILD_ARGP_STANDALONE"
 echo "fusermount           : $BUILD_FUSERMOUNT"
 echo "readline             : $BUILD_READLINE"
 echo "georeplication       : $BUILD_SYNCDAEMON"
 echo "Linux-AIO            : $BUILD_LIBAIO"
 echo "Enable Debug         : $BUILD_DEBUG"
-echo "systemtap            : $BUILD_SYSTEMTAP"
-echo "Block Device xlator  : $BUILD_BD_XLATOR"
-echo "glupy                : $BUILD_GLUPY"
+echo "Run with Valgrind    : $VALGRIND_TOOL"
+echo "Sanitizer enabled    : $SANITIZER"
 echo "Use syslog           : $USE_SYSLOG"
 echo "XML output           : $BUILD_XML_OUTPUT"
-echo "QEMU Block formats   : $BUILD_QEMU_BLOCK"
-echo "Encryption xlator    : $BUILD_CRYPT_XLATOR"
+echo "Unit Tests           : $BUILD_UNITTEST"
+echo "Track priv ports     : $TRACK_PRIVPORTS"
+echo "POSIX ACLs           : $BUILD_POSIX_ACLS"
+echo "SELinux features     : $USE_SELINUX"
+echo "firewalld-config     : $BUILD_FIREWALLD"
+echo "Events               : $BUILD_EVENTS"
+echo "EC dynamic support   : $EC_DYNAMIC_SUPPORT"
+echo "Use memory pools     : $USE_MEMPOOL"
+echo "Nanosecond m/atimes  : $BUILD_NANOSECOND_TIMESTAMPS"
+echo "Server components    : $with_server"
+echo "Legacy gNFS server   : $BUILD_GNFS"
+echo "IPV6 default         : $with_ipv6_default"
+echo "Use TIRPC            : $with_libtirpc"
+echo "With Python          : ${PYTHON_VERSION}"
+echo "Cloudsync            : $BUILD_CLOUDSYNC"
+echo "Metadata dispersal   : $BUILD_METADISP"
+echo "Link with TCMALLOC   : $BUILD_TCMALLOC"
 echo
+
+# dnl Note: ${X^^} capitalization assumes bash >= 4.x
+if test "x$SANITIZER" != "xnone"; then
+        echo "Note: since glusterfs processes are daemon processes, use"
+        echo "'export ${SANITIZER^^}_OPTIONS=log_path=/path/to/xxx.log' to collect"
+        echo "sanitizer output. Further details and more options can be"
+        echo "found at https://github.com/google/sanitizers."
+fi
diff --git a/contrib/argp-standalone/Makefile.am b/contrib/argp-standalone/Makefile.am
deleted file mode 100644
index 4775d4876aa..00000000000
--- a/contrib/argp-standalone/Makefile.am
+++ /dev/null
@@ -1,38 +0,0 @@
-# From glibc
-
-# Copyright (C) 1997, 2003, 2004 Free Software Foundation, Inc.
-# This file is part of the GNU C Library.
-
-# The GNU C Library is free software; you can redistribute it and/or
-# modify it under the terms of the GNU Library General Public License as
-# published by the Free Software Foundation; either version 2 of the
-# License, or (at your option) any later version.
-
-# The GNU C Library is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-# Library General Public License for more details.
-
-# You should have received a copy of the GNU Library General Public
-# License along with the GNU C Library; see the file COPYING.LIB.  If
-# not, write to the Free Software Foundation, Inc.,
-# 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
-AUTOMAKE_OPTIONS = foreign
-SUBDIRS = .
-
-LIBOBJS = @LIBOBJS@
-
-noinst_LIBRARIES = libargp.a
-
-noinst_HEADERS = argp.h argp-fmtstream.h argp-namefrob.h
-
-EXTRA_DIST = mempcpy.c strchrnul.c strndup.c strcasecmp.c vsnprintf.c autogen.sh
-
-# Leaves out argp-fs-xinl.c and argp-xinl.c 
-libargp_a_SOURCES = argp-ba.c argp-eexst.c argp-fmtstream.c \
-		    argp-help.c argp-parse.c argp-pv.c \
-		    argp-pvh.c
-
-libargp_a_LIBADD = $(LIBOBJS)
-
-
diff --git a/contrib/argp-standalone/acinclude.m4 b/contrib/argp-standalone/acinclude.m4
deleted file mode 100644
index fb61e957dfa..00000000000
--- a/contrib/argp-standalone/acinclude.m4
+++ /dev/null
@@ -1,1084 +0,0 @@
-dnl Try to detect the type of the third arg to getsockname() et al
-AC_DEFUN([LSH_TYPE_SOCKLEN_T],
-[AH_TEMPLATE([socklen_t], [Length type used by getsockopt])
-AC_CACHE_CHECK([for socklen_t in sys/socket.h], ac_cv_type_socklen_t,
-[AC_EGREP_HEADER(socklen_t, sys/socket.h,
-  [ac_cv_type_socklen_t=yes], [ac_cv_type_socklen_t=no])])
-if test $ac_cv_type_socklen_t = no; then
-        AC_MSG_CHECKING(for AIX)
-        AC_EGREP_CPP(yes, [
-#ifdef _AIX
- yes
-#endif
-],[
-AC_MSG_RESULT(yes)
-AC_DEFINE(socklen_t, size_t)
-],[
-AC_MSG_RESULT(no)
-AC_DEFINE(socklen_t, int)
-])
-fi
-])
-
-dnl Choose cc flags for compiling position independent code
-AC_DEFUN([LSH_CCPIC],
-[AC_MSG_CHECKING(CCPIC)
-AC_CACHE_VAL(lsh_cv_sys_ccpic,[
-  if test -z "$CCPIC" ; then
-    if test "$GCC" = yes ; then
-      case `uname -sr` in
-	BSD/OS*)
-         case `uname -r` in
-           4.*) CCPIC="-fPIC";;
-           *) CCPIC="";;
-         esac
-	;;
-	Darwin*)
-	  CCPIC="-fPIC"
-	;;
-	SunOS\ 5.*)
-	  # Could also use -fPIC, if there are a large number of symbol reference
-	  CCPIC="-fPIC"
-	;;
-	CYGWIN*)
-	  CCPIC=""
-	;;
-	*)
-	  CCPIC="-fpic"
-	;;
-      esac
-    else
-      case `uname -sr` in
-	Darwin*)
-	  CCPIC="-fPIC"
-	;;
-        IRIX*)
-          CCPIC="-share"
-        ;;
-	hp*|HP*) CCPIC="+z"; ;;
-	FreeBSD*) CCPIC="-fpic";;
-	SCO_SV*) CCPIC="-KPIC -dy -Bdynamic";;
-        UnixWare*|OpenUNIX*) CCPIC="-KPIC -dy -Bdynamic";;
-	Solaris*) CCPIC="-KPIC -Bdynamic";;
-	Windows_NT*) CCPIC="-shared" ;;
-      esac
-    fi
-  fi
-  OLD_CFLAGS="$CFLAGS"
-  CFLAGS="$CFLAGS $CCPIC"
-  AC_TRY_COMPILE([], [exit(0);],
-    lsh_cv_sys_ccpic="$CCPIC", lsh_cv_sys_ccpic='')
-  CFLAGS="$OLD_CFLAGS"
-])
-CCPIC="$lsh_cv_sys_ccpic"
-AC_MSG_RESULT($CCPIC)
-AC_SUBST([CCPIC])])
-
-dnl LSH_PATH_ADD(path-id, directory)
-AC_DEFUN([LSH_PATH_ADD],
-[AC_MSG_CHECKING($2)
-ac_exists=no
-if test -d "$2/." ; then
-  ac_real_dir=`cd $2 && pwd`
-  if test -n "$ac_real_dir" ; then
-    ac_exists=yes
-    for old in $1_REAL_DIRS ; do
-      ac_found=no
-      if test x$ac_real_dir = x$old ; then
-        ac_found=yes;
-	break;
-      fi
-    done
-    if test $ac_found = yes ; then
-      AC_MSG_RESULT(already added)
-    else
-      AC_MSG_RESULT(added)
-      # LDFLAGS="$LDFLAGS -L $2"
-      $1_REAL_DIRS="$ac_real_dir [$]$1_REAL_DIRS"
-      $1_DIRS="$2 [$]$1_DIRS"
-    fi
-  fi
-fi
-if test $ac_exists = no ; then
-  AC_MSG_RESULT(not found)
-fi
-])
-
-dnl LSH_RPATH_ADD(dir)
-AC_DEFUN([LSH_RPATH_ADD], [LSH_PATH_ADD(RPATH_CANDIDATE, $1)])
-
-dnl LSH_RPATH_INIT(candidates)
-AC_DEFUN([LSH_RPATH_INIT],
-[AC_MSG_CHECKING([for -R flag])
-RPATHFLAG=''
-case `uname -sr` in
-  OSF1\ V4.*)
-    RPATHFLAG="-rpath "
-    ;;
-  IRIX\ 6.*)
-    RPATHFLAG="-rpath "
-    ;;
-  IRIX\ 5.*)
-    RPATHFLAG="-rpath "
-    ;;
-  SunOS\ 5.*)
-    if test "$TCC" = "yes"; then
-      # tcc doesn't know about -R
-      RPATHFLAG="-Wl,-R,"
-    else
-      RPATHFLAG=-R
-    fi
-    ;;
-  Linux\ 2.*)
-    RPATHFLAG="-Wl,-rpath,"
-    ;;
-  *)
-    :
-    ;;
-esac
-
-if test x$RPATHFLAG = x ; then
-  AC_MSG_RESULT(none)
-else
-  AC_MSG_RESULT([using $RPATHFLAG])
-fi
-
-RPATH_CANDIDATE_REAL_DIRS=''
-RPATH_CANDIDATE_DIRS=''
-
-AC_MSG_RESULT([Searching for libraries])
-
-for d in $1 ; do
-  LSH_RPATH_ADD($d)
-done
-])    
-
-dnl Try to execute a main program, and if it fails, try adding some
-dnl -R flag.
-dnl LSH_RPATH_FIX
-AC_DEFUN([LSH_RPATH_FIX],
-[if test $cross_compiling = no -a "x$RPATHFLAG" != x ; then
-  ac_success=no
-  AC_TRY_RUN([int main(int argc, char **argv) { return 0; }],
-    ac_success=yes, ac_success=no, :)
-  
-  if test $ac_success = no ; then
-    AC_MSG_CHECKING([Running simple test program failed. Trying -R flags])
-dnl echo RPATH_CANDIDATE_DIRS = $RPATH_CANDIDATE_DIRS
-    ac_remaining_dirs=''
-    ac_rpath_save_LDFLAGS="$LDFLAGS"
-    for d in $RPATH_CANDIDATE_DIRS ; do
-      if test $ac_success = yes ; then
-  	ac_remaining_dirs="$ac_remaining_dirs $d"
-      else
-  	LDFLAGS="$RPATHFLAG$d $LDFLAGS"
-dnl echo LDFLAGS = $LDFLAGS
-  	AC_TRY_RUN([int main(int argc, char **argv) { return 0; }],
-  	  [ac_success=yes
-  	  ac_rpath_save_LDFLAGS="$LDFLAGS"
-  	  AC_MSG_RESULT([adding $RPATHFLAG$d])
-  	  ],
-  	  [ac_remaining_dirs="$ac_remaining_dirs $d"], :)
-  	LDFLAGS="$ac_rpath_save_LDFLAGS"
-      fi
-    done
-    RPATH_CANDIDATE_DIRS=$ac_remaining_dirs
-  fi
-  if test $ac_success = no ; then
-    AC_MSG_RESULT(failed)
-  fi
-fi
-])
-
-dnl Like AC_CHECK_LIB, but uses $KRB_LIBS rather than $LIBS.
-dnl LSH_CHECK_KRB_LIB(LIBRARY, FUNCTION, [, ACTION-IF-FOUND [,
-dnl                  ACTION-IF-NOT-FOUND [, OTHER-LIBRARIES]]])
-
-AC_DEFUN([LSH_CHECK_KRB_LIB],
-[AC_CHECK_LIB([$1], [$2],
-  ifelse([$3], ,
-      [[ac_tr_lib=HAVE_LIB`echo $1 | sed -e 's/[^a-zA-Z0-9_]/_/g' \
-     	    -e 'y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/'`
-        AC_DEFINE_UNQUOTED($ac_tr_lib)
-        KRB_LIBS="-l$1 $KRB_LIBS"
-      ]], [$3]),
-  ifelse([$4], , , [$4
-])dnl
-, [$5 $KRB_LIBS])
-])
-
-dnl LSH_LIB_ARGP(ACTION-IF-OK, ACTION-IF-BAD)
-AC_DEFUN([LSH_LIB_ARGP],
-[ ac_argp_save_LIBS="$LIBS"
-  ac_argp_save_LDFLAGS="$LDFLAGS"
-  ac_argp_ok=no
-  # First check if we can link with argp.
-  AC_SEARCH_LIBS(argp_parse, argp,
-  [ LSH_RPATH_FIX
-    AC_CACHE_CHECK([for working argp],
-      lsh_cv_lib_argp_works,
-      [ AC_TRY_RUN(
-[#include <argp.h>
-#include <stdlib.h>
-
-static const struct argp_option
-options[] =
-{
-  { NULL, 0, NULL, 0, NULL, 0 }
-};
-
-struct child_state
-{
-  int n;
-};
-
-static error_t
-child_parser(int key, char *arg, struct argp_state *state)
-{
-  struct child_state *input = (struct child_state *) state->input;
-  
-  switch(key)
-    {
-    default:
-      return ARGP_ERR_UNKNOWN;
-    case ARGP_KEY_END:
-      if (!input->n)
-	input->n = 1;
-      break;
-    }
-  return 0;
-}
-
-const struct argp child_argp =
-{
-  options,
-  child_parser,
-  NULL, NULL, NULL, NULL, NULL
-};
-
-struct main_state
-{
-  struct child_state child;
-  int m;
-};
-
-static error_t
-main_parser(int key, char *arg, struct argp_state *state)
-{
-  struct main_state *input = (struct main_state *) state->input;
-
-  switch(key)
-    {
-    default:
-      return ARGP_ERR_UNKNOWN;
-    case ARGP_KEY_INIT:
-      state->child_inputs[0] = &input->child;
-      break;
-    case ARGP_KEY_END:
-      if (!input->m)
-	input->m = input->child.n;
-      
-      break;
-    }
-  return 0;
-}
-
-static const struct argp_child
-main_children[] =
-{
-  { &child_argp, 0, "", 0 },
-  { NULL, 0, NULL, 0}
-};
-
-static const struct argp
-main_argp =
-{ options, main_parser, 
-  NULL,
-  NULL,
-  main_children,
-  NULL, NULL
-};
-
-int main(int argc, char **argv)
-{
-  struct main_state input = { { 0 }, 0 };
-  char *v[2] = { "foo", NULL };
-
-  argp_parse(&main_argp, 1, v, 0, NULL, &input);
-
-  if ( (input.m == 1) && (input.child.n == 1) )
-    return 0;
-  else
-    return 1;
-}
-], lsh_cv_lib_argp_works=yes,
-   lsh_cv_lib_argp_works=no,
-   lsh_cv_lib_argp_works=no)])
-
-  if test x$lsh_cv_lib_argp_works = xyes ; then
-    ac_argp_ok=yes
-  else
-    # Reset link flags
-    LIBS="$ac_argp_save_LIBS"
-    LDFLAGS="$ac_argp_save_LDFLAGS"
-  fi])
-
-  if test x$ac_argp_ok = xyes ; then
-    ifelse([$1],, true, [$1])
-  else
-    ifelse([$2],, true, [$2])
-  fi   
-])
-
-dnl LSH_GCC_ATTRIBUTES
-dnl Check for gcc's __attribute__ construction
-
-AC_DEFUN([LSH_GCC_ATTRIBUTES],
-[AC_CACHE_CHECK(for __attribute__,
-	       lsh_cv_c_attribute,
-[ AC_TRY_COMPILE([
-#include <stdlib.h>
-],
-[
-static void foo(void) __attribute__ ((noreturn));
-
-static void __attribute__ ((noreturn))
-foo(void)
-{
-  exit(1);
-}
-],
-lsh_cv_c_attribute=yes,
-lsh_cv_c_attribute=no)])
-
-AH_TEMPLATE([HAVE_GCC_ATTRIBUTE], [Define if the compiler understands __attribute__])
-if test "x$lsh_cv_c_attribute" = "xyes"; then
-  AC_DEFINE(HAVE_GCC_ATTRIBUTE)
-fi
-
-AH_BOTTOM(
-[#if __GNUC__ || HAVE_GCC_ATTRIBUTE
-# define NORETURN __attribute__ ((__noreturn__))
-# define PRINTF_STYLE(f, a) __attribute__ ((__format__ (__printf__, f, a)))
-# define UNUSED __attribute__ ((__unused__))
-#else
-# define NORETURN
-# define PRINTF_STYLE(f, a)
-# define UNUSED
-#endif
-])])
-
-AC_DEFUN([LSH_GCC_FUNCTION_NAME],
-[# Check for gcc's __FUNCTION__ variable
-AH_TEMPLATE([HAVE_GCC_FUNCTION],
-	    [Define if the compiler understands __FUNCTION__])
-AH_BOTTOM(
-[#if HAVE_GCC_FUNCTION
-# define FUNCTION_NAME __FUNCTION__
-#else
-# define FUNCTION_NAME "Unknown"
-#endif
-])
-
-AC_CACHE_CHECK(for __FUNCTION__,
-	       lsh_cv_c_FUNCTION,
-  [ AC_TRY_COMPILE(,
-      [ #if __GNUC__ == 3
-	#  error __FUNCTION__ is broken in gcc-3
-	#endif
-        void foo(void) { char c = __FUNCTION__[0]; } ],
-      lsh_cv_c_FUNCTION=yes,
-      lsh_cv_c_FUNCTION=no)])
-
-if test "x$lsh_cv_c_FUNCTION" = "xyes"; then
-  AC_DEFINE(HAVE_GCC_FUNCTION)
-fi
-])
-
-# Check for alloca, and include the standard blurb in config.h
-AC_DEFUN([LSH_FUNC_ALLOCA],
-[AC_FUNC_ALLOCA
-AC_CHECK_HEADERS([malloc.h])
-AH_BOTTOM(
-[/* AIX requires this to be the first thing in the file.  */
-#ifndef __GNUC__
-# if HAVE_ALLOCA_H
-#  include <alloca.h>
-# else
-#  ifdef _AIX
- #pragma alloca
-#  else
-#   ifndef alloca /* predefined by HP cc +Olibcalls */
-char *alloca ();
-#   endif
-#  endif
-# endif
-#else /* defined __GNUC__ */
-# if HAVE_ALLOCA_H
-#  include <alloca.h>
-# endif
-#endif
-/* Needed for alloca on windows */
-#if HAVE_MALLOC_H
-# include <malloc.h>
-#endif
-])])
-
-AC_DEFUN([LSH_FUNC_STRERROR],
-[AC_CHECK_FUNCS(strerror)
-AH_BOTTOM(
-[#if HAVE_STRERROR
-#define STRERROR strerror
-#else
-#define STRERROR(x) (sys_errlist[x])
-#endif
-])])
-
-AC_DEFUN([LSH_FUNC_STRSIGNAL],
-[AC_CHECK_FUNCS(strsignal)
-AC_CHECK_DECLS([sys_siglist, _sys_siglist])
-AH_BOTTOM(
-[#if HAVE_STRSIGNAL
-# define STRSIGNAL strsignal
-#else /* !HAVE_STRSIGNAL */
-# if HAVE_DECL_SYS_SIGLIST
-#  define STRSIGNAL(x) (sys_siglist[x])
-# else
-#  if HAVE_DECL__SYS_SIGLIST
-#   define STRSIGNAL(x) (_sys_siglist[x])
-#  else
-#   define STRSIGNAL(x) "Unknown signal"
-#   if __GNUC__
-#    warning Using dummy STRSIGNAL
-#   endif
-#  endif
-# endif
-#endif /* !HAVE_STRSIGNAL */
-])])
-
-dnl LSH_MAKE_CONDITIONAL(symbol, test)
-AC_DEFUN([LSH_MAKE_CONDITIONAL],
-[if $2 ; then
-  IF_$1=''
-  UNLESS_$1='# '
-else
-  IF_$1='# '
-  UNLESS_$1=''
-fi 
-AC_SUBST(IF_$1)
-AC_SUBST(UNLESS_$1)])
-
-dnl LSH_DEPENDENCY_TRACKING
-
-dnl Defines compiler flags DEP_FLAGS to generate dependency
-dnl information, and DEP_PROCESS that is any shell commands needed for
-dnl massaging the dependency information further. Dependencies are
-dnl generated as a side effect of compilation. Dependency files
-dnl themselves are not treated as targets.
-
-AC_DEFUN([LSH_DEPENDENCY_TRACKING],
-[AC_ARG_ENABLE(dependency_tracking,
-  AC_HELP_STRING([--disable-dependency-tracking],
-    [Disable dependency tracking. Dependency tracking doesn't work with BSD make]),,
-  [enable_dependency_tracking=yes])
-
-DEP_FLAGS=''
-DEP_PROCESS='true'
-if test x$enable_dependency_tracking = xyes ; then
-  if test x$GCC = xyes ; then
-    gcc_version=`gcc --version | head -1`
-    case "$gcc_version" in
-      2.*|*[[!0-9.]]2.*)
-        enable_dependency_tracking=no
-        AC_MSG_WARN([Dependency tracking disabled, gcc-3.x is needed])
-      ;;
-      *)
-        DEP_FLAGS='-MT $[]@ -MD -MP -MF $[]@.d'
-        DEP_PROCESS='true'
-      ;;
-    esac
-  else
-    enable_dependency_tracking=no
-    AC_MSG_WARN([Dependency tracking disabled])
-  fi
-fi
-
-if test x$enable_dependency_tracking = xyes ; then
-  DEP_INCLUDE='include '
-else
-  DEP_INCLUDE='# '
-fi
-
-AC_SUBST([DEP_INCLUDE])
-AC_SUBST([DEP_FLAGS])
-AC_SUBST([DEP_PROCESS])])
-
-dnl @synopsis AX_CREATE_STDINT_H [( HEADER-TO-GENERATE [, HEADERS-TO-CHECK])]
-dnl
-dnl the "ISO C9X: 7.18 Integer types <stdint.h>" section requires the
-dnl existence of an include file <stdint.h> that defines a set of 
-dnl typedefs, especially uint8_t,int32_t,uintptr_t.
-dnl Many older installations will not provide this file, but some will
-dnl have the very same definitions in <inttypes.h>. In other enviroments
-dnl we can use the inet-types in <sys/types.h> which would define the
-dnl typedefs int8_t and u_int8_t respectivly.
-dnl
-dnl This macros will create a local "_stdint.h" or the headerfile given as 
-dnl an argument. In many cases that file will just "#include <stdint.h>" 
-dnl or "#include <inttypes.h>", while in other environments it will provide 
-dnl the set of basic 'stdint's definitions/typedefs: 
-dnl   int8_t,uint8_t,int16_t,uint16_t,int32_t,uint32_t,intptr_t,uintptr_t
-dnl   int_least32_t.. int_fast32_t.. intmax_t
-dnl which may or may not rely on the definitions of other files,
-dnl or using the AC_CHECK_SIZEOF macro to determine the actual
-dnl sizeof each type.
-dnl
-dnl if your header files require the stdint-types you will want to create an
-dnl installable file mylib-int.h that all your other installable header
-dnl may include. So if you have a library package named "mylib", just use
-dnl      AX_CREATE_STDINT_H(mylib-int.h) 
-dnl in configure.ac and go to install that very header file in Makefile.am
-dnl along with the other headers (mylib.h) - and the mylib-specific headers
-dnl can simply use "#include <mylib-int.h>" to obtain the stdint-types.
-dnl
-dnl Remember, if the system already had a valid <stdint.h>, the generated
-dnl file will include it directly. No need for fuzzy HAVE_STDINT_H things...
-dnl
-dnl @, (status: used on new platforms) (see http://ac-archive.sf.net/gstdint/)
-dnl @version $Id: acinclude.m4,v 1.27 2004/11/23 21:27:35 nisse Exp $
-dnl @author  Guido Draheim <guidod@gmx.de> 
-
-AC_DEFUN([AX_CREATE_STDINT_H],
-[# ------ AX CREATE STDINT H -------------------------------------
-AC_MSG_CHECKING([for stdint types])
-ac_stdint_h=`echo ifelse($1, , _stdint.h, $1)`
-# try to shortcircuit - if the default include path of the compiler
-# can find a "stdint.h" header then we assume that all compilers can.
-AC_CACHE_VAL([ac_cv_header_stdint_t],[
-old_CXXFLAGS="$CXXFLAGS" ; CXXFLAGS=""
-old_CPPFLAGS="$CPPFLAGS" ; CPPFLAGS=""
-old_CFLAGS="$CFLAGS"     ; CFLAGS=""
-AC_TRY_COMPILE([#include <stdint.h>],[int_least32_t v = 0;],
-[ac_cv_stdint_result="(assuming C99 compatible system)"
- ac_cv_header_stdint_t="stdint.h"; ],
-[ac_cv_header_stdint_t=""])
-CXXFLAGS="$old_CXXFLAGS"
-CPPFLAGS="$old_CPPFLAGS"
-CFLAGS="$old_CFLAGS" ])
-
-v="... $ac_cv_header_stdint_h"
-if test "$ac_stdint_h" = "stdint.h" ; then
- AC_MSG_RESULT([(are you sure you want them in ./stdint.h?)])
-elif test "$ac_stdint_h" = "inttypes.h" ; then
- AC_MSG_RESULT([(are you sure you want them in ./inttypes.h?)])
-elif test "_$ac_cv_header_stdint_t" = "_" ; then
- AC_MSG_RESULT([(putting them into $ac_stdint_h)$v])
-else
- ac_cv_header_stdint="$ac_cv_header_stdint_t"
- AC_MSG_RESULT([$ac_cv_header_stdint (shortcircuit)])
-fi
-
-if test "_$ac_cv_header_stdint_t" = "_" ; then # can not shortcircuit..
-
-dnl .....intro message done, now do a few system checks.....
-dnl btw, all CHECK_TYPE macros do automatically "DEFINE" a type, therefore
-dnl we use the autoconf implementation detail _AC CHECK_TYPE_NEW instead
-
-inttype_headers=`echo $2 | sed -e 's/,/ /g'`
-
-ac_cv_stdint_result="(no helpful system typedefs seen)"
-AC_CACHE_CHECK([for stdint uintptr_t], [ac_cv_header_stdint_x],[
- ac_cv_header_stdint_x="" # the 1997 typedefs (inttypes.h)
-  AC_MSG_RESULT([(..)])
-  for i in stdint.h inttypes.h sys/inttypes.h $inttype_headers ; do
-   unset ac_cv_type_uintptr_t 
-   unset ac_cv_type_uint64_t
-   _AC_CHECK_TYPE_NEW(uintptr_t,[ac_cv_header_stdint_x=$i],dnl
-     continue,[#include <$i>])
-   AC_CHECK_TYPE(uint64_t,[and64="/uint64_t"],[and64=""],[#include<$i>])
-   ac_cv_stdint_result="(seen uintptr_t$and64 in $i)"
-   break;
-  done
-  AC_MSG_CHECKING([for stdint uintptr_t])
- ])
-
-if test "_$ac_cv_header_stdint_x" = "_" ; then
-AC_CACHE_CHECK([for stdint uint32_t], [ac_cv_header_stdint_o],[
- ac_cv_header_stdint_o="" # the 1995 typedefs (sys/inttypes.h)
-  AC_MSG_RESULT([(..)])
-  for i in inttypes.h sys/inttypes.h stdint.h $inttype_headers ; do
-   unset ac_cv_type_uint32_t
-   unset ac_cv_type_uint64_t
-   AC_CHECK_TYPE(uint32_t,[ac_cv_header_stdint_o=$i],dnl
-     continue,[#include <$i>])
-   AC_CHECK_TYPE(uint64_t,[and64="/uint64_t"],[and64=""],[#include<$i>])
-   ac_cv_stdint_result="(seen uint32_t$and64 in $i)"
-   break;
-  done
-  AC_MSG_CHECKING([for stdint uint32_t])
- ])
-fi
-
-if test "_$ac_cv_header_stdint_x" = "_" ; then
-if test "_$ac_cv_header_stdint_o" = "_" ; then
-AC_CACHE_CHECK([for stdint u_int32_t], [ac_cv_header_stdint_u],[
- ac_cv_header_stdint_u="" # the BSD typedefs (sys/types.h)
-  AC_MSG_RESULT([(..)])
-  for i in sys/types.h inttypes.h sys/inttypes.h $inttype_headers ; do
-   unset ac_cv_type_u_int32_t
-   unset ac_cv_type_u_int64_t
-   AC_CHECK_TYPE(u_int32_t,[ac_cv_header_stdint_u=$i],dnl
-     continue,[#include <$i>])
-   AC_CHECK_TYPE(u_int64_t,[and64="/u_int64_t"],[and64=""],[#include<$i>])
-   ac_cv_stdint_result="(seen u_int32_t$and64 in $i)"
-   break;
-  done
-  AC_MSG_CHECKING([for stdint u_int32_t])
- ])
-fi fi
-
-dnl if there was no good C99 header file, do some typedef checks...
-if test "_$ac_cv_header_stdint_x" = "_" ; then
-   AC_MSG_CHECKING([for stdint datatype model])
-   AC_MSG_RESULT([(..)])
-   AC_CHECK_SIZEOF(char)
-   AC_CHECK_SIZEOF(short)
-   AC_CHECK_SIZEOF(int)
-   AC_CHECK_SIZEOF(long)
-   AC_CHECK_SIZEOF(void*)
-   ac_cv_stdint_char_model=""
-   ac_cv_stdint_char_model="$ac_cv_stdint_char_model$ac_cv_sizeof_char"
-   ac_cv_stdint_char_model="$ac_cv_stdint_char_model$ac_cv_sizeof_short"
-   ac_cv_stdint_char_model="$ac_cv_stdint_char_model$ac_cv_sizeof_int"
-   ac_cv_stdint_long_model=""
-   ac_cv_stdint_long_model="$ac_cv_stdint_long_model$ac_cv_sizeof_int"
-   ac_cv_stdint_long_model="$ac_cv_stdint_long_model$ac_cv_sizeof_long"
-   ac_cv_stdint_long_model="$ac_cv_stdint_long_model$ac_cv_sizeof_voidp"
-   name="$ac_cv_stdint_long_model"
-   case "$ac_cv_stdint_char_model/$ac_cv_stdint_long_model" in
-    122/242)     name="$name,  IP16 (standard 16bit machine)" ;;
-    122/244)     name="$name,  LP32 (standard 32bit mac/win)" ;;
-    122/*)       name="$name        (unusual int16 model)" ;; 
-    124/444)     name="$name, ILP32 (standard 32bit unixish)" ;;
-    124/488)     name="$name,  LP64 (standard 64bit unixish)" ;;
-    124/448)     name="$name, LLP64 (unusual  64bit unixish)" ;;
-    124/*)       name="$name        (unusual int32 model)" ;; 
-    128/888)     name="$name, ILP64 (unusual  64bit numeric)" ;;
-    128/*)       name="$name        (unusual int64 model)" ;; 
-    222/*|444/*) name="$name        (unusual dsptype)" ;;
-     *)          name="$name        (very unusal model)" ;;
-   esac
-   AC_MSG_RESULT([combined for stdint datatype model...  $name])
-fi
-
-if test "_$ac_cv_header_stdint_x" != "_" ; then
-   ac_cv_header_stdint="$ac_cv_header_stdint_x"
-elif  test "_$ac_cv_header_stdint_o" != "_" ; then
-   ac_cv_header_stdint="$ac_cv_header_stdint_o"
-elif  test "_$ac_cv_header_stdint_u" != "_" ; then
-   ac_cv_header_stdint="$ac_cv_header_stdint_u"
-else
-   ac_cv_header_stdint="stddef.h"
-fi
-
-AC_MSG_CHECKING([for extra inttypes in chosen header])
-AC_MSG_RESULT([($ac_cv_header_stdint)])
-dnl see if int_least and int_fast types are present in _this_ header.
-unset ac_cv_type_int_least32_t
-unset ac_cv_type_int_fast32_t
-AC_CHECK_TYPE(int_least32_t,,,[#include <$ac_cv_header_stdint>])
-AC_CHECK_TYPE(int_fast32_t,,,[#include<$ac_cv_header_stdint>])
-AC_CHECK_TYPE(intmax_t,,,[#include <$ac_cv_header_stdint>])
-
-fi # shortcircut to system "stdint.h"
-# ------------------ PREPARE VARIABLES ------------------------------
-if test "$GCC" = "yes" ; then
-ac_cv_stdint_message="using gnu compiler "`$CC --version | head -1` 
-else
-ac_cv_stdint_message="using $CC"
-fi
-
-AC_MSG_RESULT([make use of $ac_cv_header_stdint in $ac_stdint_h dnl
-$ac_cv_stdint_result])
-
-# ----------------- DONE inttypes.h checks START header -------------
-AC_CONFIG_COMMANDS([$ac_stdint_h],[
-AC_MSG_NOTICE(creating $ac_stdint_h : $_ac_stdint_h)
-ac_stdint=$tmp/_stdint.h
-
-echo "#ifndef" $_ac_stdint_h >$ac_stdint
-echo "#define" $_ac_stdint_h "1" >>$ac_stdint
-echo "#ifndef" _GENERATED_STDINT_H >>$ac_stdint
-echo "#define" _GENERATED_STDINT_H '"'$PACKAGE $VERSION'"' >>$ac_stdint
-echo "/* generated $ac_cv_stdint_message */" >>$ac_stdint
-if test "_$ac_cv_header_stdint_t" != "_" ; then 
-echo "#define _STDINT_HAVE_STDINT_H" "1" >>$ac_stdint
-fi
-
-cat >>$ac_stdint <<STDINT_EOF
-
-/* ................... shortcircuit part ........................... */
-
-#if defined HAVE_STDINT_H || defined _STDINT_HAVE_STDINT_H
-#include <stdint.h>
-#else
-#include <stddef.h>
-
-/* .................... configured part ............................ */
-
-STDINT_EOF
-
-echo "/* whether we have a C99 compatible stdint header file */" >>$ac_stdint
-if test "_$ac_cv_header_stdint_x" != "_" ; then
-  ac_header="$ac_cv_header_stdint_x"
-  echo "#define _STDINT_HEADER_INTPTR" '"'"$ac_header"'"' >>$ac_stdint
-else
-  echo "/* #undef _STDINT_HEADER_INTPTR */" >>$ac_stdint
-fi
-
-echo "/* whether we have a C96 compatible inttypes header file */" >>$ac_stdint
-if  test "_$ac_cv_header_stdint_o" != "_" ; then
-  ac_header="$ac_cv_header_stdint_o"
-  echo "#define _STDINT_HEADER_UINT32" '"'"$ac_header"'"' >>$ac_stdint
-else
-  echo "/* #undef _STDINT_HEADER_UINT32 */" >>$ac_stdint
-fi
-
-echo "/* whether we have a BSD compatible inet types header */" >>$ac_stdint
-if  test "_$ac_cv_header_stdint_u" != "_" ; then
-  ac_header="$ac_cv_header_stdint_u"
-  echo "#define _STDINT_HEADER_U_INT32" '"'"$ac_header"'"' >>$ac_stdint
-else
-  echo "/* #undef _STDINT_HEADER_U_INT32 */" >>$ac_stdint
-fi
-
-echo "" >>$ac_stdint
-
-if test "_$ac_header" != "_" ; then if test "$ac_header" != "stddef.h" ; then
-  echo "#include <$ac_header>" >>$ac_stdint
-  echo "" >>$ac_stdint
-fi fi
-
-echo "/* which 64bit typedef has been found */" >>$ac_stdint
-if test "$ac_cv_type_uint64_t" = "yes" ; then
-echo "#define   _STDINT_HAVE_UINT64_T" "1"  >>$ac_stdint
-else
-echo "/* #undef _STDINT_HAVE_UINT64_T */" >>$ac_stdint
-fi
-if test "$ac_cv_type_u_int64_t" = "yes" ; then
-echo "#define   _STDINT_HAVE_U_INT64_T" "1"  >>$ac_stdint
-else
-echo "/* #undef _STDINT_HAVE_U_INT64_T */" >>$ac_stdint
-fi
-echo "" >>$ac_stdint
-
-echo "/* which type model has been detected */" >>$ac_stdint
-if test "_$ac_cv_stdint_char_model" != "_" ; then
-echo "#define   _STDINT_CHAR_MODEL" "$ac_cv_stdint_char_model" >>$ac_stdint
-echo "#define   _STDINT_LONG_MODEL" "$ac_cv_stdint_long_model" >>$ac_stdint
-else
-echo "/* #undef _STDINT_CHAR_MODEL // skipped */" >>$ac_stdint
-echo "/* #undef _STDINT_LONG_MODEL // skipped */" >>$ac_stdint
-fi
-echo "" >>$ac_stdint
-
-echo "/* whether int_least types were detected */" >>$ac_stdint
-if test "$ac_cv_type_int_least32_t" = "yes"; then
-echo "#define   _STDINT_HAVE_INT_LEAST32_T" "1"  >>$ac_stdint
-else
-echo "/* #undef _STDINT_HAVE_INT_LEAST32_T */" >>$ac_stdint
-fi
-echo "/* whether int_fast types were detected */" >>$ac_stdint
-if test "$ac_cv_type_int_fast32_t" = "yes"; then
-echo "#define   _STDINT_HAVE_INT_FAST32_T" "1" >>$ac_stdint
-else
-echo "/* #undef _STDINT_HAVE_INT_FAST32_T */" >>$ac_stdint
-fi
-echo "/* whether intmax_t type was detected */" >>$ac_stdint
-if test "$ac_cv_type_intmax_t" = "yes"; then
-echo "#define   _STDINT_HAVE_INTMAX_T" "1" >>$ac_stdint
-else
-echo "/* #undef _STDINT_HAVE_INTMAX_T */" >>$ac_stdint
-fi
-echo "" >>$ac_stdint
-
-  cat >>$ac_stdint <<STDINT_EOF
-/* .................... detections part ............................ */
-
-/* whether we need to define bitspecific types from compiler base types */
-#ifndef _STDINT_HEADER_INTPTR
-#ifndef _STDINT_HEADER_UINT32
-#ifndef _STDINT_HEADER_U_INT32
-#define _STDINT_NEED_INT_MODEL_T
-#else
-#define _STDINT_HAVE_U_INT_TYPES
-#endif
-#endif
-#endif
-
-#ifdef _STDINT_HAVE_U_INT_TYPES
-#undef _STDINT_NEED_INT_MODEL_T
-#endif
-
-#ifdef  _STDINT_CHAR_MODEL
-#if     _STDINT_CHAR_MODEL+0 == 122 || _STDINT_CHAR_MODEL+0 == 124
-#ifndef _STDINT_BYTE_MODEL
-#define _STDINT_BYTE_MODEL 12
-#endif
-#endif
-#endif
-
-#ifndef _STDINT_HAVE_INT_LEAST32_T
-#define _STDINT_NEED_INT_LEAST_T
-#endif
-
-#ifndef _STDINT_HAVE_INT_FAST32_T
-#define _STDINT_NEED_INT_FAST_T
-#endif
-
-#ifndef _STDINT_HEADER_INTPTR
-#define _STDINT_NEED_INTPTR_T
-#ifndef _STDINT_HAVE_INTMAX_T
-#define _STDINT_NEED_INTMAX_T
-#endif
-#endif
-
-
-/* .................... definition part ............................ */
-
-/* some system headers have good uint64_t */
-#ifndef _HAVE_UINT64_T
-#if     defined _STDINT_HAVE_UINT64_T  || defined HAVE_UINT64_T
-#define _HAVE_UINT64_T
-#elif   defined _STDINT_HAVE_U_INT64_T || defined HAVE_U_INT64_T
-#define _HAVE_UINT64_T
-typedef u_int64_t uint64_t;
-#endif
-#endif
-
-#ifndef _HAVE_UINT64_T
-/* .. here are some common heuristics using compiler runtime specifics */
-#if defined __STDC_VERSION__ && defined __STDC_VERSION__ >= 199901L
-#define _HAVE_UINT64_T
-typedef long long int64_t;
-typedef unsigned long long uint64_t;
-
-#elif !defined __STRICT_ANSI__
-#if defined _MSC_VER || defined __WATCOMC__ || defined __BORLANDC__
-#define _HAVE_UINT64_T
-typedef __int64 int64_t;
-typedef unsigned __int64 uint64_t;
-
-#elif defined __GNUC__ || defined __MWERKS__ || defined __ELF__
-/* note: all ELF-systems seem to have loff-support which needs 64-bit */
-#if !defined _NO_LONGLONG
-#define _HAVE_UINT64_T
-typedef long long int64_t;
-typedef unsigned long long uint64_t;
-#endif
-
-#elif defined __alpha || (defined __mips && defined _ABIN32)
-#if !defined _NO_LONGLONG
-typedef long int64_t;
-typedef unsigned long uint64_t;
-#endif
-  /* compiler/cpu type to define int64_t */
-#endif
-#endif
-#endif
-
-#if defined _STDINT_HAVE_U_INT_TYPES
-/* int8_t int16_t int32_t defined by inet code, redeclare the u_intXX types */
-typedef u_int8_t uint8_t;
-typedef u_int16_t uint16_t;
-typedef u_int32_t uint32_t;
-
-/* glibc compatibility */
-#ifndef __int8_t_defined
-#define __int8_t_defined
-#endif
-#endif
-
-#ifdef _STDINT_NEED_INT_MODEL_T
-/* we must guess all the basic types. Apart from byte-adressable system, */
-/* there a few 32-bit-only dsp-systems that we guard with BYTE_MODEL 8-} */
-/* (btw, those nibble-addressable systems are way off, or so we assume) */
-
-dnl   /* have a look at "64bit and data size neutrality" at */
-dnl   /* http://unix.org/version2/whatsnew/login_64bit.html */
-dnl   /* (the shorthand "ILP" types always have a "P" part) */
-
-#if defined _STDINT_BYTE_MODEL
-#if _STDINT_LONG_MODEL+0 == 242
-/* 2:4:2 =  IP16 = a normal 16-bit system                */
-typedef unsigned char   uint8_t;
-typedef unsigned short  uint16_t;
-typedef unsigned long   uint32_t;
-#ifndef __int8_t_defined
-#define __int8_t_defined
-typedef          char    int8_t;
-typedef          short   int16_t;
-typedef          long    int32_t;
-#endif
-#elif _STDINT_LONG_MODEL+0 == 244 || _STDINT_LONG_MODEL == 444
-/* 2:4:4 =  LP32 = a 32-bit system derived from a 16-bit */
-/* 4:4:4 = ILP32 = a normal 32-bit system                */
-typedef unsigned char   uint8_t;
-typedef unsigned short  uint16_t;
-typedef unsigned int    uint32_t;
-#ifndef __int8_t_defined
-#define __int8_t_defined
-typedef          char    int8_t;
-typedef          short   int16_t;
-typedef          int     int32_t;
-#endif
-#elif _STDINT_LONG_MODEL+0 == 484 || _STDINT_LONG_MODEL+0 == 488
-/* 4:8:4 =  IP32 = a 32-bit system prepared for 64-bit    */
-/* 4:8:8 =  LP64 = a normal 64-bit system                 */
-typedef unsigned char   uint8_t;
-typedef unsigned short  uint16_t;
-typedef unsigned int    uint32_t;
-#ifndef __int8_t_defined
-#define __int8_t_defined
-typedef          char    int8_t;
-typedef          short   int16_t;
-typedef          int     int32_t;
-#endif
-/* this system has a "long" of 64bit */
-#ifndef _HAVE_UINT64_T
-#define _HAVE_UINT64_T
-typedef unsigned long   uint64_t;
-typedef          long    int64_t;
-#endif
-#elif _STDINT_LONG_MODEL+0 == 448
-/*      LLP64   a 64-bit system derived from a 32-bit system */
-typedef unsigned char   uint8_t;
-typedef unsigned short  uint16_t;
-typedef unsigned int    uint32_t;
-#ifndef __int8_t_defined
-#define __int8_t_defined
-typedef          char    int8_t;
-typedef          short   int16_t;
-typedef          int     int32_t;
-#endif
-/* assuming the system has a "long long" */
-#ifndef _HAVE_UINT64_T
-#define _HAVE_UINT64_T
-typedef unsigned long long uint64_t;
-typedef          long long  int64_t;
-#endif
-#else
-#define _STDINT_NO_INT32_T
-#endif
-#else
-#define _STDINT_NO_INT8_T
-#define _STDINT_NO_INT32_T
-#endif
-#endif
-
-/*
- * quote from SunOS-5.8 sys/inttypes.h:
- * Use at your own risk.  As of February 1996, the committee is squarely
- * behind the fixed sized types; the "least" and "fast" types are still being
- * discussed.  The probability that the "fast" types may be removed before
- * the standard is finalized is high enough that they are not currently
- * implemented.
- */
-
-#if defined _STDINT_NEED_INT_LEAST_T
-typedef  int8_t    int_least8_t;
-typedef  int16_t   int_least16_t;
-typedef  int32_t   int_least32_t;
-#ifdef _HAVE_UINT64_T
-typedef  int64_t   int_least64_t;
-#endif
-
-typedef uint8_t   uint_least8_t;
-typedef uint16_t  uint_least16_t;
-typedef uint32_t  uint_least32_t;
-#ifdef _HAVE_UINT64_T
-typedef uint64_t  uint_least64_t;
-#endif
-  /* least types */
-#endif
-
-#if defined _STDINT_NEED_INT_FAST_T
-typedef  int8_t    int_fast8_t; 
-typedef  int       int_fast16_t;
-typedef  int32_t   int_fast32_t;
-#ifdef _HAVE_UINT64_T
-typedef  int64_t   int_fast64_t;
-#endif
-
-typedef uint8_t   uint_fast8_t; 
-typedef unsigned  uint_fast16_t;
-typedef uint32_t  uint_fast32_t;
-#ifdef _HAVE_UINT64_T
-typedef uint64_t  uint_fast64_t;
-#endif
-  /* fast types */
-#endif
-
-#ifdef _STDINT_NEED_INTMAX_T
-#ifdef _HAVE_UINT64_T
-typedef  int64_t       intmax_t;
-typedef uint64_t      uintmax_t;
-#else
-typedef          long  intmax_t;
-typedef unsigned long uintmax_t;
-#endif
-#endif
-
-#ifdef _STDINT_NEED_INTPTR_T
-#ifndef __intptr_t_defined
-#define __intptr_t_defined
-/* we encourage using "long" to store pointer values, never use "int" ! */
-#if   _STDINT_LONG_MODEL+0 == 242 || _STDINT_LONG_MODEL+0 == 484
-typedef  unsinged int   uintptr_t;
-typedef           int    intptr_t;
-#elif _STDINT_LONG_MODEL+0 == 244 || _STDINT_LONG_MODEL+0 == 444
-typedef  unsigned long  uintptr_t;
-typedef           long   intptr_t;
-#elif _STDINT_LONG_MODEL+0 == 448 && defined _HAVE_UINT64_T
-typedef        uint64_t uintptr_t;
-typedef         int64_t  intptr_t;
-#else /* matches typical system types ILP32 and LP64 - but not IP16 or LLP64 */
-typedef  unsigned long  uintptr_t;
-typedef           long   intptr_t;
-#endif
-#endif
-#endif
-
-  /* shortcircuit*/
-#endif
-  /* once */
-#endif
-#endif
-STDINT_EOF
-    if cmp -s $ac_stdint_h $ac_stdint 2>/dev/null; then
-      AC_MSG_NOTICE([$ac_stdint_h is unchanged])
-    else
-      ac_dir=`AS_DIRNAME(["$ac_stdint_h"])`
-      AS_MKDIR_P(["$ac_dir"])
-      rm -f $ac_stdint_h
-      mv $ac_stdint $ac_stdint_h
-    fi
-],[# variables for create stdint.h replacement
-PACKAGE="$PACKAGE"
-VERSION="$VERSION"
-ac_stdint_h="$ac_stdint_h"
-_ac_stdint_h=AS_TR_CPP(_$PACKAGE-$ac_stdint_h)
-ac_cv_stdint_message="$ac_cv_stdint_message"
-ac_cv_header_stdint_t="$ac_cv_header_stdint_t"
-ac_cv_header_stdint_x="$ac_cv_header_stdint_x"
-ac_cv_header_stdint_o="$ac_cv_header_stdint_o"
-ac_cv_header_stdint_u="$ac_cv_header_stdint_u"
-ac_cv_type_uint64_t="$ac_cv_type_uint64_t"
-ac_cv_type_u_int64_t="$ac_cv_type_u_int64_t"
-ac_cv_stdint_char_model="$ac_cv_stdint_char_model"
-ac_cv_stdint_long_model="$ac_cv_stdint_long_model"
-ac_cv_type_int_least32_t="$ac_cv_type_int_least32_t"
-ac_cv_type_int_fast32_t="$ac_cv_type_int_fast32_t"
-ac_cv_type_intmax_t="$ac_cv_type_intmax_t"
-])
-])
diff --git a/contrib/argp-standalone/argp-ba.c b/contrib/argp-standalone/argp-ba.c
deleted file mode 100644
index 0d3958c1151..00000000000
--- a/contrib/argp-standalone/argp-ba.c
+++ /dev/null
@@ -1,26 +0,0 @@
-/* Default definition for ARGP_PROGRAM_BUG_ADDRESS.
-   Copyright (C) 1996, 1997, 1999, 2004 Free Software Foundation, Inc.
-   This file is part of the GNU C Library.
-   Written by Miles Bader <miles@gnu.ai.mit.edu>.
-
-   The GNU C Library is free software; you can redistribute it and/or
-   modify it under the terms of the GNU Library General Public License as
-   published by the Free Software Foundation; either version 2 of the
-   License, or (at your option) any later version.
-
-   The GNU C Library is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-   Library General Public License for more details.
-
-   You should have received a copy of the GNU Library General Public
-   License along with the GNU C Library; see the file COPYING.LIB.  If not,
-   write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
-   Boston, MA 02111-1307, USA.  */
-
-/* If set by the user program, it should point to string that is the
-   bug-reporting address for the program.  It will be printed by argp_help if
-   the ARGP_HELP_BUG_ADDR flag is set (as it is by various standard help
-   messages), embedded in a sentence that says something like `Report bugs to
-   ADDR.'.  */
-const char *argp_program_bug_address = 0;
diff --git a/contrib/argp-standalone/argp-eexst.c b/contrib/argp-standalone/argp-eexst.c
deleted file mode 100644
index 46b27847ad4..00000000000
--- a/contrib/argp-standalone/argp-eexst.c
+++ /dev/null
@@ -1,36 +0,0 @@
-/* Default definition for ARGP_ERR_EXIT_STATUS
-   Copyright (C) 1997 Free Software Foundation, Inc.
-   This file is part of the GNU C Library.
-   Written by Miles Bader <miles@gnu.ai.mit.edu>.
-
-   The GNU C Library is free software; you can redistribute it and/or
-   modify it under the terms of the GNU Library General Public License as
-   published by the Free Software Foundation; either version 2 of the
-   License, or (at your option) any later version.
-
-   The GNU C Library is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-   Library General Public License for more details.
-
-   You should have received a copy of the GNU Library General Public
-   License along with the GNU C Library; see the file COPYING.LIB.  If not,
-   write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
-   Boston, MA 02111-1307, USA.  */
-
-#ifdef HAVE_CONFIG_H
-#include <config.h>
-#endif
-
-#if HAVE_SYSEXITS_H
-# include <sysexits.h>
-#else
-# define EX_USAGE 64
-#endif
-
-#include "argp.h"
-
-/* The exit status that argp will use when exiting due to a parsing error.
-   If not defined or set by the user program, this defaults to EX_USAGE from
-   <sysexits.h>.  */
-error_t argp_err_exit_status = EX_USAGE;
diff --git a/contrib/argp-standalone/argp-fmtstream.c b/contrib/argp-standalone/argp-fmtstream.c
deleted file mode 100644
index 494b6b31d12..00000000000
--- a/contrib/argp-standalone/argp-fmtstream.c
+++ /dev/null
@@ -1,477 +0,0 @@
-/* Word-wrapping and line-truncating streams
-   Copyright (C) 1997, 1998, 1999, 2001 Free Software Foundation, Inc.
-   This file is part of the GNU C Library.
-   Written by Miles Bader <miles@gnu.ai.mit.edu>.
-
-   The GNU C Library is free software; you can redistribute it and/or
-   modify it under the terms of the GNU Library General Public License as
-   published by the Free Software Foundation; either version 2 of the
-   License, or (at your option) any later version.
-
-   The GNU C Library is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-   Library General Public License for more details.
-
-   You should have received a copy of the GNU Library General Public
-   License along with the GNU C Library; see the file COPYING.LIB.  If not,
-   write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
-   Boston, MA 02111-1307, USA.  */
-
-/* This package emulates glibc `line_wrap_stream' semantics for systems that
-   don't have that.  */
-
-#ifdef HAVE_CONFIG_H
-#include <config.h>
-#endif
-
-#include <stdlib.h>
-#include <string.h>
-#include <errno.h>
-#include <stdarg.h>
-#include <ctype.h>
-
-#include "argp-fmtstream.h"
-#include "argp-namefrob.h"
-
-#ifndef ARGP_FMTSTREAM_USE_LINEWRAP
-
-#ifndef isblank
-#define isblank(ch) ((ch)==' ' || (ch)=='\t')
-#endif
-
-#if defined _LIBC && defined USE_IN_LIBIO
-# include <libio/libioP.h>
-# define __vsnprintf(s, l, f, a) _IO_vsnprintf (s, l, f, a)
-#endif
-
-#define INIT_BUF_SIZE 200
-#define PRINTF_SIZE_GUESS 150
-
-/* Return an argp_fmtstream that outputs to STREAM, and which prefixes lines
-   written on it with LMARGIN spaces and limits them to RMARGIN columns
-   total.  If WMARGIN >= 0, words that extend past RMARGIN are wrapped by
-   replacing the whitespace before them with a newline and WMARGIN spaces.
-   Otherwise, chars beyond RMARGIN are simply dropped until a newline.
-   Returns NULL if there was an error.  */
-argp_fmtstream_t
-__argp_make_fmtstream (FILE *stream,
-		       size_t lmargin, size_t rmargin, ssize_t wmargin)
-{
-  argp_fmtstream_t fs = malloc (sizeof (struct argp_fmtstream));
-  if (fs)
-    {
-      fs->stream = stream;
-
-      fs->lmargin = lmargin;
-      fs->rmargin = rmargin;
-      fs->wmargin = wmargin;
-      fs->point_col = 0;
-      fs->point_offs = 0;
-
-      fs->buf = malloc (INIT_BUF_SIZE);
-      if (! fs->buf)
-	{
-	  free (fs);
-	  fs = 0;
-	}
-      else
-	{
-	  fs->p = fs->buf;
-	  fs->end = fs->buf + INIT_BUF_SIZE;
-	}
-    }
-
-  return fs;
-}
-#ifdef weak_alias
-weak_alias (__argp_make_fmtstream, argp_make_fmtstream)
-#endif
-
-/* Flush FS to its stream, and free it (but don't close the stream).  */
-void
-__argp_fmtstream_free (argp_fmtstream_t fs)
-{
-  __argp_fmtstream_update (fs);
-  if (fs->p > fs->buf)
-    FWRITE_UNLOCKED (fs->buf, 1, fs->p - fs->buf, fs->stream);
-  free (fs->buf);
-  free (fs);
-}
-#ifdef weak_alias
-weak_alias (__argp_fmtstream_free, argp_fmtstream_free)
-#endif
-
-/* Process FS's buffer so that line wrapping is done from POINT_OFFS to the
-   end of its buffer.  This code is mostly from glibc stdio/linewrap.c.  */
-void
-__argp_fmtstream_update (argp_fmtstream_t fs)
-{
-  char *buf, *nl;
-  size_t len;
-
-  /* Scan the buffer for newlines.  */
-  buf = fs->buf + fs->point_offs;
-  while (buf < fs->p)
-    {
-      size_t r;
-
-      if (fs->point_col == 0 && fs->lmargin != 0)
-	{
-	  /* We are starting a new line.  Print spaces to the left margin.  */
-	  const size_t pad = fs->lmargin;
-	  if (fs->p + pad < fs->end)
-	    {
-	      /* We can fit in them in the buffer by moving the
-		 buffer text up and filling in the beginning.  */
-	      memmove (buf + pad, buf, fs->p - buf);
-	      fs->p += pad; /* Compensate for bigger buffer. */
-	      memset (buf, ' ', pad); /* Fill in the spaces.  */
-	      buf += pad; /* Don't bother searching them.  */
-	    }
-	  else
-	    {
-	      /* No buffer space for spaces.  Must flush.  */
-	      size_t i;
-	      for (i = 0; i < pad; i++)
-		PUTC_UNLOCKED (' ', fs->stream);
-	    }
-	  fs->point_col = pad;
-	}
-
-      len = fs->p - buf;
-      nl = memchr (buf, '\n', len);
-
-      if (fs->point_col < 0)
-	fs->point_col = 0;
-
-      if (!nl)
-	{
-	  /* The buffer ends in a partial line.  */
-
-	  if (fs->point_col + len < fs->rmargin)
-	    {
-	      /* The remaining buffer text is a partial line and fits
-		 within the maximum line width.  Advance point for the
-		 characters to be written and stop scanning.  */
-	      fs->point_col += len;
-	      break;
-	    }
-	  else
-	    /* Set the end-of-line pointer for the code below to
-	       the end of the buffer.  */
-	    nl = fs->p;
-	}
-      else if (fs->point_col + (nl - buf) < (ssize_t) fs->rmargin)
-	{
-	  /* The buffer contains a full line that fits within the maximum
-	     line width.  Reset point and scan the next line.  */
-	  fs->point_col = 0;
-	  buf = nl + 1;
-	  continue;
-	}
-
-      /* This line is too long.  */
-      r = fs->rmargin - 1;
-
-      if (fs->wmargin < 0)
-	{
-	  /* Truncate the line by overwriting the excess with the
-	     newline and anything after it in the buffer.  */
-	  if (nl < fs->p)
-	    {
-	      memmove (buf + (r - fs->point_col), nl, fs->p - nl);
-	      fs->p -= buf + (r - fs->point_col) - nl;
-	      /* Reset point for the next line and start scanning it.  */
-	      fs->point_col = 0;
-	      buf += r + 1; /* Skip full line plus \n. */
-	    }
-	  else
-	    {
-	      /* The buffer ends with a partial line that is beyond the
-		 maximum line width.  Advance point for the characters
-		 written, and discard those past the max from the buffer.  */
-	      fs->point_col += len;
-	      fs->p -= fs->point_col - r;
-	      break;
-	    }
-	}
-      else
-	{
-	  /* Do word wrap.  Go to the column just past the maximum line
-	     width and scan back for the beginning of the word there.
-	     Then insert a line break.  */
-
-	  char *p, *nextline;
-	  int i;
-
-	  p = buf + (r + 1 - fs->point_col);
-	  while (p >= buf && !isblank (*p))
-	    --p;
-	  nextline = p + 1;	/* This will begin the next line.  */
-
-	  if (nextline > buf)
-	    {
-	      /* Swallow separating blanks.  */
-	      if (p >= buf)
-		do
-		  --p;
-		while (p >= buf && isblank (*p));
-	      nl = p + 1;	/* The newline will replace the first blank. */
-	    }
-	  else
-	    {
-	      /* A single word that is greater than the maximum line width.
-		 Oh well.  Put it on an overlong line by itself.  */
-	      p = buf + (r + 1 - fs->point_col);
-	      /* Find the end of the long word.  */
-	      do
-		++p;
-	      while (p < nl && !isblank (*p));
-	      if (p == nl)
-		{
-		  /* It already ends a line.  No fussing required.  */
-		  fs->point_col = 0;
-		  buf = nl + 1;
-		  continue;
-		}
-	      /* We will move the newline to replace the first blank.  */
-	      nl = p;
-	      /* Swallow separating blanks.  */
-	      do
-		++p;
-	      while (isblank (*p));
-	      /* The next line will start here.  */
-	      nextline = p;
-	    }
-
-	  /* Note: There are a bunch of tests below for
-	     NEXTLINE == BUF + LEN + 1; this case is where NL happens to fall
-	     at the end of the buffer, and NEXTLINE is in fact empty (and so
-	     we need not be careful to maintain its contents).  */
-
-	  if (nextline == buf + len + 1
-	      ? fs->end - nl < fs->wmargin + 1
-	      : nextline - (nl + 1) < fs->wmargin)
-	    {
-	      /* The margin needs more blanks than we removed.  */
-	      if (fs->end - fs->p > fs->wmargin + 1)
-		/* Make some space for them.  */
-		{
-		  size_t mv = fs->p - nextline;
-		  memmove (nl + 1 + fs->wmargin, nextline, mv);
-		  nextline = nl + 1 + fs->wmargin;
-		  len = nextline + mv - buf;
-		  *nl++ = '\n';
-		}
-	      else
-		/* Output the first line so we can use the space.  */
-		{
-		  if (nl > fs->buf)
-		    FWRITE_UNLOCKED (fs->buf, 1, nl - fs->buf, fs->stream);
-		  PUTC_UNLOCKED ('\n', fs->stream);
-		  len += buf - fs->buf;
-		  nl = buf = fs->buf;
-		}
-	    }
-	  else
-	    /* We can fit the newline and blanks in before
-	       the next word.  */
-	    *nl++ = '\n';
-
-	  if (nextline - nl >= fs->wmargin
-	      || (nextline == buf + len + 1 && fs->end - nextline >= fs->wmargin))
-	    /* Add blanks up to the wrap margin column.  */
-	    for (i = 0; i < fs->wmargin; ++i)
-	      *nl++ = ' ';
-	  else
-	    for (i = 0; i < fs->wmargin; ++i)
-	      PUTC_UNLOCKED (' ', fs->stream);
-
-	  /* Copy the tail of the original buffer into the current buffer
-	     position.  */
-	  if (nl < nextline)
-	    memmove (nl, nextline, buf + len - nextline);
-	  len -= nextline - buf;
-
-	  /* Continue the scan on the remaining lines in the buffer.  */
-	  buf = nl;
-
-	  /* Restore bufp to include all the remaining text.  */
-	  fs->p = nl + len;
-
-	  /* Reset the counter of what has been output this line.  If wmargin
-	     is 0, we want to avoid the lmargin getting added, so we set
-	     point_col to a magic value of -1 in that case.  */
-	  fs->point_col = fs->wmargin ? fs->wmargin : -1;
-	}
-    }
-
-  /* Remember that we've scanned as far as the end of the buffer.  */
-  fs->point_offs = fs->p - fs->buf;
-}
-
-/* Ensure that FS has space for AMOUNT more bytes in its buffer, either by
-   growing the buffer, or by flushing it.  True is returned iff we succeed. */
-int
-__argp_fmtstream_ensure (struct argp_fmtstream *fs, size_t amount)
-{
-  if ((size_t) (fs->end - fs->p) < amount)
-    {
-      ssize_t wrote;
-
-      /* Flush FS's buffer.  */
-      __argp_fmtstream_update (fs);
-
-      wrote = FWRITE_UNLOCKED (fs->buf, 1, fs->p - fs->buf, fs->stream);
-      if (wrote == fs->p - fs->buf)
-	{
-	  fs->p = fs->buf;
-	  fs->point_offs = 0;
-	}
-      else
-	{
-	  fs->p -= wrote;
-	  fs->point_offs -= wrote;
-	  memmove (fs->buf, fs->buf + wrote, fs->p - fs->buf);
-	  return 0;
-	}
-
-      if ((size_t) (fs->end - fs->buf) < amount)
-	/* Gotta grow the buffer.  */
-	{
-	  size_t new_size = fs->end - fs->buf + amount;
-	  char *new_buf = realloc (fs->buf, new_size);
-
-	  if (! new_buf)
-	    {
-	      __set_errno (ENOMEM);
-	      return 0;
-	    }
-
-	  fs->buf = new_buf;
-	  fs->end = new_buf + new_size;
-	  fs->p = fs->buf;
-	}
-    }
-
-  return 1;
-}
-
-ssize_t
-__argp_fmtstream_printf (struct argp_fmtstream *fs, const char *fmt, ...)
-{
-  size_t out;
-  size_t avail;
-  size_t size_guess = PRINTF_SIZE_GUESS; /* How much space to reserve. */
-
-  do
-    {
-      va_list args;
-
-      if (! __argp_fmtstream_ensure (fs, size_guess))
-	return -1;
-
-      va_start (args, fmt);
-      avail = fs->end - fs->p;
-      out = __vsnprintf (fs->p, avail, fmt, args);
-      va_end (args);
-      if (out >= avail)
-	size_guess = out + 1;
-    }
-  while (out >= avail);
-
-  fs->p += out;
-
-  return out;
-}
-#ifdef weak_alias
-weak_alias (__argp_fmtstream_printf, argp_fmtstream_printf)
-#endif
-
-#if __STDC_VERSION__ - 199900L < 1
-/* Duplicate the inline definitions in argp-fmtstream.h, for compilers
- * that don't do inlining. */
-size_t
-__argp_fmtstream_write (argp_fmtstream_t __fs,
-			__const char *__str, size_t __len)
-{
-  if (__fs->p + __len <= __fs->end || __argp_fmtstream_ensure (__fs, __len))
-    {
-      memcpy (__fs->p, __str, __len);
-      __fs->p += __len;
-      return __len;
-    }
-  else
-    return 0;
-}
-
-int
-__argp_fmtstream_puts (argp_fmtstream_t __fs, __const char *__str)
-{
-  size_t __len = strlen (__str);
-  if (__len)
-    {
-      size_t __wrote = __argp_fmtstream_write (__fs, __str, __len);
-      return __wrote == __len ? 0 : -1;
-    }
-  else
-    return 0;
-}
-
-int
-__argp_fmtstream_putc (argp_fmtstream_t __fs, int __ch)
-{
-  if (__fs->p < __fs->end || __argp_fmtstream_ensure (__fs, 1))
-    return *__fs->p++ = __ch;
-  else
-    return EOF;
-}
-
-/* Set __FS's left margin to __LMARGIN and return the old value.  */
-size_t
-__argp_fmtstream_set_lmargin (argp_fmtstream_t __fs, size_t __lmargin)
-{
-  size_t __old;
-  if ((size_t) (__fs->p - __fs->buf) > __fs->point_offs)
-    __argp_fmtstream_update (__fs);
-  __old = __fs->lmargin;
-  __fs->lmargin = __lmargin;
-  return __old;
-}
-
-/* Set __FS's right margin to __RMARGIN and return the old value.  */
-size_t
-__argp_fmtstream_set_rmargin (argp_fmtstream_t __fs, size_t __rmargin)
-{
-  size_t __old;
-  if ((size_t) (__fs->p - __fs->buf) > __fs->point_offs)
-    __argp_fmtstream_update (__fs);
-  __old = __fs->rmargin;
-  __fs->rmargin = __rmargin;
-  return __old;
-}
-
-/* Set FS's wrap margin to __WMARGIN and return the old value.  */
-size_t
-__argp_fmtstream_set_wmargin (argp_fmtstream_t __fs, size_t __wmargin)
-{
-  size_t __old;
-  if ((size_t) (__fs->p - __fs->buf) > __fs->point_offs)
-    __argp_fmtstream_update (__fs);
-  __old = __fs->wmargin;
-  __fs->wmargin = __wmargin;
-  return __old;
-}
-
-/* Return the column number of the current output point in __FS.  */
-size_t
-__argp_fmtstream_point (argp_fmtstream_t __fs)
-{
-  if ((size_t) (__fs->p - __fs->buf) > __fs->point_offs)
-    __argp_fmtstream_update (__fs);
-  return __fs->point_col >= 0 ? __fs->point_col : 0;
-}
-#endif /* __STDC_VERSION__ - 199900L < 1 */
-
-#endif /* !ARGP_FMTSTREAM_USE_LINEWRAP */
diff --git a/contrib/argp-standalone/argp-fmtstream.h b/contrib/argp-standalone/argp-fmtstream.h
deleted file mode 100644
index 828f4357d56..00000000000
--- a/contrib/argp-standalone/argp-fmtstream.h
+++ /dev/null
@@ -1,327 +0,0 @@
-/* Word-wrapping and line-truncating streams.
-   Copyright (C) 1997, 2003 Free Software Foundation, Inc.
-   This file is part of the GNU C Library.
-   Written by Miles Bader <miles@gnu.ai.mit.edu>.
-
-   The GNU C Library is free software; you can redistribute it and/or
-   modify it under the terms of the GNU Library General Public License as
-   published by the Free Software Foundation; either version 2 of the
-   License, or (at your option) any later version.
-
-   The GNU C Library is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-   Library General Public License for more details.
-
-   You should have received a copy of the GNU Library General Public
-   License along with the GNU C Library; see the file COPYING.LIB.  If not,
-   write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
-   Boston, MA 02111-1307, USA.  */
-
-/* This package emulates glibc `line_wrap_stream' semantics for systems that
-   don't have that.  If the system does have it, it is just a wrapper for
-   that.  This header file is only used internally while compiling argp, and
-   shouldn't be installed.  */
-
-#ifndef _ARGP_FMTSTREAM_H
-#define _ARGP_FMTSTREAM_H
-
-#include <stdio.h>
-#include <string.h>
-
-#if HAVE_UNISTD_H
-# include <unistd.h>
-#else
-/* This is a kludge to make the code compile on windows. Perhaps it
-   would be better to just replace ssize_t with int through out the
-   code. */
-# define ssize_t int
-#endif
-
-#if _LIBC || (defined (HAVE_FLOCKFILE) && defined(HAVE_PUTC_UNLOCKED) \
-     && defined (HAVE_FPUTS_UNLOCKED) && defined (HAVE_FWRITE_UNLOCKED) )
-/* Use locking funxtions */
-# define FLOCKFILE(f) flockfile(f)
-# define FUNLOCKFILE(f) funlockfile(f)
-# define PUTC_UNLOCKED(c, f) putc_unlocked((c), (f))
-# define FPUTS_UNLOCKED(s, f) fputs_unlocked((s), (f))
-# define FWRITE_UNLOCKED(b, s, n, f) fwrite_unlocked((b), (s), (n), (f))
-#else
-/* Disable stdio locking */
-# define FLOCKFILE(f)
-# define FUNLOCKFILE(f)
-# define PUTC_UNLOCKED(c, f) putc((c), (f))
-# define FPUTS_UNLOCKED(s, f) fputs((s), (f))
-# define FWRITE_UNLOCKED(b, s, n, f) fwrite((b), (s), (n), (f))
-#endif /* No thread safe i/o */
-
-#if    (_LIBC - 0 && !defined (USE_IN_LIBIO)) \
-    || (defined (__GNU_LIBRARY__) && defined (HAVE_LINEWRAP_H))
-/* line_wrap_stream is available, so use that.  */
-#define ARGP_FMTSTREAM_USE_LINEWRAP
-#endif
-
-#ifdef ARGP_FMTSTREAM_USE_LINEWRAP
-/* Just be a simple wrapper for line_wrap_stream; the semantics are
-   *slightly* different, as line_wrap_stream doesn't actually make a new
-   object, it just modifies the given stream (reversibly) to do
-   line-wrapping.  Since we control who uses this code, it doesn't matter.  */
-
-#include <linewrap.h>
-
-typedef FILE *argp_fmtstream_t;
-
-#define argp_make_fmtstream line_wrap_stream
-#define __argp_make_fmtstream line_wrap_stream
-#define argp_fmtstream_free line_unwrap_stream
-#define __argp_fmtstream_free line_unwrap_stream
-
-#define __argp_fmtstream_putc(fs,ch) putc(ch,fs)
-#define argp_fmtstream_putc(fs,ch) putc(ch,fs)
-#define __argp_fmtstream_puts(fs,str) fputs(str,fs)
-#define argp_fmtstream_puts(fs,str) fputs(str,fs)
-#define __argp_fmtstream_write(fs,str,len) fwrite(str,1,len,fs)
-#define argp_fmtstream_write(fs,str,len) fwrite(str,1,len,fs)
-#define __argp_fmtstream_printf fprintf
-#define argp_fmtstream_printf fprintf
-
-#define __argp_fmtstream_lmargin line_wrap_lmargin
-#define argp_fmtstream_lmargin line_wrap_lmargin
-#define __argp_fmtstream_set_lmargin line_wrap_set_lmargin
-#define argp_fmtstream_set_lmargin line_wrap_set_lmargin
-#define __argp_fmtstream_rmargin line_wrap_rmargin
-#define argp_fmtstream_rmargin line_wrap_rmargin
-#define __argp_fmtstream_set_rmargin line_wrap_set_rmargin
-#define argp_fmtstream_set_rmargin line_wrap_set_rmargin
-#define __argp_fmtstream_wmargin line_wrap_wmargin
-#define argp_fmtstream_wmargin line_wrap_wmargin
-#define __argp_fmtstream_set_wmargin line_wrap_set_wmargin
-#define argp_fmtstream_set_wmargin line_wrap_set_wmargin
-#define __argp_fmtstream_point line_wrap_point
-#define argp_fmtstream_point line_wrap_point
-
-#else /* !ARGP_FMTSTREAM_USE_LINEWRAP */
-/* Guess we have to define our own version.  */
-
-#ifndef __const
-#define __const const
-#endif
-
-
-struct argp_fmtstream
-{
-  FILE *stream;			/* The stream we're outputting to.  */
-
-  size_t lmargin, rmargin;	/* Left and right margins.  */
-  ssize_t wmargin;		/* Margin to wrap to, or -1 to truncate.  */
-
-  /* Point in buffer to which we've processed for wrapping, but not output.  */
-  size_t point_offs;
-  /* Output column at POINT_OFFS, or -1 meaning 0 but don't add lmargin.  */
-  ssize_t point_col;
-
-  char *buf;			/* Output buffer.  */
-  char *p;			/* Current end of text in BUF. */
-  char *end;			/* Absolute end of BUF.  */
-};
-
-typedef struct argp_fmtstream *argp_fmtstream_t;
-
-/* Return an argp_fmtstream that outputs to STREAM, and which prefixes lines
-   written on it with LMARGIN spaces and limits them to RMARGIN columns
-   total.  If WMARGIN >= 0, words that extend past RMARGIN are wrapped by
-   replacing the whitespace before them with a newline and WMARGIN spaces.
-   Otherwise, chars beyond RMARGIN are simply dropped until a newline.
-   Returns NULL if there was an error.  */
-extern argp_fmtstream_t __argp_make_fmtstream (FILE *__stream,
-					       size_t __lmargin,
-					       size_t __rmargin,
-					       ssize_t __wmargin);
-extern argp_fmtstream_t argp_make_fmtstream (FILE *__stream,
-					     size_t __lmargin,
-					     size_t __rmargin,
-					     ssize_t __wmargin);
-
-/* Flush __FS to its stream, and free it (but don't close the stream).  */
-extern void __argp_fmtstream_free (argp_fmtstream_t __fs);
-extern void argp_fmtstream_free (argp_fmtstream_t __fs);
-
-extern ssize_t __argp_fmtstream_printf (argp_fmtstream_t __fs,
-				       __const char *__fmt, ...)
-     PRINTF_STYLE(2,3);
-extern ssize_t argp_fmtstream_printf (argp_fmtstream_t __fs,
-				      __const char *__fmt, ...)
-     PRINTF_STYLE(2,3);
-
-#if __STDC_VERSION__ - 199900L < 1
-extern int __argp_fmtstream_putc (argp_fmtstream_t __fs, int __ch);
-extern int argp_fmtstream_putc (argp_fmtstream_t __fs, int __ch);
-
-extern int __argp_fmtstream_puts (argp_fmtstream_t __fs, __const char *__str);
-extern int argp_fmtstream_puts (argp_fmtstream_t __fs, __const char *__str);
-
-extern size_t __argp_fmtstream_write (argp_fmtstream_t __fs,
-				      __const char *__str, size_t __len);
-extern size_t argp_fmtstream_write (argp_fmtstream_t __fs,
-				    __const char *__str, size_t __len);
-#endif /* __STDC_VERSION__ - 199900L < 1 */
-
-/* Access macros for various bits of state.  */
-#define argp_fmtstream_lmargin(__fs) ((__fs)->lmargin)
-#define argp_fmtstream_rmargin(__fs) ((__fs)->rmargin)
-#define argp_fmtstream_wmargin(__fs) ((__fs)->wmargin)
-#define __argp_fmtstream_lmargin argp_fmtstream_lmargin
-#define __argp_fmtstream_rmargin argp_fmtstream_rmargin
-#define __argp_fmtstream_wmargin argp_fmtstream_wmargin
-
-#if __STDC_VERSION__ - 199900L < 1
-/* Set __FS's left margin to LMARGIN and return the old value.  */
-extern size_t argp_fmtstream_set_lmargin (argp_fmtstream_t __fs,
-					  size_t __lmargin);
-extern size_t __argp_fmtstream_set_lmargin (argp_fmtstream_t __fs,
-					    size_t __lmargin);
-
-/* Set __FS's right margin to __RMARGIN and return the old value.  */
-extern size_t argp_fmtstream_set_rmargin (argp_fmtstream_t __fs,
-					  size_t __rmargin);
-extern size_t __argp_fmtstream_set_rmargin (argp_fmtstream_t __fs,
-					    size_t __rmargin);
-
-/* Set __FS's wrap margin to __WMARGIN and return the old value.  */
-extern size_t argp_fmtstream_set_wmargin (argp_fmtstream_t __fs,
-					  size_t __wmargin);
-extern size_t __argp_fmtstream_set_wmargin (argp_fmtstream_t __fs,
-					    size_t __wmargin);
-
-/* Return the column number of the current output point in __FS.  */
-extern size_t argp_fmtstream_point (argp_fmtstream_t __fs);
-extern size_t __argp_fmtstream_point (argp_fmtstream_t __fs);
-#endif /* __STDC_VERSION__ - 199900L < 1 */
-
-/* Internal routines.  */
-extern void _argp_fmtstream_update (argp_fmtstream_t __fs);
-extern void __argp_fmtstream_update (argp_fmtstream_t __fs);
-extern int _argp_fmtstream_ensure (argp_fmtstream_t __fs, size_t __amount);
-extern int __argp_fmtstream_ensure (argp_fmtstream_t __fs, size_t __amount);
-
-#ifdef __OPTIMIZE__
-/* Inline versions of above routines.  */
-
-#if !_LIBC
-#define __argp_fmtstream_putc argp_fmtstream_putc
-#define __argp_fmtstream_puts argp_fmtstream_puts
-#define __argp_fmtstream_write argp_fmtstream_write
-#define __argp_fmtstream_set_lmargin argp_fmtstream_set_lmargin
-#define __argp_fmtstream_set_rmargin argp_fmtstream_set_rmargin
-#define __argp_fmtstream_set_wmargin argp_fmtstream_set_wmargin
-#define __argp_fmtstream_point argp_fmtstream_point
-#define __argp_fmtstream_update _argp_fmtstream_update
-#define __argp_fmtstream_ensure _argp_fmtstream_ensure
-#endif
-
-#ifndef ARGP_FS_EI
-#if defined(__GNUC__) && !defined(__GNUC_STDC_INLINE__)
-#define ARGP_FS_EI extern inline
-#else
-#define ARGP_FS_EI inline
-#endif
-#endif
-
-ARGP_FS_EI size_t
-__argp_fmtstream_write (argp_fmtstream_t __fs,
-			__const char *__str, size_t __len)
-{
-  if (__fs->p + __len <= __fs->end || __argp_fmtstream_ensure (__fs, __len))
-    {
-      memcpy (__fs->p, __str, __len);
-      __fs->p += __len;
-      return __len;
-    }
-  else
-    return 0;
-}
-
-ARGP_FS_EI int
-__argp_fmtstream_puts (argp_fmtstream_t __fs, __const char *__str)
-{
-  size_t __len = strlen (__str);
-  if (__len)
-    {
-      size_t __wrote = __argp_fmtstream_write (__fs, __str, __len);
-      return __wrote == __len ? 0 : -1;
-    }
-  else
-    return 0;
-}
-
-ARGP_FS_EI int
-__argp_fmtstream_putc (argp_fmtstream_t __fs, int __ch)
-{
-  if (__fs->p < __fs->end || __argp_fmtstream_ensure (__fs, 1))
-    return *__fs->p++ = __ch;
-  else
-    return EOF;
-}
-
-/* Set __FS's left margin to __LMARGIN and return the old value.  */
-ARGP_FS_EI size_t
-__argp_fmtstream_set_lmargin (argp_fmtstream_t __fs, size_t __lmargin)
-{
-  size_t __old;
-  if ((size_t) (__fs->p - __fs->buf) > __fs->point_offs)
-    __argp_fmtstream_update (__fs);
-  __old = __fs->lmargin;
-  __fs->lmargin = __lmargin;
-  return __old;
-}
-
-/* Set __FS's right margin to __RMARGIN and return the old value.  */
-ARGP_FS_EI size_t
-__argp_fmtstream_set_rmargin (argp_fmtstream_t __fs, size_t __rmargin)
-{
-  size_t __old;
-  if ((size_t) (__fs->p - __fs->buf) > __fs->point_offs)
-    __argp_fmtstream_update (__fs);
-  __old = __fs->rmargin;
-  __fs->rmargin = __rmargin;
-  return __old;
-}
-
-/* Set FS's wrap margin to __WMARGIN and return the old value.  */
-ARGP_FS_EI size_t
-__argp_fmtstream_set_wmargin (argp_fmtstream_t __fs, size_t __wmargin)
-{
-  size_t __old;
-  if ((size_t) (__fs->p - __fs->buf) > __fs->point_offs)
-    __argp_fmtstream_update (__fs);
-  __old = __fs->wmargin;
-  __fs->wmargin = __wmargin;
-  return __old;
-}
-
-/* Return the column number of the current output point in __FS.  */
-ARGP_FS_EI size_t
-__argp_fmtstream_point (argp_fmtstream_t __fs)
-{
-  if ((size_t) (__fs->p - __fs->buf) > __fs->point_offs)
-    __argp_fmtstream_update (__fs);
-  return __fs->point_col >= 0 ? __fs->point_col : 0;
-}
-
-#if !_LIBC
-#undef __argp_fmtstream_putc
-#undef __argp_fmtstream_puts
-#undef __argp_fmtstream_write
-#undef __argp_fmtstream_set_lmargin
-#undef __argp_fmtstream_set_rmargin
-#undef __argp_fmtstream_set_wmargin
-#undef __argp_fmtstream_point
-#undef __argp_fmtstream_update
-#undef __argp_fmtstream_ensure
-#endif
-
-#endif /* __OPTIMIZE__ */
-
-#endif /* ARGP_FMTSTREAM_USE_LINEWRAP */
-
-#endif /* argp-fmtstream.h */
diff --git a/contrib/argp-standalone/argp-help.c b/contrib/argp-standalone/argp-help.c
deleted file mode 100644
index ced78c4cb26..00000000000
--- a/contrib/argp-standalone/argp-help.c
+++ /dev/null
@@ -1,1849 +0,0 @@
-/* Hierarchial argument parsing help output
-   Copyright (C) 1995,96,97,98,99,2000, 2003 Free Software Foundation, Inc.
-   This file is part of the GNU C Library.
-   Written by Miles Bader <miles@gnu.ai.mit.edu>.
-
-   The GNU C Library is free software; you can redistribute it and/or
-   modify it under the terms of the GNU Library General Public License as
-   published by the Free Software Foundation; either version 2 of the
-   License, or (at your option) any later version.
-
-   The GNU C Library is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-   Library General Public License for more details.
-
-   You should have received a copy of the GNU Library General Public
-   License along with the GNU C Library; see the file COPYING.LIB.  If not,
-   write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
-   Boston, MA 02111-1307, USA.  */
-
-#ifndef _GNU_SOURCE
-# define _GNU_SOURCE	1
-#endif
-
-#ifdef HAVE_CONFIG_H
-#include <config.h>
-#endif
-
-#if HAVE_ALLOCA_H
-#include <alloca.h>
-#endif
-
-#include <stddef.h>
-#include <stdlib.h>
-#include <string.h>
-#include <assert.h>
-#include <stdarg.h>
-#include <ctype.h>
-#if HAVE_MALLOC_H
-/* Needed, for alloca on windows */
-# include <malloc.h>
-#endif
-
-#ifndef _
-/* This is for other GNU distributions with internationalized messages.  */
-# if defined HAVE_LIBINTL_H || defined _LIBC
-#  include <libintl.h>
-#  ifdef _LIBC
-#   undef dgettext
-#   define dgettext(domain, msgid) __dcgettext (domain, msgid, LC_MESSAGES)
-#  endif
-# else
-#  define dgettext(domain, msgid) (msgid)
-# endif
-#endif
-
-#include "argp.h"
-#include "argp-fmtstream.h"
-#include "argp-namefrob.h"
-
-
-#ifndef _LIBC
-# ifndef __strchrnul
-#  define __strchrnul strchrnul
-# endif
-# ifndef __mempcpy
-#  define __mempcpy mempcpy
-# endif
-/* We need to use a different name, as __strndup is likely a macro. */
-# define STRNDUP strndup
-# if HAVE_STRERROR
-#  define STRERROR strerror
-# else
-#  define STRERROR(x) (sys_errlist[x])
-# endif
-#else /* _LIBC */
-# define FLOCKFILE __flockfile
-# define FUNLOCKFILE __funlockfile
-# define STRNDUP __strndup
-# define STRERROR strerror
-#endif
-
-#if !_LIBC
-# if !HAVE_STRNDUP
-char *strndup (const char *s, size_t size);
-# endif /* !HAVE_STRNDUP */
-
-# if !HAVE_MEMPCPY
-void *mempcpy (void *to, const void *from, size_t size);
-# endif /* !HAVE_MEMPCPY */
-
-# if !HAVE_STRCHRNUL
-char *strchrnul(const char *s, int c);
-# endif /* !HAVE_STRCHRNUL */
-
-# if !HAVE_STRCASECMP
-int strcasecmp(const char *s1, const char *s2);
-#endif
-
-#endif /* !_LIBC */
-
-
-/* User-selectable (using an environment variable) formatting parameters.
-
-   These may be specified in an environment variable called `ARGP_HELP_FMT',
-   with a contents like:  VAR1=VAL1,VAR2=VAL2,BOOLVAR2,no-BOOLVAR2
-   Where VALn must be a positive integer.  The list of variables is in the
-   UPARAM_NAMES vector, below.  */
-
-/* Default parameters.  */
-#define DUP_ARGS      0		/* True if option argument can be duplicated. */
-#define DUP_ARGS_NOTE 1		/* True to print a note about duplicate args. */
-#define SHORT_OPT_COL 2		/* column in which short options start */
-#define LONG_OPT_COL  6		/* column in which long options start */
-#define DOC_OPT_COL   2		/* column in which doc options start */
-#define OPT_DOC_COL  29		/* column in which option text starts */
-#define HEADER_COL    1		/* column in which group headers are printed */
-#define USAGE_INDENT 12		/* indentation of wrapped usage lines */
-#define RMARGIN      79		/* right margin used for wrapping */
-
-/* User-selectable (using an environment variable) formatting parameters.
-   They must all be of type `int' for the parsing code to work.  */
-struct uparams
-{
-  /* If true, arguments for an option are shown with both short and long
-     options, even when a given option has both, e.g. `-x ARG, --longx=ARG'.
-     If false, then if an option has both, the argument is only shown with
-     the long one, e.g., `-x, --longx=ARG', and a message indicating that
-     this really means both is printed below the options.  */
-  int dup_args;
-
-  /* This is true if when DUP_ARGS is false, and some duplicate arguments have
-     been suppressed, an explanatory message should be printed.  */
-  int dup_args_note;
-
-  /* Various output columns.  */
-  int short_opt_col;
-  int long_opt_col;
-  int doc_opt_col;
-  int opt_doc_col;
-  int header_col;
-  int usage_indent;
-  int rmargin;
-
-  int valid;			/* True when the values in here are valid.  */
-};
-
-/* This is a global variable, as user options are only ever read once.  */
-static struct uparams uparams = {
-  DUP_ARGS, DUP_ARGS_NOTE,
-  SHORT_OPT_COL, LONG_OPT_COL, DOC_OPT_COL, OPT_DOC_COL, HEADER_COL,
-  USAGE_INDENT, RMARGIN,
-  0
-};
-
-/* A particular uparam, and what the user name is.  */
-struct uparam_name
-{
-  const char *name;		/* User name.  */
-  int is_bool;			/* Whether it's `boolean'.  */
-  size_t uparams_offs;		/* Location of the (int) field in UPARAMS.  */
-};
-
-/* The name-field mappings we know about.  */
-static const struct uparam_name uparam_names[] =
-{
-  { "dup-args",       1, offsetof (struct uparams, dup_args) },
-  { "dup-args-note",  1, offsetof (struct uparams, dup_args_note) },
-  { "short-opt-col",  0, offsetof (struct uparams, short_opt_col) },
-  { "long-opt-col",   0, offsetof (struct uparams, long_opt_col) },
-  { "doc-opt-col",    0, offsetof (struct uparams, doc_opt_col) },
-  { "opt-doc-col",    0, offsetof (struct uparams, opt_doc_col) },
-  { "header-col",     0, offsetof (struct uparams, header_col) },
-  { "usage-indent",   0, offsetof (struct uparams, usage_indent) },
-  { "rmargin",        0, offsetof (struct uparams, rmargin) },
-  { 0, 0, 0 }
-};
-
-/* Read user options from the environment, and fill in UPARAMS appropiately.  */
-static void
-fill_in_uparams (const struct argp_state *state)
-{
-
-  const char *var = getenv ("ARGP_HELP_FMT");
-
-#define SKIPWS(p) do { while (isspace (*p)) p++; } while (0);
-
-  if (var)
-    /* Parse var. */
-    while (*var)
-      {
-	SKIPWS (var);
-
-	if (isalpha (*var))
-	  {
-	    size_t var_len;
-	    const struct uparam_name *un;
-	    int unspec = 0, val = 0;
-	    const char *arg = var;
-
-	    while (isalnum (*arg) || *arg == '-' || *arg == '_')
-	      arg++;
-	    var_len = arg - var;
-
-	    SKIPWS (arg);
-
-	    if (*arg == '\0' || *arg == ',')
-	      unspec = 1;
-	    else if (*arg == '=')
-	      {
-		arg++;
-		SKIPWS (arg);
-	      }
-
-	    if (unspec)
-	      {
-		if (var[0] == 'n' && var[1] == 'o' && var[2] == '-')
-		  {
-		    val = 0;
-		    var += 3;
-		    var_len -= 3;
-		  }
-		else
-		  val = 1;
-	      }
-	    else if (isdigit (*arg))
-	      {
-		val = atoi (arg);
-		while (isdigit (*arg))
-		  arg++;
-		SKIPWS (arg);
-	      }
-
-	    for (un = uparam_names; un->name; un++)
-	      if (strlen (un->name) == var_len
-		  && strncmp (var, un->name, var_len) == 0)
-		{
-		  if (unspec && !un->is_bool)
-		    __argp_failure (state, 0, 0,
-				    dgettext (state->root_argp->argp_domain, "\
-%.*s: ARGP_HELP_FMT parameter requires a value"),
-				    (int) var_len, var);
-		  else
-		    *(int *)((char *)&uparams + un->uparams_offs) = val;
-		  break;
-		}
-	    if (! un->name)
-	      __argp_failure (state, 0, 0,
-			      dgettext (state->root_argp->argp_domain, "\
-%.*s: Unknown ARGP_HELP_FMT parameter"),
-			      (int) var_len, var);
-
-	    var = arg;
-	    if (*var == ',')
-	      var++;
-	  }
-	else if (*var)
-	  {
-	    __argp_failure (state, 0, 0,
-			    dgettext (state->root_argp->argp_domain,
-				      "Garbage in ARGP_HELP_FMT: %s"), var);
-	    break;
-	  }
-      }
-}
-
-/* Returns true if OPT hasn't been marked invisible.  Visibility only affects
-   whether OPT is displayed or used in sorting, not option shadowing.  */
-#define ovisible(opt) (! ((opt)->flags & OPTION_HIDDEN))
-
-/* Returns true if OPT is an alias for an earlier option.  */
-#define oalias(opt) ((opt)->flags & OPTION_ALIAS)
-
-/* Returns true if OPT is an documentation-only entry.  */
-#define odoc(opt) ((opt)->flags & OPTION_DOC)
-
-/* Returns true if OPT is the end-of-list marker for a list of options.  */
-#define oend(opt) __option_is_end (opt)
-
-/* Returns true if OPT has a short option.  */
-#define oshort(opt) __option_is_short (opt)
-
-/*
-   The help format for a particular option is like:
-
-     -xARG, -yARG, --long1=ARG, --long2=ARG        Documentation...
-
-   Where ARG will be omitted if there's no argument, for this option, or
-   will be surrounded by "[" and "]" appropiately if the argument is
-   optional.  The documentation string is word-wrapped appropiately, and if
-   the list of options is long enough, it will be started on a separate line.
-   If there are no short options for a given option, the first long option is
-   indented slighly in a way that's supposed to make most long options appear
-   to be in a separate column.
-
-   For example, the following output (from ps):
-
-     -p PID, --pid=PID          List the process PID
-	 --pgrp=PGRP            List processes in the process group PGRP
-     -P, -x, --no-parent        Include processes without parents
-     -Q, --all-fields           Don't elide unusable fields (normally if there's
-				some reason ps can't print a field for any
-				process, it's removed from the output entirely)
-     -r, --reverse, --gratuitously-long-reverse-option
-				Reverse the order of any sort
-	 --session[=SID]        Add the processes from the session SID (which
-				defaults to the sid of the current process)
-
-    Here are some more options:
-     -f ZOT, --foonly=ZOT       Glork a foonly
-     -z, --zaza                 Snit a zar
-
-     -?, --help                 Give this help list
-	 --usage                Give a short usage message
-     -V, --version              Print program version
-
-   The struct argp_option array for the above could look like:
-
-   {
-     {"pid",       'p',      "PID",  0, "List the process PID"},
-     {"pgrp",      OPT_PGRP, "PGRP", 0, "List processes in the process group PGRP"},
-     {"no-parent", 'P',	      0,     0, "Include processes without parents"},
-     {0,           'x',       0,     OPTION_ALIAS},
-     {"all-fields",'Q',       0,     0, "Don't elide unusable fields (normally"
-                                        " if there's some reason ps can't"
-					" print a field for any process, it's"
-                                        " removed from the output entirely)" },
-     {"reverse",   'r',       0,     0, "Reverse the order of any sort"},
-     {"gratuitously-long-reverse-option", 0, 0, OPTION_ALIAS},
-     {"session",   OPT_SESS,  "SID", OPTION_ARG_OPTIONAL,
-                                        "Add the processes from the session"
-					" SID (which defaults to the sid of"
-					" the current process)" },
-
-     {0,0,0,0, "Here are some more options:"},
-     {"foonly", 'f', "ZOT", 0, "Glork a foonly"},
-     {"zaza", 'z', 0, 0, "Snit a zar"},
-
-     {0}
-   }
-
-   Note that the last three options are automatically supplied by argp_parse,
-   unless you tell it not to with ARGP_NO_HELP.
-
-*/
-
-/* Returns true if CH occurs between BEG and END.  */
-static int
-find_char (char ch, char *beg, char *end)
-{
-  while (beg < end)
-    if (*beg == ch)
-      return 1;
-    else
-      beg++;
-  return 0;
-}
-
-struct hol_cluster;		/* fwd decl */
-
-struct hol_entry
-{
-  /* First option.  */
-  const struct argp_option *opt;
-  /* Number of options (including aliases).  */
-  unsigned num;
-
-  /* A pointers into the HOL's short_options field, to the first short option
-     letter for this entry.  The order of the characters following this point
-     corresponds to the order of options pointed to by OPT, and there are at
-     most NUM.  A short option recorded in a option following OPT is only
-     valid if it occurs in the right place in SHORT_OPTIONS (otherwise it's
-     probably been shadowed by some other entry).  */
-  char *short_options;
-
-  /* Entries are sorted by their group first, in the order:
-       1, 2, ..., n, 0, -m, ..., -2, -1
-     and then alphabetically within each group.  The default is 0.  */
-  int group;
-
-  /* The cluster of options this entry belongs to, or 0 if none.  */
-  struct hol_cluster *cluster;
-
-  /* The argp from which this option came.  */
-  const struct argp *argp;
-};
-
-/* A cluster of entries to reflect the argp tree structure.  */
-struct hol_cluster
-{
-  /* A descriptive header printed before options in this cluster.  */
-  const char *header;
-
-  /* Used to order clusters within the same group with the same parent,
-     according to the order in which they occurred in the parent argp's child
-     list.  */
-  int index;
-
-  /* How to sort this cluster with respect to options and other clusters at the
-     same depth (clusters always follow options in the same group).  */
-  int group;
-
-  /* The cluster to which this cluster belongs, or 0 if it's at the base
-     level.  */
-  struct hol_cluster *parent;
-
-  /* The argp from which this cluster is (eventually) derived.  */
-  const struct argp *argp;
-
-  /* The distance this cluster is from the root.  */
-  int depth;
-
-  /* Clusters in a given hol are kept in a linked list, to make freeing them
-     possible.  */
-  struct hol_cluster *next;
-};
-
-/* A list of options for help.  */
-struct hol
-{
-  /* An array of hol_entry's.  */
-  struct hol_entry *entries;
-  /* The number of entries in this hol.  If this field is zero, the others
-     are undefined.  */
-  unsigned num_entries;
-
-  /* A string containing all short options in this HOL.  Each entry contains
-     pointers into this string, so the order can't be messed with blindly.  */
-  char *short_options;
-
-  /* Clusters of entries in this hol.  */
-  struct hol_cluster *clusters;
-};
-
-/* Create a struct hol from the options in ARGP.  CLUSTER is the
-   hol_cluster in which these entries occur, or 0, if at the root.  */
-static struct hol *
-make_hol (const struct argp *argp, struct hol_cluster *cluster)
-{
-  char *so;
-  const struct argp_option *o;
-  const struct argp_option *opts = argp->options;
-  struct hol_entry *entry;
-  unsigned num_short_options = 0;
-  struct hol *hol = malloc (sizeof (struct hol));
-
-  assert (hol);
-
-  hol->num_entries = 0;
-  hol->clusters = 0;
-
-  if (opts)
-    {
-      int cur_group = 0;
-
-      /* The first option must not be an alias.  */
-      assert (! oalias (opts));
-
-      /* Calculate the space needed.  */
-      for (o = opts; ! oend (o); o++)
-	{
-	  if (! oalias (o))
-	    hol->num_entries++;
-	  if (oshort (o))
-	    num_short_options++;	/* This is an upper bound.  */
-	}
-
-      hol->entries = malloc (sizeof (struct hol_entry) * hol->num_entries);
-      hol->short_options = malloc (num_short_options + 1);
-
-      assert (hol->entries && hol->short_options);
-
-      /* Fill in the entries.  */
-      so = hol->short_options;
-      for (o = opts, entry = hol->entries; ! oend (o); entry++)
-	{
-	  entry->opt = o;
-	  entry->num = 0;
-	  entry->short_options = so;
-	  entry->group = cur_group =
-	    o->group
-	    ? o->group
-	    : ((!o->name && !o->key)
-	       ? cur_group + 1
-	       : cur_group);
-	  entry->cluster = cluster;
-	  entry->argp = argp;
-
-	  do
-	    {
-	      entry->num++;
-	      if (oshort (o) && ! find_char (o->key, hol->short_options, so))
-		/* O has a valid short option which hasn't already been used.*/
-		*so++ = o->key;
-	      o++;
-	    }
-	  while (! oend (o) && oalias (o));
-	}
-      *so = '\0';		/* null terminated so we can find the length */
-    }
-
-  return hol;
-}
-
-/* Add a new cluster to HOL, with the given GROUP and HEADER (taken from the
-   associated argp child list entry), INDEX, and PARENT, and return a pointer
-   to it.  ARGP is the argp that this cluster results from.  */
-static struct hol_cluster *
-hol_add_cluster (struct hol *hol, int group, const char *header, int index,
-		 struct hol_cluster *parent, const struct argp *argp)
-{
-  struct hol_cluster *cl = malloc (sizeof (struct hol_cluster));
-  if (cl)
-    {
-      cl->group = group;
-      cl->header = header;
-
-      cl->index = index;
-      cl->parent = parent;
-      cl->argp = argp;
-      cl->depth = parent ? parent->depth + 1 : 0;
-
-      cl->next = hol->clusters;
-      hol->clusters = cl;
-    }
-  return cl;
-}
-
-/* Free HOL and any resources it uses.  */
-static void
-hol_free (struct hol *hol)
-{
-  struct hol_cluster *cl = hol->clusters;
-
-  while (cl)
-    {
-      struct hol_cluster *next = cl->next;
-      free (cl);
-      cl = next;
-    }
-
-  if (hol->num_entries > 0)
-    {
-      free (hol->entries);
-      free (hol->short_options);
-    }
-
-  free (hol);
-}
-
-static inline int
-hol_entry_short_iterate (const struct hol_entry *entry,
-			 int (*func)(const struct argp_option *opt,
-				     const struct argp_option *real,
-				     const char *domain, void *cookie),
-			 const char *domain, void *cookie)
-{
-  unsigned nopts;
-  int val = 0;
-  const struct argp_option *opt, *real = entry->opt;
-  char *so = entry->short_options;
-
-  for (opt = real, nopts = entry->num; nopts > 0 && !val; opt++, nopts--)
-    if (oshort (opt) && *so == opt->key)
-      {
-	if (!oalias (opt))
-	  real = opt;
-	if (ovisible (opt))
-	  val = (*func)(opt, real, domain, cookie);
-	so++;
-      }
-
-  return val;
-}
-
-static inline int
-hol_entry_long_iterate (const struct hol_entry *entry,
-			int (*func)(const struct argp_option *opt,
-				    const struct argp_option *real,
-				    const char *domain, void *cookie),
-			const char *domain, void *cookie)
-{
-  unsigned nopts;
-  int val = 0;
-  const struct argp_option *opt, *real = entry->opt;
-
-  for (opt = real, nopts = entry->num; nopts > 0 && !val; opt++, nopts--)
-    if (opt->name)
-      {
-	if (!oalias (opt))
-	  real = opt;
-	if (ovisible (opt))
-	  val = (*func)(opt, real, domain, cookie);
-      }
-
-  return val;
-}
-
-/* Iterator that returns true for the first short option.  */
-static inline int
-until_short (const struct argp_option *opt, const struct argp_option *real UNUSED,
-	     const char *domain UNUSED, void *cookie UNUSED)
-{
-  return oshort (opt) ? opt->key : 0;
-}
-
-/* Returns the first valid short option in ENTRY, or 0 if there is none.  */
-static char
-hol_entry_first_short (const struct hol_entry *entry)
-{
-  return hol_entry_short_iterate (entry, until_short,
-				  entry->argp->argp_domain, 0);
-}
-
-/* Returns the first valid long option in ENTRY, or 0 if there is none.  */
-static const char *
-hol_entry_first_long (const struct hol_entry *entry)
-{
-  const struct argp_option *opt;
-  unsigned num;
-  for (opt = entry->opt, num = entry->num; num > 0; opt++, num--)
-    if (opt->name && ovisible (opt))
-      return opt->name;
-  return 0;
-}
-
-/* Returns the entry in HOL with the long option name NAME, or 0 if there is
-   none.  */
-static struct hol_entry *
-hol_find_entry (struct hol *hol, const char *name)
-{
-  struct hol_entry *entry = hol->entries;
-  unsigned num_entries = hol->num_entries;
-
-  while (num_entries-- > 0)
-    {
-      const struct argp_option *opt = entry->opt;
-      unsigned num_opts = entry->num;
-
-      while (num_opts-- > 0)
-	if (opt->name && ovisible (opt) && strcmp (opt->name, name) == 0)
-	  return entry;
-	else
-	  opt++;
-
-      entry++;
-    }
-
-  return 0;
-}
-
-/* If an entry with the long option NAME occurs in HOL, set it's special
-   sort position to GROUP.  */
-static void
-hol_set_group (struct hol *hol, const char *name, int group)
-{
-  struct hol_entry *entry = hol_find_entry (hol, name);
-  if (entry)
-    entry->group = group;
-}
-
-/* Order by group:  0, 1, 2, ..., n, -m, ..., -2, -1.
-   EQ is what to return if GROUP1 and GROUP2 are the same.  */
-static int
-group_cmp (int group1, int group2, int eq)
-{
-  if (group1 == group2)
-    return eq;
-  else if ((group1 < 0 && group2 < 0) || (group1 >= 0 && group2 >= 0))
-    return group1 - group2;
-  else
-    return group2 - group1;
-}
-
-/* Compare clusters CL1 & CL2 by the order that they should appear in
-   output.  */
-static int
-hol_cluster_cmp (const struct hol_cluster *cl1, const struct hol_cluster *cl2)
-{
-  /* If one cluster is deeper than the other, use its ancestor at the same
-     level, so that finding the common ancestor is straightforward.  */
-  while (cl1->depth < cl2->depth)
-    cl1 = cl1->parent;
-  while (cl2->depth < cl1->depth)
-    cl2 = cl2->parent;
-
-  /* Now reduce both clusters to their ancestors at the point where both have
-     a common parent; these can be directly compared.  */
-  while (cl1->parent != cl2->parent)
-    cl1 = cl1->parent, cl2 = cl2->parent;
-
-  return group_cmp (cl1->group, cl2->group, cl2->index - cl1->index);
-}
-
-/* Return the ancestor of CL that's just below the root (i.e., has a parent
-   of 0).  */
-static struct hol_cluster *
-hol_cluster_base (struct hol_cluster *cl)
-{
-  while (cl->parent)
-    cl = cl->parent;
-  return cl;
-}
-
-/* Return true if CL1 is a child of CL2.  */
-static int
-hol_cluster_is_child (const struct hol_cluster *cl1,
-		      const struct hol_cluster *cl2)
-{
-  while (cl1 && cl1 != cl2)
-    cl1 = cl1->parent;
-  return cl1 == cl2;
-}
-
-/* Given the name of a OPTION_DOC option, modifies NAME to start at the tail
-   that should be used for comparisons, and returns true iff it should be
-   treated as a non-option.  */
-
-/* FIXME: Can we use unsigned char * for the argument? */
-static int
-canon_doc_option (const char **name)
-{
-  int non_opt;
-  /* Skip initial whitespace.  */
-  while (isspace ( (unsigned char) **name))
-    (*name)++;
-  /* Decide whether this looks like an option (leading `-') or not.  */
-  non_opt = (**name != '-');
-  /* Skip until part of name used for sorting.  */
-  while (**name && !isalnum ( (unsigned char) **name))
-    (*name)++;
-  return non_opt;
-}
-
-/* Order ENTRY1 & ENTRY2 by the order which they should appear in a help
-   listing.  */
-static int
-hol_entry_cmp (const struct hol_entry *entry1,
-	       const struct hol_entry *entry2)
-{
-  /* The group numbers by which the entries should be ordered; if either is
-     in a cluster, then this is just the group within the cluster.  */
-  int group1 = entry1->group, group2 = entry2->group;
-
-  if (entry1->cluster != entry2->cluster)
-    {
-      /* The entries are not within the same cluster, so we can't compare them
-	 directly, we have to use the appropiate clustering level too.  */
-      if (! entry1->cluster)
-	/* ENTRY1 is at the `base level', not in a cluster, so we have to
-	   compare it's group number with that of the base cluster in which
-	   ENTRY2 resides.  Note that if they're in the same group, the
-	   clustered option always comes laster.  */
-	return group_cmp (group1, hol_cluster_base (entry2->cluster)->group, -1);
-      else if (! entry2->cluster)
-	/* Likewise, but ENTRY2's not in a cluster.  */
-	return group_cmp (hol_cluster_base (entry1->cluster)->group, group2, 1);
-      else
-	/* Both entries are in clusters, we can just compare the clusters.  */
-	return hol_cluster_cmp (entry1->cluster, entry2->cluster);
-    }
-  else if (group1 == group2)
-    /* The entries are both in the same cluster and group, so compare them
-       alphabetically.  */
-    {
-      int short1 = hol_entry_first_short (entry1);
-      int short2 = hol_entry_first_short (entry2);
-      int doc1 = odoc (entry1->opt);
-      int doc2 = odoc (entry2->opt);
-      /* FIXME: Can we use unsigned char * instead? */
-      const char *long1 = hol_entry_first_long (entry1);
-      const char *long2 = hol_entry_first_long (entry2);
-
-      if (doc1)
-	doc1 = canon_doc_option (&long1);
-      if (doc2)
-	doc2 = canon_doc_option (&long2);
-
-      if (doc1 != doc2)
-	/* `documentation' options always follow normal options (or
-	   documentation options that *look* like normal options).  */
-	return doc1 - doc2;
-      else if (!short1 && !short2 && long1 && long2)
-	/* Only long options.  */
-	return __strcasecmp (long1, long2);
-      else
-	/* Compare short/short, long/short, short/long, using the first
-	   character of long options.  Entries without *any* valid
-	   options (such as options with OPTION_HIDDEN set) will be put
-	   first, but as they're not displayed, it doesn't matter where
-	   they are.  */
-	{
-	  unsigned char first1 = short1 ? short1 : long1 ? *long1 : 0;
-	  unsigned char first2 = short2 ? short2 : long2 ? *long2 : 0;
-#ifdef _tolower
-	  int lower_cmp = _tolower (first1) - _tolower (first2);
-#else
-	  int lower_cmp = tolower (first1) - tolower (first2);
-#endif
-	  /* Compare ignoring case, except when the options are both the
-	     same letter, in which case lower-case always comes first.  */
-	  /* NOTE: The subtraction below does the right thing
-	     even with eight-bit chars: first1 and first2 are
-	     converted to int *before* the subtraction. */
-	  return lower_cmp ? lower_cmp : first2 - first1;
-	}
-    }
-  else
-    /* Within the same cluster, but not the same group, so just compare
-       groups.  */
-    return group_cmp (group1, group2, 0);
-}
-
-/* Version of hol_entry_cmp with correct signature for qsort.  */
-static int
-hol_entry_qcmp (const void *entry1_v, const void *entry2_v)
-{
-  return hol_entry_cmp (entry1_v, entry2_v);
-}
-
-/* Sort HOL by group and alphabetically by option name (with short options
-   taking precedence over long).  Since the sorting is for display purposes
-   only, the shadowing of options isn't effected.  */
-static void
-hol_sort (struct hol *hol)
-{
-  if (hol->num_entries > 0)
-    qsort (hol->entries, hol->num_entries, sizeof (struct hol_entry),
-	   hol_entry_qcmp);
-}
-
-/* Append MORE to HOL, destroying MORE in the process.  Options in HOL shadow
-   any in MORE with the same name.  */
-static void
-hol_append (struct hol *hol, struct hol *more)
-{
-  struct hol_cluster **cl_end = &hol->clusters;
-
-  /* Steal MORE's cluster list, and add it to the end of HOL's.  */
-  while (*cl_end)
-    cl_end = &(*cl_end)->next;
-  *cl_end = more->clusters;
-  more->clusters = 0;
-
-  /* Merge entries.  */
-  if (more->num_entries > 0)
-    {
-      if (hol->num_entries == 0)
-	{
-	  hol->num_entries = more->num_entries;
-	  hol->entries = more->entries;
-	  hol->short_options = more->short_options;
-	  more->num_entries = 0;	/* Mark MORE's fields as invalid.  */
-	}
-      else
-	/* Append the entries in MORE to those in HOL, taking care to only add
-	   non-shadowed SHORT_OPTIONS values.  */
-	{
-	  unsigned left;
-	  char *so, *more_so;
-	  struct hol_entry *e;
-	  unsigned num_entries = hol->num_entries + more->num_entries;
-	  struct hol_entry *entries =
-	    malloc (num_entries * sizeof (struct hol_entry));
-	  unsigned hol_so_len = strlen (hol->short_options);
-	  char *short_options =
-	    malloc (hol_so_len + strlen (more->short_options) + 1);
-
-	  __mempcpy (__mempcpy (entries, hol->entries,
-				hol->num_entries * sizeof (struct hol_entry)),
-		     more->entries,
-		     more->num_entries * sizeof (struct hol_entry));
-
-	  __mempcpy (short_options, hol->short_options, hol_so_len);
-
-	  /* Fix up the short options pointers from HOL.  */
-	  for (e = entries, left = hol->num_entries; left > 0; e++, left--)
-	    e->short_options += (short_options - hol->short_options);
-
-	  /* Now add the short options from MORE, fixing up its entries
-	     too.  */
-	  so = short_options + hol_so_len;
-	  more_so = more->short_options;
-	  for (left = more->num_entries; left > 0; e++, left--)
-	    {
-	      int opts_left;
-	      const struct argp_option *opt;
-
-	      e->short_options = so;
-
-	      for (opts_left = e->num, opt = e->opt; opts_left; opt++, opts_left--)
-		{
-		  int ch = *more_so;
-		  if (oshort (opt) && ch == opt->key)
-		    /* The next short option in MORE_SO, CH, is from OPT.  */
-		    {
-		      if (! find_char (ch, short_options,
-				       short_options + hol_so_len))
-			/* The short option CH isn't shadowed by HOL's options,
-			   so add it to the sum.  */
-			*so++ = ch;
-		      more_so++;
-		    }
-		}
-	    }
-
-	  *so = '\0';
-
-	  free (hol->entries);
-	  free (hol->short_options);
-
-	  hol->entries = entries;
-	  hol->num_entries = num_entries;
-	  hol->short_options = short_options;
-	}
-    }
-
-  hol_free (more);
-}
-
-/* Inserts enough spaces to make sure STREAM is at column COL.  */
-static void
-indent_to (argp_fmtstream_t stream, unsigned col)
-{
-  int needed = col - __argp_fmtstream_point (stream);
-  while (needed-- > 0)
-    __argp_fmtstream_putc (stream, ' ');
-}
-
-/* Output to STREAM either a space, or a newline if there isn't room for at
-   least ENSURE characters before the right margin.  */
-static void
-space (argp_fmtstream_t stream, size_t ensure)
-{
-  if (__argp_fmtstream_point (stream) + ensure
-      >= __argp_fmtstream_rmargin (stream))
-    __argp_fmtstream_putc (stream, '\n');
-  else
-    __argp_fmtstream_putc (stream, ' ');
-}
-
-/* If the option REAL has an argument, we print it in using the printf
-   format REQ_FMT or OPT_FMT depending on whether it's a required or
-   optional argument.  */
-static void
-arg (const struct argp_option *real, const char *req_fmt, const char *opt_fmt,
-     const char *domain UNUSED, argp_fmtstream_t stream)
-{
-  if (real->arg)
-    {
-      if (real->flags & OPTION_ARG_OPTIONAL)
-	__argp_fmtstream_printf (stream, opt_fmt,
-				 dgettext (domain, real->arg));
-      else
-	__argp_fmtstream_printf (stream, req_fmt,
-				 dgettext (domain, real->arg));
-    }
-}
-
-/* Helper functions for hol_entry_help.  */
-
-/* State used during the execution of hol_help.  */
-struct hol_help_state
-{
-  /* PREV_ENTRY should contain the previous entry printed, or 0.  */
-  struct hol_entry *prev_entry;
-
-  /* If an entry is in a different group from the previous one, and SEP_GROUPS
-     is true, then a blank line will be printed before any output. */
-  int sep_groups;
-
-  /* True if a duplicate option argument was suppressed (only ever set if
-     UPARAMS.dup_args is false).  */
-  int suppressed_dup_arg;
-};
-
-/* Some state used while printing a help entry (used to communicate with
-   helper functions).  See the doc for hol_entry_help for more info, as most
-   of the fields are copied from its arguments.  */
-struct pentry_state
-{
-  const struct hol_entry *entry;
-  argp_fmtstream_t stream;
-  struct hol_help_state *hhstate;
-
-  /* True if nothing's been printed so far.  */
-  int first;
-
-  /* If non-zero, the state that was used to print this help.  */
-  const struct argp_state *state;
-};
-
-/* If a user doc filter should be applied to DOC, do so.  */
-static const char *
-filter_doc (const char *doc, int key, const struct argp *argp,
-	    const struct argp_state *state)
-{
-  if (argp->help_filter)
-    /* We must apply a user filter to this output.  */
-    {
-      void *input = __argp_input (argp, state);
-      return (*argp->help_filter) (key, doc, input);
-    }
-  else
-    /* No filter.  */
-    return doc;
-}
-
-/* Prints STR as a header line, with the margin lines set appropiately, and
-   notes the fact that groups should be separated with a blank line.  ARGP is
-   the argp that should dictate any user doc filtering to take place.  Note
-   that the previous wrap margin isn't restored, but the left margin is reset
-   to 0.  */
-static void
-print_header (const char *str, const struct argp *argp,
-	      struct pentry_state *pest)
-{
-  const char *tstr = dgettext (argp->argp_domain, str);
-  const char *fstr = filter_doc (tstr, ARGP_KEY_HELP_HEADER, argp, pest->state);
-
-  if (fstr)
-    {
-      if (*fstr)
-	{
-	  if (pest->hhstate->prev_entry)
-	    /* Precede with a blank line.  */
-	    __argp_fmtstream_putc (pest->stream, '\n');
-	  indent_to (pest->stream, uparams.header_col);
-	  __argp_fmtstream_set_lmargin (pest->stream, uparams.header_col);
-	  __argp_fmtstream_set_wmargin (pest->stream, uparams.header_col);
-	  __argp_fmtstream_puts (pest->stream, fstr);
-	  __argp_fmtstream_set_lmargin (pest->stream, 0);
-	  __argp_fmtstream_putc (pest->stream, '\n');
-	}
-
-      pest->hhstate->sep_groups = 1; /* Separate subsequent groups. */
-    }
-
-  if (fstr != tstr)
-    free ((char *) fstr);
-}
-
-/* Inserts a comma if this isn't the first item on the line, and then makes
-   sure we're at least to column COL.  If this *is* the first item on a line,
-   prints any pending whitespace/headers that should precede this line. Also
-   clears FIRST.  */
-static void
-comma (unsigned col, struct pentry_state *pest)
-{
-  if (pest->first)
-    {
-      const struct hol_entry *pe = pest->hhstate->prev_entry;
-      const struct hol_cluster *cl = pest->entry->cluster;
-
-      if (pest->hhstate->sep_groups && pe && pest->entry->group != pe->group)
-	__argp_fmtstream_putc (pest->stream, '\n');
-
-      if (cl && cl->header && *cl->header
-	  && (!pe
-	      || (pe->cluster != cl
-		  && !hol_cluster_is_child (pe->cluster, cl))))
-	/* If we're changing clusters, then this must be the start of the
-	   ENTRY's cluster unless that is an ancestor of the previous one
-	   (in which case we had just popped into a sub-cluster for a bit).
-	   If so, then print the cluster's header line.  */
-	{
-	  int old_wm = __argp_fmtstream_wmargin (pest->stream);
-	  print_header (cl->header, cl->argp, pest);
-	  __argp_fmtstream_set_wmargin (pest->stream, old_wm);
-	}
-
-      pest->first = 0;
-    }
-  else
-    __argp_fmtstream_puts (pest->stream, ", ");
-
-  indent_to (pest->stream, col);
-}
-
-/* Print help for ENTRY to STREAM.  */
-static void
-hol_entry_help (struct hol_entry *entry, const struct argp_state *state,
-		argp_fmtstream_t stream, struct hol_help_state *hhstate)
-{
-  unsigned num;
-  const struct argp_option *real = entry->opt, *opt;
-  char *so = entry->short_options;
-  int have_long_opt = 0;	/* We have any long options.  */
-  /* Saved margins.  */
-  int old_lm = __argp_fmtstream_set_lmargin (stream, 0);
-  int old_wm = __argp_fmtstream_wmargin (stream);
-  /* PEST is a state block holding some of our variables that we'd like to
-     share with helper functions.  */
-
-  /* Decent initializers are a GNU extension, so don't use it here. */
-  struct pentry_state pest;
-  pest.entry = entry;
-  pest.stream = stream;
-  pest.hhstate = hhstate;
-  pest.first = 1;
-  pest.state = state;
-
-  if (! odoc (real))
-    for (opt = real, num = entry->num; num > 0; opt++, num--)
-      if (opt->name && ovisible (opt))
-	{
-	  have_long_opt = 1;
-	  break;
-	}
-
-  /* First emit short options.  */
-  __argp_fmtstream_set_wmargin (stream, uparams.short_opt_col); /* For truly bizarre cases. */
-  for (opt = real, num = entry->num; num > 0; opt++, num--)
-    if (oshort (opt) && opt->key == *so)
-      /* OPT has a valid (non shadowed) short option.  */
-      {
-	if (ovisible (opt))
-	  {
-	    comma (uparams.short_opt_col, &pest);
-	    __argp_fmtstream_putc (stream, '-');
-	    __argp_fmtstream_putc (stream, *so);
-	    if (!have_long_opt || uparams.dup_args)
-	      arg (real, " %s", "[%s]", state->root_argp->argp_domain, stream);
-	    else if (real->arg)
-	      hhstate->suppressed_dup_arg = 1;
-	  }
-	so++;
-      }
-
-  /* Now, long options.  */
-  if (odoc (real))
-    /* A `documentation' option.  */
-    {
-      __argp_fmtstream_set_wmargin (stream, uparams.doc_opt_col);
-      for (opt = real, num = entry->num; num > 0; opt++, num--)
-	if (opt->name && ovisible (opt))
-	  {
-	    comma (uparams.doc_opt_col, &pest);
-	    /* Calling gettext here isn't quite right, since sorting will
-	       have been done on the original; but documentation options
-	       should be pretty rare anyway...  */
-	    __argp_fmtstream_puts (stream,
-				   dgettext (state->root_argp->argp_domain,
-					     opt->name));
-	  }
-    }
-  else
-    /* A real long option.  */
-    {
-      int first_long_opt = 1;
-
-      __argp_fmtstream_set_wmargin (stream, uparams.long_opt_col);
-      for (opt = real, num = entry->num; num > 0; opt++, num--)
-	if (opt->name && ovisible (opt))
-	  {
-	    comma (uparams.long_opt_col, &pest);
-	    __argp_fmtstream_printf (stream, "--%s", opt->name);
-	    if (first_long_opt || uparams.dup_args)
-	      arg (real, "=%s", "[=%s]", state->root_argp->argp_domain,
-		   stream);
-	    else if (real->arg)
-	      hhstate->suppressed_dup_arg = 1;
-	  }
-    }
-
-  /* Next, documentation strings.  */
-  __argp_fmtstream_set_lmargin (stream, 0);
-
-  if (pest.first)
-    {
-      /* Didn't print any switches, what's up?  */
-      if (!oshort (real) && !real->name)
-	/* This is a group header, print it nicely.  */
-	print_header (real->doc, entry->argp, &pest);
-      else
-	/* Just a totally shadowed option or null header; print nothing.  */
-	goto cleanup;		/* Just return, after cleaning up.  */
-    }
-  else
-    {
-      const char *tstr = real->doc ? dgettext (state->root_argp->argp_domain,
-					       real->doc) : 0;
-      const char *fstr = filter_doc (tstr, real->key, entry->argp, state);
-      if (fstr && *fstr)
-	{
-	  unsigned int col = __argp_fmtstream_point (stream);
-
-	  __argp_fmtstream_set_lmargin (stream, uparams.opt_doc_col);
-	  __argp_fmtstream_set_wmargin (stream, uparams.opt_doc_col);
-
-	  if (col > (unsigned int) (uparams.opt_doc_col + 3))
-	    __argp_fmtstream_putc (stream, '\n');
-	  else if (col >= (unsigned int) uparams.opt_doc_col)
-	    __argp_fmtstream_puts (stream, "   ");
-	  else
-	    indent_to (stream, uparams.opt_doc_col);
-
-	  __argp_fmtstream_puts (stream, fstr);
-	}
-      if (fstr && fstr != tstr)
-	free ((char *) fstr);
-
-      /* Reset the left margin.  */
-      __argp_fmtstream_set_lmargin (stream, 0);
-      __argp_fmtstream_putc (stream, '\n');
-    }
-
-  hhstate->prev_entry = entry;
-
-cleanup:
-  __argp_fmtstream_set_lmargin (stream, old_lm);
-  __argp_fmtstream_set_wmargin (stream, old_wm);
-}
-
-/* Output a long help message about the options in HOL to STREAM.  */
-static void
-hol_help (struct hol *hol, const struct argp_state *state,
-	  argp_fmtstream_t stream)
-{
-  unsigned num;
-  struct hol_entry *entry;
-  struct hol_help_state hhstate = { 0, 0, 0 };
-
-  for (entry = hol->entries, num = hol->num_entries; num > 0; entry++, num--)
-    hol_entry_help (entry, state, stream, &hhstate);
-
-  if (hhstate.suppressed_dup_arg && uparams.dup_args_note)
-    {
-      const char *tstr = dgettext (state->root_argp->argp_domain, "\
-Mandatory or optional arguments to long options are also mandatory or \
-optional for any corresponding short options.");
-      const char *fstr = filter_doc (tstr, ARGP_KEY_HELP_DUP_ARGS_NOTE,
-				     state ? state->root_argp : 0, state);
-      if (fstr && *fstr)
-	{
-	  __argp_fmtstream_putc (stream, '\n');
-	  __argp_fmtstream_puts (stream, fstr);
-	  __argp_fmtstream_putc (stream, '\n');
-	}
-      if (fstr && fstr != tstr)
-	free ((char *) fstr);
-    }
-}
-
-/* Helper functions for hol_usage.  */
-
-/* If OPT is a short option without an arg, append its key to the string
-   pointer pointer to by COOKIE, and advance the pointer.  */
-static int
-add_argless_short_opt (const struct argp_option *opt,
-		       const struct argp_option *real,
-		       const char *domain UNUSED, void *cookie)
-{
-  char **snao_end = cookie;
-  if (!(opt->arg || real->arg)
-      && !((opt->flags | real->flags) & OPTION_NO_USAGE))
-    *(*snao_end)++ = opt->key;
-  return 0;
-}
-
-/* If OPT is a short option with an arg, output a usage entry for it to the
-   stream pointed at by COOKIE.  */
-static int
-usage_argful_short_opt (const struct argp_option *opt,
-			const struct argp_option *real,
-			const char *domain UNUSED, void *cookie)
-{
-  argp_fmtstream_t stream = cookie;
-  const char *arg = opt->arg;
-  int flags = opt->flags | real->flags;
-
-  if (! arg)
-    arg = real->arg;
-
-  if (arg && !(flags & OPTION_NO_USAGE))
-    {
-      arg = dgettext (domain, arg);
-
-      if (flags & OPTION_ARG_OPTIONAL)
-	__argp_fmtstream_printf (stream, " [-%c[%s]]", opt->key, arg);
-      else
-	{
-	  /* Manually do line wrapping so that it (probably) won't
-	     get wrapped at the embedded space.  */
-	  space (stream, 6 + strlen (arg));
-	  __argp_fmtstream_printf (stream, "[-%c %s]", opt->key, arg);
-	}
-    }
-
-  return 0;
-}
-
-/* Output a usage entry for the long option opt to the stream pointed at by
-   COOKIE.  */
-static int
-usage_long_opt (const struct argp_option *opt,
-		const struct argp_option *real,
-		const char *domain UNUSED, void *cookie)
-{
-  argp_fmtstream_t stream = cookie;
-  const char *arg = opt->arg;
-  int flags = opt->flags | real->flags;
-
-  if (! arg)
-    arg = real->arg;
-
-  if (! (flags & OPTION_NO_USAGE))
-    {
-      if (arg)
-	{
-	  arg = dgettext (domain, arg);
-	  if (flags & OPTION_ARG_OPTIONAL)
-	    __argp_fmtstream_printf (stream, " [--%s[=%s]]", opt->name, arg);
-	  else
-	    __argp_fmtstream_printf (stream, " [--%s=%s]", opt->name, arg);
-	}
-      else
-	__argp_fmtstream_printf (stream, " [--%s]", opt->name);
-    }
-
-  return 0;
-}
-
-/* Print a short usage description for the arguments in HOL to STREAM.  */
-static void
-hol_usage (struct hol *hol, argp_fmtstream_t stream)
-{
-  if (hol->num_entries > 0)
-    {
-      unsigned nentries;
-      struct hol_entry *entry;
-      char *short_no_arg_opts = alloca (strlen (hol->short_options) + 1);
-      char *snao_end = short_no_arg_opts;
-
-      /* First we put a list of short options without arguments.  */
-      for (entry = hol->entries, nentries = hol->num_entries
-	   ; nentries > 0
-	   ; entry++, nentries--)
-	hol_entry_short_iterate (entry, add_argless_short_opt,
-				 entry->argp->argp_domain, &snao_end);
-      if (snao_end > short_no_arg_opts)
-	{
-	  *snao_end++ = 0;
-	  __argp_fmtstream_printf (stream, " [-%s]", short_no_arg_opts);
-	}
-
-      /* Now a list of short options *with* arguments.  */
-      for (entry = hol->entries, nentries = hol->num_entries
-	   ; nentries > 0
-	   ; entry++, nentries--)
-	hol_entry_short_iterate (entry, usage_argful_short_opt,
-				 entry->argp->argp_domain, stream);
-
-      /* Finally, a list of long options (whew!).  */
-      for (entry = hol->entries, nentries = hol->num_entries
-	   ; nentries > 0
-	   ; entry++, nentries--)
-	hol_entry_long_iterate (entry, usage_long_opt,
-				entry->argp->argp_domain, stream);
-    }
-}
-
-/* Make a HOL containing all levels of options in ARGP.  CLUSTER is the
-   cluster in which ARGP's entries should be clustered, or 0.  */
-static struct hol *
-argp_hol (const struct argp *argp, struct hol_cluster *cluster)
-{
-  const struct argp_child *child = argp->children;
-  struct hol *hol = make_hol (argp, cluster);
-  if (child)
-    while (child->argp)
-      {
-	struct hol_cluster *child_cluster =
-	  ((child->group || child->header)
-	   /* Put CHILD->argp within its own cluster.  */
-	   ? hol_add_cluster (hol, child->group, child->header,
-			      child - argp->children, cluster, argp)
-	   /* Just merge it into the parent's cluster.  */
-	   : cluster);
-	hol_append (hol, argp_hol (child->argp, child_cluster)) ;
-	child++;
-      }
-  return hol;
-}
-
-/* Calculate how many different levels with alternative args strings exist in
-   ARGP.  */
-static size_t
-argp_args_levels (const struct argp *argp)
-{
-  size_t levels = 0;
-  const struct argp_child *child = argp->children;
-
-  if (argp->args_doc && strchr (argp->args_doc, '\n'))
-    levels++;
-
-  if (child)
-    while (child->argp)
-      levels += argp_args_levels ((child++)->argp);
-
-  return levels;
-}
-
-/* Print all the non-option args documented in ARGP to STREAM.  Any output is
-   preceded by a space.  LEVELS is a pointer to a byte vector the length
-   returned by argp_args_levels; it should be initialized to zero, and
-   updated by this routine for the next call if ADVANCE is true.  True is
-   returned as long as there are more patterns to output.  */
-static int
-argp_args_usage (const struct argp *argp, const struct argp_state *state,
-		 char **levels, int advance, argp_fmtstream_t stream)
-{
-  char *our_level = *levels;
-  int multiple = 0;
-  const struct argp_child *child = argp->children;
-  const char *tdoc = dgettext (argp->argp_domain, argp->args_doc), *nl = 0;
-  const char *fdoc = filter_doc (tdoc, ARGP_KEY_HELP_ARGS_DOC, argp, state);
-
-  if (fdoc)
-    {
-      const char *cp = fdoc;
-      nl = __strchrnul (cp, '\n');
-      if (*nl != '\0')
-	/* This is a `multi-level' args doc; advance to the correct position
-	   as determined by our state in LEVELS, and update LEVELS.  */
-	{
-	  int i;
-	  multiple = 1;
-	  for (i = 0; i < *our_level; i++)
-	    cp = nl + 1, nl = __strchrnul (cp, '\n');
-	  (*levels)++;
-	}
-
-      /* Manually do line wrapping so that it (probably) won't get wrapped at
-	 any embedded spaces.  */
-      space (stream, 1 + nl - cp);
-
-      __argp_fmtstream_write (stream, cp, nl - cp);
-    }
-  if (fdoc && fdoc != tdoc)
-    free ((char *)fdoc);	/* Free user's modified doc string.  */
-
-  if (child)
-    while (child->argp)
-      advance = !argp_args_usage ((child++)->argp, state, levels, advance, stream);
-
-  if (advance && multiple)
-    {
-      /* Need to increment our level.  */
-      if (*nl)
-	/* There's more we can do here.  */
-	{
-	  (*our_level)++;
-	  advance = 0;		/* Our parent shouldn't advance also. */
-	}
-      else if (*our_level > 0)
-	/* We had multiple levels, but used them up; reset to zero.  */
-	*our_level = 0;
-    }
-
-  return !advance;
-}
-
-/* Print the documentation for ARGP to STREAM; if POST is false, then
-   everything preceeding a `\v' character in the documentation strings (or
-   the whole string, for those with none) is printed, otherwise, everything
-   following the `\v' character (nothing for strings without).  Each separate
-   bit of documentation is separated a blank line, and if PRE_BLANK is true,
-   then the first is as well.  If FIRST_ONLY is true, only the first
-   occurrence is output.  Returns true if anything was output.  */
-static int
-argp_doc (const struct argp *argp, const struct argp_state *state,
-	  int post, int pre_blank, int first_only,
-	  argp_fmtstream_t stream)
-{
-  const char *text;
-  const char *inp_text;
-  void *input = 0;
-  int anything = 0;
-  size_t inp_text_limit = 0;
-  const char *doc = dgettext (argp->argp_domain, argp->doc);
-  const struct argp_child *child = argp->children;
-
-  if (doc)
-    {
-      char *vt = strchr (doc, '\v');
-      inp_text = post ? (vt ? vt + 1 : 0) : doc;
-      inp_text_limit = (!post && vt) ? (vt - doc) : 0;
-    }
-  else
-    inp_text = 0;
-
-  if (argp->help_filter)
-    /* We have to filter the doc strings.  */
-    {
-      if (inp_text_limit)
-	/* Copy INP_TEXT so that it's nul-terminated.  */
-	inp_text = STRNDUP (inp_text, inp_text_limit);
-      input = __argp_input (argp, state);
-      text =
-	(*argp->help_filter) (post
-			      ? ARGP_KEY_HELP_POST_DOC
-			      : ARGP_KEY_HELP_PRE_DOC,
-			      inp_text, input);
-    }
-  else
-    text = (const char *) inp_text;
-
-  if (text)
-    {
-      if (pre_blank)
-	__argp_fmtstream_putc (stream, '\n');
-
-      if (text == inp_text && inp_text_limit)
-	__argp_fmtstream_write (stream, inp_text, inp_text_limit);
-      else
-	__argp_fmtstream_puts (stream, text);
-
-      if (__argp_fmtstream_point (stream) > __argp_fmtstream_lmargin (stream))
-	__argp_fmtstream_putc (stream, '\n');
-
-      anything = 1;
-    }
-
-  if (text && text != inp_text)
-    free ((char *) text);	/* Free TEXT returned from the help filter.  */
-  if (inp_text && inp_text_limit && argp->help_filter)
-    free ((char *) inp_text);	/* We copied INP_TEXT, so free it now.  */
-
-  if (post && argp->help_filter)
-    /* Now see if we have to output a ARGP_KEY_HELP_EXTRA text.  */
-    {
-      text = (*argp->help_filter) (ARGP_KEY_HELP_EXTRA, 0, input);
-      if (text)
-	{
-	  if (anything || pre_blank)
-	    __argp_fmtstream_putc (stream, '\n');
-	  __argp_fmtstream_puts (stream, text);
-	  free ((char *) text);
-	  if (__argp_fmtstream_point (stream)
-	      > __argp_fmtstream_lmargin (stream))
-	    __argp_fmtstream_putc (stream, '\n');
-	  anything = 1;
-	}
-    }
-
-  if (child)
-    while (child->argp && !(first_only && anything))
-      anything |=
-	argp_doc ((child++)->argp, state,
-		  post, anything || pre_blank, first_only,
-		  stream);
-
-  return anything;
-}
-
-/* Output a usage message for ARGP to STREAM.  If called from
-   argp_state_help, STATE is the relevent parsing state.  FLAGS are from the
-   set ARGP_HELP_*.  NAME is what to use wherever a `program name' is
-   needed. */
-
-static void
-_help (const struct argp *argp, const struct argp_state *state, FILE *stream,
-       unsigned flags, const char *name)
-{
-  int anything = 0;		/* Whether we've output anything.  */
-  struct hol *hol = 0;
-  argp_fmtstream_t fs;
-
-  if (! stream)
-    return;
-
-  FLOCKFILE (stream);
-
-  if (! uparams.valid)
-    fill_in_uparams (state);
-
-  fs = __argp_make_fmtstream (stream, 0, uparams.rmargin, 0);
-  if (! fs)
-    {
-      FUNLOCKFILE (stream);
-      return;
-    }
-
-  if (flags & (ARGP_HELP_USAGE | ARGP_HELP_SHORT_USAGE | ARGP_HELP_LONG))
-    {
-      hol = argp_hol (argp, 0);
-
-      /* If present, these options always come last.  */
-      hol_set_group (hol, "help", -1);
-      hol_set_group (hol, "version", -1);
-
-      hol_sort (hol);
-    }
-
-  if (flags & (ARGP_HELP_USAGE | ARGP_HELP_SHORT_USAGE))
-    /* Print a short `Usage:' message.  */
-    {
-      int first_pattern = 1, more_patterns;
-      size_t num_pattern_levels = argp_args_levels (argp);
-      char *pattern_levels = alloca (num_pattern_levels);
-
-      memset (pattern_levels, 0, num_pattern_levels);
-
-      do
-	{
-	  int old_lm;
-	  int old_wm = __argp_fmtstream_set_wmargin (fs, uparams.usage_indent);
-	  char *levels = pattern_levels;
-
-	  if (first_pattern)
-	    __argp_fmtstream_printf (fs, "%s %s",
-				     dgettext (argp->argp_domain, "Usage:"),
-				     name);
-	  else
-	    __argp_fmtstream_printf (fs, "%s %s",
-				     dgettext (argp->argp_domain, "  or: "),
-				     name);
-
-	  /* We set the lmargin as well as the wmargin, because hol_usage
-	     manually wraps options with newline to avoid annoying breaks.  */
-	  old_lm = __argp_fmtstream_set_lmargin (fs, uparams.usage_indent);
-
-	  if (flags & ARGP_HELP_SHORT_USAGE)
-	    /* Just show where the options go.  */
-	    {
-	      if (hol->num_entries > 0)
-		__argp_fmtstream_puts (fs, dgettext (argp->argp_domain,
-						     " [OPTION...]"));
-	    }
-	  else
-	    /* Actually print the options.  */
-	    {
-	      hol_usage (hol, fs);
-	      flags |= ARGP_HELP_SHORT_USAGE; /* But only do so once.  */
-	    }
-
-	  more_patterns = argp_args_usage (argp, state, &levels, 1, fs);
-
-	  __argp_fmtstream_set_wmargin (fs, old_wm);
-	  __argp_fmtstream_set_lmargin (fs, old_lm);
-
-	  __argp_fmtstream_putc (fs, '\n');
-	  anything = 1;
-
-	  first_pattern = 0;
-	}
-      while (more_patterns);
-    }
-
-  if (flags & ARGP_HELP_PRE_DOC)
-    anything |= argp_doc (argp, state, 0, 0, 1, fs);
-
-  if (flags & ARGP_HELP_SEE)
-    {
-      __argp_fmtstream_printf (fs, dgettext (argp->argp_domain, "\
-Try `%s --help' or `%s --usage' for more information.\n"),
-			       name, name);
-      anything = 1;
-    }
-
-  if (flags & ARGP_HELP_LONG)
-    /* Print a long, detailed help message.  */
-    {
-      /* Print info about all the options.  */
-      if (hol->num_entries > 0)
-	{
-	  if (anything)
-	    __argp_fmtstream_putc (fs, '\n');
-	  hol_help (hol, state, fs);
-	  anything = 1;
-	}
-    }
-
-  if (flags & ARGP_HELP_POST_DOC)
-    /* Print any documentation strings at the end.  */
-    anything |= argp_doc (argp, state, 1, anything, 0, fs);
-
-  if ((flags & ARGP_HELP_BUG_ADDR) && argp_program_bug_address)
-    {
-      if (anything)
-	__argp_fmtstream_putc (fs, '\n');
-      __argp_fmtstream_printf (fs, dgettext (argp->argp_domain,
-					     "Report bugs to %s.\n"),
- 			       argp_program_bug_address);
-      anything = 1;
-    }
-
-  FUNLOCKFILE (stream);
-
-  if (hol)
-    hol_free (hol);
-
-  __argp_fmtstream_free (fs);
-}
-
-/* Output a usage message for ARGP to STREAM.  FLAGS are from the set
-   ARGP_HELP_*.  NAME is what to use wherever a `program name' is needed. */
-void __argp_help (const struct argp *argp, FILE *stream,
-		  unsigned flags, char *name)
-{
-  _help (argp, 0, stream, flags, name);
-}
-#ifdef weak_alias
-weak_alias (__argp_help, argp_help)
-#endif
-
-char *__argp_basename(char *name)
-{
-  char *short_name = strrchr(name, '/');
-  return short_name ? short_name + 1 : name;
-}
-
-char *
-__argp_short_program_name(const struct argp_state *state)
-{
-  if (state)
-    return state->name;
-#if HAVE_DECL_PROGRAM_INVOCATION_SHORT_NAME
-  return program_invocation_short_name;
-#elif HAVE_DECL_PROGRAM_INVOCATION_NAME
-  return __argp_basename(program_invocation_name);
-#else /* !HAVE_DECL_PROGRAM_INVOCATION_NAME */
-  /* FIXME: What now? Miles suggests that it is better to use NULL,
-     but currently the value is passed on directly to fputs_unlocked,
-     so that requires more changes. */
-# if __GNUC__
-  return "";
-# endif /* __GNUC__ */
-#endif /* !HAVE_DECL_PROGRAM_INVOCATION_NAME */
-}
-
-/* Output, if appropriate, a usage message for STATE to STREAM.  FLAGS are
-   from the set ARGP_HELP_*.  */
-void
-__argp_state_help (const struct argp_state *state, FILE *stream, unsigned flags)
-{
-  if ((!state || ! (state->flags & ARGP_NO_ERRS)) && stream)
-    {
-      if (state && (state->flags & ARGP_LONG_ONLY))
-	flags |= ARGP_HELP_LONG_ONLY;
-
-      _help (state ? state->root_argp : 0, state, stream, flags,
-	     __argp_short_program_name(state));
-
-      if (!state || ! (state->flags & ARGP_NO_EXIT))
-	{
-	  if (flags & ARGP_HELP_EXIT_ERR)
-	    exit (argp_err_exit_status);
-	  if (flags & ARGP_HELP_EXIT_OK)
-	    exit (0);
-	}
-  }
-}
-#ifdef weak_alias
-weak_alias (__argp_state_help, argp_state_help)
-#endif
-
-/* If appropriate, print the printf string FMT and following args, preceded
-   by the program name and `:', to stderr, and followed by a `Try ... --help'
-   message, then exit (1).  */
-void
-__argp_error (const struct argp_state *state, const char *fmt, ...)
-{
-  if (!state || !(state->flags & ARGP_NO_ERRS))
-    {
-      FILE *stream = state ? state->err_stream : stderr;
-
-      if (stream)
-	{
-	  va_list ap;
-
-	  FLOCKFILE (stream);
-
-	  FPUTS_UNLOCKED (__argp_short_program_name(state),
-			  stream);
-	  PUTC_UNLOCKED (':', stream);
-	  PUTC_UNLOCKED (' ', stream);
-
-	  va_start (ap, fmt);
-	  vfprintf (stream, fmt, ap);
-	  va_end (ap);
-
-	  PUTC_UNLOCKED ('\n', stream);
-
-	  __argp_state_help (state, stream, ARGP_HELP_STD_ERR);
-
-	  FUNLOCKFILE (stream);
-	}
-    }
-}
-#ifdef weak_alias
-weak_alias (__argp_error, argp_error)
-#endif
-
-/* Similar to the standard gnu error-reporting function error(), but will
-   respect the ARGP_NO_EXIT and ARGP_NO_ERRS flags in STATE, and will print
-   to STATE->err_stream.  This is useful for argument parsing code that is
-   shared between program startup (when exiting is desired) and runtime
-   option parsing (when typically an error code is returned instead).  The
-   difference between this function and argp_error is that the latter is for
-   *parsing errors*, and the former is for other problems that occur during
-   parsing but don't reflect a (syntactic) problem with the input.  */
-void
-__argp_failure (const struct argp_state *state, int status, int errnum,
-		const char *fmt, ...)
-{
-  if (!state || !(state->flags & ARGP_NO_ERRS))
-    {
-      FILE *stream = state ? state->err_stream : stderr;
-
-      if (stream)
-	{
-	  FLOCKFILE (stream);
-
-	  FPUTS_UNLOCKED (__argp_short_program_name(state),
-			  stream);
-
-	  if (fmt)
-	    {
-	      va_list ap;
-
-	      PUTC_UNLOCKED (':', stream);
-	      PUTC_UNLOCKED (' ', stream);
-
-	      va_start (ap, fmt);
-	      vfprintf (stream, fmt, ap);
-	      va_end (ap);
-	    }
-
-	  if (errnum)
-	    {
-	      PUTC_UNLOCKED (':', stream);
-	      PUTC_UNLOCKED (' ', stream);
-	      fputs (STRERROR (errnum), stream);
-	    }
-
-	  PUTC_UNLOCKED ('\n', stream);
-
-	  FUNLOCKFILE (stream);
-
-	  if (status && (!state || !(state->flags & ARGP_NO_EXIT)))
-	    exit (status);
-	}
-    }
-}
-#ifdef weak_alias
-weak_alias (__argp_failure, argp_failure)
-#endif
diff --git a/contrib/argp-standalone/argp-namefrob.h b/contrib/argp-standalone/argp-namefrob.h
deleted file mode 100644
index 0ce11481a7b..00000000000
--- a/contrib/argp-standalone/argp-namefrob.h
+++ /dev/null
@@ -1,96 +0,0 @@
-/* Name frobnication for compiling argp outside of glibc
-   Copyright (C) 1997 Free Software Foundation, Inc.
-   This file is part of the GNU C Library.
-   Written by Miles Bader <miles@gnu.ai.mit.edu>.
-
-   The GNU C Library is free software; you can redistribute it and/or
-   modify it under the terms of the GNU Library General Public License as
-   published by the Free Software Foundation; either version 2 of the
-   License, or (at your option) any later version.
-
-   The GNU C Library is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-   Library General Public License for more details.
-
-   You should have received a copy of the GNU Library General Public
-   License along with the GNU C Library; see the file COPYING.LIB.  If not,
-   write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
-   Boston, MA 02111-1307, USA.  */
-
-#if !_LIBC
-/* This code is written for inclusion in gnu-libc, and uses names in the
-   namespace reserved for libc.  If we're not compiling in libc, define those
-   names to be the normal ones instead.  */
-
-/* argp-parse functions */
-#undef __argp_parse
-#define __argp_parse argp_parse
-#undef __option_is_end
-#define __option_is_end _option_is_end
-#undef __option_is_short
-#define __option_is_short _option_is_short
-#undef __argp_input
-#define __argp_input _argp_input
-
-/* argp-help functions */
-#undef __argp_help
-#define __argp_help argp_help
-#undef __argp_error
-#define __argp_error argp_error
-#undef __argp_failure
-#define __argp_failure argp_failure
-#undef __argp_state_help
-#define __argp_state_help argp_state_help
-#undef __argp_usage
-#define __argp_usage argp_usage
-#undef __argp_basename
-#define __argp_basename _argp_basename
-#undef __argp_short_program_name
-#define __argp_short_program_name _argp_short_program_name
-
-/* argp-fmtstream functions */
-#undef __argp_make_fmtstream
-#define __argp_make_fmtstream argp_make_fmtstream
-#undef __argp_fmtstream_free
-#define __argp_fmtstream_free argp_fmtstream_free
-#undef __argp_fmtstream_putc
-#define __argp_fmtstream_putc argp_fmtstream_putc
-#undef __argp_fmtstream_puts
-#define __argp_fmtstream_puts argp_fmtstream_puts
-#undef __argp_fmtstream_write
-#define __argp_fmtstream_write argp_fmtstream_write
-#undef __argp_fmtstream_printf
-#define __argp_fmtstream_printf argp_fmtstream_printf
-#undef __argp_fmtstream_set_lmargin
-#define __argp_fmtstream_set_lmargin argp_fmtstream_set_lmargin
-#undef __argp_fmtstream_set_rmargin
-#define __argp_fmtstream_set_rmargin argp_fmtstream_set_rmargin
-#undef __argp_fmtstream_set_wmargin
-#define __argp_fmtstream_set_wmargin argp_fmtstream_set_wmargin
-#undef __argp_fmtstream_point
-#define __argp_fmtstream_point argp_fmtstream_point
-#undef __argp_fmtstream_update
-#define __argp_fmtstream_update _argp_fmtstream_update
-#undef __argp_fmtstream_ensure
-#define __argp_fmtstream_ensure _argp_fmtstream_ensure
-#undef __argp_fmtstream_lmargin
-#define __argp_fmtstream_lmargin argp_fmtstream_lmargin
-#undef __argp_fmtstream_rmargin
-#define __argp_fmtstream_rmargin argp_fmtstream_rmargin
-#undef __argp_fmtstream_wmargin
-#define __argp_fmtstream_wmargin argp_fmtstream_wmargin
-
-/* normal libc functions we call */
-#undef __sleep
-#define __sleep sleep
-#undef __strcasecmp
-#define __strcasecmp strcasecmp
-#undef __vsnprintf
-#define __vsnprintf vsnprintf
-
-#endif /* !_LIBC */
-
-#ifndef __set_errno
-#define __set_errno(e) (errno = (e))
-#endif
diff --git a/contrib/argp-standalone/argp-parse.c b/contrib/argp-standalone/argp-parse.c
deleted file mode 100644
index 78f7bf139b6..00000000000
--- a/contrib/argp-standalone/argp-parse.c
+++ /dev/null
@@ -1,1305 +0,0 @@
-/* Hierarchial argument parsing
-   Copyright (C) 1995, 96, 97, 98, 99, 2000,2003 Free Software Foundation, Inc.
-   This file is part of the GNU C Library.
-   Written by Miles Bader <miles@gnu.ai.mit.edu>.
-
-   The GNU C Library is free software; you can redistribute it and/or
-   modify it under the terms of the GNU Library General Public License as
-   published by the Free Software Foundation; either version 2 of the
-   License, or (at your option) any later version.
-
-   The GNU C Library is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-   Library General Public License for more details.
-
-   You should have received a copy of the GNU Library General Public
-   License along with the GNU C Library; see the file COPYING.LIB.  If not,
-   write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
-   Boston, MA 02111-1307, USA.  */
-
-#ifndef _GNU_SOURCE
-# define _GNU_SOURCE	1
-#endif
-
-#ifdef HAVE_CONFIG_H
-#include <config.h>
-#endif
-
-#if HAVE_ALLOCA_H
-#include <alloca.h>
-#endif
-
-#include <stdlib.h>
-#include <string.h>
-#if HAVE_UNISTD_H
-# include <unistd.h>
-#endif
-#include <limits.h>
-#include <assert.h>
-
-#if HAVE_MALLOC_H
-/* Needed, for alloca on windows */
-# include <malloc.h>
-#endif
-
-#ifndef _
-/* This is for other GNU distributions with internationalized messages.
-   When compiling libc, the _ macro is predefined.  */
-# if defined HAVE_LIBINTL_H || defined _LIBC
-#  include <libintl.h>
-#  ifdef _LIBC
-#   undef dgettext
-#   define dgettext(domain, msgid) __dcgettext (domain, msgid, LC_MESSAGES)
-#  endif
-# else
-#  define dgettext(domain, msgid) (msgid)
-#  define gettext(msgid) (msgid)
-# endif
-#endif
-#ifndef N_
-# define N_(msgid) (msgid)
-#endif
-
-#if _LIBC - 0
-#include <bits/libc-lock.h>
-#else
-#ifdef HAVE_CTHREADS_H
-#include <cthreads.h>
-#endif
-#endif /* _LIBC */
-
-#include "argp.h"
-#include "argp-namefrob.h"
-
-
-/* The meta-argument used to prevent any further arguments being interpreted
-   as options.  */
-#define QUOTE "--"
-
-/* EZ alias for ARGP_ERR_UNKNOWN.  */
-#define EBADKEY ARGP_ERR_UNKNOWN
-
-
-/* Default options.  */
-
-/* When argp is given the --HANG switch, _ARGP_HANG is set and argp will sleep
-   for one second intervals, decrementing _ARGP_HANG until it's zero.  Thus
-   you can force the program to continue by attaching a debugger and setting
-   it to 0 yourself.  */
-volatile int _argp_hang;
-
-#define OPT_PROGNAME	-2
-#define OPT_USAGE	-3
-#if HAVE_SLEEP && HAVE_GETPID
-#define OPT_HANG	-4
-#endif
-
-static const struct argp_option argp_default_options[] =
-{
-  {"help",	  '?',    	0, 0,  N_("Give this help list"), -1},
-  {"usage",	  OPT_USAGE,	0, 0,  N_("Give a short usage message"), 0 },
-  {"program-name",OPT_PROGNAME,"NAME", OPTION_HIDDEN,
-     N_("Set the program name"), 0},
-#if OPT_HANG
-  {"HANG",	  OPT_HANG,    "SECS", OPTION_ARG_OPTIONAL | OPTION_HIDDEN,
-     N_("Hang for SECS seconds (default 3600)"), 0 },
-#endif
-  {0, 0, 0, 0, 0, 0}
-};
-
-static error_t
-argp_default_parser (int key, char *arg, struct argp_state *state)
-{
-  switch (key)
-    {
-    case '?':
-      __argp_state_help (state, state->out_stream, ARGP_HELP_STD_HELP);
-      break;
-    case OPT_USAGE:
-      __argp_state_help (state, state->out_stream,
-		       ARGP_HELP_USAGE | ARGP_HELP_EXIT_OK);
-      break;
-
-    case OPT_PROGNAME:		/* Set the program name.  */
-#if HAVE_DECL_PROGRAM_INVOCATION_NAME
-      program_invocation_name = arg;
-#endif
-      /* [Note that some systems only have PROGRAM_INVOCATION_SHORT_NAME (aka
-	 __PROGNAME), in which case, PROGRAM_INVOCATION_NAME is just defined
-	 to be that, so we have to be a bit careful here.]  */
-
-      /* Update what we use for messages.  */
-
-      state->name = __argp_basename(arg);
-      
-#if HAVE_DECL_PROGRAM_INVOCATION_SHORT_NAME
-      program_invocation_short_name = state->name;
-#endif
-
-      if ((state->flags & (ARGP_PARSE_ARGV0 | ARGP_NO_ERRS))
-	  == ARGP_PARSE_ARGV0)
-	/* Update what getopt uses too.  */
-	state->argv[0] = arg;
-
-      break;
-      
-#if OPT_HANG
-    case OPT_HANG:
-      _argp_hang = atoi (arg ? arg : "3600");
-      fprintf(state->err_stream, "%s: pid = %ld\n",
-	      state->name, (long) getpid());
-      while (_argp_hang-- > 0)
-	__sleep (1);
-      break;
-#endif
-      
-    default:
-      return EBADKEY;
-    }
-  return 0;
-}
-
-static const struct argp argp_default_argp =
-  {argp_default_options, &argp_default_parser, NULL, NULL, NULL, NULL, "libc"};
-
-
-static const struct argp_option argp_version_options[] =
-{
-  {"version",	  'V',    	0, 0,  N_("Print program version"), -1},
-  {0, 0, 0, 0, 0, 0 }
-};
-
-static error_t
-argp_version_parser (int key, char *arg UNUSED, struct argp_state *state)
-{
-  switch (key)
-    {
-    case 'V':
-      if (argp_program_version_hook)
-	(*argp_program_version_hook) (state->out_stream, state);
-      else if (argp_program_version)
-	fprintf (state->out_stream, "%s\n", argp_program_version);
-      else
-	__argp_error (state, dgettext (state->root_argp->argp_domain,
-				       "(PROGRAM ERROR) No version known!?"));
-      if (! (state->flags & ARGP_NO_EXIT))
-	exit (0);
-      break;
-    default:
-      return EBADKEY;
-    }
-  return 0;
-}
-
-static const struct argp argp_version_argp =
-  {argp_version_options, &argp_version_parser, NULL, NULL, NULL, NULL, "libc"};
-
-
-
-/* The state of a `group' during parsing.  Each group corresponds to a
-   particular argp structure from the tree of such descending from the top
-   level argp passed to argp_parse.  */
-struct group
-{
-  /* This group's parsing function.  */
-  argp_parser_t parser;
-
-  /* Which argp this group is from.  */
-  const struct argp *argp;
-
-  /* The number of non-option args sucessfully handled by this parser.  */
-  unsigned args_processed;
-
-  /* This group's parser's parent's group.  */
-  struct group *parent;
-  unsigned parent_index;	/* And the our position in the parent.   */
-
-  /* These fields are swapped into and out of the state structure when
-     calling this group's parser.  */
-  void *input, **child_inputs;
-  void *hook;
-};
-
-/* Call GROUP's parser with KEY and ARG, swapping any group-specific info
-   from STATE before calling, and back into state afterwards.  If GROUP has
-   no parser, EBADKEY is returned.  */
-static error_t
-group_parse (struct group *group, struct argp_state *state, int key, char *arg)
-{
-  if (group->parser)
-    {
-      error_t err;
-      state->hook = group->hook;
-      state->input = group->input;
-      state->child_inputs = group->child_inputs;
-      state->arg_num = group->args_processed;
-      err = (*group->parser)(key, arg, state);
-      group->hook = state->hook;
-      return err;
-    }
-  else
-    return EBADKEY;
-}
-
-struct parser
-{
-  const struct argp *argp;
-
-  const char *posixly_correct;
-  
-  /* True if there are only no-option arguments left, which are just
-     passed verbatim with ARGP_KEY_ARG. This is set if we encounter a
-     quote, or the end of the proper options, but may be cleared again
-     if the user moves the next argument pointer backwards. */
-  int args_only;
-
-  /* Describe how to deal with options that follow non-option ARGV-elements.
-
-     If the caller did not specify anything, the default is
-     REQUIRE_ORDER if the environment variable POSIXLY_CORRECT is
-     defined, PERMUTE otherwise.
-
-     REQUIRE_ORDER means don't recognize them as options; stop option
-     processing when the first non-option is seen. This is what Unix
-     does. This mode of operation is selected by either setting the
-     environment variable POSIXLY_CORRECT, or using `+' as the first
-     character of the list of option characters.
-
-     PERMUTE is the default. We permute the contents of ARGV as we
-     scan, so that eventually all the non-options are at the end. This
-     allows options to be given in any order, even with programs that
-     were not written to expect this.
-
-     RETURN_IN_ORDER is an option available to programs that were
-     written to expect options and other ARGV-elements in any order
-     and that care about the ordering of the two. We describe each
-     non-option ARGV-element as if it were the argument of an option
-     with character code 1. Using `-' as the first character of the
-     list of option characters selects this mode of operation.
-
-  */
-  enum { REQUIRE_ORDER, PERMUTE, RETURN_IN_ORDER } ordering;
-
-  /* A segment of non-option arguments that have been skipped for
-     later processing, after all options. `first_nonopt' is the index
-     in ARGV of the first of them; `last_nonopt' is the index after
-     the last of them.
-
-     If quoted or args_only is non-zero, this segment should be empty. */
-
-  /* FIXME: I'd prefer to use unsigned, but it's more consistent to
-     use the same type as for state.next. */
-  int first_nonopt;
-  int last_nonopt;
-  
-  /* String of all recognized short options. Needed for ARGP_LONG_ONLY. */
-  /* FIXME: Perhaps change to a pointer to a suitable bitmap instead? */
-  char *short_opts;
-
-  /* For parsing combined short options. */
-  char *nextchar;
-  
-  /* States of the various parsing groups.  */
-  struct group *groups;
-  /* The end of the GROUPS array.  */
-  struct group *egroup;
-  /* An vector containing storage for the CHILD_INPUTS field in all groups.  */
-  void **child_inputs;
-  
-  /* State block supplied to parsing routines.  */
-  struct argp_state state;
-
-  /* Memory used by this parser.  */
-  void *storage;
-};
-
-/* Search for a group defining a short option. */
-static const struct argp_option *
-find_short_option(struct parser *parser, int key, struct group **p)
-{
-  struct group *group;
-  
-  assert(key >= 0);
-  assert(isascii(key));
-
-  for (group = parser->groups; group < parser->egroup; group++)
-    {
-      const struct argp_option *opts;
-
-      for (opts = group->argp->options; !__option_is_end(opts); opts++)
-	if (opts->key == key)
-	  {
-	    *p = group;
-	    return opts;
-	  }
-    }
-  return NULL;
-}
-
-enum match_result { MATCH_EXACT, MATCH_PARTIAL, MATCH_NO };
-
-/* If defined, allow complete.el-like abbreviations of long options. */
-#ifndef ARGP_COMPLETE
-#define ARGP_COMPLETE 0
-#endif
-
-/* Matches an encountern long-option argument ARG against an option NAME.
- * ARG is terminated by NUL or '='. */
-static enum match_result
-match_option(const char *arg, const char *name)
-{
-  unsigned i, j;
-  for (i = j = 0;; i++, j++)
-    {
-      switch(arg[i])
-	{
-	case '\0':
-	case '=':
-	  return name[j] ? MATCH_PARTIAL : MATCH_EXACT;
-#if ARGP_COMPLETE
-	case '-':
-	  while (name[j] != '-')
-	    if (!name[j++])
-	      return MATCH_NO;
-	  break;
-#endif
-	default:
-	  if (arg[i] != name[j])
-	    return MATCH_NO;
-	}
-    }
-}
-
-static const struct argp_option *
-find_long_option(struct parser *parser,
-		 const char *arg,
-		 struct group **p)
-{
-  struct group *group;
-
-  /* Partial match found so far. */
-  struct group *matched_group = NULL;
-  const struct argp_option *matched_option = NULL;
-
-  /* Number of partial matches. */
-  int num_partial = 0;
-
-  for (group = parser->groups; group < parser->egroup; group++)
-    {
-      const struct argp_option *opts;
-
-      for (opts = group->argp->options; !__option_is_end(opts); opts++)
-	{
-	  if (!opts->name)
-	    continue;
-	  switch (match_option(arg, opts->name))
-	    {
-	    case MATCH_NO:
-	      break;
-	    case MATCH_PARTIAL:
-	      num_partial++;
-
-	      matched_group = group;
-	      matched_option = opts;
-
-	      break;
-	    case MATCH_EXACT:
-	      /* Exact match. */
-	      *p = group;
-	      return opts;
-	    }
-	}
-    }
-  if (num_partial == 1)
-    {
-      *p = matched_group;
-      return matched_option;
-    }
-
-  return NULL;
-}
-
-
-/* The next usable entries in the various parser tables being filled in by
-   convert_options.  */
-struct parser_convert_state
-{
-  struct parser *parser;
-  char *short_end;
-  void **child_inputs_end;
-};
-
-/* Initialize GROUP from ARGP. If CVT->SHORT_END is non-NULL, short
-   options are recorded in the short options string. Returns the next
-   unused group entry. CVT holds state used during the conversion. */
-static struct group *
-convert_options (const struct argp *argp,
-		 struct group *parent, unsigned parent_index,
-		 struct group *group, struct parser_convert_state *cvt)
-{
-  const struct argp_option *opt = argp->options;
-  const struct argp_child *children = argp->children;
-
-  if (opt || argp->parser)
-    {
-      /* This parser needs a group. */
-      if (cvt->short_end)
-	{
-	  /* Record any short options. */
-	  for ( ; !__option_is_end (opt); opt++)
-	    if (__option_is_short(opt))
-	      *cvt->short_end++ = opt->key;
-	}
-      
-      group->parser = argp->parser;
-      group->argp = argp;
-      group->args_processed = 0;
-      group->parent = parent;
-      group->parent_index = parent_index;
-      group->input = 0;
-      group->hook = 0;
-      group->child_inputs = 0;
-      
-      if (children)
-	/* Assign GROUP's CHILD_INPUTS field some space from
-	   CVT->child_inputs_end.*/
-	{
-	  unsigned num_children = 0;
-	  while (children[num_children].argp)
-	    num_children++;
-	  group->child_inputs = cvt->child_inputs_end;
-	  cvt->child_inputs_end += num_children;
-	}
-      parent = group++;
-    }
-  else
-    parent = 0;
-
-  if (children)
-    {
-      unsigned index = 0;
-      while (children->argp)
-	group =
-	  convert_options (children++->argp, parent, index++, group, cvt);
-    }
-
-  return group;
-}
-/* Allocate and initialize the group structures, so that they are
-   ordered as if by traversing the corresponding argp parser tree in
-   pre-order. Also build the list of short options, if that is needed. */
-static void
-parser_convert (struct parser *parser, const struct argp *argp)
-{
-  struct parser_convert_state cvt;
-
-  cvt.parser = parser;
-  cvt.short_end = parser->short_opts;
-  cvt.child_inputs_end = parser->child_inputs;
-
-  parser->argp = argp;
-
-  if (argp)
-    parser->egroup = convert_options (argp, 0, 0, parser->groups, &cvt);
-  else
-    parser->egroup = parser->groups; /* No parsers at all! */
-
-  if (parser->short_opts)
-    *cvt.short_end ='\0';
-}
-
-/* Lengths of various parser fields which we will allocated.  */
-struct parser_sizes
-{
-  /* Needed only ARGP_LONG_ONLY */
-  size_t short_len;		/* Number of short options.  */
-
-  size_t num_groups;		/* Group structures we allocate.  */
-  size_t num_child_inputs;	/* Child input slots.  */
-};
-
-/* For ARGP, increments the NUM_GROUPS field in SZS by the total
-   number of argp structures descended from it, and the SHORT_LEN by
-   the total number of short options. */
-static void
-calc_sizes (const struct argp *argp,  struct parser_sizes *szs)
-{
-  const struct argp_child *child = argp->children;
-  const struct argp_option *opt = argp->options;
-
-  if (opt || argp->parser)
-    {
-      /* This parser needs a group. */
-      szs->num_groups++;
-      if (opt)
-	{
-	  while (__option_is_short (opt++))
-	    szs->short_len++;
-	}
-    }
-
-  if (child)
-    while (child->argp)
-      {
-	calc_sizes ((child++)->argp, szs);
-	szs->num_child_inputs++;
-      }
-}
-
-/* Initializes PARSER to parse ARGP in a manner described by FLAGS.  */
-static error_t
-parser_init (struct parser *parser, const struct argp *argp,
-	     int argc, char **argv, int flags, void *input)
-{
-  error_t err = 0;
-  struct group *group;
-  struct parser_sizes szs;
-
-  parser->posixly_correct = getenv ("POSIXLY_CORRECT");
-
-  if (flags & ARGP_IN_ORDER)
-    parser->ordering = RETURN_IN_ORDER;
-  else if (flags & ARGP_NO_ARGS)
-    parser->ordering = REQUIRE_ORDER;
-  else if (parser->posixly_correct)
-    parser->ordering = REQUIRE_ORDER;
-  else
-    parser->ordering = PERMUTE;
-  
-  szs.short_len = 0;
-  szs.num_groups = 0;
-  szs.num_child_inputs = 0;
-
-  if (argp)
-    calc_sizes (argp, &szs);
-
-  if (!(flags & ARGP_LONG_ONLY))
-    /* We have no use for the short option array. */
-    szs.short_len = 0;
-  
-  /* Lengths of the various bits of storage used by PARSER.  */
-#define GLEN (szs.num_groups + 1) * sizeof (struct group)
-#define CLEN (szs.num_child_inputs * sizeof (void *))
-#define SLEN (szs.short_len + 1)
-#define STORAGE(offset) ((void *) (((char *) parser->storage) + (offset)))
-
-  parser->storage = malloc (GLEN + CLEN + SLEN);
-  if (! parser->storage)
-    return ENOMEM;
-
-  parser->groups = parser->storage;
-
-  parser->child_inputs = STORAGE(GLEN);
-  memset (parser->child_inputs, 0, szs.num_child_inputs * sizeof (void *));
-
-  if (flags & ARGP_LONG_ONLY)
-    parser->short_opts = STORAGE(GLEN + CLEN);
-  else
-    parser->short_opts = NULL;
-
-  parser_convert (parser, argp);
-
-  memset (&parser->state, 0, sizeof (struct argp_state));
-  
-  parser->state.root_argp = parser->argp;
-  parser->state.argc = argc;
-  parser->state.argv = argv;
-  parser->state.flags = flags;
-  parser->state.err_stream = stderr;
-  parser->state.out_stream = stdout;
-  parser->state.pstate = parser;
-
-  parser->args_only = 0;
-  parser->nextchar = NULL;
-  parser->first_nonopt = parser->last_nonopt = 0;
-    
-  /* Call each parser for the first time, giving it a chance to propagate
-     values to child parsers.  */
-  if (parser->groups < parser->egroup)
-    parser->groups->input = input;
-  for (group = parser->groups;
-       group < parser->egroup && (!err || err == EBADKEY);
-       group++)
-    {
-      if (group->parent)
-	/* If a child parser, get the initial input value from the parent. */
-	group->input = group->parent->child_inputs[group->parent_index];
-
-      if (!group->parser
-	  && group->argp->children && group->argp->children->argp)
-	/* For the special case where no parsing function is supplied for an
-	   argp, propagate its input to its first child, if any (this just
-	   makes very simple wrapper argps more convenient).  */
-	group->child_inputs[0] = group->input;
-
-      err = group_parse (group, &parser->state, ARGP_KEY_INIT, 0);
-    }
-  if (err == EBADKEY)
-    err = 0;			/* Some parser didn't understand.  */
-
-  if (err)
-    return err;
-
-  if (argv[0] && !(parser->state.flags & ARGP_PARSE_ARGV0))
-    /* There's an argv[0]; use it for messages.  */
-    {
-      parser->state.name = __argp_basename(argv[0]);
-
-      /* Don't parse it as an argument. */
-      parser->state.next = 1;
-    }
-  else
-    parser->state.name = __argp_short_program_name(NULL);
-  
-  return 0;
-}
-
-/* Free any storage consumed by PARSER (but not PARSER itself).  */
-static error_t
-parser_finalize (struct parser *parser,
-		 error_t err, int arg_ebadkey, int *end_index)
-{
-  struct group *group;
-
-  if (err == EBADKEY && arg_ebadkey)
-    /* Suppress errors generated by unparsed arguments.  */
-    err = 0;
-
-  if (! err)
-    {
-      if (parser->state.next == parser->state.argc)
-	/* We successfully parsed all arguments!  Call all the parsers again,
-	   just a few more times... */
-	{
-	  for (group = parser->groups;
-	       group < parser->egroup && (!err || err==EBADKEY);
-	       group++)
-	    if (group->args_processed == 0)
-	      err = group_parse (group, &parser->state, ARGP_KEY_NO_ARGS, 0);
-	  for (group = parser->egroup - 1;
-	       group >= parser->groups && (!err || err==EBADKEY);
-	       group--)
-	    err = group_parse (group, &parser->state, ARGP_KEY_END, 0);
-
-	  if (err == EBADKEY)
-	    err = 0;		/* Some parser didn't understand.  */
-
-	  /* Tell the user that all arguments are parsed.  */
-	  if (end_index)
-	    *end_index = parser->state.next;
-	}
-      else if (end_index)
-	/* Return any remaining arguments to the user.  */
-	*end_index = parser->state.next;
-      else
-	/* No way to return the remaining arguments, they must be bogus. */
-	{
-	  if (!(parser->state.flags & ARGP_NO_ERRS)
-	      && parser->state.err_stream)
-	    fprintf (parser->state.err_stream,
-		     dgettext (parser->argp->argp_domain,
-			       "%s: Too many arguments\n"),
-		     parser->state.name);
-	  err = EBADKEY;
-	}
-    }
-
-  /* Okay, we're all done, with either an error or success; call the parsers
-     to indicate which one.  */
-
-  if (err)
-    {
-      /* Maybe print an error message.  */
-      if (err == EBADKEY)
-	/* An appropriate message describing what the error was should have
-	   been printed earlier.  */
-	__argp_state_help (&parser->state, parser->state.err_stream,
-			   ARGP_HELP_STD_ERR);
-
-      /* Since we didn't exit, give each parser an error indication.  */
-      for (group = parser->groups; group < parser->egroup; group++)
-	group_parse (group, &parser->state, ARGP_KEY_ERROR, 0);
-    }
-  else
-    /* Notify parsers of success, and propagate back values from parsers.  */
-    {
-      /* We pass over the groups in reverse order so that child groups are
-	 given a chance to do there processing before passing back a value to
-	 the parent.  */
-      for (group = parser->egroup - 1
-	   ; group >= parser->groups && (!err || err == EBADKEY)
-	   ; group--)
-	err = group_parse (group, &parser->state, ARGP_KEY_SUCCESS, 0);
-      if (err == EBADKEY)
-	err = 0;		/* Some parser didn't understand.  */
-    }
-
-  /* Call parsers once more, to do any final cleanup.  Errors are ignored.  */
-  for (group = parser->egroup - 1; group >= parser->groups; group--)
-    group_parse (group, &parser->state, ARGP_KEY_FINI, 0);
-
-  if (err == EBADKEY)
-    err = EINVAL;
-
-  free (parser->storage);
-
-  return err;
-}
-
-/* Call the user parsers to parse the non-option argument VAL, at the
-   current position, returning any error. The state NEXT pointer
-   should point to the argument; this function will adjust it
-   correctly to reflect however many args actually end up being
-   consumed. */
-static error_t
-parser_parse_arg (struct parser *parser, char *val)
-{
-  /* Save the starting value of NEXT */
-  int index = parser->state.next;
-  error_t err = EBADKEY;
-  struct group *group;
-  int key = 0;			/* Which of ARGP_KEY_ARG[S] we used.  */
-
-  /* Try to parse the argument in each parser.  */
-  for (group = parser->groups
-       ; group < parser->egroup && err == EBADKEY
-       ; group++)
-    {
-      parser->state.next++;	/* For ARGP_KEY_ARG, consume the arg.  */
-      key = ARGP_KEY_ARG;
-      err = group_parse (group, &parser->state, key, val);
-
-      if (err == EBADKEY)
-	/* This parser doesn't like ARGP_KEY_ARG; try ARGP_KEY_ARGS instead. */
-	{
-	  parser->state.next--;	/* For ARGP_KEY_ARGS, put back the arg.  */
-	  key = ARGP_KEY_ARGS;
-	  err = group_parse (group, &parser->state, key, 0);
-	}
-    }
-
-  if (! err)
-    {
-      if (key == ARGP_KEY_ARGS)
-	/* The default for ARGP_KEY_ARGS is to assume that if NEXT isn't
-	   changed by the user, *all* arguments should be considered
-	   consumed.  */
-	parser->state.next = parser->state.argc;
-
-      if (parser->state.next > index)
-	/* Remember that we successfully processed a non-option
-	   argument -- but only if the user hasn't gotten tricky and set
-	   the clock back.  */
-	(--group)->args_processed += (parser->state.next - index);
-      else
-	/* The user wants to reparse some args, so try looking for options again.  */
-	parser->args_only = 0;
-    }
-
-  return err;
-}
-
-/* Exchange two adjacent subsequences of ARGV.
-   One subsequence is elements [first_nonopt,last_nonopt)
-   which contains all the non-options that have been skipped so far.
-   The other is elements [last_nonopt,next), which contains all
-   the options processed since those non-options were skipped.
-
-   `first_nonopt' and `last_nonopt' are relocated so that they describe
-   the new indices of the non-options in ARGV after they are moved.  */
-
-static void
-exchange (struct parser *parser)
-{
-  int bottom = parser->first_nonopt;
-  int middle = parser->last_nonopt;
-  int top = parser->state.next;
-  char **argv = parser->state.argv;
-  
-  char *tem;
-
-  /* Exchange the shorter segment with the far end of the longer segment.
-     That puts the shorter segment into the right place.
-     It leaves the longer segment in the right place overall,
-     but it consists of two parts that need to be swapped next.  */
-
-  while (top > middle && middle > bottom)
-    {
-      if (top - middle > middle - bottom)
-	{
-	  /* Bottom segment is the short one.  */
-	  int len = middle - bottom;
-	  register int i;
-
-	  /* Swap it with the top part of the top segment.  */
-	  for (i = 0; i < len; i++)
-	    {
-	      tem = argv[bottom + i];
-	      argv[bottom + i] = argv[top - (middle - bottom) + i];
-	      argv[top - (middle - bottom) + i] = tem;
-	    }
-	  /* Exclude the moved bottom segment from further swapping.  */
-	  top -= len;
-	}
-      else
-	{
-	  /* Top segment is the short one.  */
-	  int len = top - middle;
-	  register int i;
-
-	  /* Swap it with the bottom part of the bottom segment.  */
-	  for (i = 0; i < len; i++)
-	    {
-	      tem = argv[bottom + i];
-	      argv[bottom + i] = argv[middle + i];
-	      argv[middle + i] = tem;
-	    }
-	  /* Exclude the moved top segment from further swapping.  */
-	  bottom += len;
-	}
-    }
-
-  /* Update records for the slots the non-options now occupy.  */
-
-  parser->first_nonopt += (parser->state.next - parser->last_nonopt);
-  parser->last_nonopt = parser->state.next;
-}
-
-
-
-enum arg_type { ARG_ARG, ARG_SHORT_OPTION,
-		ARG_LONG_OPTION, ARG_LONG_ONLY_OPTION,
-		ARG_QUOTE };
-
-static enum arg_type
-classify_arg(struct parser *parser, char *arg, char **opt)
-{
-  if (arg[0] == '-')
-    /* Looks like an option... */
-    switch (arg[1])
-      {
-      case '\0':
-	/* "-" is not an option. */
-	return ARG_ARG;
-      case '-':
-	/* Long option, or quote. */
-	if (!arg[2])
-	  return ARG_QUOTE;
-	  
-	/* A long option. */
-	if (opt)
-	  *opt = arg + 2;
-	return ARG_LONG_OPTION;
-
-      default:
-	/* Short option. But if ARGP_LONG_ONLY, it can also be a long option. */
-
-	if (opt)
-	  *opt = arg + 1;
-
-	if (parser->state.flags & ARGP_LONG_ONLY)
-	  {
-	    /* Rules from getopt.c:
-
-	       If long_only and the ARGV-element has the form "-f",
-	       where f is a valid short option, don't consider it an
-	       abbreviated form of a long option that starts with f.
-	       Otherwise there would be no way to give the -f short
-	       option.
-
-	       On the other hand, if there's a long option "fubar" and
-	       the ARGV-element is "-fu", do consider that an
-	       abbreviation of the long option, just like "--fu", and
-	       not "-f" with arg "u".
-
-	       This distinction seems to be the most useful approach. */
-
-	    assert(parser->short_opts);
-	    
-	    if (arg[2] || !strchr(parser->short_opts, arg[1]))
-	      return ARG_LONG_ONLY_OPTION;
-	  }
-
-	return ARG_SHORT_OPTION;
-      }
-  
-  else
-    return ARG_ARG;
-}
-
-/* Parse the next argument in PARSER (as indicated by PARSER->state.next).
-   Any error from the parsers is returned, and *ARGP_EBADKEY indicates
-   whether a value of EBADKEY is due to an unrecognized argument (which is
-   generally not fatal).  */
-static error_t
-parser_parse_next (struct parser *parser, int *arg_ebadkey)
-{
-  if (parser->state.quoted && parser->state.next < parser->state.quoted)
-    /* The next argument pointer has been moved to before the quoted
-       region, so pretend we never saw the quoting `--', and start
-       looking for options again. If the `--' is still there we'll just
-       process it one more time. */
-    parser->state.quoted = parser->args_only = 0;
-
-  /* Give FIRST_NONOPT & LAST_NONOPT rational values if NEXT has been
-     moved back by the user (who may also have changed the arguments).  */
-  if (parser->last_nonopt > parser->state.next)
-    parser->last_nonopt = parser->state.next;
-  if (parser->first_nonopt > parser->state.next)
-    parser->first_nonopt = parser->state.next;
-
-  if (parser->nextchar)
-    /* Deal with short options. */
-    {
-      struct group *group;
-      char c;
-      const struct argp_option *option;
-      char *value = NULL;;
-
-      assert(!parser->args_only);
-
-      c = *parser->nextchar++;
-      
-      option = find_short_option(parser, c, &group);
-      if (!option)
-	{
-	  if (parser->posixly_correct)
-	    /* 1003.2 specifies the format of this message.  */
-	    fprintf (parser->state.err_stream,
-		     dgettext(parser->state.root_argp->argp_domain,
-			      "%s: illegal option -- %c\n"),
-		     parser->state.name, c);
-	  else
-	    fprintf (parser->state.err_stream,
-		     dgettext(parser->state.root_argp->argp_domain,
-			      "%s: invalid option -- %c\n"),
-		     parser->state.name, c);
-
-	  *arg_ebadkey = 0;
-	  return EBADKEY;
-	}
-
-      if (!*parser->nextchar)
-	parser->nextchar = NULL;
-
-      if (option->arg)
-	{
-	  value = parser->nextchar;
-	  parser->nextchar = NULL;
-	      
-	  if (!value
-	      && !(option->flags & OPTION_ARG_OPTIONAL))
-	    /* We need an mandatory argument. */
-	    {
-	      if (parser->state.next == parser->state.argc)
-		/* Missing argument */
-		{
-		  /* 1003.2 specifies the format of this message.  */
-		  fprintf (parser->state.err_stream,
-			   dgettext(parser->state.root_argp->argp_domain,
-				    "%s: option requires an argument -- %c\n"),
-			   parser->state.name, c);
-
-		  *arg_ebadkey = 0;
-		  return EBADKEY;
-		}
-	      value = parser->state.argv[parser->state.next++];
-	    }
-	}
-      return group_parse(group, &parser->state,
-			 option->key, value);
-    }
-  else
-    /* Advance to the next ARGV-element.  */
-    {
-      if (parser->args_only)
-	{
-	  *arg_ebadkey = 1;
-	  if (parser->state.next >= parser->state.argc)
-	    /* We're done. */
-	    return EBADKEY;
-	  else
-	    return parser_parse_arg(parser,
-				    parser->state.argv[parser->state.next]);
-	}
-      
-      if (parser->state.next >= parser->state.argc)
-	/* Almost done. If there are non-options that we skipped
-	   previously, we should process them now. */
-	{
-	  *arg_ebadkey = 1;
-	  if (parser->first_nonopt != parser->last_nonopt)
-	    {
-	      exchange(parser);
-	      
-	      /* Start processing the arguments we skipped previously. */
-	      parser->state.next = parser->first_nonopt;
-	      
-	      parser->first_nonopt = parser->last_nonopt = 0;
-
-	      parser->args_only = 1;
-	      return 0;
-	    }
-	  else
-	    /* Indicate that we're really done. */
-	    return EBADKEY;
-	}
-      else
-	/* Look for options. */
-	{
-	  char *arg = parser->state.argv[parser->state.next];
-
-	  char *optstart;
-	  enum arg_type token = classify_arg(parser, arg, &optstart);
-	  
-	  switch (token)
-	    {
-	    case ARG_ARG:
-	      switch (parser->ordering)
-		{
-		case PERMUTE:
-		  if (parser->first_nonopt == parser->last_nonopt)
-		    /* Skipped sequence is empty; start a new one. */
-		    parser->first_nonopt = parser->last_nonopt = parser->state.next;
-
-		  else if (parser->last_nonopt != parser->state.next)
-		    /* We have a non-empty skipped sequence, and
-		       we're not at the end-point, so move it. */
-		    exchange(parser);
-
-		  assert(parser->last_nonopt == parser->state.next);
-		  
-		  /* Skip this argument for now. */
-		  parser->state.next++;
-		  parser->last_nonopt = parser->state.next; 
-		  
-		  return 0;
-
-		case REQUIRE_ORDER:
-		  /* Implicit quote before the first argument. */
-		   parser->args_only = 1;
-		   return 0;
-		   
-		case RETURN_IN_ORDER:
-		  *arg_ebadkey = 1;
-		  return parser_parse_arg(parser, arg);
-
-		default:
-		  abort();
-		}
-	    case ARG_QUOTE:
-	      /* Skip it, then exchange with any previous non-options. */
-	      parser->state.next++;
-	      assert (parser->last_nonopt != parser->state.next);
-
-	      if (parser->first_nonopt != parser->last_nonopt)
-		{
-		  exchange(parser);
-		  
-		  /* Start processing the skipped and the quoted
-		     arguments. */
-
-		  parser->state.quoted = parser->state.next = parser->first_nonopt;
-
-		  /* Also empty the skipped-list, to avoid confusion
-		     if the user resets the next pointer. */
-		  parser->first_nonopt = parser->last_nonopt = 0;
-		}
-	      else
-		parser->state.quoted = parser->state.next;
-
-	      parser->args_only = 1;	      
-	      return 0;
-
-	    case ARG_LONG_ONLY_OPTION:
-	    case ARG_LONG_OPTION:
-	      {
-		struct group *group;
-		const struct argp_option *option;
-		char *value;
-
-		parser->state.next++;
-		option = find_long_option(parser, optstart, &group);
-		
-		if (!option)
-		  {
-		    /* NOTE: This includes any "=something" in the output. */
-		    fprintf (parser->state.err_stream,
-			     dgettext(parser->state.root_argp->argp_domain,
-				      "%s: unrecognized option `%s'\n"),
-			     parser->state.name, arg);
-		    *arg_ebadkey = 0;
-		    return EBADKEY;
-		  }
-
-		value = strchr(optstart, '=');
-		if (value)
-		  value++;
-		
-		if (value && !option->arg)
-		  /* Unexpected argument. */
-		  {
-		    if (token == ARG_LONG_OPTION)
-		      /* --option */
-		      fprintf (parser->state.err_stream,
-			       dgettext(parser->state.root_argp->argp_domain,
-					"%s: option `--%s' doesn't allow an argument\n"),
-			       parser->state.name, option->name);
-		    else
-		      /* +option or -option */
-		      fprintf (parser->state.err_stream,
-			       dgettext(parser->state.root_argp->argp_domain,
-					"%s: option `%c%s' doesn't allow an argument\n"),
-			       parser->state.name, arg[0], option->name);
-
-		    *arg_ebadkey = 0;
-		    return EBADKEY;
-		  }
-		
-		if (option->arg && !value
-		    && !(option->flags & OPTION_ARG_OPTIONAL))
-		  /* We need an mandatory argument. */
-		  {
-		    if (parser->state.next == parser->state.argc)
-		      /* Missing argument */
-		      {
-			if (token == ARG_LONG_OPTION)
-			  /* --option */
-			  fprintf (parser->state.err_stream,
-				   dgettext(parser->state.root_argp->argp_domain,
-					    "%s: option `--%s' requires an argument\n"),
-				 parser->state.name, option->name);
-			else
-			  /* +option or -option */
-			  fprintf (parser->state.err_stream,
-				   dgettext(parser->state.root_argp->argp_domain,
-					    "%s: option `%c%s' requires an argument\n"),
-				   parser->state.name, arg[0], option->name);
-
-			*arg_ebadkey = 0;
-			return EBADKEY;
-		      }
-
-		    value = parser->state.argv[parser->state.next++];
-		  }
-		*arg_ebadkey = 0;
-		return group_parse(group, &parser->state,
-				   option->key, value);
-	      }
-	    case ARG_SHORT_OPTION:
-	      parser->state.next++;
-	      parser->nextchar = optstart;
-	      return 0;
-
-	    default:
-	      abort();
-	    }
-	}
-    }
-}
-
-/* Parse the options strings in ARGC & ARGV according to the argp in ARGP.
-   FLAGS is one of the ARGP_ flags above.  If END_INDEX is non-NULL, the
-   index in ARGV of the first unparsed option is returned in it.  If an
-   unknown option is present, EINVAL is returned; if some parser routine
-   returned a non-zero value, it is returned; otherwise 0 is returned.  */
-error_t
-__argp_parse (const struct argp *argp, int argc, char **argv, unsigned flags,
-	      int *end_index, void *input)
-{
-  error_t err;
-  struct parser parser;
-
-  /* If true, then err == EBADKEY is a result of a non-option argument failing
-     to be parsed (which in some cases isn't actually an error).  */
-  int arg_ebadkey = 0;
-
-  if (! (flags & ARGP_NO_HELP))
-    /* Add our own options.  */
-    {
-      struct argp_child *child = alloca (4 * sizeof (struct argp_child));
-      struct argp *top_argp = alloca (sizeof (struct argp));
-
-      /* TOP_ARGP has no options, it just serves to group the user & default
-	 argps.  */
-      memset (top_argp, 0, sizeof (*top_argp));
-      top_argp->children = child;
-
-      memset (child, 0, 4 * sizeof (struct argp_child));
-
-      if (argp)
-	(child++)->argp = argp;
-      (child++)->argp = &argp_default_argp;
-      if (argp_program_version || argp_program_version_hook)
-	(child++)->argp = &argp_version_argp;
-      child->argp = 0;
-
-      argp = top_argp;
-    }
-
-  /* Construct a parser for these arguments.  */
-  err = parser_init (&parser, argp, argc, argv, flags, input);
-
-  if (! err)
-    /* Parse! */
-    {
-      while (! err)
-	err = parser_parse_next (&parser, &arg_ebadkey);
-      err = parser_finalize (&parser, err, arg_ebadkey, end_index);
-    }
-
-  return err;
-}
-#ifdef weak_alias
-weak_alias (__argp_parse, argp_parse)
-#endif
-
-/* Return the input field for ARGP in the parser corresponding to STATE; used
-   by the help routines.  */
-void *
-__argp_input (const struct argp *argp, const struct argp_state *state)
-{
-  if (state)
-    {
-      struct group *group;
-      struct parser *parser = state->pstate;
-
-      for (group = parser->groups; group < parser->egroup; group++)
-	if (group->argp == argp)
-	  return group->input;
-    }
-
-  return 0;
-}
-#ifdef weak_alias
-weak_alias (__argp_input, _argp_input)
-#endif
-
-/* Defined here, in case a user is not inlining the definitions in
- * argp.h */
-void
-__argp_usage (__const struct argp_state *__state)
-{
-  __argp_state_help (__state, stderr, ARGP_HELP_STD_USAGE);
-}
-
-int
-__option_is_short (__const struct argp_option *__opt)
-{
-  if (__opt->flags & OPTION_DOC)
-    return 0;
-  else
-    {
-      int __key = __opt->key;
-      /* FIXME: whether or not a particular key implies a short option
-       * ought not to be locale dependent. */
-      return __key > 0 && isprint (__key);
-    }
-}
-
-int
-__option_is_end (__const struct argp_option *__opt)
-{
-  return !__opt->key && !__opt->name && !__opt->doc && !__opt->group;
-}
diff --git a/contrib/argp-standalone/argp-pv.c b/contrib/argp-standalone/argp-pv.c
deleted file mode 100644
index d7d374a66bd..00000000000
--- a/contrib/argp-standalone/argp-pv.c
+++ /dev/null
@@ -1,25 +0,0 @@
-/* Default definition for ARGP_PROGRAM_VERSION.
-   Copyright (C) 1996, 1997, 1999, 2004 Free Software Foundation, Inc.
-   This file is part of the GNU C Library.
-   Written by Miles Bader <miles@gnu.ai.mit.edu>.
-
-   The GNU C Library is free software; you can redistribute it and/or
-   modify it under the terms of the GNU Library General Public License as
-   published by the Free Software Foundation; either version 2 of the
-   License, or (at your option) any later version.
-
-   The GNU C Library is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-   Library General Public License for more details.
-
-   You should have received a copy of the GNU Library General Public
-   License along with the GNU C Library; see the file COPYING.LIB.  If not,
-   write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
-   Boston, MA 02111-1307, USA.  */
-
-/* If set by the user program to a non-zero value, then a default option
-   --version is added (unless the ARGP_NO_HELP flag is used), which will
-   print this this string followed by a newline and exit (unless the
-   ARGP_NO_EXIT flag is used).  Overridden by ARGP_PROGRAM_VERSION_HOOK.  */
-const char *argp_program_version = 0;
diff --git a/contrib/argp-standalone/argp-pvh.c b/contrib/argp-standalone/argp-pvh.c
deleted file mode 100644
index 829a1cda80d..00000000000
--- a/contrib/argp-standalone/argp-pvh.c
+++ /dev/null
@@ -1,32 +0,0 @@
-/* Default definition for ARGP_PROGRAM_VERSION_HOOK.
-   Copyright (C) 1996, 1997, 1999, 2004 Free Software Foundation, Inc.
-   This file is part of the GNU C Library.
-   Written by Miles Bader <miles@gnu.ai.mit.edu>.
-
-   The GNU C Library is free software; you can redistribute it and/or
-   modify it under the terms of the GNU Library General Public License as
-   published by the Free Software Foundation; either version 2 of the
-   License, or (at your option) any later version.
-
-   The GNU C Library is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-   Library General Public License for more details.
-
-   You should have received a copy of the GNU Library General Public
-   License along with the GNU C Library; see the file COPYING.LIB.  If not,
-   write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
-   Boston, MA 02111-1307, USA.  */
-
-#ifdef HAVE_CONFIG_H
-#include <config.h>
-#endif
-
-#include "argp.h"
-
-/* If set by the user program to a non-zero value, then a default option
-   --version is added (unless the ARGP_NO_HELP flag is used), which calls
-   this function with a stream to print the version to and a pointer to the
-   current parsing state, and then exits (unless the ARGP_NO_EXIT flag is
-   used).  This variable takes precedent over ARGP_PROGRAM_VERSION.  */
-void (*argp_program_version_hook) (FILE *stream, struct argp_state *state) = 0;
diff --git a/contrib/argp-standalone/argp.h b/contrib/argp-standalone/argp.h
deleted file mode 100644
index 29d3dfe9720..00000000000
--- a/contrib/argp-standalone/argp.h
+++ /dev/null
@@ -1,602 +0,0 @@
-/* Hierarchial argument parsing.
-   Copyright (C) 1995, 96, 97, 98, 99, 2003 Free Software Foundation, Inc.
-   This file is part of the GNU C Library.
-   Written by Miles Bader <miles@gnu.ai.mit.edu>.
-
-   The GNU C Library is free software; you can redistribute it and/or
-   modify it under the terms of the GNU Library General Public License as
-   published by the Free Software Foundation; either version 2 of the
-   License, or (at your option) any later version.
-
-   The GNU C Library is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-   Library General Public License for more details.
-
-   You should have received a copy of the GNU Library General Public
-   License along with the GNU C Library; see the file COPYING.LIB.  If not,
-   write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
-   Boston, MA 02111-1307, USA.  */
-
-#ifndef _ARGP_H
-#define _ARGP_H
-
-#include <stdio.h>
-#include <ctype.h>
-
-#define __need_error_t
-#include <errno.h>
-
-#ifndef __THROW
-# define __THROW
-#endif
-
-#ifndef __const
-# define __const const
-#endif
-
-#ifndef __error_t_defined
-typedef int error_t;
-# define __error_t_defined
-#endif
-
-/* FIXME: What's the right way to check for __restrict? Sun's cc seems
-   not to have it. Perhaps it's easiest to just delete the use of
-   __restrict from the prototypes. */
-#ifndef __restrict
-# ifndef __GNUC___
-#  define __restrict
-# endif
-#endif
-
-/* NOTE: We can't use the autoconf tests, since this is supposed to be
-   an installed header file and argp's config.h is of course not
-   installed. */
-#ifndef PRINTF_STYLE
-# if __GNUC__ >= 2
-#  define PRINTF_STYLE(f, a) __attribute__ ((__format__ (__printf__, f, a)))
-# else
-#  define PRINTF_STYLE(f, a)
-# endif
-#endif
-
-
-#ifdef  __cplusplus
-extern "C" {
-#endif
-
-/* A description of a particular option.  A pointer to an array of
-   these is passed in the OPTIONS field of an argp structure.  Each option
-   entry can correspond to one long option and/or one short option; more
-   names for the same option can be added by following an entry in an option
-   array with options having the OPTION_ALIAS flag set.  */
-struct argp_option
-{
-  /* The long option name.  For more than one name for the same option, you
-     can use following options with the OPTION_ALIAS flag set.  */
-  __const char *name;
-
-  /* What key is returned for this option.  If > 0 and printable, then it's
-     also accepted as a short option.  */
-  int key;
-
-  /* If non-NULL, this is the name of the argument associated with this
-     option, which is required unless the OPTION_ARG_OPTIONAL flag is set. */
-  __const char *arg;
-
-  /* OPTION_ flags.  */
-  int flags;
-
-  /* The doc string for this option.  If both NAME and KEY are 0, This string
-     will be printed outdented from the normal option column, making it
-     useful as a group header (it will be the first thing printed in its
-     group); in this usage, it's conventional to end the string with a `:'.  */
-  __const char *doc;
-
-  /* The group this option is in.  In a long help message, options are sorted
-     alphabetically within each group, and the groups presented in the order
-     0, 1, 2, ..., n, -m, ..., -2, -1.  Every entry in an options array with
-     if this field 0 will inherit the group number of the previous entry, or
-     zero if it's the first one, unless its a group header (NAME and KEY both
-     0), in which case, the previous entry + 1 is the default.  Automagic
-     options such as --help are put into group -1.  */
-  int group;
-};
-
-/* The argument associated with this option is optional.  */
-#define OPTION_ARG_OPTIONAL	0x1
-
-/* This option isn't displayed in any help messages.  */
-#define OPTION_HIDDEN	       	0x2
-
-/* This option is an alias for the closest previous non-alias option.  This
-   means that it will be displayed in the same help entry, and will inherit
-   fields other than NAME and KEY from the aliased option.  */
-#define OPTION_ALIAS		0x4
-
-/* This option isn't actually an option (and so should be ignored by the
-   actual option parser), but rather an arbitrary piece of documentation that
-   should be displayed in much the same manner as the options.  If this flag
-   is set, then the option NAME field is displayed unmodified (e.g., no `--'
-   prefix is added) at the left-margin (where a *short* option would normally
-   be displayed), and the documentation string in the normal place.  For
-   purposes of sorting, any leading whitespace and puncuation is ignored,
-   except that if the first non-whitespace character is not `-', this entry
-   is displayed after all options (and OPTION_DOC entries with a leading `-')
-   in the same group.  */
-#define OPTION_DOC		0x8
-
-/* This option shouldn't be included in `long' usage messages (but is still
-   included in help messages).  This is mainly intended for options that are
-   completely documented in an argp's ARGS_DOC field, in which case including
-   the option in the generic usage list would be redundant.  For instance,
-   if ARGS_DOC is "FOO BAR\n-x BLAH", and the `-x' option's purpose is to
-   distinguish these two cases, -x should probably be marked
-   OPTION_NO_USAGE.  */
-#define OPTION_NO_USAGE		0x10
-
-struct argp;			/* fwd declare this type */
-struct argp_state;		/* " */
-struct argp_child;		/* " */
-
-/* The type of a pointer to an argp parsing function.  */
-typedef error_t (*argp_parser_t) (int key, char *arg,
-				  struct argp_state *state);
-
-/* What to return for unrecognized keys.  For special ARGP_KEY_ keys, such
-   returns will simply be ignored.  For user keys, this error will be turned
-   into EINVAL (if the call to argp_parse is such that errors are propagated
-   back to the user instead of exiting); returning EINVAL itself would result
-   in an immediate stop to parsing in *all* cases.  */
-#define ARGP_ERR_UNKNOWN	E2BIG /* Hurd should never need E2BIG.  XXX */
-
-/* Special values for the KEY argument to an argument parsing function.
-   ARGP_ERR_UNKNOWN should be returned if they aren't understood.
-
-   The sequence of keys to a parsing function is either (where each
-   uppercased word should be prefixed by `ARGP_KEY_' and opt is a user key):
-
-       INIT opt... NO_ARGS END SUCCESS  -- No non-option arguments at all
-   or  INIT (opt | ARG)... END SUCCESS  -- All non-option args parsed
-   or  INIT (opt | ARG)... SUCCESS      -- Some non-option arg unrecognized
-
-   The third case is where every parser returned ARGP_KEY_UNKNOWN for an
-   argument, in which case parsing stops at that argument (returning the
-   unparsed arguments to the caller of argp_parse if requested, or stopping
-   with an error message if not).
-
-   If an error occurs (either detected by argp, or because the parsing
-   function returned an error value), then the parser is called with
-   ARGP_KEY_ERROR, and no further calls are made.  */
-
-/* This is not an option at all, but rather a command line argument.  If a
-   parser receiving this key returns success, the fact is recorded, and the
-   ARGP_KEY_NO_ARGS case won't be used.  HOWEVER, if while processing the
-   argument, a parser function decrements the NEXT field of the state it's
-   passed, the option won't be considered processed; this is to allow you to
-   actually modify the argument (perhaps into an option), and have it
-   processed again.  */
-#define ARGP_KEY_ARG		0
-/* There are remaining arguments not parsed by any parser, which may be found
-   starting at (STATE->argv + STATE->next).  If success is returned, but
-   STATE->next left untouched, it's assumed that all arguments were consume,
-   otherwise, the parser should adjust STATE->next to reflect any arguments
-   consumed.  */
-#define ARGP_KEY_ARGS		0x1000006
-/* There are no more command line arguments at all.  */
-#define ARGP_KEY_END		0x1000001
-/* Because it's common to want to do some special processing if there aren't
-   any non-option args, user parsers are called with this key if they didn't
-   successfully process any non-option arguments.  Called just before
-   ARGP_KEY_END (where more general validity checks on previously parsed
-   arguments can take place).  */
-#define ARGP_KEY_NO_ARGS	0x1000002
-/* Passed in before any parsing is done.  Afterwards, the values of each
-   element of the CHILD_INPUT field, if any, in the state structure is
-   copied to each child's state to be the initial value of the INPUT field.  */
-#define ARGP_KEY_INIT		0x1000003
-/* Use after all other keys, including SUCCESS & END.  */
-#define ARGP_KEY_FINI		0x1000007
-/* Passed in when parsing has successfully been completed (even if there are
-   still arguments remaining).  */
-#define ARGP_KEY_SUCCESS	0x1000004
-/* Passed in if an error occurs.  */
-#define ARGP_KEY_ERROR		0x1000005
-
-/* An argp structure contains a set of options declarations, a function to
-   deal with parsing one, documentation string, a possible vector of child
-   argp's, and perhaps a function to filter help output.  When actually
-   parsing options, getopt is called with the union of all the argp
-   structures chained together through their CHILD pointers, with conflicts
-   being resolved in favor of the first occurrence in the chain.  */
-struct argp
-{
-  /* An array of argp_option structures, terminated by an entry with both
-     NAME and KEY having a value of 0.  */
-  __const struct argp_option *options;
-
-  /* What to do with an option from this structure.  KEY is the key
-     associated with the option, and ARG is any associated argument (NULL if
-     none was supplied).  If KEY isn't understood, ARGP_ERR_UNKNOWN should be
-     returned.  If a non-zero, non-ARGP_ERR_UNKNOWN value is returned, then
-     parsing is stopped immediately, and that value is returned from
-     argp_parse().  For special (non-user-supplied) values of KEY, see the
-     ARGP_KEY_ definitions below.  */
-  argp_parser_t parser;
-
-  /* A string describing what other arguments are wanted by this program.  It
-     is only used by argp_usage to print the `Usage:' message.  If it
-     contains newlines, the strings separated by them are considered
-     alternative usage patterns, and printed on separate lines (lines after
-     the first are prefix by `  or: ' instead of `Usage:').  */
-  __const char *args_doc;
-
-  /* If non-NULL, a string containing extra text to be printed before and
-     after the options in a long help message (separated by a vertical tab
-     `\v' character).  */
-  __const char *doc;
-
-  /* A vector of argp_children structures, terminated by a member with a 0
-     argp field, pointing to child argps should be parsed with this one.  Any
-     conflicts are resolved in favor of this argp, or early argps in the
-     CHILDREN list.  This field is useful if you use libraries that supply
-     their own argp structure, which you want to use in conjunction with your
-     own.  */
-  __const struct argp_child *children;
-
-  /* If non-zero, this should be a function to filter the output of help
-     messages.  KEY is either a key from an option, in which case TEXT is
-     that option's help text, or a special key from the ARGP_KEY_HELP_
-     defines, below, describing which other help text TEXT is.  The function
-     should return either TEXT, if it should be used as-is, a replacement
-     string, which should be malloced, and will be freed by argp, or NULL,
-     meaning `print nothing'.  The value for TEXT is *after* any translation
-     has been done, so if any of the replacement text also needs translation,
-     that should be done by the filter function.  INPUT is either the input
-     supplied to argp_parse, or NULL, if argp_help was called directly.  */
-  char *(*help_filter) (int __key, __const char *__text, void *__input);
-
-  /* If non-zero the strings used in the argp library are translated using
-     the domain described by this string.  Otherwise the currently installed
-     default domain is used.  */
-  const char *argp_domain;
-};
-
-/* Possible KEY arguments to a help filter function.  */
-#define ARGP_KEY_HELP_PRE_DOC	0x2000001 /* Help text preceeding options. */
-#define ARGP_KEY_HELP_POST_DOC	0x2000002 /* Help text following options. */
-#define ARGP_KEY_HELP_HEADER	0x2000003 /* Option header string. */
-#define ARGP_KEY_HELP_EXTRA	0x2000004 /* After all other documentation;
-					     TEXT is NULL for this key.  */
-/* Explanatory note emitted when duplicate option arguments have been
-   suppressed.  */
-#define ARGP_KEY_HELP_DUP_ARGS_NOTE 0x2000005
-#define ARGP_KEY_HELP_ARGS_DOC	0x2000006 /* Argument doc string.  */
-
-/* When an argp has a non-zero CHILDREN field, it should point to a vector of
-   argp_child structures, each of which describes a subsidiary argp.  */
-struct argp_child
-{
-  /* The child parser.  */
-  __const struct argp *argp;
-
-  /* Flags for this child.  */
-  int flags;
-
-  /* If non-zero, an optional header to be printed in help output before the
-     child options.  As a side-effect, a non-zero value forces the child
-     options to be grouped together; to achieve this effect without actually
-     printing a header string, use a value of "".  */
-  __const char *header;
-
-  /* Where to group the child options relative to the other (`consolidated')
-     options in the parent argp; the values are the same as the GROUP field
-     in argp_option structs, but all child-groupings follow parent options at
-     a particular group level.  If both this field and HEADER are zero, then
-     they aren't grouped at all, but rather merged with the parent options
-     (merging the child's grouping levels with the parents).  */
-  int group;
-};
-
-/* Parsing state.  This is provided to parsing functions called by argp,
-   which may examine and, as noted, modify fields.  */
-struct argp_state
-{
-  /* The top level ARGP being parsed.  */
-  __const struct argp *root_argp;
-
-  /* The argument vector being parsed.  May be modified.  */
-  int argc;
-  char **argv;
-
-  /* The index in ARGV of the next arg that to be parsed.  May be modified. */
-  int next;
-
-  /* The flags supplied to argp_parse.  May be modified.  */
-  unsigned flags;
-
-  /* While calling a parsing function with a key of ARGP_KEY_ARG, this is the
-     number of the current arg, starting at zero, and incremented after each
-     such call returns.  At all other times, this is the number of such
-     arguments that have been processed.  */
-  unsigned arg_num;
-
-  /* If non-zero, the index in ARGV of the first argument following a special
-     `--' argument (which prevents anything following being interpreted as an
-     option).  Only set once argument parsing has proceeded past this point. */
-  int quoted;
-
-  /* An arbitrary pointer passed in from the user.  */
-  void *input;
-  /* Values to pass to child parsers.  This vector will be the same length as
-     the number of children for the current parser.  */
-  void **child_inputs;
-
-  /* For the parser's use.  Initialized to 0.  */
-  void *hook;
-
-  /* The name used when printing messages.  This is initialized to ARGV[0],
-     or PROGRAM_INVOCATION_NAME if that is unavailable.  */
-  char *name;
-
-  /* Streams used when argp prints something.  */
-  FILE *err_stream;		/* For errors; initialized to stderr. */
-  FILE *out_stream;		/* For information; initialized to stdout. */
-
-  void *pstate;			/* Private, for use by argp.  */
-};
-
-/* Flags for argp_parse (note that the defaults are those that are
-   convenient for program command line parsing): */
-
-/* Don't ignore the first element of ARGV.  Normally (and always unless
-   ARGP_NO_ERRS is set) the first element of the argument vector is
-   skipped for option parsing purposes, as it corresponds to the program name
-   in a command line.  */
-#define ARGP_PARSE_ARGV0  0x01
-
-/* Don't print error messages for unknown options to stderr; unless this flag
-   is set, ARGP_PARSE_ARGV0 is ignored, as ARGV[0] is used as the program
-   name in the error messages.  This flag implies ARGP_NO_EXIT (on the
-   assumption that silent exiting upon errors is bad behaviour).  */
-#define ARGP_NO_ERRS	0x02
-
-/* Don't parse any non-option args.  Normally non-option args are parsed by
-   calling the parse functions with a key of ARGP_KEY_ARG, and the actual arg
-   as the value.  Since it's impossible to know which parse function wants to
-   handle it, each one is called in turn, until one returns 0 or an error
-   other than ARGP_ERR_UNKNOWN; if an argument is handled by no one, the
-   argp_parse returns prematurely (but with a return value of 0).  If all
-   args have been parsed without error, all parsing functions are called one
-   last time with a key of ARGP_KEY_END.  This flag needn't normally be set,
-   as the normal behavior is to stop parsing as soon as some argument can't
-   be handled.  */
-#define ARGP_NO_ARGS	0x04
-
-/* Parse options and arguments in the same order they occur on the command
-   line -- normally they're rearranged so that all options come first. */
-#define ARGP_IN_ORDER	0x08
-
-/* Don't provide the standard long option --help, which causes usage and
-      option help information to be output to stdout, and exit (0) called. */
-#define ARGP_NO_HELP	0x10
-
-/* Don't exit on errors (they may still result in error messages).  */
-#define ARGP_NO_EXIT	0x20
-
-/* Use the gnu getopt `long-only' rules for parsing arguments.  */
-#define ARGP_LONG_ONLY	0x40
-
-/* Turns off any message-printing/exiting options.  */
-#define ARGP_SILENT    (ARGP_NO_EXIT | ARGP_NO_ERRS | ARGP_NO_HELP)
-
-/* Parse the options strings in ARGC & ARGV according to the options in ARGP.
-   FLAGS is one of the ARGP_ flags above.  If ARG_INDEX is non-NULL, the
-   index in ARGV of the first unparsed option is returned in it.  If an
-   unknown option is present, ARGP_ERR_UNKNOWN is returned; if some parser
-   routine returned a non-zero value, it is returned; otherwise 0 is
-   returned.  This function may also call exit unless the ARGP_NO_HELP flag
-   is set.  INPUT is a pointer to a value to be passed in to the parser.  */
-extern error_t argp_parse (__const struct argp *__restrict argp,
-			   int argc, char **__restrict argv,
-			   unsigned flags, int *__restrict arg_index,
-			   void *__restrict input) __THROW;
-extern error_t __argp_parse (__const struct argp *__restrict argp,
-			     int argc, char **__restrict argv,
-			     unsigned flags, int *__restrict arg_index,
-			     void *__restrict input) __THROW;
-
-/* Global variables.  */
-
-/* If defined or set by the user program to a non-zero value, then a default
-   option --version is added (unless the ARGP_NO_HELP flag is used), which
-   will print this string followed by a newline and exit (unless the
-   ARGP_NO_EXIT flag is used).  Overridden by ARGP_PROGRAM_VERSION_HOOK.  */
-extern __const char *argp_program_version;
-
-/* If defined or set by the user program to a non-zero value, then a default
-   option --version is added (unless the ARGP_NO_HELP flag is used), which
-   calls this function with a stream to print the version to and a pointer to
-   the current parsing state, and then exits (unless the ARGP_NO_EXIT flag is
-   used).  This variable takes precedent over ARGP_PROGRAM_VERSION.  */
-extern void (*argp_program_version_hook) (FILE *__restrict __stream,
-					  struct argp_state *__restrict
-					  __state);
-
-/* If defined or set by the user program, it should point to string that is
-   the bug-reporting address for the program.  It will be printed by
-   argp_help if the ARGP_HELP_BUG_ADDR flag is set (as it is by various
-   standard help messages), embedded in a sentence that says something like
-   `Report bugs to ADDR.'.  */
-extern __const char *argp_program_bug_address;
-
-/* The exit status that argp will use when exiting due to a parsing error.
-   If not defined or set by the user program, this defaults to EX_USAGE from
-   <sysexits.h>.  */
-extern error_t argp_err_exit_status;
-
-/* Flags for argp_help.  */
-#define ARGP_HELP_USAGE		0x01 /* a Usage: message. */
-#define ARGP_HELP_SHORT_USAGE	0x02 /*  " but don't actually print options. */
-#define ARGP_HELP_SEE		0x04 /* a `Try ... for more help' message. */
-#define ARGP_HELP_LONG		0x08 /* a long help message. */
-#define ARGP_HELP_PRE_DOC	0x10 /* doc string preceding long help.  */
-#define ARGP_HELP_POST_DOC	0x20 /* doc string following long help.  */
-#define ARGP_HELP_DOC		(ARGP_HELP_PRE_DOC | ARGP_HELP_POST_DOC)
-#define ARGP_HELP_BUG_ADDR	0x40 /* bug report address */
-#define ARGP_HELP_LONG_ONLY	0x80 /* modify output appropriately to
-					reflect ARGP_LONG_ONLY mode.  */
-
-/* These ARGP_HELP flags are only understood by argp_state_help.  */
-#define ARGP_HELP_EXIT_ERR	0x100 /* Call exit(1) instead of returning.  */
-#define ARGP_HELP_EXIT_OK	0x200 /* Call exit(0) instead of returning.  */
-
-/* The standard thing to do after a program command line parsing error, if an
-   error message has already been printed.  */
-#define ARGP_HELP_STD_ERR \
-  (ARGP_HELP_SEE | ARGP_HELP_EXIT_ERR)
-/* The standard thing to do after a program command line parsing error, if no
-   more specific error message has been printed.  */
-#define ARGP_HELP_STD_USAGE \
-  (ARGP_HELP_SHORT_USAGE | ARGP_HELP_SEE | ARGP_HELP_EXIT_ERR)
-/* The standard thing to do in response to a --help option.  */
-#define ARGP_HELP_STD_HELP \
-  (ARGP_HELP_SHORT_USAGE | ARGP_HELP_LONG | ARGP_HELP_EXIT_OK \
-   | ARGP_HELP_DOC | ARGP_HELP_BUG_ADDR)
-
-/* Output a usage message for ARGP to STREAM.  FLAGS are from the set
-   ARGP_HELP_*.  */
-extern void argp_help (__const struct argp *__restrict __argp,
-		       FILE *__restrict __stream,
-		       unsigned __flags, char *__restrict __name) __THROW;
-extern void __argp_help (__const struct argp *__restrict __argp,
-			 FILE *__restrict __stream, unsigned __flags,
-			 char *__name) __THROW;
-
-/* The following routines are intended to be called from within an argp
-   parsing routine (thus taking an argp_state structure as the first
-   argument).  They may or may not print an error message and exit, depending
-   on the flags in STATE -- in any case, the caller should be prepared for
-   them *not* to exit, and should return an appropiate error after calling
-   them.  [argp_usage & argp_error should probably be called argp_state_...,
-   but they're used often enough that they should be short]  */
-
-/* Output, if appropriate, a usage message for STATE to STREAM.  FLAGS are
-   from the set ARGP_HELP_*.  */
-extern void argp_state_help (__const struct argp_state *__restrict __state,
-			     FILE *__restrict __stream,
-			     unsigned int __flags) __THROW;
-extern void __argp_state_help (__const struct argp_state *__restrict __state,
-			       FILE *__restrict __stream,
-			       unsigned int __flags) __THROW;
-
-/* Possibly output the standard usage message for ARGP to stderr and exit.  */
-extern void argp_usage (__const struct argp_state *__state) __THROW;
-extern void __argp_usage (__const struct argp_state *__state) __THROW;
-
-/* If appropriate, print the printf string FMT and following args, preceded
-   by the program name and `:', to stderr, and followed by a `Try ... --help'
-   message, then exit (1).  */
-extern void argp_error (__const struct argp_state *__restrict __state,
-			__const char *__restrict __fmt, ...) __THROW
-     PRINTF_STYLE(2,3);
-extern void __argp_error (__const struct argp_state *__restrict __state,
-			  __const char *__restrict __fmt, ...) __THROW
-     PRINTF_STYLE(2,3);
-
-/* Similar to the standard gnu error-reporting function error(), but will
-   respect the ARGP_NO_EXIT and ARGP_NO_ERRS flags in STATE, and will print
-   to STATE->err_stream.  This is useful for argument parsing code that is
-   shared between program startup (when exiting is desired) and runtime
-   option parsing (when typically an error code is returned instead).  The
-   difference between this function and argp_error is that the latter is for
-   *parsing errors*, and the former is for other problems that occur during
-   parsing but don't reflect a (syntactic) problem with the input.  */
-extern void argp_failure (__const struct argp_state *__restrict __state,
-			  int __status, int __errnum,
-			  __const char *__restrict __fmt, ...) __THROW
-     PRINTF_STYLE(4,5);
-extern void __argp_failure (__const struct argp_state *__restrict __state,
-			    int __status, int __errnum,
-			    __const char *__restrict __fmt, ...) __THROW
-     PRINTF_STYLE(4,5);
-
-/* Returns true if the option OPT is a valid short option.  */
-extern int _option_is_short (__const struct argp_option *__opt) __THROW;
-extern int __option_is_short (__const struct argp_option *__opt) __THROW;
-
-/* Returns true if the option OPT is in fact the last (unused) entry in an
-   options array.  */
-extern int _option_is_end (__const struct argp_option *__opt) __THROW;
-extern int __option_is_end (__const struct argp_option *__opt) __THROW;
-
-/* Return the input field for ARGP in the parser corresponding to STATE; used
-   by the help routines.  */
-extern void *_argp_input (__const struct argp *__restrict __argp,
-			  __const struct argp_state *__restrict __state)
-     __THROW;
-extern void *__argp_input (__const struct argp *__restrict __argp,
-			   __const struct argp_state *__restrict __state)
-     __THROW;
-
-/* Used for extracting the program name from argv[0] */
-extern char *_argp_basename(char *name) __THROW;
-extern char *__argp_basename(char *name) __THROW;
-
-/* Getting the program name given an argp state */
-extern char *
-_argp_short_program_name(const struct argp_state *state) __THROW;
-extern char *
-__argp_short_program_name(const struct argp_state *state) __THROW;
-
-
-#ifdef __USE_EXTERN_INLINES
-
-# if !_LIBC
-#  define __argp_usage argp_usage
-#  define __argp_state_help argp_state_help
-#  define __option_is_short _option_is_short
-#  define __option_is_end _option_is_end
-# endif
-
-# ifndef ARGP_EI
-#  define ARGP_EI extern __inline__
-# endif
-
-ARGP_EI void
-__argp_usage (__const struct argp_state *__state)
-{
-  __argp_state_help (__state, stderr, ARGP_HELP_STD_USAGE);
-}
-
-ARGP_EI int
-__option_is_short (__const struct argp_option *__opt)
-{
-  if (__opt->flags & OPTION_DOC)
-    return 0;
-  else
-    {
-      int __key = __opt->key;
-      return __key > 0 && isprint (__key);
-    }
-}
-
-ARGP_EI int
-__option_is_end (__const struct argp_option *__opt)
-{
-  return !__opt->key && !__opt->name && !__opt->doc && !__opt->group;
-}
-
-# if !_LIBC
-#  undef __argp_usage
-#  undef __argp_state_help
-#  undef __option_is_short
-#  undef __option_is_end
-# endif
-#endif /* Use extern inlines.  */
-
-#ifdef  __cplusplus
-}
-#endif
-
-#endif /* argp.h */
diff --git a/contrib/argp-standalone/autogen.sh b/contrib/argp-standalone/autogen.sh
deleted file mode 100755
index 8337353b5ae..00000000000
--- a/contrib/argp-standalone/autogen.sh
+++ /dev/null
@@ -1,6 +0,0 @@
-#!/bin/sh
-
-aclocal -I .
-autoheader
-autoconf
-automake --add-missing --copy --foreign
diff --git a/contrib/argp-standalone/configure.ac b/contrib/argp-standalone/configure.ac
deleted file mode 100644
index c0867eb5b35..00000000000
--- a/contrib/argp-standalone/configure.ac
+++ /dev/null
@@ -1,105 +0,0 @@
-dnl Process this file with autoconf to produce a configure script.
-
-dnl This configure.ac is only for building a standalone argp library.
-AC_INIT([argp], [standalone-1.3])
-AC_PREREQ(2.54)
-AC_CONFIG_SRCDIR([argp-ba.c])
-# Needed to stop autoconf from looking for files in parent directories.
-AC_CONFIG_AUX_DIR([.])
-
-AM_INIT_AUTOMAKE
-AC_CONFIG_HEADERS(config.h)
-
-m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES(yes)])
-
-# GNU libc defaults to supplying the ISO C library functions only. The
-# _GNU_SOURCE define enables these extensions, in particular we want
-# errno.h to declare program_invocation_name. Enable it on all
-# systems; no problems have been reported with it so far.
-AC_GNU_SOURCE
-
-# Checks for programs.
-AC_PROG_CC
-AC_PROG_MAKE_SET
-AC_PROG_RANLIB
-AC_PROG_CC
-
-if test "x$am_cv_prog_cc_stdc" = xno ; then
-  AC_ERROR([the C compiler doesn't handle ANSI-C])
-fi
-
-# Checks for libraries.
-
-# Checks for header files.
-AC_HEADER_STDC
-AC_CHECK_HEADERS(libintl.h limits.h malloc.h unistd.h sysexits.h stdarg.h)
-
-# Checks for typedefs, structures, and compiler characteristics.
-AC_C_CONST
-AC_C_INLINE
-AC_TYPE_SIZE_T
-
-LSH_GCC_ATTRIBUTES
-
-# Checks for library functions.
-AC_FUNC_ALLOCA
-AC_FUNC_VPRINTF
-AC_CHECK_FUNCS(strerror sleep getpid snprintf)
-
-AC_REPLACE_FUNCS(mempcpy strndup strchrnul strcasecmp vsnprintf)
-
-dnl ARGP_CHECK_FUNC(includes, function-call [, if-found [, if-not-found]])
-AC_DEFUN([ARGP_CHECK_FUNC],
-  [AS_VAR_PUSHDEF([ac_func], m4_substr([$2], 0, m4_index([$2], [(])))
-   AS_VAR_PUSHDEF([ac_var], [ac_cv_func_call_]ac_func)
-   AH_TEMPLATE(AS_TR_CPP(HAVE_[]ac_func),
-               [Define to 1 if you have the `]ac_func[' function.])
-   AC_CACHE_CHECK([for $2], ac_var,
-     [AC_TRY_LINK([$1], [$2],
-                  [AS_VAR_SET(ac_var, yes)],
-                  [AS_VAR_SET(ac_var, no)])])
-   if test AS_VAR_GET(ac_var) = yes ; then
-     ifelse([$3],,
-            [AC_DEFINE_UNQUOTED(AS_TR_CPP(HAVE_[]ac_func))],
-            [$3
-])
-   else
-     ifelse([$4],, true, [$4])
-   fi
-   AS_VAR_POPDEF([ac_var])
-   AS_VAR_POPDEF([ac_func])
-   ])
-
-# At least on freebsd, putc_unlocked is a macro, so the standard
-# AC_CHECK_FUNCS doesn't work well.
-ARGP_CHECK_FUNC([#include <stdio.h>], [putc_unlocked('x', stdout)])
-
-AC_CHECK_FUNCS(flockfile)
-AC_CHECK_FUNCS(fputs_unlocked fwrite_unlocked)
-
-# Used only by argp-test.c, so don't use AC_REPLACE_FUNCS.
-AC_CHECK_FUNCS(strdup asprintf)
-
-AC_CHECK_DECLS([program_invocation_name, program_invocation_short_name],
-               [], [], [[#include <errno.h>]])
-
-# Set these flags *last*, or else the test programs won't compile
-if test x$GCC = xyes ; then
-  # Using -ggdb3 makes (some versions of) Redhat's gcc-2.96 dump core
-  if "$CC" --version | grep '^2\.96$' 1>/dev/null 2>&1; then
-    true
-  else
-    CFLAGS="$CFLAGS -ggdb3"
-  fi
-  CFLAGS="$CFLAGS -Wall -W \
- -Wmissing-prototypes -Wmissing-declarations -Wstrict-prototypes \
- -Waggregate-return \
- -Wpointer-arith -Wbad-function-cast -Wnested-externs"
-fi
-
-CPPFLAGS="$CPPFLAGS -I$srcdir"
-
-dnl Added for C99 standards
-CFLAGS="$CFLAGS -std=gnu89 -static"
-
-AC_OUTPUT(Makefile)
diff --git a/contrib/argp-standalone/mempcpy.c b/contrib/argp-standalone/mempcpy.c
deleted file mode 100644
index 21d8bd2ed94..00000000000
--- a/contrib/argp-standalone/mempcpy.c
+++ /dev/null
@@ -1,21 +0,0 @@
-/* strndup.c
- *
- */
-
-/* Written by Niels M�ller <nisse@lysator.liu.se>
- *
- * This file is hereby placed in the public domain.
- */
-
-#include <string.h>
-
-void *
-mempcpy (void *, const void *, size_t) ;
-
-void *
-mempcpy (void *to, const void *from, size_t size)
-{
-  memcpy(to, from, size);
-  return (char *) to + size;
-}
-
diff --git a/contrib/argp-standalone/strcasecmp.c b/contrib/argp-standalone/strcasecmp.c
deleted file mode 100644
index 9c1637232fd..00000000000
--- a/contrib/argp-standalone/strcasecmp.c
+++ /dev/null
@@ -1,29 +0,0 @@
-/* strcasecmp.c
- *
- */
-
-/* Written by Niels M�ller <nisse@lysator.liu.se>
- *
- * This file is hereby placed in the public domain.
- */
-
-#include <ctype.h>
-int strcasecmp(const char *, const char *);
-
-int strcasecmp(const char *s1, const char *s2)
-{
-  unsigned i;
-  
-  for (i = 0; s1[i] && s2[i]; i++)
-    {
-      unsigned char c1 = tolower( (unsigned char) s1[i]);
-      unsigned char c2 = tolower( (unsigned char) s2[i]);
-
-      if (c1 < c2)
-	return -1;
-      else if (c1 > c2)
-	return 1;
-    }
-
-  return !s2[i] - !s1[i];	
-}
diff --git a/contrib/argp-standalone/strchrnul.c b/contrib/argp-standalone/strchrnul.c
deleted file mode 100644
index ee4145e4eda..00000000000
--- a/contrib/argp-standalone/strchrnul.c
+++ /dev/null
@@ -1,23 +0,0 @@
-/* strchrnul.c
- *
- */
-
-/* Written by Niels M�ller <nisse@lysator.liu.se>
- *
- * This file is hereby placed in the public domain.
- */
-
-/* FIXME: What is this function supposed to do? My guess is that it is
- * like strchr, but returns a pointer to the NUL character, not a NULL
- * pointer, if the character isn't found. */
-
-char *strchrnul(const char *, int );
-
-char *strchrnul(const char *s, int c)
-{
-  const char *p = s;
-  while (*p && (*p != c))
-    p++;
-
-  return (char *) p;
-}
diff --git a/contrib/argp-standalone/strndup.c b/contrib/argp-standalone/strndup.c
deleted file mode 100644
index 4147b7a2051..00000000000
--- a/contrib/argp-standalone/strndup.c
+++ /dev/null
@@ -1,34 +0,0 @@
-/* strndup.c
- *
- */
-
-/* Written by Niels M�ller <nisse@lysator.liu.se>
- *
- * This file is hereby placed in the public domain.
- */
-
-#include <stdlib.h>
-#include <string.h>
-
-char *
-strndup (const char *, size_t);
-
-char *
-strndup (const char *s, size_t size)
-{
-  char *r;
-  char *end = memchr(s, 0, size);
-  
-  if (end)
-    /* Length + 1 */
-    size = end - s + 1;
-  
-  r = malloc(size);
-
-  if (size)
-    {
-      memcpy(r, s, size-1);
-      r[size-1] = '\0';
-    }
-  return r;
-}
diff --git a/contrib/argp-standalone/vsnprintf.c b/contrib/argp-standalone/vsnprintf.c
deleted file mode 100644
index 33c9a5d0042..00000000000
--- a/contrib/argp-standalone/vsnprintf.c
+++ /dev/null
@@ -1,839 +0,0 @@
-/* Copied from http://www.fiction.net/blong/programs/snprintf.c */
-
-/*
- * Copyright Patrick Powell 1995
- * This code is based on code written by Patrick Powell (papowell@astart.com)
- * It may be used for any purpose as long as this notice remains intact
- * on all source code distributions
- */
-
-/**************************************************************
- * Original:
- * Patrick Powell Tue Apr 11 09:48:21 PDT 1995
- * A bombproof version of doprnt (dopr) included.
- * Sigh.  This sort of thing is always nasty do deal with.  Note that
- * the version here does not include floating point...
- *
- * snprintf() is used instead of sprintf() as it does limit checks
- * for string length.  This covers a nasty loophole.
- *
- * The other functions are there to prevent NULL pointers from
- * causing nast effects.
- *
- * More Recently:
- *  Brandon Long <blong@fiction.net> 9/15/96 for mutt 0.43
- *  This was ugly.  It is still ugly.  I opted out of floating point
- *  numbers, but the formatter understands just about everything
- *  from the normal C string format, at least as far as I can tell from
- *  the Solaris 2.5 printf(3S) man page.
- *
- *  Brandon Long <blong@fiction.net> 10/22/97 for mutt 0.87.1
- *    Ok, added some minimal floating point support, which means this
- *    probably requires libm on most operating systems.  Don't yet
- *    support the exponent (e,E) and sigfig (g,G).  Also, fmtint()
- *    was pretty badly broken, it just wasn't being exercised in ways
- *    which showed it, so that's been fixed.  Also, formated the code
- *    to mutt conventions, and removed dead code left over from the
- *    original.  Also, there is now a builtin-test, just compile with:
- *           gcc -DTEST_SNPRINTF -o snprintf snprintf.c -lm
- *    and run snprintf for results.
- * 
- *  Thomas Roessler <roessler@guug.de> 01/27/98 for mutt 0.89i
- *    The PGP code was using unsigned hexadecimal formats. 
- *    Unfortunately, unsigned formats simply didn't work.
- *
- *  Michael Elkins <me@cs.hmc.edu> 03/05/98 for mutt 0.90.8
- *    The original code assumed that both snprintf() and vsnprintf() were
- *    missing.  Some systems only have snprintf() but not vsnprintf(), so
- *    the code is now broken down under HAVE_SNPRINTF and HAVE_VSNPRINTF.
- *
- *  Andrew Tridgell (tridge@samba.org) Oct 1998
- *    fixed handling of %.0f
- *    added test for HAVE_LONG_DOUBLE
- *
- *  Russ Allbery <rra@stanford.edu> 2000-08-26
- *    fixed return value to comply with C99
- *    fixed handling of snprintf(NULL, ...)
- *
- * Niels M�ller <nisse@lysator.liu.se> 2004-03-05
- *    fixed calls to isdigit to use unsigned char.
- *    fixed calls to va_arg; short arguments are always passed as int.
- *
- **************************************************************/
-
-#if HAVE_CONFIG_H
-# include "config.h"
-#endif
-
-#if !defined(HAVE_SNPRINTF) || !defined(HAVE_VSNPRINTF)
-
-#include <string.h>
-#include <ctype.h>
-#include <sys/types.h>
-
-/* Define this as a fall through, HAVE_STDARG_H is probably already set */
-
-#define HAVE_VARARGS_H
-
-
-/* varargs declarations: */
-
-#if defined(HAVE_STDARG_H)
-# include <stdarg.h>
-# define HAVE_STDARGS    /* let's hope that works everywhere (mj) */
-# define VA_LOCAL_DECL   va_list ap
-# define VA_START(f)     va_start(ap, f)
-# define VA_SHIFT(v,t)  ;   /* no-op for ANSI */
-# define VA_END          va_end(ap)
-#else
-# if defined(HAVE_VARARGS_H)
-#  include <varargs.h>
-#  undef HAVE_STDARGS
-#  define VA_LOCAL_DECL   va_list ap
-#  define VA_START(f)     va_start(ap)      /* f is ignored! */
-#  define VA_SHIFT(v,t) v = va_arg(ap,t)
-#  define VA_END        va_end(ap)
-# else
-/*XX ** NO VARARGS ** XX*/
-# endif
-#endif
-
-#ifdef HAVE_LONG_DOUBLE
-#define LDOUBLE long double
-#else
-#define LDOUBLE double
-#endif
-
-int snprintf (char *str, size_t count, const char *fmt, ...);
-int vsnprintf (char *str, size_t count, const char *fmt, va_list arg);
-
-static int dopr (char *buffer, size_t maxlen, const char *format, 
-                 va_list args);
-static int fmtstr (char *buffer, size_t *currlen, size_t maxlen,
-		   char *value, int flags, int min, int max);
-static int fmtint (char *buffer, size_t *currlen, size_t maxlen,
-		   long value, int base, int min, int max, int flags);
-static int fmtfp (char *buffer, size_t *currlen, size_t maxlen,
-		  LDOUBLE fvalue, int min, int max, int flags);
-static int dopr_outch (char *buffer, size_t *currlen, size_t maxlen, char c );
-
-/*
- * dopr(): poor man's version of doprintf
- */
-
-/* format read states */
-#define DP_S_DEFAULT 0
-#define DP_S_FLAGS   1
-#define DP_S_MIN     2
-#define DP_S_DOT     3
-#define DP_S_MAX     4
-#define DP_S_MOD     5
-#define DP_S_CONV    6
-#define DP_S_DONE    7
-
-/* format flags - Bits */
-#define DP_F_MINUS 	(1 << 0)
-#define DP_F_PLUS  	(1 << 1)
-#define DP_F_SPACE 	(1 << 2)
-#define DP_F_NUM   	(1 << 3)
-#define DP_F_ZERO  	(1 << 4)
-#define DP_F_UP    	(1 << 5)
-#define DP_F_UNSIGNED 	(1 << 6)
-
-/* Conversion Flags */
-#define DP_C_SHORT   1
-#define DP_C_LONG    2
-#define DP_C_LDOUBLE 3
-
-#define char_to_int(p) (p - '0')
-#define MAX(p,q) ((p >= q) ? p : q)
-#define MIN(p,q) ((p <= q) ? p : q)
-
-static int dopr (char *buffer, size_t maxlen, const char *format, va_list args)
-{
-  unsigned char ch;
-  long value;
-  LDOUBLE fvalue;
-  char *strvalue;
-  int min;
-  int max;
-  int state;
-  int flags;
-  int cflags;
-  int total;
-  size_t currlen;
-  
-  state = DP_S_DEFAULT;
-  currlen = flags = cflags = min = 0;
-  max = -1;
-  ch = *format++;
-  total = 0;
-
-  while (state != DP_S_DONE)
-  {
-    if (ch == '\0')
-      state = DP_S_DONE;
-
-    switch(state) 
-    {
-    case DP_S_DEFAULT:
-      if (ch == '%') 
-	state = DP_S_FLAGS;
-      else 
-	total += dopr_outch (buffer, &currlen, maxlen, ch);
-      ch = *format++;
-      break;
-    case DP_S_FLAGS:
-      switch (ch) 
-      {
-      case '-':
-	flags |= DP_F_MINUS;
-        ch = *format++;
-	break;
-      case '+':
-	flags |= DP_F_PLUS;
-        ch = *format++;
-	break;
-      case ' ':
-	flags |= DP_F_SPACE;
-        ch = *format++;
-	break;
-      case '#':
-	flags |= DP_F_NUM;
-        ch = *format++;
-	break;
-      case '0':
-	flags |= DP_F_ZERO;
-        ch = *format++;
-	break;
-      default:
-	state = DP_S_MIN;
-	break;
-      }
-      break;
-    case DP_S_MIN:
-      if (isdigit(ch)) 
-      {
-	min = 10*min + char_to_int (ch);
-	ch = *format++;
-      } 
-      else if (ch == '*') 
-      {
-	min = va_arg (args, int);
-	ch = *format++;
-	state = DP_S_DOT;
-      } 
-      else 
-	state = DP_S_DOT;
-      break;
-    case DP_S_DOT:
-      if (ch == '.') 
-      {
-	state = DP_S_MAX;
-	ch = *format++;
-      } 
-      else 
-	state = DP_S_MOD;
-      break;
-    case DP_S_MAX:
-      if (isdigit(ch)) 
-      {
-	if (max < 0)
-	  max = 0;
-	max = 10*max + char_to_int (ch);
-	ch = *format++;
-      } 
-      else if (ch == '*') 
-      {
-	max = va_arg (args, int);
-	ch = *format++;
-	state = DP_S_MOD;
-      } 
-      else 
-	state = DP_S_MOD;
-      break;
-    case DP_S_MOD:
-      /* Currently, we don't support Long Long, bummer */
-      switch (ch) 
-      {
-      case 'h':
-	cflags = DP_C_SHORT;
-	ch = *format++;
-	break;
-      case 'l':
-	cflags = DP_C_LONG;
-	ch = *format++;
-	break;
-      case 'L':
-	cflags = DP_C_LDOUBLE;
-	ch = *format++;
-	break;
-      default:
-	break;
-      }
-      state = DP_S_CONV;
-      break;
-    case DP_S_CONV:
-      switch (ch) 
-      {
-      case 'd':
-      case 'i':
-	if (cflags == DP_C_SHORT) 
-	  value = (short) va_arg (args, int);
-	else if (cflags == DP_C_LONG)
-	  value = va_arg (args, long int);
-	else
-	  value = va_arg (args, int);
-	total += fmtint (buffer, &currlen, maxlen, value, 10, min, max, flags);
-	break;
-      case 'o':
-	flags |= DP_F_UNSIGNED;
-	if (cflags == DP_C_SHORT)
-	  value = (unsigned short) va_arg (args, unsigned);
-	else if (cflags == DP_C_LONG)
-	  value = va_arg (args, unsigned long int);
-	else
-	  value = va_arg (args, unsigned int);
-	total += fmtint (buffer, &currlen, maxlen, value, 8, min, max, flags);
-	break;
-      case 'u':
-	flags |= DP_F_UNSIGNED;
-	if (cflags == DP_C_SHORT)
-	  value = (unsigned short) va_arg (args, unsigned);
-	else if (cflags == DP_C_LONG)
-	  value = va_arg (args, unsigned long int);
-	else
-	  value = va_arg (args, unsigned int);
-	total += fmtint (buffer, &currlen, maxlen, value, 10, min, max, flags);
-	break;
-      case 'X':
-	flags |= DP_F_UP;
-      case 'x':
-	flags |= DP_F_UNSIGNED;
-	if (cflags == DP_C_SHORT)
-	  value = (unsigned short) va_arg (args, unsigned);
-	else if (cflags == DP_C_LONG)
-	  value = va_arg (args, unsigned long int);
-	else
-	  value = va_arg (args, unsigned int);
-	total += fmtint (buffer, &currlen, maxlen, value, 16, min, max, flags);
-	break;
-      case 'f':
-	if (cflags == DP_C_LDOUBLE)
-	  fvalue = va_arg (args, LDOUBLE);
-	else
-	  fvalue = va_arg (args, double);
-	/* um, floating point? */
-	total += fmtfp (buffer, &currlen, maxlen, fvalue, min, max, flags);
-	break;
-      case 'E':
-	flags |= DP_F_UP;
-      case 'e':
-	if (cflags == DP_C_LDOUBLE)
-	  fvalue = va_arg (args, LDOUBLE);
-	else
-	  fvalue = va_arg (args, double);
-	break;
-      case 'G':
-	flags |= DP_F_UP;
-      case 'g':
-	if (cflags == DP_C_LDOUBLE)
-	  fvalue = va_arg (args, LDOUBLE);
-	else
-	  fvalue = va_arg (args, double);
-	break;
-      case 'c':
-	total += dopr_outch (buffer, &currlen, maxlen, va_arg (args, int));
-	break;
-      case 's':
-	strvalue = va_arg (args, char *);
-	total += fmtstr (buffer, &currlen, maxlen, strvalue, flags, min, max);
-	break;
-      case 'p':
-	strvalue = va_arg (args, void *);
-	total += fmtint (buffer, &currlen, maxlen, (long) strvalue, 16, min,
-                         max, flags);
-	break;
-      case 'n':
-	if (cflags == DP_C_SHORT) 
-	{
-	  short int *num;
-	  num = va_arg (args, short int *);
-	  *num = currlen;
-        } 
-	else if (cflags == DP_C_LONG) 
-	{
-	  long int *num;
-	  num = va_arg (args, long int *);
-	  *num = currlen;
-        } 
-	else 
-	{
-	  int *num;
-	  num = va_arg (args, int *);
-	  *num = currlen;
-        }
-	break;
-      case '%':
-	total += dopr_outch (buffer, &currlen, maxlen, ch);
-	break;
-      case 'w':
-	/* not supported yet, treat as next char */
-	ch = *format++;
-	break;
-      default:
-	/* Unknown, skip */
-	break;
-      }
-      ch = *format++;
-      state = DP_S_DEFAULT;
-      flags = cflags = min = 0;
-      max = -1;
-      break;
-    case DP_S_DONE:
-      break;
-    default:
-      /* hmm? */
-      break; /* some picky compilers need this */
-    }
-  }
-  if (buffer != NULL)
-  {
-    if (currlen < maxlen - 1) 
-      buffer[currlen] = '\0';
-    else 
-      buffer[maxlen - 1] = '\0';
-  }
-  return total;
-}
-
-static int fmtstr (char *buffer, size_t *currlen, size_t maxlen,
-                   char *value, int flags, int min, int max)
-{
-  int padlen, strln;     /* amount to pad */
-  int cnt = 0;
-  int total = 0;
-  
-  if (value == 0)
-  {
-    value = "<NULL>";
-  }
-
-  for (strln = 0; value[strln]; ++strln); /* strlen */
-  if (max >= 0 && max < strln)
-    strln = max;
-  padlen = min - strln;
-  if (padlen < 0) 
-    padlen = 0;
-  if (flags & DP_F_MINUS) 
-    padlen = -padlen; /* Left Justify */
-
-  while (padlen > 0)
-  {
-    total += dopr_outch (buffer, currlen, maxlen, ' ');
-    --padlen;
-  }
-  while (*value && ((max < 0) || (cnt < max)))
-  {
-    total += dopr_outch (buffer, currlen, maxlen, *value++);
-    ++cnt;
-  }
-  while (padlen < 0)
-  {
-    total += dopr_outch (buffer, currlen, maxlen, ' ');
-    ++padlen;
-  }
-  return total;
-}
-
-/* Have to handle DP_F_NUM (ie 0x and 0 alternates) */
-
-static int fmtint (char *buffer, size_t *currlen, size_t maxlen,
-		   long value, int base, int min, int max, int flags)
-{
-  int signvalue = 0;
-  unsigned long uvalue;
-  char convert[20];
-  int place = 0;
-  int spadlen = 0; /* amount to space pad */
-  int zpadlen = 0; /* amount to zero pad */
-  int caps = 0;
-  int total = 0;
-  
-  if (max < 0)
-    max = 0;
-
-  uvalue = value;
-
-  if(!(flags & DP_F_UNSIGNED))
-  {
-    if( value < 0 ) {
-      signvalue = '-';
-      uvalue = -value;
-    }
-    else
-      if (flags & DP_F_PLUS)  /* Do a sign (+/i) */
-	signvalue = '+';
-    else
-      if (flags & DP_F_SPACE)
-	signvalue = ' ';
-  }
-  
-  if (flags & DP_F_UP) caps = 1; /* Should characters be upper case? */
-
-  do {
-    convert[place++] =
-      (caps? "0123456789ABCDEF":"0123456789abcdef")
-      [uvalue % (unsigned)base  ];
-    uvalue = (uvalue / (unsigned)base );
-  } while(uvalue && (place < 20));
-  if (place == 20) place--;
-  convert[place] = 0;
-
-  zpadlen = max - place;
-  spadlen = min - MAX (max, place) - (signvalue ? 1 : 0);
-  if (zpadlen < 0) zpadlen = 0;
-  if (spadlen < 0) spadlen = 0;
-  if (flags & DP_F_ZERO)
-  {
-    zpadlen = MAX(zpadlen, spadlen);
-    spadlen = 0;
-  }
-  if (flags & DP_F_MINUS) 
-    spadlen = -spadlen; /* Left Justifty */
-
-#ifdef DEBUG_SNPRINTF
-  dprint (1, (debugfile, "zpad: %d, spad: %d, min: %d, max: %d, place: %d\n",
-      zpadlen, spadlen, min, max, place));
-#endif
-
-  /* Spaces */
-  while (spadlen > 0) 
-  {
-    total += dopr_outch (buffer, currlen, maxlen, ' ');
-    --spadlen;
-  }
-
-  /* Sign */
-  if (signvalue) 
-    total += dopr_outch (buffer, currlen, maxlen, signvalue);
-
-  /* Zeros */
-  if (zpadlen > 0) 
-  {
-    while (zpadlen > 0)
-    {
-      total += dopr_outch (buffer, currlen, maxlen, '0');
-      --zpadlen;
-    }
-  }
-
-  /* Digits */
-  while (place > 0) 
-    total += dopr_outch (buffer, currlen, maxlen, convert[--place]);
-  
-  /* Left Justified spaces */
-  while (spadlen < 0) {
-    total += dopr_outch (buffer, currlen, maxlen, ' ');
-    ++spadlen;
-  }
-
-  return total;
-}
-
-static LDOUBLE abs_val (LDOUBLE value)
-{
-  LDOUBLE result = value;
-
-  if (value < 0)
-    result = -value;
-
-  return result;
-}
-
-static LDOUBLE pow10_argp (int exp)
-{
-  LDOUBLE result = 1;
-
-  while (exp)
-  {
-    result *= 10;
-    exp--;
-  }
-  
-  return result;
-}
-
-static long round_argp (LDOUBLE value)
-{
-  long intpart;
-
-  intpart = value;
-  value = value - intpart;
-  if (value >= 0.5)
-    intpart++;
-
-  return intpart;
-}
-
-static int fmtfp (char *buffer, size_t *currlen, size_t maxlen,
-		  LDOUBLE fvalue, int min, int max, int flags)
-{
-  int signvalue = 0;
-  LDOUBLE ufvalue;
-  char iconvert[20];
-  char fconvert[20];
-  int iplace = 0;
-  int fplace = 0;
-  int padlen = 0; /* amount to pad */
-  int zpadlen = 0; 
-  int caps = 0;
-  int total = 0;
-  long intpart;
-  long fracpart;
-  
-  /* 
-   * AIX manpage says the default is 0, but Solaris says the default
-   * is 6, and sprintf on AIX defaults to 6
-   */
-  if (max < 0)
-    max = 6;
-
-  ufvalue = abs_val (fvalue);
-
-  if (fvalue < 0)
-    signvalue = '-';
-  else
-    if (flags & DP_F_PLUS)  /* Do a sign (+/i) */
-      signvalue = '+';
-    else
-      if (flags & DP_F_SPACE)
-	signvalue = ' ';
-
-#if 0
-  if (flags & DP_F_UP) caps = 1; /* Should characters be upper case? */
-#endif
-
-  intpart = ufvalue;
-
-  /* 
-   * Sorry, we only support 9 digits past the decimal because of our 
-   * conversion method
-   */
-  if (max > 9)
-    max = 9;
-
-  /* We "cheat" by converting the fractional part to integer by
-   * multiplying by a factor of 10
-   */
-  fracpart = round_argp ((pow10_argp (max)) * (ufvalue - intpart));
-
-  if (fracpart >= pow10_argp (max))
-  {
-    intpart++;
-    fracpart -= pow10_argp (max);
-  }
-
-#ifdef DEBUG_SNPRINTF
-  dprint (1, (debugfile, "fmtfp: %f =? %d.%d\n", fvalue, intpart, fracpart));
-#endif
-
-  /* Convert integer part */
-  do {
-    iconvert[iplace++] =
-      (caps? "0123456789ABCDEF":"0123456789abcdef")[intpart % 10];
-    intpart = (intpart / 10);
-  } while(intpart && (iplace < 20));
-  if (iplace == 20) iplace--;
-  iconvert[iplace] = 0;
-
-  /* Convert fractional part */
-  do {
-    fconvert[fplace++] =
-      (caps? "0123456789ABCDEF":"0123456789abcdef")[fracpart % 10];
-    fracpart = (fracpart / 10);
-  } while(fracpart && (fplace < 20));
-  if (fplace == 20) fplace--;
-  fconvert[fplace] = 0;
-
-  /* -1 for decimal point, another -1 if we are printing a sign */
-  padlen = min - iplace - max - 1 - ((signvalue) ? 1 : 0); 
-  zpadlen = max - fplace;
-  if (zpadlen < 0)
-    zpadlen = 0;
-  if (padlen < 0) 
-    padlen = 0;
-  if (flags & DP_F_MINUS) 
-    padlen = -padlen; /* Left Justifty */
-
-  if ((flags & DP_F_ZERO) && (padlen > 0)) 
-  {
-    if (signvalue) 
-    {
-      total += dopr_outch (buffer, currlen, maxlen, signvalue);
-      --padlen;
-      signvalue = 0;
-    }
-    while (padlen > 0)
-    {
-      total += dopr_outch (buffer, currlen, maxlen, '0');
-      --padlen;
-    }
-  }
-  while (padlen > 0)
-  {
-    total += dopr_outch (buffer, currlen, maxlen, ' ');
-    --padlen;
-  }
-  if (signvalue) 
-    total += dopr_outch (buffer, currlen, maxlen, signvalue);
-
-  while (iplace > 0) 
-    total += dopr_outch (buffer, currlen, maxlen, iconvert[--iplace]);
-
-  /*
-   * Decimal point.  This should probably use locale to find the correct
-   * char to print out.
-   */
-  if (max > 0)
-  {
-    total += dopr_outch (buffer, currlen, maxlen, '.');
-
-    while (fplace > 0) 
-      total += dopr_outch (buffer, currlen, maxlen, fconvert[--fplace]);
-  }
-
-  while (zpadlen > 0)
-  {
-    total += dopr_outch (buffer, currlen, maxlen, '0');
-    --zpadlen;
-  }
-
-  while (padlen < 0) 
-  {
-    total += dopr_outch (buffer, currlen, maxlen, ' ');
-    ++padlen;
-  }
-
-  return total;
-}
-
-static int dopr_outch (char *buffer, size_t *currlen, size_t maxlen, char c)
-{
-  if (*currlen + 1 < maxlen)
-    buffer[(*currlen)++] = c;
-  return 1;
-}
-
-#ifndef HAVE_VSNPRINTF
-int vsnprintf (char *str, size_t count, const char *fmt, va_list args)
-{
-  if (str != NULL)
-    str[0] = 0;
-  return dopr(str, count, fmt, args);
-}
-#endif /* !HAVE_VSNPRINTF */
-
-#ifndef HAVE_SNPRINTF
-/* VARARGS3 */
-#ifdef HAVE_STDARGS
-int snprintf (char *str,size_t count,const char *fmt,...)
-#else
-int snprintf (va_alist) va_dcl
-#endif
-{
-#ifndef HAVE_STDARGS
-  char *str;
-  size_t count;
-  char *fmt;
-#endif
-  VA_LOCAL_DECL;
-  int total;
-    
-  VA_START (fmt);
-  VA_SHIFT (str, char *);
-  VA_SHIFT (count, size_t );
-  VA_SHIFT (fmt, char *);
-  total = vsnprintf(str, count, fmt, ap);
-  VA_END;
-  return total;
-}
-#endif /* !HAVE_SNPRINTF */
-
-#ifdef TEST_SNPRINTF
-#ifndef LONG_STRING
-#define LONG_STRING 1024
-#endif
-int main (void)
-{
-  char buf1[LONG_STRING];
-  char buf2[LONG_STRING];
-  char *fp_fmt[] = {
-    "%-1.5f",
-    "%1.5f",
-    "%123.9f",
-    "%10.5f",
-    "% 10.5f",
-    "%+22.9f",
-    "%+4.9f",
-    "%01.3f",
-    "%4f",
-    "%3.1f",
-    "%3.2f",
-    "%.0f",
-    "%.1f",
-    NULL
-  };
-  double fp_nums[] = { -1.5, 134.21, 91340.2, 341.1234, 0203.9, 0.96, 0.996, 
-    0.9996, 1.996, 4.136, 0};
-  char *int_fmt[] = {
-    "%-1.5d",
-    "%1.5d",
-    "%123.9d",
-    "%5.5d",
-    "%10.5d",
-    "% 10.5d",
-    "%+22.33d",
-    "%01.3d",
-    "%4d",
-    NULL
-  };
-  long int_nums[] = { -1, 134, 91340, 341, 0203, 0};
-  int x, y;
-  int fail = 0;
-  int num = 0;
-
-  printf ("Testing snprintf format codes against system sprintf...\n");
-
-  for (x = 0; fp_fmt[x] != NULL ; x++)
-    for (y = 0; fp_nums[y] != 0 ; y++)
-    {
-      snprintf (buf1, sizeof (buf1), fp_fmt[x], fp_nums[y]);
-      sprintf (buf2, fp_fmt[x], fp_nums[y]);
-      if (strcmp (buf1, buf2))
-      {
-	printf("snprintf doesn't match Format: %s\n\tsnprintf = %s\n\tsprintf  = %s\n", 
-	    fp_fmt[x], buf1, buf2);
-	fail++;
-      }
-      num++;
-    }
-
-  for (x = 0; int_fmt[x] != NULL ; x++)
-    for (y = 0; int_nums[y] != 0 ; y++)
-    {
-      snprintf (buf1, sizeof (buf1), int_fmt[x], int_nums[y]);
-      sprintf (buf2, int_fmt[x], int_nums[y]);
-      if (strcmp (buf1, buf2))
-      {
-	printf("snprintf doesn't match Format: %s\n\tsnprintf = %s\n\tsprintf  = %s\n", 
-	    int_fmt[x], buf1, buf2);
-	fail++;
-      }
-      num++;
-    }
-  printf ("%d tests failed out of %d.\n", fail, num);
-}
-#endif /* SNPRINTF_TEST */
-
-#endif /* !HAVE_SNPRINTF */
diff --git a/contrib/fuse-include/fuse-mount.h b/contrib/fuse-include/fuse-mount.h
index 9358ac810e1..7d28462de47 100644
--- a/contrib/fuse-include/fuse-mount.h
+++ b/contrib/fuse-include/fuse-mount.h
@@ -8,6 +8,6 @@
 */
 
 void gf_fuse_unmount (const char *mountpoint, int fd);
-int gf_fuse_mount (const char *mountpoint, char *fsname,
-                   unsigned long mountflags, char *mnt_param,
+int gf_fuse_unmount_daemon (const char *mountpoint, int fd);
+int gf_fuse_mount (const char *mountpoint, char *fsname, char *mnt_param,
                    pid_t *mtab_pid, int status_fd);
diff --git a/contrib/fuse-include/fuse_kernel.h b/contrib/fuse-include/fuse_kernel.h
index 60bb2f9f7b7..1e41e237e6f 100644
--- a/contrib/fuse-include/fuse_kernel.h
+++ b/contrib/fuse-include/fuse_kernel.h
@@ -93,6 +93,18 @@
  *
  * 7.22
  *  - add FUSE_ASYNC_DIO
+ *
+ * 7.23
+ *  - add FUSE_WRITEBACK_CACHE
+ *  - add time_gran to fuse_init_out
+ *  - add reserved space to fuse_init_out
+ *  - add FATTR_CTIME
+ *  - add ctime and ctimensec to fuse_setattr_in
+ *  - add FUSE_RENAME2 request
+ *  - add FUSE_NO_OPEN_SUPPORT flag
+ *
+ *  7.24
+ *  - add FUSE_LSEEK for SEEK_HOLE and SEEK_DATA support
  */
 
 #ifndef _LINUX_FUSE_H
@@ -128,7 +140,7 @@
 #define FUSE_KERNEL_VERSION 7
 
 /** Minor version number of this interface */
-#define FUSE_KERNEL_MINOR_VERSION 22
+#define FUSE_KERNEL_MINOR_VERSION 24
 
 /** The node ID of the root inode */
 #define FUSE_ROOT_ID 1
@@ -188,6 +200,7 @@ struct fuse_file_lock {
 #define FATTR_ATIME_NOW	(1 << 7)
 #define FATTR_MTIME_NOW	(1 << 8)
 #define FATTR_LOCKOWNER	(1 << 9)
+#define FATTR_CTIME	(1 << 10)
 
 /**
  * Flags returned by the OPEN request
@@ -219,6 +232,8 @@ struct fuse_file_lock {
  * FUSE_DO_READDIRPLUS: do READDIRPLUS (READDIR+LOOKUP in one)
  * FUSE_READDIRPLUS_AUTO: adaptive readdirplus
  * FUSE_ASYNC_DIO: asynchronous direct I/O submission
+ * FUSE_WRITEBACK_CACHE: use writeback cache for buffered writes
+ * FUSE_NO_OPEN_SUPPORT: kernel supports zero-message opens
  */
 #define FUSE_ASYNC_READ		(1 << 0)
 #define FUSE_POSIX_LOCKS	(1 << 1)
@@ -236,6 +251,8 @@ struct fuse_file_lock {
 #define FUSE_DO_READDIRPLUS	(1 << 13)
 #define FUSE_READDIRPLUS_AUTO	(1 << 14)
 #define FUSE_ASYNC_DIO		(1 << 15)
+#define FUSE_WRITEBACK_CACHE	(1 << 16)
+#define FUSE_NO_OPEN_SUPPORT	(1 << 17)
 
 /**
  * CUSE INIT request/reply flags
@@ -343,6 +360,8 @@ enum fuse_opcode {
 	FUSE_BATCH_FORGET  = 42,
 	FUSE_FALLOCATE     = 43,
 	FUSE_READDIRPLUS   = 44,
+	FUSE_RENAME2       = 45,
+	FUSE_LSEEK         = 46,
 
 	/* CUSE specific operations */
 	CUSE_INIT          = 4096,
@@ -421,6 +440,12 @@ struct fuse_rename_in {
 	uint64_t	newdir;
 };
 
+struct fuse_rename2_in {
+	uint64_t	newdir;
+	uint32_t	flags;
+	uint32_t	padding;
+};
+
 struct fuse_link_in {
 	uint64_t	oldnodeid;
 };
@@ -433,10 +458,10 @@ struct fuse_setattr_in {
 	uint64_t	lock_owner;
 	uint64_t	atime;
 	uint64_t	mtime;
-	uint64_t	unused2;
+	uint64_t	ctime;
 	uint32_t	atimensec;
 	uint32_t	mtimensec;
-	uint32_t	unused3;
+	uint32_t	ctimensec;
 	uint32_t	mode;
 	uint32_t	unused4;
 	uint32_t	uid;
@@ -554,6 +579,9 @@ struct fuse_init_in {
 	uint32_t	flags;
 };
 
+#define FUSE_COMPAT_INIT_OUT_SIZE 8
+#define FUSE_COMPAT_22_INIT_OUT_SIZE 24
+
 struct fuse_init_out {
 	uint32_t	major;
 	uint32_t	minor;
@@ -562,6 +590,8 @@ struct fuse_init_out {
 	uint16_t	max_background;
 	uint16_t	congestion_threshold;
 	uint32_t	max_write;
+	uint32_t	time_gran;
+	uint32_t	unused[9];
 };
 
 #define CUSE_INIT_INFO_MAX 4096
@@ -729,4 +759,15 @@ struct fuse_notify_retrieve_in {
 	uint64_t	dummy4;
 };
 
+struct fuse_lseek_in {
+	uint64_t	fh;
+	uint64_t	offset;
+	int32_t		whence;
+	uint32_t	padding;
+};
+
+struct fuse_lseek_out {
+	uint64_t	offset;
+};
+
 #endif /* _LINUX_FUSE_H */
diff --git a/contrib/fuse-lib/misc.c b/contrib/fuse-lib/misc.c
index 0c41b1a1917..1a9b418e511 100644
--- a/contrib/fuse-lib/misc.c
+++ b/contrib/fuse-lib/misc.c
@@ -10,7 +10,7 @@
 #include <string.h>
 #include <limits.h>
 #include <fcntl.h>
-#include "glusterfs.h"
+#include "glusterfs/glusterfs.h"
 #include "fuse_kernel.h"
 #include "fuse-misc.h"
 
@@ -41,7 +41,6 @@ void
 convert_fuse_file_lock (struct fuse_file_lock *fl, struct gf_flock *flock,
                         uint64_t lk_owner)
 {
-        memset (flock, 0, sizeof (struct flock));
         flock->l_type = fl->type;
         flock->l_whence = SEEK_SET;
         flock->l_start = fl->start;
diff --git a/contrib/fuse-lib/mount-common.c b/contrib/fuse-lib/mount-common.c
index cd226fa3141..cffd4c01ed5 100644
--- a/contrib/fuse-lib/mount-common.c
+++ b/contrib/fuse-lib/mount-common.c
@@ -32,7 +32,7 @@ mtab_needs_update (const char *mnt)
         struct stat stbuf;
 
         /* If mtab is within new mount, don't touch it */
-        if (strncmp (mnt, _PATH_MOUNTED, strlen (mnt)) == 0 &&
+        if (strncmp (mnt, _PATH_MOUNTED, sizeof (_PATH_MOUNTED) - 1) == 0 &&
             _PATH_MOUNTED[strlen (mnt)] == '/')
                 return 0;
 
@@ -105,7 +105,11 @@ fuse_mnt_add_mount (const char *progname, const char *fsname,
                 char *tmp;
 
                 sigprocmask (SIG_SETMASK, &oldmask, NULL);
-                setuid (geteuid ());
+                res = setuid (geteuid ());
+                if (res != 0) {
+                        GFFUSE_LOGERR ("%s: setuid: %s", progname, strerror (errno));
+                        exit (1);
+                }
 
                 /*
                  * hide in a directory, where mount isn't able to resolve
@@ -245,16 +249,22 @@ fuse_mnt_umount (const char *progname, const char *abs_mnt,
         }
         if (res == 0) {
                 sigprocmask (SIG_SETMASK, &oldmask, NULL);
-                setuid (geteuid ());
+                res = setuid (geteuid ());
+                if (res != 0) {
+                        GFFUSE_LOGERR ("%s: setuid: %s", progname, strerror (errno));
+                        exit (1);
+                }
 #ifdef GF_LINUX_HOST_OS
-                execl ("/bin/umount", "/bin/umount", "-i", rel_mnt,
+                execl ("umount", "umount", "-i", rel_mnt,
                        lazy ? "-l" : NULL, NULL);
-                GFFUSE_LOGERR ("%s: failed to execute /bin/umount: %s",
+                GFFUSE_LOGERR ("%s: failed to execute umount: %s",
                                progname, strerror (errno));
+#elif __NetBSD__
+                /* exitting the filesystem causes the umount */
+                exit (0);
 #else
-                execl ("/sbin/umount", "/sbin/umount", rel_mnt,
-                       lazy ? "-l" : NULL, NULL);
-                GFFUSE_LOGERR ("%s: failed to execute /sbin/umount: %s",
+                execl ("umount", "umount", "-f", rel_mnt, NULL);
+                GFFUSE_LOGERR ("%s: failed to execute umount: %s",
                                progname, strerror (errno));
 #endif /* GF_LINUX_HOST_OS */
                 exit (1);
diff --git a/contrib/fuse-lib/mount-gluster-compat.h b/contrib/fuse-lib/mount-gluster-compat.h
index 562f089dd1f..d3646d08d8e 100644
--- a/contrib/fuse-lib/mount-gluster-compat.h
+++ b/contrib/fuse-lib/mount-gluster-compat.h
@@ -30,17 +30,59 @@
 #include <sys/wait.h>
 #include <sys/mount.h>
 
+#ifdef GF_LINUX_HOST_OS
+typedef unsigned long mount_flag_t;
+#endif
+
 #if defined(__NetBSD__)
 #include <perfuse.h>
 #define umount2(dir, flags) unmount(dir, ((flags) != 0) ? MNT_FORCE : 0)
 #define MS_RDONLY MNT_RDONLY
+#define MS_NOSUID MNT_NOSUID
+#define MS_NODEV  MNT_NODEV
+#define MS_NOATIME MNT_NOATIME
+#define MS_NOEXEC MNT_NOEXEC
+typedef int mount_flag_t;
 #endif
 
 #if defined(GF_DARWIN_HOST_OS) || defined(__FreeBSD__)
 #include <sys/param.h>
 #include <sys/mount.h>
 #define umount2(dir, flags) unmount(dir, ((flags) != 0) ? MNT_FORCE : 0)
+#endif
+
+#if defined(__FreeBSD__)
 #define MS_RDONLY MNT_RDONLY
+#define MS_NOSUID MNT_NOSUID
+/* "nodev"/MNT_NODEV was removed from FreBSD, as it became unneeded because "As
+ * of FreeBSD 6.0 device nodes may be created in regular file systems but such
+ * nodes cannot be used to access devices." (See
+ * https://freebsd.org/cgi/man.cgi?query=mknod&sektion=8 .
+ * Also see:
+ * - https://github.com/freebsd/freebsd/commit/266790a
+ * - https://github.com/freebsd/freebsd/commit/a5e716d
+ * - 700008 in
+ *   https://www.freebsd.org/doc/en/books/porters-handbook/versions-7.html .)
+ */
+#if __FreeBSD_version < 700008
+#define MS_NODEV  MNT_NODEV
+#else
+#define MS_NODEV  0
+#endif
+#define MS_NOATIME MNT_NOATIME
+#define MS_NOEXEC MNT_NOEXEC
+#if __FreeBSD_version < 1000715
+typedef int  mount_flag_t;
+#else
+/* __FreeBSD_version was not bumped for this type change. Anyway, see
+ * https://github.com/freebsd/freebsd/commit/e8d76f8
+ * and respective __FreeBSD_version:
+ * https://github.com/freebsd/freebsd/blob/e8d76f8/sys/sys/param.h#L61 .
+ * We use the subsequent value, 1000715, to switch. (Also see:
+ * https://www.freebsd.org/doc/en/books/porters-handbook/versions-10.html .)
+ */
+typedef long long mount_flag_t;
+#endif
 #endif
 
 #ifdef GF_LINUX_HOST_OS
@@ -54,9 +96,9 @@
 #define FREE(ptr) free (ptr)
 #define GFFUSE_LOGERR(...) fprintf (stderr, ## __VA_ARGS__)
 #else /* FUSE_UTIL */
-#include "glusterfs.h"
-#include "logging.h"
-#include "common-utils.h"
+#include "glusterfs/glusterfs.h"
+#include "glusterfs/logging.h"
+#include "glusterfs/common-utils.h"
 
 #define GFFUSE_LOGERR(...) \
         gf_log ("glusterfs-fuse", GF_LOG_ERROR, ## __VA_ARGS__)
diff --git a/contrib/fuse-lib/mount.c b/contrib/fuse-lib/mount.c
index 1edde86014a..06ff191f542 100644
--- a/contrib/fuse-lib/mount.c
+++ b/contrib/fuse-lib/mount.c
@@ -52,12 +52,16 @@ gf_fuse_unmount (const char *mountpoint, int fd)
         if (geteuid () == 0) {
                 fuse_mnt_umount ("fuse", mountpoint, mountpoint, 1);
                 return;
+        } else {
+            GFFUSE_LOGERR ("fuse: Effective-uid: %d", geteuid());
         }
 
         res = umount2 (mountpoint, 2);
         if (res == 0)
                 return;
 
+        GFFUSE_LOGERR ("fuse: failed to unmount %s: %s",
+                       mountpoint, strerror (errno));
         pid = fork ();
         if (pid == -1)
                 return;
@@ -67,6 +71,8 @@ gf_fuse_unmount (const char *mountpoint, int fd)
                                        "--", mountpoint, NULL };
 
                 execvp (FUSERMOUNT_PROG, (char **)argv);
+                GFFUSE_LOGERR ("fuse: failed to execute fuserumount: %s",
+                               strerror (errno));
                 _exit (1);
         }
         waitpid (pid, NULL, 0);
@@ -75,6 +81,54 @@ gf_fuse_unmount (const char *mountpoint, int fd)
 
 /* gluster-specific routines */
 
+/* Unmounting in a daemon that lurks 'till main process exits */
+int
+gf_fuse_unmount_daemon (const char *mountpoint, int fd)
+{
+        int   ret = -1;
+        pid_t pid = -1;
+
+        if (fd == -1)
+                return -1;
+
+        int ump[2] = {0,};
+
+        ret = pipe(ump);
+        if (ret == -1) {
+                close (fd);
+                return -1;
+        }
+
+        pid = fork ();
+        switch (pid) {
+        case 0:
+        {
+                char c = 0;
+                sigset_t sigset;
+
+                close_fds_except (ump, 1);
+
+                setsid();
+                (void)chdir("/");
+                sigfillset(&sigset);
+                sigprocmask(SIG_BLOCK, &sigset, NULL);
+
+                read (ump[0], &c, 1);
+
+                gf_fuse_unmount (mountpoint, fd);
+                exit (0);
+        }
+        case -1:
+                close (fd);
+                fd = -1;
+                ret = -1;
+                close (ump[1]);
+        }
+        close (ump[0]);
+
+        return ret;
+}
+
 static char *
 escape (char *s)
 {
@@ -106,8 +160,7 @@ escape (char *s)
 
 static int
 fuse_mount_fusermount (const char *mountpoint, char *fsname,
-                       unsigned long mountflags, char *mnt_param,
-                       int fd)
+                       char *mnt_param, int fd)
 {
         int  pid = -1;
         int  res = 0;
@@ -130,8 +183,7 @@ fuse_mount_fusermount (const char *mountpoint, char *fsname,
                 return -1;
         }
         ret = asprintf (&fm_mnt_params,
-                        "%s%s,fsname=%s,nonempty,subtype=glusterfs",
-                        (mountflags & MS_RDONLY) ? "ro," : "",
+                        "%s,fsname=%s,nonempty,subtype=glusterfs",
                         mnt_param, efsname);
         FREE (efsname);
         if (ret == -1) {
@@ -224,19 +276,101 @@ build_iovec_argf(struct iovec **iov, int *iovlen, const char *name,
 }
 #endif /* __FreeBSD__ */
 
+struct mount_flags {
+        const char *opt;
+        mount_flag_t flag;
+        int on;
+} mount_flags[] = {
+        /* We provide best effort cross platform support for mount flags by
+         * defining the ones which are commonly used in Unix-like OS-es.
+         */
+        {"ro",      MS_RDONLY,      1},
+        {"nosuid",  MS_NOSUID,      1},
+        {"nodev",   MS_NODEV,       1},
+        {"noatime", MS_NOATIME,     1},
+        {"noexec",  MS_NOEXEC,      1},
+#ifdef GF_LINUX_HOST_OS
+        {"rw",      MS_RDONLY,      0},
+        {"suid",    MS_NOSUID,      0},
+        {"dev",     MS_NODEV,       0},
+        {"exec",    MS_NOEXEC,      0},
+        {"async",   MS_SYNCHRONOUS, 0},
+        {"sync",    MS_SYNCHRONOUS, 1},
+        {"atime",   MS_NOATIME,     0},
+        {"dirsync", MS_DIRSYNC,     1},
+#endif
+        {NULL,      0,              0}
+};
+
+static int
+mount_param_to_flag (char *mnt_param, mount_flag_t *mntflags,
+                     char **mnt_param_new)
+{
+        gf_boolean_t found = _gf_false;
+        struct mount_flags *flag = NULL;
+        char *param_tok = NULL;
+        token_iter_t tit = {0,};
+        gf_boolean_t iter_end = _gf_false;
+
+        /* Allocate a buffer that will hold the mount parameters remaining
+         * after the ones corresponding to mount flags are processed and
+         * removed.The length of the original params are a good upper bound
+         * of the size needed.
+         */
+        *mnt_param_new = strdup (mnt_param);
+        if (!*mnt_param_new)
+                return -1;
+
+        for (param_tok = token_iter_init (*mnt_param_new, ',', &tit) ;;) {
+                iter_end = next_token (&param_tok, &tit);
+
+                found = _gf_false;
+                for (flag = mount_flags; flag->opt; flag++) {
+                        /* Compare the mount flag name to the param
+                         * name at hand.
+                         */
+                        if (strcmp (flag->opt, param_tok) == 0) {
+                                /* If there is a match, adjust mntflags
+                                 * accordingly and break.
+                                 */
+                                if (flag->on) {
+                                        *mntflags |= flag->flag;
+                                } else {
+                                        *mntflags &= ~flag->flag;
+                                }
+                                found = _gf_true;
+                                break;
+                        }
+                }
+                /* Exclude flag names from new parameter list. */
+                if (found)
+                        drop_token (param_tok, &tit);
+
+                if (iter_end)
+                        break;
+        }
+
+        return 0;
+}
+
 static int
 fuse_mount_sys (const char *mountpoint, char *fsname,
-                unsigned long mountflags, char *mnt_param, int fd)
+                char *mnt_param, int fd)
 {
         int ret = -1;
         unsigned mounted = 0;
         char *mnt_param_mnt = NULL;
         char *fstype = "fuse.glusterfs";
         char *source = fsname;
-
-        ret = asprintf (&mnt_param_mnt,
-                        "%s,fd=%i,rootmode=%o,user_id=%i,group_id=%i",
-                        mnt_param, fd, S_IFDIR, getuid (), getgid ());
+        mount_flag_t mountflags = 0;
+        char *mnt_param_new = NULL;
+
+        ret = mount_param_to_flag (mnt_param, &mountflags, &mnt_param_new);
+        if (ret == 0)
+                ret = asprintf (&mnt_param_mnt,
+                                "%s,fd=%i,rootmode=%o,user_id=%i,group_id=%i",
+                                mnt_param_new, fd, S_IFDIR, getuid (),
+                                getgid ());
         if (ret == -1) {
                 GFFUSE_LOGERR ("Out of memory");
 
@@ -256,6 +390,7 @@ fuse_mount_sys (const char *mountpoint, char *fsname,
         build_iovec (&iov, &iovlen, "from", "/dev/fuse", -1);
         build_iovec (&iov, &iovlen, "volname", source, -1);
         build_iovec (&iov, &iovlen, "fd", fdstr, -1);
+        build_iovec (&iov, &iovlen, "allow_other", NULL, -1);
         ret = nmount (iov, iovlen, mountflags);
 #else
         ret = mount (source, mountpoint, fstype, mountflags,
@@ -295,7 +430,7 @@ fuse_mount_sys (const char *mountpoint, char *fsname,
 
                 ret = asprintf (&mnt_param_mtab, "%s%s",
                                 mountflags & MS_RDONLY ? "ro," : "",
-                                mnt_param);
+                                mnt_param_new);
                 if (ret == -1)
                         GFFUSE_LOGERR ("Out of memory");
                 else {
@@ -320,6 +455,7 @@ out:
                         umount2 (mountpoint, 2); /* lazy umount */
         }
         FREE (mnt_param_mnt);
+        FREE (mnt_param_new);
         if (source != fsname)
                 FREE (source);
 
@@ -328,8 +464,7 @@ out:
 
 int
 gf_fuse_mount (const char *mountpoint, char *fsname,
-               unsigned long mountflags, char *mnt_param,
-               pid_t *mnt_pid, int status_fd)
+               char *mnt_param, pid_t *mnt_pid, int status_fd)
 {
         int   fd  = -1;
         pid_t pid = -1;
@@ -356,16 +491,19 @@ gf_fuse_mount (const char *mountpoint, char *fsname,
                                 exit (pid == -1 ? 1 : 0);
                 }
 
-                ret = fuse_mount_sys (mountpoint, fsname, mountflags, mnt_param,
-                                      fd);
+                ret = fuse_mount_sys (mountpoint, fsname, mnt_param, fd);
                 if (ret == -1) {
                         gf_log ("glusterfs-fuse", GF_LOG_INFO,
-                                "direct mount failed (%s) errno %d, "
-                                "retry to mount via fusermount",
+                                "direct mount failed (%s) errno %d",
                                 strerror (errno), errno);
 
-                        ret = fuse_mount_fusermount (mountpoint, fsname,
-                                                     mountflags, mnt_param, fd);
+                        if (errno == EPERM) {
+                                gf_log ("glusterfs-fuse", GF_LOG_INFO,
+                                        "retry to mount via fusermount");
+
+                                ret = fuse_mount_fusermount (mountpoint, fsname,
+                                                             mnt_param, fd);
+                        }
                 }
 
                 if (ret == -1)
diff --git a/contrib/fuse-util/fusermount.c b/contrib/fuse-util/fusermount.c
index a64d8e102ff..ff743f75a21 100644
--- a/contrib/fuse-util/fusermount.c
+++ b/contrib/fuse-util/fusermount.c
@@ -520,20 +520,22 @@ static void parse_line(char *line, int linenum)
 
 static void read_conf(void)
 {
+	int  len;
 	FILE *fp = fopen(FUSE_CONF, "r");
 	if (fp != NULL) {
 		int linenum = 1;
 		char line[256];
 		int isnewline = 1;
 		while (fgets(line, sizeof(line), fp) != NULL) {
+			len = strlen (line);
 			if (isnewline) {
-				if (strlen(line) && line[strlen(line)-1] == '\n') {
+				if (len && line[len-1] == '\n') {
 					strip_line(line);
 					parse_line(line, linenum);
 				} else {
 					isnewline = 0;
 				}
-			} else if(strlen(line) && line[strlen(line)-1] == '\n') {
+			} else if (len && line[len-1] == '\n') {
 				fprintf(stderr, "%s: reading %s: line %i too long\n", progname, FUSE_CONF, linenum);
 
 				isnewline = 1;
diff --git a/contrib/ipaddr-py/COPYING b/contrib/ipaddr-py/COPYING
deleted file mode 100644
index d6456956733..00000000000
--- a/contrib/ipaddr-py/COPYING
+++ /dev/null
@@ -1,202 +0,0 @@
-
-                                 Apache License
-                           Version 2.0, January 2004
-                        http://www.apache.org/licenses/
-
-   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
-
-   1. Definitions.
-
-      "License" shall mean the terms and conditions for use, reproduction,
-      and distribution as defined by Sections 1 through 9 of this document.
-
-      "Licensor" shall mean the copyright owner or entity authorized by
-      the copyright owner that is granting the License.
-
-      "Legal Entity" shall mean the union of the acting entity and all
-      other entities that control, are controlled by, or are under common
-      control with that entity. For the purposes of this definition,
-      "control" means (i) the power, direct or indirect, to cause the
-      direction or management of such entity, whether by contract or
-      otherwise, or (ii) ownership of fifty percent (50%) or more of the
-      outstanding shares, or (iii) beneficial ownership of such entity.
-
-      "You" (or "Your") shall mean an individual or Legal Entity
-      exercising permissions granted by this License.
-
-      "Source" form shall mean the preferred form for making modifications,
-      including but not limited to software source code, documentation
-      source, and configuration files.
-
-      "Object" form shall mean any form resulting from mechanical
-      transformation or translation of a Source form, including but
-      not limited to compiled object code, generated documentation,
-      and conversions to other media types.
-
-      "Work" shall mean the work of authorship, whether in Source or
-      Object form, made available under the License, as indicated by a
-      copyright notice that is included in or attached to the work
-      (an example is provided in the Appendix below).
-
-      "Derivative Works" shall mean any work, whether in Source or Object
-      form, that is based on (or derived from) the Work and for which the
-      editorial revisions, annotations, elaborations, or other modifications
-      represent, as a whole, an original work of authorship. For the purposes
-      of this License, Derivative Works shall not include works that remain
-      separable from, or merely link (or bind by name) to the interfaces of,
-      the Work and Derivative Works thereof.
-
-      "Contribution" shall mean any work of authorship, including
-      the original version of the Work and any modifications or additions
-      to that Work or Derivative Works thereof, that is intentionally
-      submitted to Licensor for inclusion in the Work by the copyright owner
-      or by an individual or Legal Entity authorized to submit on behalf of
-      the copyright owner. For the purposes of this definition, "submitted"
-      means any form of electronic, verbal, or written communication sent
-      to the Licensor or its representatives, including but not limited to
-      communication on electronic mailing lists, source code control systems,
-      and issue tracking systems that are managed by, or on behalf of, the
-      Licensor for the purpose of discussing and improving the Work, but
-      excluding communication that is conspicuously marked or otherwise
-      designated in writing by the copyright owner as "Not a Contribution."
-
-      "Contributor" shall mean Licensor and any individual or Legal Entity
-      on behalf of whom a Contribution has been received by Licensor and
-      subsequently incorporated within the Work.
-
-   2. Grant of Copyright License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      copyright license to reproduce, prepare Derivative Works of,
-      publicly display, publicly perform, sublicense, and distribute the
-      Work and such Derivative Works in Source or Object form.
-
-   3. Grant of Patent License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      (except as stated in this section) patent license to make, have made,
-      use, offer to sell, sell, import, and otherwise transfer the Work,
-      where such license applies only to those patent claims licensable
-      by such Contributor that are necessarily infringed by their
-      Contribution(s) alone or by combination of their Contribution(s)
-      with the Work to which such Contribution(s) was submitted. If You
-      institute patent litigation against any entity (including a
-      cross-claim or counterclaim in a lawsuit) alleging that the Work
-      or a Contribution incorporated within the Work constitutes direct
-      or contributory patent infringement, then any patent licenses
-      granted to You under this License for that Work shall terminate
-      as of the date such litigation is filed.
-
-   4. Redistribution. You may reproduce and distribute copies of the
-      Work or Derivative Works thereof in any medium, with or without
-      modifications, and in Source or Object form, provided that You
-      meet the following conditions:
-
-      (a) You must give any other recipients of the Work or
-          Derivative Works a copy of this License; and
-
-      (b) You must cause any modified files to carry prominent notices
-          stating that You changed the files; and
-
-      (c) You must retain, in the Source form of any Derivative Works
-          that You distribute, all copyright, patent, trademark, and
-          attribution notices from the Source form of the Work,
-          excluding those notices that do not pertain to any part of
-          the Derivative Works; and
-
-      (d) If the Work includes a "NOTICE" text file as part of its
-          distribution, then any Derivative Works that You distribute must
-          include a readable copy of the attribution notices contained
-          within such NOTICE file, excluding those notices that do not
-          pertain to any part of the Derivative Works, in at least one
-          of the following places: within a NOTICE text file distributed
-          as part of the Derivative Works; within the Source form or
-          documentation, if provided along with the Derivative Works; or,
-          within a display generated by the Derivative Works, if and
-          wherever such third-party notices normally appear. The contents
-          of the NOTICE file are for informational purposes only and
-          do not modify the License. You may add Your own attribution
-          notices within Derivative Works that You distribute, alongside
-          or as an addendum to the NOTICE text from the Work, provided
-          that such additional attribution notices cannot be construed
-          as modifying the License.
-
-      You may add Your own copyright statement to Your modifications and
-      may provide additional or different license terms and conditions
-      for use, reproduction, or distribution of Your modifications, or
-      for any such Derivative Works as a whole, provided Your use,
-      reproduction, and distribution of the Work otherwise complies with
-      the conditions stated in this License.
-
-   5. Submission of Contributions. Unless You explicitly state otherwise,
-      any Contribution intentionally submitted for inclusion in the Work
-      by You to the Licensor shall be under the terms and conditions of
-      this License, without any additional terms or conditions.
-      Notwithstanding the above, nothing herein shall supersede or modify
-      the terms of any separate license agreement you may have executed
-      with Licensor regarding such Contributions.
-
-   6. Trademarks. This License does not grant permission to use the trade
-      names, trademarks, service marks, or product names of the Licensor,
-      except as required for reasonable and customary use in describing the
-      origin of the Work and reproducing the content of the NOTICE file.
-
-   7. Disclaimer of Warranty. Unless required by applicable law or
-      agreed to in writing, Licensor provides the Work (and each
-      Contributor provides its Contributions) on an "AS IS" BASIS,
-      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-      implied, including, without limitation, any warranties or conditions
-      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
-      PARTICULAR PURPOSE. You are solely responsible for determining the
-      appropriateness of using or redistributing the Work and assume any
-      risks associated with Your exercise of permissions under this License.
-
-   8. Limitation of Liability. In no event and under no legal theory,
-      whether in tort (including negligence), contract, or otherwise,
-      unless required by applicable law (such as deliberate and grossly
-      negligent acts) or agreed to in writing, shall any Contributor be
-      liable to You for damages, including any direct, indirect, special,
-      incidental, or consequential damages of any character arising as a
-      result of this License or out of the use or inability to use the
-      Work (including but not limited to damages for loss of goodwill,
-      work stoppage, computer failure or malfunction, or any and all
-      other commercial damages or losses), even if such Contributor
-      has been advised of the possibility of such damages.
-
-   9. Accepting Warranty or Additional Liability. While redistributing
-      the Work or Derivative Works thereof, You may choose to offer,
-      and charge a fee for, acceptance of support, warranty, indemnity,
-      or other liability obligations and/or rights consistent with this
-      License. However, in accepting such obligations, You may act only
-      on Your own behalf and on Your sole responsibility, not on behalf
-      of any other Contributor, and only if You agree to indemnify,
-      defend, and hold each Contributor harmless for any liability
-      incurred by, or claims asserted against, such Contributor by reason
-      of your accepting any such warranty or additional liability.
-
-   END OF TERMS AND CONDITIONS
-
-   APPENDIX: How to apply the Apache License to your work.
-
-      To apply the Apache License to your work, attach the following
-      boilerplate notice, with the fields enclosed by brackets "[]"
-      replaced with your own identifying information. (Don't include
-      the brackets!)  The text should be enclosed in the appropriate
-      comment syntax for the file format. We also recommend that a
-      file or class name and description of purpose be included on the
-      same "printed page" as the copyright notice for easier
-      identification within third-party archives.
-
-   Copyright [yyyy] [name of copyright owner]
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.
diff --git a/contrib/ipaddr-py/MANIFEST.in b/contrib/ipaddr-py/MANIFEST.in
deleted file mode 100644
index f572804441b..00000000000
--- a/contrib/ipaddr-py/MANIFEST.in
+++ /dev/null
@@ -1,3 +0,0 @@
-include COPYING
-include ipaddr_test.py
-include RELEASENOTES
diff --git a/contrib/ipaddr-py/OWNERS b/contrib/ipaddr-py/OWNERS
deleted file mode 100644
index 501673e0395..00000000000
--- a/contrib/ipaddr-py/OWNERS
+++ /dev/null
@@ -1,4 +0,0 @@
-pmoody
-harro
-mshields
-smart
diff --git a/contrib/ipaddr-py/README b/contrib/ipaddr-py/README
deleted file mode 100644
index 1b54294bb10..00000000000
--- a/contrib/ipaddr-py/README
+++ /dev/null
@@ -1,8 +0,0 @@
-ipaddr.py is a library for working with IP addresses, both IPv4 and IPv6.
-It was developed by Google for internal use, and is now open source.
-
-Project home page: http://code.google.com/p/ipaddr-py/
-
-Please send contributions to ipaddr-py-dev@googlegroups.com.  Code should
-include unit tests and follow the Google Python style guide:
-http://code.google.com/p/soc/wiki/PythonStyleGuide
diff --git a/contrib/ipaddr-py/ipaddr.py b/contrib/ipaddr-py/ipaddr.py
deleted file mode 100644
index a89298a315d..00000000000
--- a/contrib/ipaddr-py/ipaddr.py
+++ /dev/null
@@ -1,1907 +0,0 @@
-#!/usr/bin/python
-#
-# Copyright 2007 Google Inc.
-#  Licensed to PSF under a Contributor Agreement.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-# implied. See the License for the specific language governing
-# permissions and limitations under the License.
-
-"""A fast, lightweight IPv4/IPv6 manipulation library in Python.
-
-This library is used to create/poke/manipulate IPv4 and IPv6 addresses
-and networks.
-
-"""
-
-__version__ = 'trunk'
-
-import struct
-
-IPV4LENGTH = 32
-IPV6LENGTH = 128
-
-
-class AddressValueError(ValueError):
-    """A Value Error related to the address."""
-
-
-class NetmaskValueError(ValueError):
-    """A Value Error related to the netmask."""
-
-
-def IPAddress(address, version=None):
-    """Take an IP string/int and return an object of the correct type.
-
-    Args:
-        address: A string or integer, the IP address.  Either IPv4 or
-          IPv6 addresses may be supplied; integers less than 2**32 will
-          be considered to be IPv4 by default.
-        version: An Integer, 4 or 6. If set, don't try to automatically
-          determine what the IP address type is. important for things
-          like IPAddress(1), which could be IPv4, '0.0.0.1',  or IPv6,
-          '::1'.
-
-    Returns:
-        An IPv4Address or IPv6Address object.
-
-    Raises:
-        ValueError: if the string passed isn't either a v4 or a v6
-          address.
-
-    """
-    if version:
-        if version == 4:
-            return IPv4Address(address)
-        elif version == 6:
-            return IPv6Address(address)
-
-    try:
-        return IPv4Address(address)
-    except (AddressValueError, NetmaskValueError):
-        pass
-
-    try:
-        return IPv6Address(address)
-    except (AddressValueError, NetmaskValueError):
-        pass
-
-    raise ValueError('%r does not appear to be an IPv4 or IPv6 address' %
-                     address)
-
-
-def IPNetwork(address, version=None, strict=False):
-    """Take an IP string/int and return an object of the correct type.
-
-    Args:
-        address: A string or integer, the IP address.  Either IPv4 or
-          IPv6 addresses may be supplied; integers less than 2**32 will
-          be considered to be IPv4 by default.
-        version: An Integer, if set, don't try to automatically
-          determine what the IP address type is. important for things
-          like IPNetwork(1), which could be IPv4, '0.0.0.1/32', or IPv6,
-          '::1/128'.
-
-    Returns:
-        An IPv4Network or IPv6Network object.
-
-    Raises:
-        ValueError: if the string passed isn't either a v4 or a v6
-          address. Or if a strict network was requested and a strict
-          network wasn't given.
-
-    """
-    if version:
-        if version == 4:
-            return IPv4Network(address, strict)
-        elif version == 6:
-            return IPv6Network(address, strict)
-
-    try:
-        return IPv4Network(address, strict)
-    except (AddressValueError, NetmaskValueError):
-        pass
-
-    try:
-        return IPv6Network(address, strict)
-    except (AddressValueError, NetmaskValueError):
-        pass
-
-    raise ValueError('%r does not appear to be an IPv4 or IPv6 network' %
-                     address)
-
-
-def v4_int_to_packed(address):
-    """The binary representation of this address.
-
-    Args:
-        address: An integer representation of an IPv4 IP address.
-
-    Returns:
-        The binary representation of this address.
-
-    Raises:
-        ValueError: If the integer is too large to be an IPv4 IP
-          address.
-    """
-    if address > _BaseV4._ALL_ONES:
-        raise ValueError('Address too large for IPv4')
-    return struct.pack('!I', address)
-
-
-def v6_int_to_packed(address):
-    """The binary representation of this address.
-
-    Args:
-        address: An integer representation of an IPv4 IP address.
-
-    Returns:
-        The binary representation of this address.
-    """
-    return struct.pack('!QQ', address >> 64, address & (2**64 - 1))
-
-
-def _find_address_range(addresses):
-    """Find a sequence of addresses.
-
-    Args:
-        addresses: a list of IPv4 or IPv6 addresses.
-
-    Returns:
-        A tuple containing the first and last IP addresses in the sequence.
-
-    """
-    first = last = addresses[0]
-    for ip in addresses[1:]:
-        if ip._ip == last._ip + 1:
-            last = ip
-        else:
-            break
-    return (first, last)
-
-def _get_prefix_length(number1, number2, bits):
-    """Get the number of leading bits that are same for two numbers.
-
-    Args:
-        number1: an integer.
-        number2: another integer.
-        bits: the maximum number of bits to compare.
-
-    Returns:
-        The number of leading bits that are the same for two numbers.
-
-    """
-    for i in range(bits):
-        if number1 >> i == number2 >> i:
-            return bits - i
-    return 0
-
-def _count_righthand_zero_bits(number, bits):
-    """Count the number of zero bits on the right hand side.
-
-    Args:
-        number: an integer.
-        bits: maximum number of bits to count.
-
-    Returns:
-        The number of zero bits on the right hand side of the number.
-
-    """
-    if number == 0:
-        return bits
-    for i in range(bits):
-        if (number >> i) % 2:
-            return i
-
-def summarize_address_range(first, last):
-    """Summarize a network range given the first and last IP addresses.
-
-    Example:
-        >>> summarize_address_range(IPv4Address('1.1.1.0'),
-            IPv4Address('1.1.1.130'))
-        [IPv4Network('1.1.1.0/25'), IPv4Network('1.1.1.128/31'),
-        IPv4Network('1.1.1.130/32')]
-
-    Args:
-        first: the first IPv4Address or IPv6Address in the range.
-        last: the last IPv4Address or IPv6Address in the range.
-
-    Returns:
-        The address range collapsed to a list of IPv4Network's or
-        IPv6Network's.
-
-    Raise:
-        TypeError:
-            If the first and last objects are not IP addresses.
-            If the first and last objects are not the same version.
-        ValueError:
-            If the last object is not greater than the first.
-            If the version is not 4 or 6.
-
-    """
-    if not (isinstance(first, _BaseIP) and isinstance(last, _BaseIP)):
-        raise TypeError('first and last must be IP addresses, not networks')
-    if first.version != last.version:
-        raise TypeError("%s and %s are not of the same version" % (
-                str(first), str(last)))
-    if first > last:
-        raise ValueError('last IP address must be greater than first')
-
-    networks = []
-
-    if first.version == 4:
-        ip = IPv4Network
-    elif first.version == 6:
-        ip = IPv6Network
-    else:
-        raise ValueError('unknown IP version')
-
-    ip_bits = first._max_prefixlen
-    first_int = first._ip
-    last_int = last._ip
-    while first_int <= last_int:
-        nbits = _count_righthand_zero_bits(first_int, ip_bits)
-        current = None
-        while nbits >= 0:
-            addend = 2**nbits - 1
-            current = first_int + addend
-            nbits -= 1
-            if current <= last_int:
-                break
-        prefix = _get_prefix_length(first_int, current, ip_bits)
-        net = ip('%s/%d' % (str(first), prefix))
-        networks.append(net)
-        if current == ip._ALL_ONES:
-            break
-        first_int = current + 1
-        first = IPAddress(first_int, version=first._version)
-    return networks
-
-def _collapse_address_list_recursive(addresses):
-    """Loops through the addresses, collapsing concurrent netblocks.
-
-    Example:
-
-        ip1 = IPv4Network('1.1.0.0/24')
-        ip2 = IPv4Network('1.1.1.0/24')
-        ip3 = IPv4Network('1.1.2.0/24')
-        ip4 = IPv4Network('1.1.3.0/24')
-        ip5 = IPv4Network('1.1.4.0/24')
-        ip6 = IPv4Network('1.1.0.1/22')
-
-        _collapse_address_list_recursive([ip1, ip2, ip3, ip4, ip5, ip6]) ->
-          [IPv4Network('1.1.0.0/22'), IPv4Network('1.1.4.0/24')]
-
-        This shouldn't be called directly; it is called via
-          collapse_address_list([]).
-
-    Args:
-        addresses: A list of IPv4Network's or IPv6Network's
-
-    Returns:
-        A list of IPv4Network's or IPv6Network's depending on what we were
-        passed.
-
-    """
-    ret_array = []
-    optimized = False
-
-    for cur_addr in addresses:
-        if not ret_array:
-            ret_array.append(cur_addr)
-            continue
-        if cur_addr in ret_array[-1]:
-            optimized = True
-        elif cur_addr == ret_array[-1].supernet().subnet()[1]:
-            ret_array.append(ret_array.pop().supernet())
-            optimized = True
-        else:
-            ret_array.append(cur_addr)
-
-    if optimized:
-        return _collapse_address_list_recursive(ret_array)
-
-    return ret_array
-
-
-def collapse_address_list(addresses):
-    """Collapse a list of IP objects.
-
-    Example:
-        collapse_address_list([IPv4('1.1.0.0/24'), IPv4('1.1.1.0/24')]) ->
-          [IPv4('1.1.0.0/23')]
-
-    Args:
-        addresses: A list of IPv4Network or IPv6Network objects.
-
-    Returns:
-        A list of IPv4Network or IPv6Network objects depending on what we
-        were passed.
-
-    Raises:
-        TypeError: If passed a list of mixed version objects.
-
-    """
-    i = 0
-    addrs = []
-    ips = []
-    nets = []
-
-    # split IP addresses and networks
-    for ip in addresses:
-        if isinstance(ip, _BaseIP):
-            if ips and ips[-1]._version != ip._version:
-                raise TypeError("%s and %s are not of the same version" % (
-                        str(ip), str(ips[-1])))
-            ips.append(ip)
-        elif ip._prefixlen == ip._max_prefixlen:
-            if ips and ips[-1]._version != ip._version:
-                raise TypeError("%s and %s are not of the same version" % (
-                        str(ip), str(ips[-1])))
-            ips.append(ip.ip)
-        else:
-            if nets and nets[-1]._version != ip._version:
-                raise TypeError("%s and %s are not of the same version" % (
-                        str(ip), str(ips[-1])))
-            nets.append(ip)
-
-    # sort and dedup
-    ips = sorted(set(ips))
-    nets = sorted(set(nets))
-
-    while i < len(ips):
-        (first, last) = _find_address_range(ips[i:])
-        i = ips.index(last) + 1
-        addrs.extend(summarize_address_range(first, last))
-
-    return _collapse_address_list_recursive(sorted(
-        addrs + nets, key=_BaseNet._get_networks_key))
-
-# backwards compatibility
-CollapseAddrList = collapse_address_list
-
-# Test whether this Python implementation supports byte objects that
-# are not identical to str ones.
-# We need to exclude platforms where bytes == str so that we can
-# distinguish between packed representations and strings, for example
-# b'12::' (the IPv4 address 49.50.58.58) and '12::' (an IPv6 address).
-try:
-    _compat_has_real_bytes = bytes is not str
-except NameError: # <Python2.6
-    _compat_has_real_bytes = False
-
-def get_mixed_type_key(obj):
-    """Return a key suitable for sorting between networks and addresses.
-
-    Address and Network objects are not sortable by default; they're
-    fundamentally different so the expression
-
-        IPv4Address('1.1.1.1') <= IPv4Network('1.1.1.1/24')
-
-    doesn't make any sense.  There are some times however, where you may wish
-    to have ipaddr sort these for you anyway. If you need to do this, you
-    can use this function as the key= argument to sorted().
-
-    Args:
-      obj: either a Network or Address object.
-    Returns:
-      appropriate key.
-
-    """
-    if isinstance(obj, _BaseNet):
-        return obj._get_networks_key()
-    elif isinstance(obj, _BaseIP):
-        return obj._get_address_key()
-    return NotImplemented
-
-class _IPAddrBase(object):
-
-    """The mother class."""
-
-    def __index__(self):
-        return self._ip
-
-    def __int__(self):
-        return self._ip
-
-    def __hex__(self):
-        return hex(self._ip)
-
-    @property
-    def exploded(self):
-        """Return the longhand version of the IP address as a string."""
-        return self._explode_shorthand_ip_string()
-
-    @property
-    def compressed(self):
-        """Return the shorthand version of the IP address as a string."""
-        return str(self)
-
-
-class _BaseIP(_IPAddrBase):
-
-    """A generic IP object.
-
-    This IP class contains the version independent methods which are
-    used by single IP addresses.
-
-    """
-
-    def __init__(self, address):
-        if (not (_compat_has_real_bytes and isinstance(address, bytes))
-            and '/' in str(address)):
-            raise AddressValueError(address)
-
-    def __eq__(self, other):
-        try:
-            return (self._ip == other._ip
-                    and self._version == other._version)
-        except AttributeError:
-            return NotImplemented
-
-    def __ne__(self, other):
-        eq = self.__eq__(other)
-        if eq is NotImplemented:
-            return NotImplemented
-        return not eq
-
-    def __le__(self, other):
-        gt = self.__gt__(other)
-        if gt is NotImplemented:
-            return NotImplemented
-        return not gt
-
-    def __ge__(self, other):
-        lt = self.__lt__(other)
-        if lt is NotImplemented:
-            return NotImplemented
-        return not lt
-
-    def __lt__(self, other):
-        if self._version != other._version:
-            raise TypeError('%s and %s are not of the same version' % (
-                    str(self), str(other)))
-        if not isinstance(other, _BaseIP):
-            raise TypeError('%s and %s are not of the same type' % (
-                    str(self), str(other)))
-        if self._ip != other._ip:
-            return self._ip < other._ip
-        return False
-
-    def __gt__(self, other):
-        if self._version != other._version:
-            raise TypeError('%s and %s are not of the same version' % (
-                    str(self), str(other)))
-        if not isinstance(other, _BaseIP):
-            raise TypeError('%s and %s are not of the same type' % (
-                    str(self), str(other)))
-        if self._ip != other._ip:
-            return self._ip > other._ip
-        return False
-
-    # Shorthand for Integer addition and subtraction. This is not
-    # meant to ever support addition/subtraction of addresses.
-    def __add__(self, other):
-        if not isinstance(other, int):
-            return NotImplemented
-        return IPAddress(int(self) + other, version=self._version)
-
-    def __sub__(self, other):
-        if not isinstance(other, int):
-            return NotImplemented
-        return IPAddress(int(self) - other, version=self._version)
-
-    def __repr__(self):
-        return '%s(%r)' % (self.__class__.__name__, str(self))
-
-    def __str__(self):
-        return  '%s' % self._string_from_ip_int(self._ip)
-
-    def __hash__(self):
-        return hash(hex(long(self._ip)))
-
-    def _get_address_key(self):
-        return (self._version, self)
-
-    @property
-    def version(self):
-        raise NotImplementedError('BaseIP has no version')
-
-
-class _BaseNet(_IPAddrBase):
-
-    """A generic IP object.
-
-    This IP class contains the version independent methods which are
-    used by networks.
-
-    """
-
-    def __init__(self, address):
-        self._cache = {}
-
-    def __repr__(self):
-        return '%s(%r)' % (self.__class__.__name__, str(self))
-
-    def iterhosts(self):
-        """Generate Iterator over usable hosts in a network.
-
-           This is like __iter__ except it doesn't return the network
-           or broadcast addresses.
-
-        """
-        cur = int(self.network) + 1
-        bcast = int(self.broadcast) - 1
-        while cur <= bcast:
-            cur += 1
-            yield IPAddress(cur - 1, version=self._version)
-
-    def __iter__(self):
-        cur = int(self.network)
-        bcast = int(self.broadcast)
-        while cur <= bcast:
-            cur += 1
-            yield IPAddress(cur - 1, version=self._version)
-
-    def __getitem__(self, n):
-        network = int(self.network)
-        broadcast = int(self.broadcast)
-        if n >= 0:
-            if network + n > broadcast:
-                raise IndexError
-            return IPAddress(network + n, version=self._version)
-        else:
-            n += 1
-            if broadcast + n < network:
-                raise IndexError
-            return IPAddress(broadcast + n, version=self._version)
-
-    def __lt__(self, other):
-        if self._version != other._version:
-            raise TypeError('%s and %s are not of the same version' % (
-                    str(self), str(other)))
-        if not isinstance(other, _BaseNet):
-            raise TypeError('%s and %s are not of the same type' % (
-                    str(self), str(other)))
-        if self.network != other.network:
-            return self.network < other.network
-        if self.netmask != other.netmask:
-            return self.netmask < other.netmask
-        return False
-
-    def __gt__(self, other):
-        if self._version != other._version:
-            raise TypeError('%s and %s are not of the same version' % (
-                    str(self), str(other)))
-        if not isinstance(other, _BaseNet):
-            raise TypeError('%s and %s are not of the same type' % (
-                    str(self), str(other)))
-        if self.network != other.network:
-            return self.network > other.network
-        if self.netmask != other.netmask:
-            return self.netmask > other.netmask
-        return False
-
-    def __le__(self, other):
-        gt = self.__gt__(other)
-        if gt is NotImplemented:
-            return NotImplemented
-        return not gt
-
-    def __ge__(self, other):
-        lt = self.__lt__(other)
-        if lt is NotImplemented:
-            return NotImplemented
-        return not lt
-
-    def __eq__(self, other):
-        try:
-            return (self._version == other._version
-                    and self.network == other.network
-                    and int(self.netmask) == int(other.netmask))
-        except AttributeError:
-            if isinstance(other, _BaseIP):
-                return (self._version == other._version
-                        and self._ip == other._ip)
-
-    def __ne__(self, other):
-        eq = self.__eq__(other)
-        if eq is NotImplemented:
-            return NotImplemented
-        return not eq
-
-    def __str__(self):
-        return  '%s/%s' % (str(self.ip),
-                           str(self._prefixlen))
-
-    def __hash__(self):
-        return hash(int(self.network) ^ int(self.netmask))
-
-    def __contains__(self, other):
-        # always false if one is v4 and the other is v6.
-        if self._version != other._version:
-          return False
-        # dealing with another network.
-        if isinstance(other, _BaseNet):
-            return (self.network <= other.network and
-                    self.broadcast >= other.broadcast)
-        # dealing with another address
-        else:
-            return (int(self.network) <= int(other._ip) <=
-                    int(self.broadcast))
-
-    def overlaps(self, other):
-        """Tell if self is partly contained in other."""
-        return self.network in other or self.broadcast in other or (
-            other.network in self or other.broadcast in self)
-
-    @property
-    def network(self):
-        x = self._cache.get('network')
-        if x is None:
-            x = IPAddress(self._ip & int(self.netmask), version=self._version)
-            self._cache['network'] = x
-        return x
-
-    @property
-    def broadcast(self):
-        x = self._cache.get('broadcast')
-        if x is None:
-            x = IPAddress(self._ip | int(self.hostmask), version=self._version)
-            self._cache['broadcast'] = x
-        return x
-
-    @property
-    def hostmask(self):
-        x = self._cache.get('hostmask')
-        if x is None:
-            x = IPAddress(int(self.netmask) ^ self._ALL_ONES,
-                          version=self._version)
-            self._cache['hostmask'] = x
-        return x
-
-    @property
-    def with_prefixlen(self):
-        return '%s/%d' % (str(self.ip), self._prefixlen)
-
-    @property
-    def with_netmask(self):
-        return '%s/%s' % (str(self.ip), str(self.netmask))
-
-    @property
-    def with_hostmask(self):
-        return '%s/%s' % (str(self.ip), str(self.hostmask))
-
-    @property
-    def numhosts(self):
-        """Number of hosts in the current subnet."""
-        return int(self.broadcast) - int(self.network) + 1
-
-    @property
-    def version(self):
-        raise NotImplementedError('BaseNet has no version')
-
-    @property
-    def prefixlen(self):
-        return self._prefixlen
-
-    def address_exclude(self, other):
-        """Remove an address from a larger block.
-
-        For example:
-
-            addr1 = IPNetwork('10.1.1.0/24')
-            addr2 = IPNetwork('10.1.1.0/26')
-            addr1.address_exclude(addr2) =
-                [IPNetwork('10.1.1.64/26'), IPNetwork('10.1.1.128/25')]
-
-        or IPv6:
-
-            addr1 = IPNetwork('::1/32')
-            addr2 = IPNetwork('::1/128')
-            addr1.address_exclude(addr2) = [IPNetwork('::0/128'),
-                IPNetwork('::2/127'),
-                IPNetwork('::4/126'),
-                IPNetwork('::8/125'),
-                ...
-                IPNetwork('0:0:8000::/33')]
-
-        Args:
-            other: An IPvXNetwork object of the same type.
-
-        Returns:
-            A sorted list of IPvXNetwork objects addresses which is self
-            minus other.
-
-        Raises:
-            TypeError: If self and other are of difffering address
-              versions, or if other is not a network object.
-            ValueError: If other is not completely contained by self.
-
-        """
-        if not self._version == other._version:
-            raise TypeError("%s and %s are not of the same version" % (
-                str(self), str(other)))
-
-        if not isinstance(other, _BaseNet):
-            raise TypeError("%s is not a network object" % str(other))
-
-        if other not in self:
-            raise ValueError('%s not contained in %s' % (str(other),
-                                                         str(self)))
-        if other == self:
-            return []
-
-        ret_addrs = []
-
-        # Make sure we're comparing the network of other.
-        other = IPNetwork('%s/%s' % (str(other.network), str(other.prefixlen)),
-                   version=other._version)
-
-        s1, s2 = self.subnet()
-        while s1 != other and s2 != other:
-            if other in s1:
-                ret_addrs.append(s2)
-                s1, s2 = s1.subnet()
-            elif other in s2:
-                ret_addrs.append(s1)
-                s1, s2 = s2.subnet()
-            else:
-                # If we got here, there's a bug somewhere.
-                assert True == False, ('Error performing exclusion: '
-                                       's1: %s s2: %s other: %s' %
-                                       (str(s1), str(s2), str(other)))
-        if s1 == other:
-            ret_addrs.append(s2)
-        elif s2 == other:
-            ret_addrs.append(s1)
-        else:
-            # If we got here, there's a bug somewhere.
-            assert True == False, ('Error performing exclusion: '
-                                   's1: %s s2: %s other: %s' %
-                                   (str(s1), str(s2), str(other)))
-
-        return sorted(ret_addrs, key=_BaseNet._get_networks_key)
-
-    def compare_networks(self, other):
-        """Compare two IP objects.
-
-        This is only concerned about the comparison of the integer
-        representation of the network addresses.  This means that the
-        host bits aren't considered at all in this method.  If you want
-        to compare host bits, you can easily enough do a
-        'HostA._ip < HostB._ip'
-
-        Args:
-            other: An IP object.
-
-        Returns:
-            If the IP versions of self and other are the same, returns:
-
-            -1 if self < other:
-              eg: IPv4('1.1.1.0/24') < IPv4('1.1.2.0/24')
-              IPv6('1080::200C:417A') < IPv6('1080::200B:417B')
-            0 if self == other
-              eg: IPv4('1.1.1.1/24') == IPv4('1.1.1.2/24')
-              IPv6('1080::200C:417A/96') == IPv6('1080::200C:417B/96')
-            1 if self > other
-              eg: IPv4('1.1.1.0/24') > IPv4('1.1.0.0/24')
-              IPv6('1080::1:200C:417A/112') >
-              IPv6('1080::0:200C:417A/112')
-
-            If the IP versions of self and other are different, returns:
-
-            -1 if self._version < other._version
-              eg: IPv4('10.0.0.1/24') < IPv6('::1/128')
-            1 if self._version > other._version
-              eg: IPv6('::1/128') > IPv4('255.255.255.0/24')
-
-        """
-        if self._version < other._version:
-            return -1
-        if self._version > other._version:
-            return 1
-        # self._version == other._version below here:
-        if self.network < other.network:
-            return -1
-        if self.network > other.network:
-            return 1
-        # self.network == other.network below here:
-        if self.netmask < other.netmask:
-            return -1
-        if self.netmask > other.netmask:
-            return 1
-        # self.network == other.network and self.netmask == other.netmask
-        return 0
-
-    def _get_networks_key(self):
-        """Network-only key function.
-
-        Returns an object that identifies this address' network and
-        netmask. This function is a suitable "key" argument for sorted()
-        and list.sort().
-
-        """
-        return (self._version, self.network, self.netmask)
-
-    def _ip_int_from_prefix(self, prefixlen=None):
-        """Turn the prefix length netmask into a int for comparison.
-
-        Args:
-            prefixlen: An integer, the prefix length.
-
-        Returns:
-            An integer.
-
-        """
-        if not prefixlen and prefixlen != 0:
-            prefixlen = self._prefixlen
-        return self._ALL_ONES ^ (self._ALL_ONES >> prefixlen)
-
-    def _prefix_from_ip_int(self, ip_int, mask=32):
-        """Return prefix length from the decimal netmask.
-
-        Args:
-            ip_int: An integer, the IP address.
-            mask: The netmask.  Defaults to 32.
-
-        Returns:
-            An integer, the prefix length.
-
-        """
-        while mask:
-            if ip_int & 1 == 1:
-                break
-            ip_int >>= 1
-            mask -= 1
-
-        return mask
-
-    def _ip_string_from_prefix(self, prefixlen=None):
-        """Turn a prefix length into a dotted decimal string.
-
-        Args:
-            prefixlen: An integer, the netmask prefix length.
-
-        Returns:
-            A string, the dotted decimal netmask string.
-
-        """
-        if not prefixlen:
-            prefixlen = self._prefixlen
-        return self._string_from_ip_int(self._ip_int_from_prefix(prefixlen))
-
-    def iter_subnets(self, prefixlen_diff=1, new_prefix=None):
-        """The subnets which join to make the current subnet.
-
-        In the case that self contains only one IP
-        (self._prefixlen == 32 for IPv4 or self._prefixlen == 128
-        for IPv6), return a list with just ourself.
-
-        Args:
-            prefixlen_diff: An integer, the amount the prefix length
-              should be increased by. This should not be set if
-              new_prefix is also set.
-            new_prefix: The desired new prefix length. This must be a
-              larger number (smaller prefix) than the existing prefix.
-              This should not be set if prefixlen_diff is also set.
-
-        Returns:
-            An iterator of IPv(4|6) objects.
-
-        Raises:
-            ValueError: The prefixlen_diff is too small or too large.
-                OR
-            prefixlen_diff and new_prefix are both set or new_prefix
-              is a smaller number than the current prefix (smaller
-              number means a larger network)
-
-        """
-        if self._prefixlen == self._max_prefixlen:
-            yield self
-            return
-
-        if new_prefix is not None:
-            if new_prefix < self._prefixlen:
-                raise ValueError('new prefix must be longer')
-            if prefixlen_diff != 1:
-                raise ValueError('cannot set prefixlen_diff and new_prefix')
-            prefixlen_diff = new_prefix - self._prefixlen
-
-        if prefixlen_diff < 0:
-            raise ValueError('prefix length diff must be > 0')
-        new_prefixlen = self._prefixlen + prefixlen_diff
-
-        if not self._is_valid_netmask(str(new_prefixlen)):
-            raise ValueError(
-                'prefix length diff %d is invalid for netblock %s' % (
-                    new_prefixlen, str(self)))
-
-        first = IPNetwork('%s/%s' % (str(self.network),
-                                     str(self._prefixlen + prefixlen_diff)),
-                         version=self._version)
-
-        yield first
-        current = first
-        while True:
-            broadcast = current.broadcast
-            if broadcast == self.broadcast:
-                return
-            new_addr = IPAddress(int(broadcast) + 1, version=self._version)
-            current = IPNetwork('%s/%s' % (str(new_addr), str(new_prefixlen)),
-                                version=self._version)
-
-            yield current
-
-    def masked(self):
-        """Return the network object with the host bits masked out."""
-        return IPNetwork('%s/%d' % (self.network, self._prefixlen),
-                         version=self._version)
-
-    def subnet(self, prefixlen_diff=1, new_prefix=None):
-        """Return a list of subnets, rather than an iterator."""
-        return list(self.iter_subnets(prefixlen_diff, new_prefix))
-
-    def supernet(self, prefixlen_diff=1, new_prefix=None):
-        """The supernet containing the current network.
-
-        Args:
-            prefixlen_diff: An integer, the amount the prefix length of
-              the network should be decreased by.  For example, given a
-              /24 network and a prefixlen_diff of 3, a supernet with a
-              /21 netmask is returned.
-
-        Returns:
-            An IPv4 network object.
-
-        Raises:
-            ValueError: If self.prefixlen - prefixlen_diff < 0. I.e., you have a
-              negative prefix length.
-                OR
-            If prefixlen_diff and new_prefix are both set or new_prefix is a
-              larger number than the current prefix (larger number means a
-              smaller network)
-
-        """
-        if self._prefixlen == 0:
-            return self
-
-        if new_prefix is not None:
-            if new_prefix > self._prefixlen:
-                raise ValueError('new prefix must be shorter')
-            if prefixlen_diff != 1:
-                raise ValueError('cannot set prefixlen_diff and new_prefix')
-            prefixlen_diff = self._prefixlen - new_prefix
-
-
-        if self.prefixlen - prefixlen_diff < 0:
-            raise ValueError(
-                'current prefixlen is %d, cannot have a prefixlen_diff of %d' %
-                (self.prefixlen, prefixlen_diff))
-        return IPNetwork('%s/%s' % (str(self.network),
-                                    str(self.prefixlen - prefixlen_diff)),
-                         version=self._version)
-
-    # backwards compatibility
-    Subnet = subnet
-    Supernet = supernet
-    AddressExclude = address_exclude
-    CompareNetworks = compare_networks
-    Contains = __contains__
-
-
-class _BaseV4(object):
-
-    """Base IPv4 object.
-
-    The following methods are used by IPv4 objects in both single IP
-    addresses and networks.
-
-    """
-
-    # Equivalent to 255.255.255.255 or 32 bits of 1's.
-    _ALL_ONES = (2**IPV4LENGTH) - 1
-    _DECIMAL_DIGITS = frozenset('0123456789')
-
-    def __init__(self, address):
-        self._version = 4
-        self._max_prefixlen = IPV4LENGTH
-
-    def _explode_shorthand_ip_string(self, ip_str=None):
-        if not ip_str:
-            ip_str = str(self)
-        return ip_str
-
-    def _ip_int_from_string(self, ip_str):
-        """Turn the given IP string into an integer for comparison.
-
-        Args:
-            ip_str: A string, the IP ip_str.
-
-        Returns:
-            The IP ip_str as an integer.
-
-        Raises:
-            AddressValueError: if ip_str isn't a valid IPv4 Address.
-
-        """
-        octets = ip_str.split('.')
-        if len(octets) != 4:
-            raise AddressValueError(ip_str)
-
-        packed_ip = 0
-        for oc in octets:
-            try:
-                packed_ip = (packed_ip << 8) | self._parse_octet(oc)
-            except ValueError:
-                raise AddressValueError(ip_str)
-        return packed_ip
-
-    def _parse_octet(self, octet_str):
-        """Convert a decimal octet into an integer.
-
-        Args:
-            octet_str: A string, the number to parse.
-
-        Returns:
-            The octet as an integer.
-
-        Raises:
-            ValueError: if the octet isn't strictly a decimal from [0..255].
-
-        """
-        # Whitelist the characters, since int() allows a lot of bizarre stuff.
-        if not self._DECIMAL_DIGITS.issuperset(octet_str):
-            raise ValueError
-        octet_int = int(octet_str, 10)
-        # Disallow leading zeroes, because no clear standard exists on
-        # whether these should be interpreted as decimal or octal.
-        if octet_int > 255 or (octet_str[0] == '0' and len(octet_str) > 1):
-            raise ValueError
-        return octet_int
-
-    def _string_from_ip_int(self, ip_int):
-        """Turns a 32-bit integer into dotted decimal notation.
-
-        Args:
-            ip_int: An integer, the IP address.
-
-        Returns:
-            The IP address as a string in dotted decimal notation.
-
-        """
-        octets = []
-        for _ in xrange(4):
-            octets.insert(0, str(ip_int & 0xFF))
-            ip_int >>= 8
-        return '.'.join(octets)
-
-    @property
-    def max_prefixlen(self):
-        return self._max_prefixlen
-
-    @property
-    def packed(self):
-        """The binary representation of this address."""
-        return v4_int_to_packed(self._ip)
-
-    @property
-    def version(self):
-        return self._version
-
-    @property
-    def is_reserved(self):
-       """Test if the address is otherwise IETF reserved.
-
-        Returns:
-            A boolean, True if the address is within the
-            reserved IPv4 Network range.
-
-       """
-       return self in IPv4Network('240.0.0.0/4')
-
-    @property
-    def is_private(self):
-        """Test if this address is allocated for private networks.
-
-        Returns:
-            A boolean, True if the address is reserved per RFC 1918.
-
-        """
-        return (self in IPv4Network('10.0.0.0/8') or
-                self in IPv4Network('172.16.0.0/12') or
-                self in IPv4Network('192.168.0.0/16'))
-
-    @property
-    def is_multicast(self):
-        """Test if the address is reserved for multicast use.
-
-        Returns:
-            A boolean, True if the address is multicast.
-            See RFC 3171 for details.
-
-        """
-        return self in IPv4Network('224.0.0.0/4')
-
-    @property
-    def is_unspecified(self):
-        """Test if the address is unspecified.
-
-        Returns:
-            A boolean, True if this is the unspecified address as defined in
-            RFC 5735 3.
-
-        """
-        return self in IPv4Network('0.0.0.0')
-
-    @property
-    def is_loopback(self):
-        """Test if the address is a loopback address.
-
-        Returns:
-            A boolean, True if the address is a loopback per RFC 3330.
-
-        """
-        return self in IPv4Network('127.0.0.0/8')
-
-    @property
-    def is_link_local(self):
-        """Test if the address is reserved for link-local.
-
-        Returns:
-            A boolean, True if the address is link-local per RFC 3927.
-
-        """
-        return self in IPv4Network('169.254.0.0/16')
-
-
-class IPv4Address(_BaseV4, _BaseIP):
-
-    """Represent and manipulate single IPv4 Addresses."""
-
-    def __init__(self, address):
-
-        """
-        Args:
-            address: A string or integer representing the IP
-              '192.168.1.1'
-
-              Additionally, an integer can be passed, so
-              IPv4Address('192.168.1.1') == IPv4Address(3232235777).
-              or, more generally
-              IPv4Address(int(IPv4Address('192.168.1.1'))) ==
-                IPv4Address('192.168.1.1')
-
-        Raises:
-            AddressValueError: If ipaddr isn't a valid IPv4 address.
-
-        """
-        _BaseIP.__init__(self, address)
-        _BaseV4.__init__(self, address)
-
-        # Efficient constructor from integer.
-        if isinstance(address, (int, long)):
-            self._ip = address
-            if address < 0 or address > self._ALL_ONES:
-                raise AddressValueError(address)
-            return
-
-        # Constructing from a packed address
-        if _compat_has_real_bytes:
-            if isinstance(address, bytes) and len(address) == 4:
-                self._ip = struct.unpack('!I', address)[0]
-                return
-
-        # Assume input argument to be string or any object representation
-        # which converts into a formatted IP string.
-        addr_str = str(address)
-        self._ip = self._ip_int_from_string(addr_str)
-
-
-class IPv4Network(_BaseV4, _BaseNet):
-
-    """This class represents and manipulates 32-bit IPv4 networks.
-
-    Attributes: [examples for IPv4Network('1.2.3.4/27')]
-        ._ip: 16909060
-        .ip: IPv4Address('1.2.3.4')
-        .network: IPv4Address('1.2.3.0')
-        .hostmask: IPv4Address('0.0.0.31')
-        .broadcast: IPv4Address('1.2.3.31')
-        .netmask: IPv4Address('255.255.255.224')
-        .prefixlen: 27
-
-    """
-
-    # the valid octets for host and netmasks. only useful for IPv4.
-    _valid_mask_octets = set((255, 254, 252, 248, 240, 224, 192, 128, 0))
-
-    def __init__(self, address, strict=False):
-        """Instantiate a new IPv4 network object.
-
-        Args:
-            address: A string or integer representing the IP [& network].
-              '192.168.1.1/24'
-              '192.168.1.1/255.255.255.0'
-              '192.168.1.1/0.0.0.255'
-              are all functionally the same in IPv4. Similarly,
-              '192.168.1.1'
-              '192.168.1.1/255.255.255.255'
-              '192.168.1.1/32'
-              are also functionaly equivalent. That is to say, failing to
-              provide a subnetmask will create an object with a mask of /32.
-
-              If the mask (portion after the / in the argument) is given in
-              dotted quad form, it is treated as a netmask if it starts with a
-              non-zero field (e.g. /255.0.0.0 == /8) and as a hostmask if it
-              starts with a zero field (e.g. 0.255.255.255 == /8), with the
-              single exception of an all-zero mask which is treated as a
-              netmask == /0. If no mask is given, a default of /32 is used.
-
-              Additionally, an integer can be passed, so
-              IPv4Network('192.168.1.1') == IPv4Network(3232235777).
-              or, more generally
-              IPv4Network(int(IPv4Network('192.168.1.1'))) ==
-                IPv4Network('192.168.1.1')
-
-            strict: A boolean. If true, ensure that we have been passed
-              A true network address, eg, 192.168.1.0/24 and not an
-              IP address on a network, eg, 192.168.1.1/24.
-
-        Raises:
-            AddressValueError: If ipaddr isn't a valid IPv4 address.
-            NetmaskValueError: If the netmask isn't valid for
-              an IPv4 address.
-            ValueError: If strict was True and a network address was not
-              supplied.
-
-        """
-        _BaseNet.__init__(self, address)
-        _BaseV4.__init__(self, address)
-
-        # Efficient constructor from integer.
-        if isinstance(address, (int, long)):
-            self._ip = address
-            self.ip = IPv4Address(self._ip)
-            self._prefixlen = self._max_prefixlen
-            self.netmask = IPv4Address(self._ALL_ONES)
-            if address < 0 or address > self._ALL_ONES:
-                raise AddressValueError(address)
-            return
-
-        # Constructing from a packed address
-        if _compat_has_real_bytes:
-            if isinstance(address, bytes) and len(address) == 4:
-                self._ip = struct.unpack('!I', address)[0]
-                self.ip = IPv4Address(self._ip)
-                self._prefixlen = self._max_prefixlen
-                self.netmask = IPv4Address(self._ALL_ONES)
-                return
-
-        # Assume input argument to be string or any object representation
-        # which converts into a formatted IP prefix string.
-        addr = str(address).split('/')
-
-        if len(addr) > 2:
-            raise AddressValueError(address)
-
-        self._ip = self._ip_int_from_string(addr[0])
-        self.ip = IPv4Address(self._ip)
-
-        if len(addr) == 2:
-            mask = addr[1].split('.')
-            if len(mask) == 4:
-                # We have dotted decimal netmask.
-                if self._is_valid_netmask(addr[1]):
-                    self.netmask = IPv4Address(self._ip_int_from_string(
-                            addr[1]))
-                elif self._is_hostmask(addr[1]):
-                    self.netmask = IPv4Address(
-                        self._ip_int_from_string(addr[1]) ^ self._ALL_ONES)
-                else:
-                    raise NetmaskValueError('%s is not a valid netmask'
-                                                     % addr[1])
-
-                self._prefixlen = self._prefix_from_ip_int(int(self.netmask))
-            else:
-                # We have a netmask in prefix length form.
-                if not self._is_valid_netmask(addr[1]):
-                    raise NetmaskValueError(addr[1])
-                self._prefixlen = int(addr[1])
-                self.netmask = IPv4Address(self._ip_int_from_prefix(
-                    self._prefixlen))
-        else:
-            self._prefixlen = self._max_prefixlen
-            self.netmask = IPv4Address(self._ip_int_from_prefix(
-                self._prefixlen))
-        if strict:
-            if self.ip != self.network:
-                raise ValueError('%s has host bits set' %
-                                 self.ip)
-
-    def _is_hostmask(self, ip_str):
-        """Test if the IP string is a hostmask (rather than a netmask).
-
-        Args:
-            ip_str: A string, the potential hostmask.
-
-        Returns:
-            A boolean, True if the IP string is a hostmask.
-
-        """
-        bits = ip_str.split('.')
-        try:
-            parts = [int(x) for x in bits if int(x) in self._valid_mask_octets]
-        except ValueError:
-            return False
-        if len(parts) != len(bits):
-            return False
-        if parts[0] < parts[-1]:
-            return True
-        return False
-
-    def _is_valid_netmask(self, netmask):
-        """Verify that the netmask is valid.
-
-        Args:
-            netmask: A string, either a prefix or dotted decimal
-              netmask.
-
-        Returns:
-            A boolean, True if the prefix represents a valid IPv4
-            netmask.
-
-        """
-        mask = netmask.split('.')
-        if len(mask) == 4:
-            if [x for x in mask if int(x) not in self._valid_mask_octets]:
-                return False
-            if [y for idx, y in enumerate(mask) if idx > 0 and
-                y > mask[idx - 1]]:
-                return False
-            return True
-        try:
-            netmask = int(netmask)
-        except ValueError:
-            return False
-        return 0 <= netmask <= self._max_prefixlen
-
-    # backwards compatibility
-    IsRFC1918 = lambda self: self.is_private
-    IsMulticast = lambda self: self.is_multicast
-    IsLoopback = lambda self: self.is_loopback
-    IsLinkLocal = lambda self: self.is_link_local
-
-
-class _BaseV6(object):
-
-    """Base IPv6 object.
-
-    The following methods are used by IPv6 objects in both single IP
-    addresses and networks.
-
-    """
-
-    _ALL_ONES = (2**IPV6LENGTH) - 1
-    _HEXTET_COUNT = 8
-    _HEX_DIGITS = frozenset('0123456789ABCDEFabcdef')
-
-    def __init__(self, address):
-        self._version = 6
-        self._max_prefixlen = IPV6LENGTH
-
-    def _ip_int_from_string(self, ip_str):
-        """Turn an IPv6 ip_str into an integer.
-
-        Args:
-            ip_str: A string, the IPv6 ip_str.
-
-        Returns:
-            A long, the IPv6 ip_str.
-
-        Raises:
-            AddressValueError: if ip_str isn't a valid IPv6 Address.
-
-        """
-        parts = ip_str.split(':')
-
-        # An IPv6 address needs at least 2 colons (3 parts).
-        if len(parts) < 3:
-            raise AddressValueError(ip_str)
-
-        # If the address has an IPv4-style suffix, convert it to hexadecimal.
-        if '.' in parts[-1]:
-            ipv4_int = IPv4Address(parts.pop())._ip
-            parts.append('%x' % ((ipv4_int >> 16) & 0xFFFF))
-            parts.append('%x' % (ipv4_int & 0xFFFF))
-
-        # An IPv6 address can't have more than 8 colons (9 parts).
-        if len(parts) > self._HEXTET_COUNT + 1:
-            raise AddressValueError(ip_str)
-
-        # Disregarding the endpoints, find '::' with nothing in between.
-        # This indicates that a run of zeroes has been skipped.
-        try:
-            skip_index, = (
-                [i for i in xrange(1, len(parts) - 1) if not parts[i]] or
-                [None])
-        except ValueError:
-            # Can't have more than one '::'
-            raise AddressValueError(ip_str)
-
-        # parts_hi is the number of parts to copy from above/before the '::'
-        # parts_lo is the number of parts to copy from below/after the '::'
-        if skip_index is not None:
-            # If we found a '::', then check if it also covers the endpoints.
-            parts_hi = skip_index
-            parts_lo = len(parts) - skip_index - 1
-            if not parts[0]:
-                parts_hi -= 1
-                if parts_hi:
-                    raise AddressValueError(ip_str)  # ^: requires ^::
-            if not parts[-1]:
-                parts_lo -= 1
-                if parts_lo:
-                    raise AddressValueError(ip_str)  # :$ requires ::$
-            parts_skipped = self._HEXTET_COUNT - (parts_hi + parts_lo)
-            if parts_skipped < 1:
-                raise AddressValueError(ip_str)
-        else:
-            # Otherwise, allocate the entire address to parts_hi.  The endpoints
-            # could still be empty, but _parse_hextet() will check for that.
-            if len(parts) != self._HEXTET_COUNT:
-                raise AddressValueError(ip_str)
-            parts_hi = len(parts)
-            parts_lo = 0
-            parts_skipped = 0
-
-        try:
-            # Now, parse the hextets into a 128-bit integer.
-            ip_int = 0L
-            for i in xrange(parts_hi):
-                ip_int <<= 16
-                ip_int |= self._parse_hextet(parts[i])
-            ip_int <<= 16 * parts_skipped
-            for i in xrange(-parts_lo, 0):
-                ip_int <<= 16
-                ip_int |= self._parse_hextet(parts[i])
-            return ip_int
-        except ValueError:
-            raise AddressValueError(ip_str)
-
-    def _parse_hextet(self, hextet_str):
-        """Convert an IPv6 hextet string into an integer.
-
-        Args:
-            hextet_str: A string, the number to parse.
-
-        Returns:
-            The hextet as an integer.
-
-        Raises:
-            ValueError: if the input isn't strictly a hex number from [0..FFFF].
-
-        """
-        # Whitelist the characters, since int() allows a lot of bizarre stuff.
-        if not self._HEX_DIGITS.issuperset(hextet_str):
-            raise ValueError
-        hextet_int = int(hextet_str, 16)
-        if hextet_int > 0xFFFF:
-            raise ValueError
-        return hextet_int
-
-    def _compress_hextets(self, hextets):
-        """Compresses a list of hextets.
-
-        Compresses a list of strings, replacing the longest continuous
-        sequence of "0" in the list with "" and adding empty strings at
-        the beginning or at the end of the string such that subsequently
-        calling ":".join(hextets) will produce the compressed version of
-        the IPv6 address.
-
-        Args:
-            hextets: A list of strings, the hextets to compress.
-
-        Returns:
-            A list of strings.
-
-        """
-        best_doublecolon_start = -1
-        best_doublecolon_len = 0
-        doublecolon_start = -1
-        doublecolon_len = 0
-        for index in range(len(hextets)):
-            if hextets[index] == '0':
-                doublecolon_len += 1
-                if doublecolon_start == -1:
-                    # Start of a sequence of zeros.
-                    doublecolon_start = index
-                if doublecolon_len > best_doublecolon_len:
-                    # This is the longest sequence of zeros so far.
-                    best_doublecolon_len = doublecolon_len
-                    best_doublecolon_start = doublecolon_start
-            else:
-                doublecolon_len = 0
-                doublecolon_start = -1
-
-        if best_doublecolon_len > 1:
-            best_doublecolon_end = (best_doublecolon_start +
-                                    best_doublecolon_len)
-            # For zeros at the end of the address.
-            if best_doublecolon_end == len(hextets):
-                hextets += ['']
-            hextets[best_doublecolon_start:best_doublecolon_end] = ['']
-            # For zeros at the beginning of the address.
-            if best_doublecolon_start == 0:
-                hextets = [''] + hextets
-
-        return hextets
-
-    def _string_from_ip_int(self, ip_int=None):
-        """Turns a 128-bit integer into hexadecimal notation.
-
-        Args:
-            ip_int: An integer, the IP address.
-
-        Returns:
-            A string, the hexadecimal representation of the address.
-
-        Raises:
-            ValueError: The address is bigger than 128 bits of all ones.
-
-        """
-        if not ip_int and ip_int != 0:
-            ip_int = int(self._ip)
-
-        if ip_int > self._ALL_ONES:
-            raise ValueError('IPv6 address is too large')
-
-        hex_str = '%032x' % ip_int
-        hextets = []
-        for x in range(0, 32, 4):
-            hextets.append('%x' % int(hex_str[x:x+4], 16))
-
-        hextets = self._compress_hextets(hextets)
-        return ':'.join(hextets)
-
-    def _explode_shorthand_ip_string(self, ip_str=None):
-        """Expand a shortened IPv6 address.
-
-        Args:
-            ip_str: A string, the IPv6 address.
-
-        Returns:
-            A string, the expanded IPv6 address.
-
-        """
-        if not ip_str:
-            ip_str = str(self)
-            if isinstance(self, _BaseNet):
-                ip_str = str(self.ip)
-
-        ip_int = self._ip_int_from_string(ip_str)
-        parts = []
-        for i in xrange(self._HEXTET_COUNT):
-            parts.append('%04x' % (ip_int & 0xFFFF))
-            ip_int >>= 16
-        parts.reverse()
-        return ':'.join(parts)
-
-    @property
-    def max_prefixlen(self):
-        return self._max_prefixlen
-
-    @property
-    def packed(self):
-        """The binary representation of this address."""
-        return v6_int_to_packed(self._ip)
-
-    @property
-    def version(self):
-        return self._version
-
-    @property
-    def is_multicast(self):
-        """Test if the address is reserved for multicast use.
-
-        Returns:
-            A boolean, True if the address is a multicast address.
-            See RFC 2373 2.7 for details.
-
-        """
-        return self in IPv6Network('ff00::/8')
-
-    @property
-    def is_reserved(self):
-        """Test if the address is otherwise IETF reserved.
-
-        Returns:
-            A boolean, True if the address is within one of the
-            reserved IPv6 Network ranges.
-
-        """
-        return (self in IPv6Network('::/8') or
-                self in IPv6Network('100::/8') or
-                self in IPv6Network('200::/7') or
-                self in IPv6Network('400::/6') or
-                self in IPv6Network('800::/5') or
-                self in IPv6Network('1000::/4') or
-                self in IPv6Network('4000::/3') or
-                self in IPv6Network('6000::/3') or
-                self in IPv6Network('8000::/3') or
-                self in IPv6Network('A000::/3') or
-                self in IPv6Network('C000::/3') or
-                self in IPv6Network('E000::/4') or
-                self in IPv6Network('F000::/5') or
-                self in IPv6Network('F800::/6') or
-                self in IPv6Network('FE00::/9'))
-
-    @property
-    def is_unspecified(self):
-        """Test if the address is unspecified.
-
-        Returns:
-            A boolean, True if this is the unspecified address as defined in
-            RFC 2373 2.5.2.
-
-        """
-        return self._ip == 0 and getattr(self, '_prefixlen', 128) == 128
-
-    @property
-    def is_loopback(self):
-        """Test if the address is a loopback address.
-
-        Returns:
-            A boolean, True if the address is a loopback address as defined in
-            RFC 2373 2.5.3.
-
-        """
-        return self._ip == 1 and getattr(self, '_prefixlen', 128) == 128
-
-    @property
-    def is_link_local(self):
-        """Test if the address is reserved for link-local.
-
-        Returns:
-            A boolean, True if the address is reserved per RFC 4291.
-
-        """
-        return self in IPv6Network('fe80::/10')
-
-    @property
-    def is_site_local(self):
-        """Test if the address is reserved for site-local.
-
-        Note that the site-local address space has been deprecated by RFC 3879.
-        Use is_private to test if this address is in the space of unique local
-        addresses as defined by RFC 4193.
-
-        Returns:
-            A boolean, True if the address is reserved per RFC 3513 2.5.6.
-
-        """
-        return self in IPv6Network('fec0::/10')
-
-    @property
-    def is_private(self):
-        """Test if this address is allocated for private networks.
-
-        Returns:
-            A boolean, True if the address is reserved per RFC 4193.
-
-        """
-        return self in IPv6Network('fc00::/7')
-
-    @property
-    def ipv4_mapped(self):
-        """Return the IPv4 mapped address.
-
-        Returns:
-            If the IPv6 address is a v4 mapped address, return the
-            IPv4 mapped address. Return None otherwise.
-
-        """
-        if (self._ip >> 32) != 0xFFFF:
-            return None
-        return IPv4Address(self._ip & 0xFFFFFFFF)
-
-    @property
-    def teredo(self):
-        """Tuple of embedded teredo IPs.
-
-        Returns:
-            Tuple of the (server, client) IPs or None if the address
-            doesn't appear to be a teredo address (doesn't start with
-            2001::/32)
-
-        """
-        if (self._ip >> 96) != 0x20010000:
-            return None
-        return (IPv4Address((self._ip >> 64) & 0xFFFFFFFF),
-                IPv4Address(~self._ip & 0xFFFFFFFF))
-
-    @property
-    def sixtofour(self):
-        """Return the IPv4 6to4 embedded address.
-
-        Returns:
-            The IPv4 6to4-embedded address if present or None if the
-            address doesn't appear to contain a 6to4 embedded address.
-
-        """
-        if (self._ip >> 112) != 0x2002:
-            return None
-        return IPv4Address((self._ip >> 80) & 0xFFFFFFFF)
-
-
-class IPv6Address(_BaseV6, _BaseIP):
-
-    """Represent and manipulate single IPv6 Addresses.
-    """
-
-    def __init__(self, address):
-        """Instantiate a new IPv6 address object.
-
-        Args:
-            address: A string or integer representing the IP
-
-              Additionally, an integer can be passed, so
-              IPv6Address('2001:4860::') ==
-                IPv6Address(42541956101370907050197289607612071936L).
-              or, more generally
-              IPv6Address(IPv6Address('2001:4860::')._ip) ==
-                IPv6Address('2001:4860::')
-
-        Raises:
-            AddressValueError: If address isn't a valid IPv6 address.
-
-        """
-        _BaseIP.__init__(self, address)
-        _BaseV6.__init__(self, address)
-
-        # Efficient constructor from integer.
-        if isinstance(address, (int, long)):
-            self._ip = address
-            if address < 0 or address > self._ALL_ONES:
-                raise AddressValueError(address)
-            return
-
-        # Constructing from a packed address
-        if _compat_has_real_bytes:
-            if isinstance(address, bytes) and len(address) == 16:
-                tmp = struct.unpack('!QQ', address)
-                self._ip = (tmp[0] << 64) | tmp[1]
-                return
-
-        # Assume input argument to be string or any object representation
-        # which converts into a formatted IP string.
-        addr_str = str(address)
-        if not addr_str:
-            raise AddressValueError('')
-
-        self._ip = self._ip_int_from_string(addr_str)
-
-
-class IPv6Network(_BaseV6, _BaseNet):
-
-    """This class represents and manipulates 128-bit IPv6 networks.
-
-    Attributes: [examples for IPv6('2001:658:22A:CAFE:200::1/64')]
-        .ip: IPv6Address('2001:658:22a:cafe:200::1')
-        .network: IPv6Address('2001:658:22a:cafe::')
-        .hostmask: IPv6Address('::ffff:ffff:ffff:ffff')
-        .broadcast: IPv6Address('2001:658:22a:cafe:ffff:ffff:ffff:ffff')
-        .netmask: IPv6Address('ffff:ffff:ffff:ffff::')
-        .prefixlen: 64
-
-    """
-
-
-    def __init__(self, address, strict=False):
-        """Instantiate a new IPv6 Network object.
-
-        Args:
-            address: A string or integer representing the IPv6 network or the IP
-              and prefix/netmask.
-              '2001:4860::/128'
-              '2001:4860:0000:0000:0000:0000:0000:0000/128'
-              '2001:4860::'
-              are all functionally the same in IPv6.  That is to say,
-              failing to provide a subnetmask will create an object with
-              a mask of /128.
-
-              Additionally, an integer can be passed, so
-              IPv6Network('2001:4860::') ==
-                IPv6Network(42541956101370907050197289607612071936L).
-              or, more generally
-              IPv6Network(IPv6Network('2001:4860::')._ip) ==
-                IPv6Network('2001:4860::')
-
-            strict: A boolean. If true, ensure that we have been passed
-              A true network address, eg, 192.168.1.0/24 and not an
-              IP address on a network, eg, 192.168.1.1/24.
-
-        Raises:
-            AddressValueError: If address isn't a valid IPv6 address.
-            NetmaskValueError: If the netmask isn't valid for
-              an IPv6 address.
-            ValueError: If strict was True and a network address was not
-              supplied.
-
-        """
-        _BaseNet.__init__(self, address)
-        _BaseV6.__init__(self, address)
-
-        # Efficient constructor from integer.
-        if isinstance(address, (int, long)):
-            self._ip = address
-            self.ip = IPv6Address(self._ip)
-            self._prefixlen = self._max_prefixlen
-            self.netmask = IPv6Address(self._ALL_ONES)
-            if address < 0 or address > self._ALL_ONES:
-                raise AddressValueError(address)
-            return
-
-        # Constructing from a packed address
-        if _compat_has_real_bytes:
-            if isinstance(address, bytes) and len(address) == 16:
-                tmp = struct.unpack('!QQ', address)
-                self._ip = (tmp[0] << 64) | tmp[1]
-                self.ip = IPv6Address(self._ip)
-                self._prefixlen = self._max_prefixlen
-                self.netmask = IPv6Address(self._ALL_ONES)
-                return
-
-        # Assume input argument to be string or any object representation
-        # which converts into a formatted IP prefix string.
-        addr = str(address).split('/')
-
-        if len(addr) > 2:
-            raise AddressValueError(address)
-
-        self._ip = self._ip_int_from_string(addr[0])
-        self.ip = IPv6Address(self._ip)
-
-        if len(addr) == 2:
-            if self._is_valid_netmask(addr[1]):
-                self._prefixlen = int(addr[1])
-            else:
-                raise NetmaskValueError(addr[1])
-        else:
-            self._prefixlen = self._max_prefixlen
-
-        self.netmask = IPv6Address(self._ip_int_from_prefix(self._prefixlen))
-
-        if strict:
-            if self.ip != self.network:
-                raise ValueError('%s has host bits set' %
-                                 self.ip)
-
-    def _is_valid_netmask(self, prefixlen):
-        """Verify that the netmask/prefixlen is valid.
-
-        Args:
-            prefixlen: A string, the netmask in prefix length format.
-
-        Returns:
-            A boolean, True if the prefix represents a valid IPv6
-            netmask.
-
-        """
-        try:
-            prefixlen = int(prefixlen)
-        except ValueError:
-            return False
-        return 0 <= prefixlen <= self._max_prefixlen
-
-    @property
-    def with_netmask(self):
-        return self.with_prefixlen
diff --git a/contrib/ipaddr-py/ipaddr_test.py b/contrib/ipaddr-py/ipaddr_test.py
deleted file mode 100755
index 09bece0e751..00000000000
--- a/contrib/ipaddr-py/ipaddr_test.py
+++ /dev/null
@@ -1,1099 +0,0 @@
-#!/usr/bin/python
-#
-# Copyright 2007 Google Inc.
-#  Licensed to PSF under a Contributor Agreement.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""Unittest for ipaddr module."""
-
-
-import unittest
-import time
-import ipaddr
-
-# Compatibility function to cast str to bytes objects
-if ipaddr._compat_has_real_bytes:
-    _cb = lambda bytestr: bytes(bytestr, 'charmap')
-else:
-    _cb = str
-
-class IpaddrUnitTest(unittest.TestCase):
-
-    def setUp(self):
-        self.ipv4 = ipaddr.IPv4Network('1.2.3.4/24')
-        self.ipv4_hostmask = ipaddr.IPv4Network('10.0.0.1/0.255.255.255')
-        self.ipv6 = ipaddr.IPv6Network('2001:658:22a:cafe:200:0:0:1/64')
-
-    def tearDown(self):
-        del(self.ipv4)
-        del(self.ipv4_hostmask)
-        del(self.ipv6)
-        del(self)
-
-    def testRepr(self):
-        self.assertEqual("IPv4Network('1.2.3.4/32')",
-                         repr(ipaddr.IPv4Network('1.2.3.4')))
-        self.assertEqual("IPv6Network('::1/128')",
-                         repr(ipaddr.IPv6Network('::1')))
-
-    def testAutoMasking(self):
-        addr1 = ipaddr.IPv4Network('1.1.1.255/24')
-        addr1_masked = ipaddr.IPv4Network('1.1.1.0/24')
-        self.assertEqual(addr1_masked, addr1.masked())
-
-        addr2 = ipaddr.IPv6Network('2000:cafe::efac:100/96')
-        addr2_masked = ipaddr.IPv6Network('2000:cafe::/96')
-        self.assertEqual(addr2_masked, addr2.masked())
-
-    # issue57
-    def testAddressIntMath(self):
-        self.assertEqual(ipaddr.IPv4Address('1.1.1.1') + 255,
-                         ipaddr.IPv4Address('1.1.2.0'))
-        self.assertEqual(ipaddr.IPv4Address('1.1.1.1') - 256,
-                         ipaddr.IPv4Address('1.1.0.1'))
-        self.assertEqual(ipaddr.IPv6Address('::1') + (2**16 - 2),
-                         ipaddr.IPv6Address('::ffff'))
-        self.assertEqual(ipaddr.IPv6Address('::ffff') - (2**16 - 2),
-                         ipaddr.IPv6Address('::1'))
-
-    def testInvalidStrings(self):
-        def AssertInvalidIP(ip_str):
-            self.assertRaises(ValueError, ipaddr.IPAddress, ip_str)
-        AssertInvalidIP("")
-        AssertInvalidIP("016.016.016.016")
-        AssertInvalidIP("016.016.016")
-        AssertInvalidIP("016.016")
-        AssertInvalidIP("016")
-        AssertInvalidIP("000.000.000.000")
-        AssertInvalidIP("000")
-        AssertInvalidIP("0x0a.0x0a.0x0a.0x0a")
-        AssertInvalidIP("0x0a.0x0a.0x0a")
-        AssertInvalidIP("0x0a.0x0a")
-        AssertInvalidIP("0x0a")
-        AssertInvalidIP("42.42.42.42.42")
-        AssertInvalidIP("42.42.42")
-        AssertInvalidIP("42.42")
-        AssertInvalidIP("42")
-        AssertInvalidIP("42..42.42")
-        AssertInvalidIP("42..42.42.42")
-        AssertInvalidIP("42.42.42.42.")
-        AssertInvalidIP("42.42.42.42...")
-        AssertInvalidIP(".42.42.42.42")
-        AssertInvalidIP("...42.42.42.42")
-        AssertInvalidIP("42.42.42.-0")
-        AssertInvalidIP("42.42.42.+0")
-        AssertInvalidIP(".")
-        AssertInvalidIP("...")
-        AssertInvalidIP("bogus")
-        AssertInvalidIP("bogus.com")
-        AssertInvalidIP("192.168.0.1.com")
-        AssertInvalidIP("12345.67899.-54321.-98765")
-        AssertInvalidIP("257.0.0.0")
-        AssertInvalidIP("42.42.42.-42")
-        AssertInvalidIP("3ffe::1.net")
-        AssertInvalidIP("3ffe::1::1")
-        AssertInvalidIP("1::2::3::4:5")
-        AssertInvalidIP("::7:6:5:4:3:2:")
-        AssertInvalidIP(":6:5:4:3:2:1::")
-        AssertInvalidIP("2001::db:::1")
-        AssertInvalidIP("FEDC:9878")
-        AssertInvalidIP("+1.+2.+3.4")
-        AssertInvalidIP("1.2.3.4e0")
-        AssertInvalidIP("::7:6:5:4:3:2:1:0")
-        AssertInvalidIP("7:6:5:4:3:2:1:0::")
-        AssertInvalidIP("9:8:7:6:5:4:3::2:1")
-        AssertInvalidIP("0:1:2:3::4:5:6:7")
-        AssertInvalidIP("3ffe:0:0:0:0:0:0:0:1")
-        AssertInvalidIP("3ffe::10000")
-        AssertInvalidIP("3ffe::goog")
-        AssertInvalidIP("3ffe::-0")
-        AssertInvalidIP("3ffe::+0")
-        AssertInvalidIP("3ffe::-1")
-        AssertInvalidIP(":")
-        AssertInvalidIP(":::")
-        AssertInvalidIP("::1.2.3")
-        AssertInvalidIP("::1.2.3.4.5")
-        AssertInvalidIP("::1.2.3.4:")
-        AssertInvalidIP("1.2.3.4::")
-        AssertInvalidIP("2001:db8::1:")
-        AssertInvalidIP(":2001:db8::1")
-        AssertInvalidIP(":1:2:3:4:5:6:7")
-        AssertInvalidIP("1:2:3:4:5:6:7:")
-        AssertInvalidIP(":1:2:3:4:5:6:")
-
-        self.assertRaises(ipaddr.AddressValueError, ipaddr.IPv4Network, '')
-        self.assertRaises(ipaddr.AddressValueError, ipaddr.IPv4Network,
-                          'google.com')
-        self.assertRaises(ipaddr.AddressValueError, ipaddr.IPv4Network,
-                          '::1.2.3.4')
-        self.assertRaises(ipaddr.AddressValueError, ipaddr.IPv6Network, '')
-        self.assertRaises(ipaddr.AddressValueError, ipaddr.IPv6Network,
-                          'google.com')
-        self.assertRaises(ipaddr.AddressValueError, ipaddr.IPv6Network,
-                          '1.2.3.4')
-        self.assertRaises(ipaddr.AddressValueError, ipaddr.IPv6Network,
-                          'cafe:cafe::/128/190')
-        self.assertRaises(ipaddr.AddressValueError, ipaddr.IPv6Network,
-                          '1234:axy::b')
-        self.assertRaises(ipaddr.AddressValueError, ipaddr.IPv6Address,
-                          '1234:axy::b')
-        self.assertRaises(ipaddr.AddressValueError, ipaddr.IPv6Address,
-                          '2001:db8:::1')
-        self.assertRaises(ipaddr.AddressValueError, ipaddr.IPv6Address,
-                          '2001:888888::1')
-        self.assertRaises(ipaddr.AddressValueError,
-                          ipaddr.IPv4Address(1)._ip_int_from_string,
-                          '1.a.2.3')
-        self.assertEqual(False, ipaddr.IPv4Network(1)._is_hostmask('1.a.2.3'))
-
-    def testGetNetwork(self):
-        self.assertEqual(int(self.ipv4.network), 16909056)
-        self.assertEqual(str(self.ipv4.network), '1.2.3.0')
-        self.assertEqual(str(self.ipv4_hostmask.network), '10.0.0.0')
-
-        self.assertEqual(int(self.ipv6.network),
-                         42540616829182469433403647294022090752)
-        self.assertEqual(str(self.ipv6.network),
-                         '2001:658:22a:cafe::')
-        self.assertEqual(str(self.ipv6.hostmask),
-                         '::ffff:ffff:ffff:ffff')
-
-    def testBadVersionComparison(self):
-        # These should always raise TypeError
-        v4addr = ipaddr.IPAddress('1.1.1.1')
-        v4net = ipaddr.IPNetwork('1.1.1.1')
-        v6addr = ipaddr.IPAddress('::1')
-        v6net = ipaddr.IPAddress('::1')
-
-        self.assertRaises(TypeError, v4addr.__lt__, v6addr)
-        self.assertRaises(TypeError, v4addr.__gt__, v6addr)
-        self.assertRaises(TypeError, v4net.__lt__, v6net)
-        self.assertRaises(TypeError, v4net.__gt__, v6net)
-
-        self.assertRaises(TypeError, v6addr.__lt__, v4addr)
-        self.assertRaises(TypeError, v6addr.__gt__, v4addr)
-        self.assertRaises(TypeError, v6net.__lt__, v4net)
-        self.assertRaises(TypeError, v6net.__gt__, v4net)
-
-    def testMixedTypeComparison(self):
-        v4addr = ipaddr.IPAddress('1.1.1.1')
-        v4net = ipaddr.IPNetwork('1.1.1.1/32')
-        v6addr = ipaddr.IPAddress('::1')
-        v6net = ipaddr.IPNetwork('::1/128')
-
-        self.assertFalse(v4net.__contains__(v6net))
-        self.assertFalse(v6net.__contains__(v4net))
-
-        self.assertRaises(TypeError, lambda: v4addr < v4net)
-        self.assertRaises(TypeError, lambda: v4addr > v4net)
-        self.assertRaises(TypeError, lambda: v4net < v4addr)
-        self.assertRaises(TypeError, lambda: v4net > v4addr)
-
-        self.assertRaises(TypeError, lambda: v6addr < v6net)
-        self.assertRaises(TypeError, lambda: v6addr > v6net)
-        self.assertRaises(TypeError, lambda: v6net < v6addr)
-        self.assertRaises(TypeError, lambda: v6net > v6addr)
-
-        # with get_mixed_type_key, you can sort addresses and network.
-        self.assertEqual([v4addr, v4net], sorted([v4net, v4addr],
-                                                 key=ipaddr.get_mixed_type_key))
-        self.assertEqual([v6addr, v6net], sorted([v6net, v6addr],
-                                                 key=ipaddr.get_mixed_type_key))
-
-    def testIpFromInt(self):
-        self.assertEqual(self.ipv4.ip, ipaddr.IPv4Network(16909060).ip)
-        self.assertRaises(ipaddr.AddressValueError,
-                          ipaddr.IPv4Network, 2**32)
-        self.assertRaises(ipaddr.AddressValueError,
-                          ipaddr.IPv4Network, -1)
-
-        ipv4 = ipaddr.IPNetwork('1.2.3.4')
-        ipv6 = ipaddr.IPNetwork('2001:658:22a:cafe:200:0:0:1')
-        self.assertEqual(ipv4, ipaddr.IPNetwork(int(ipv4)))
-        self.assertEqual(ipv6, ipaddr.IPNetwork(int(ipv6)))
-
-        v6_int = 42540616829182469433547762482097946625
-        self.assertEqual(self.ipv6.ip, ipaddr.IPv6Network(v6_int).ip)
-        self.assertRaises(ipaddr.AddressValueError,
-                          ipaddr.IPv6Network, 2**128)
-        self.assertRaises(ipaddr.AddressValueError,
-                          ipaddr.IPv6Network, -1)
-
-        self.assertEqual(ipaddr.IPNetwork(self.ipv4.ip).version, 4)
-        self.assertEqual(ipaddr.IPNetwork(self.ipv6.ip).version, 6)
-
-    if ipaddr._compat_has_real_bytes: # on python3+
-        def testIpFromPacked(self):
-            ip = ipaddr.IPNetwork
-
-            self.assertEqual(self.ipv4.ip,
-                             ip(_cb('\x01\x02\x03\x04')).ip)
-            self.assertEqual(ip('255.254.253.252'),
-                             ip(_cb('\xff\xfe\xfd\xfc')))
-            self.assertRaises(ValueError, ipaddr.IPNetwork, _cb('\x00' * 3))
-            self.assertRaises(ValueError, ipaddr.IPNetwork, _cb('\x00' * 5))
-            self.assertEqual(self.ipv6.ip,
-                             ip(_cb('\x20\x01\x06\x58\x02\x2a\xca\xfe'
-                               '\x02\x00\x00\x00\x00\x00\x00\x01')).ip)
-            self.assertEqual(ip('ffff:2:3:4:ffff::'),
-                             ip(_cb('\xff\xff\x00\x02\x00\x03\x00\x04' +
-                                   '\xff\xff' + '\x00' * 6)))
-            self.assertEqual(ip('::'),
-                             ip(_cb('\x00' * 16)))
-            self.assertRaises(ValueError, ip, _cb('\x00' * 15))
-            self.assertRaises(ValueError, ip, _cb('\x00' * 17))
-
-    def testGetIp(self):
-        self.assertEqual(int(self.ipv4.ip), 16909060)
-        self.assertEqual(str(self.ipv4.ip), '1.2.3.4')
-        self.assertEqual(str(self.ipv4_hostmask.ip), '10.0.0.1')
-
-        self.assertEqual(int(self.ipv6.ip),
-                         42540616829182469433547762482097946625)
-        self.assertEqual(str(self.ipv6.ip),
-                         '2001:658:22a:cafe:200::1')
-
-    def testGetNetmask(self):
-        self.assertEqual(int(self.ipv4.netmask), 4294967040L)
-        self.assertEqual(str(self.ipv4.netmask), '255.255.255.0')
-        self.assertEqual(str(self.ipv4_hostmask.netmask), '255.0.0.0')
-        self.assertEqual(int(self.ipv6.netmask),
-                         340282366920938463444927863358058659840)
-        self.assertEqual(self.ipv6.prefixlen, 64)
-
-    def testZeroNetmask(self):
-        ipv4_zero_netmask = ipaddr.IPv4Network('1.2.3.4/0')
-        self.assertEqual(int(ipv4_zero_netmask.netmask), 0)
-        self.assertTrue(ipv4_zero_netmask._is_valid_netmask(str(0)))
-
-        ipv6_zero_netmask = ipaddr.IPv6Network('::1/0')
-        self.assertEqual(int(ipv6_zero_netmask.netmask), 0)
-        self.assertTrue(ipv6_zero_netmask._is_valid_netmask(str(0)))
-
-    def testGetBroadcast(self):
-        self.assertEqual(int(self.ipv4.broadcast), 16909311L)
-        self.assertEqual(str(self.ipv4.broadcast), '1.2.3.255')
-
-        self.assertEqual(int(self.ipv6.broadcast),
-                         42540616829182469451850391367731642367)
-        self.assertEqual(str(self.ipv6.broadcast),
-                         '2001:658:22a:cafe:ffff:ffff:ffff:ffff')
-
-    def testGetPrefixlen(self):
-        self.assertEqual(self.ipv4.prefixlen, 24)
-
-        self.assertEqual(self.ipv6.prefixlen, 64)
-
-    def testGetSupernet(self):
-        self.assertEqual(self.ipv4.supernet().prefixlen, 23)
-        self.assertEqual(str(self.ipv4.supernet().network), '1.2.2.0')
-        self.assertEqual(ipaddr.IPv4Network('0.0.0.0/0').supernet(),
-                         ipaddr.IPv4Network('0.0.0.0/0'))
-
-        self.assertEqual(self.ipv6.supernet().prefixlen, 63)
-        self.assertEqual(str(self.ipv6.supernet().network),
-                         '2001:658:22a:cafe::')
-        self.assertEqual(ipaddr.IPv6Network('::0/0').supernet(),
-                         ipaddr.IPv6Network('::0/0'))
-
-    def testGetSupernet3(self):
-        self.assertEqual(self.ipv4.supernet(3).prefixlen, 21)
-        self.assertEqual(str(self.ipv4.supernet(3).network), '1.2.0.0')
-
-        self.assertEqual(self.ipv6.supernet(3).prefixlen, 61)
-        self.assertEqual(str(self.ipv6.supernet(3).network),
-                         '2001:658:22a:caf8::')
-
-    def testGetSupernet4(self):
-        self.assertRaises(ValueError, self.ipv4.supernet, prefixlen_diff=2,
-                          new_prefix=1)
-        self.assertRaises(ValueError, self.ipv4.supernet, new_prefix=25)
-        self.assertEqual(self.ipv4.supernet(prefixlen_diff=2),
-                         self.ipv4.supernet(new_prefix=22))
-
-        self.assertRaises(ValueError, self.ipv6.supernet, prefixlen_diff=2,
-                          new_prefix=1)
-        self.assertRaises(ValueError, self.ipv6.supernet, new_prefix=65)
-        self.assertEqual(self.ipv6.supernet(prefixlen_diff=2),
-                         self.ipv6.supernet(new_prefix=62))
-
-    def testIterSubnets(self):
-        self.assertEqual(self.ipv4.subnet(), list(self.ipv4.iter_subnets()))
-        self.assertEqual(self.ipv6.subnet(), list(self.ipv6.iter_subnets()))
-
-    def testFancySubnetting(self):
-        self.assertEqual(sorted(self.ipv4.subnet(prefixlen_diff=3)),
-                         sorted(self.ipv4.subnet(new_prefix=27)))
-        self.assertRaises(ValueError, self.ipv4.subnet, new_prefix=23)
-        self.assertRaises(ValueError, self.ipv4.subnet,
-                          prefixlen_diff=3, new_prefix=27)
-        self.assertEqual(sorted(self.ipv6.subnet(prefixlen_diff=4)),
-                         sorted(self.ipv6.subnet(new_prefix=68)))
-        self.assertRaises(ValueError, self.ipv6.subnet, new_prefix=63)
-        self.assertRaises(ValueError, self.ipv6.subnet,
-                          prefixlen_diff=4, new_prefix=68)
-
-    def testGetSubnet(self):
-        self.assertEqual(self.ipv4.subnet()[0].prefixlen, 25)
-        self.assertEqual(str(self.ipv4.subnet()[0].network), '1.2.3.0')
-        self.assertEqual(str(self.ipv4.subnet()[1].network), '1.2.3.128')
-
-        self.assertEqual(self.ipv6.subnet()[0].prefixlen, 65)
-
-    def testGetSubnetForSingle32(self):
-        ip = ipaddr.IPv4Network('1.2.3.4/32')
-        subnets1 = [str(x) for x in ip.subnet()]
-        subnets2 = [str(x) for x in ip.subnet(2)]
-        self.assertEqual(subnets1, ['1.2.3.4/32'])
-        self.assertEqual(subnets1, subnets2)
-
-    def testGetSubnetForSingle128(self):
-        ip = ipaddr.IPv6Network('::1/128')
-        subnets1 = [str(x) for x in ip.subnet()]
-        subnets2 = [str(x) for x in ip.subnet(2)]
-        self.assertEqual(subnets1, ['::1/128'])
-        self.assertEqual(subnets1, subnets2)
-
-    def testSubnet2(self):
-        ips = [str(x) for x in self.ipv4.subnet(2)]
-        self.assertEqual(
-            ips,
-            ['1.2.3.0/26', '1.2.3.64/26', '1.2.3.128/26', '1.2.3.192/26'])
-
-        ipsv6 = [str(x) for x in self.ipv6.subnet(2)]
-        self.assertEqual(
-            ipsv6,
-            ['2001:658:22a:cafe::/66',
-             '2001:658:22a:cafe:4000::/66',
-             '2001:658:22a:cafe:8000::/66',
-             '2001:658:22a:cafe:c000::/66'])
-
-    def testSubnetFailsForLargeCidrDiff(self):
-        self.assertRaises(ValueError, self.ipv4.subnet, 9)
-        self.assertRaises(ValueError, self.ipv6.subnet, 65)
-
-    def testSupernetFailsForLargeCidrDiff(self):
-        self.assertRaises(ValueError, self.ipv4.supernet, 25)
-        self.assertRaises(ValueError, self.ipv6.supernet, 65)
-
-    def testSubnetFailsForNegativeCidrDiff(self):
-        self.assertRaises(ValueError, self.ipv4.subnet, -1)
-        self.assertRaises(ValueError, self.ipv6.subnet, -1)
-
-    def testGetNumHosts(self):
-        self.assertEqual(self.ipv4.numhosts, 256)
-        self.assertEqual(self.ipv4.subnet()[0].numhosts, 128)
-        self.assertEqual(self.ipv4.supernet().numhosts, 512)
-
-        self.assertEqual(self.ipv6.numhosts, 18446744073709551616)
-        self.assertEqual(self.ipv6.subnet()[0].numhosts, 9223372036854775808)
-        self.assertEqual(self.ipv6.supernet().numhosts, 36893488147419103232)
-
-    def testContains(self):
-        self.assertTrue(ipaddr.IPv4Network('1.2.3.128/25') in self.ipv4)
-        self.assertFalse(ipaddr.IPv4Network('1.2.4.1/24') in self.ipv4)
-        self.assertTrue(self.ipv4 in self.ipv4)
-        self.assertTrue(self.ipv6 in self.ipv6)
-        # We can test addresses and string as well.
-        addr1 = ipaddr.IPv4Address('1.2.3.37')
-        self.assertTrue(addr1 in self.ipv4)
-        # issue 61, bad network comparison on like-ip'd network objects
-        # with identical broadcast addresses.
-        self.assertFalse(ipaddr.IPv4Network('1.1.0.0/16').__contains__(
-                ipaddr.IPv4Network('1.0.0.0/15')))
-
-    def testBadAddress(self):
-        self.assertRaises(ipaddr.AddressValueError, ipaddr.IPv4Network,
-                          'poop')
-        self.assertRaises(ipaddr.AddressValueError,
-                          ipaddr.IPv4Network, '1.2.3.256')
-
-        self.assertRaises(ipaddr.AddressValueError, ipaddr.IPv6Network,
-                          'poopv6')
-        self.assertRaises(ipaddr.AddressValueError,
-                          ipaddr.IPv4Network, '1.2.3.4/32/24')
-        self.assertRaises(ipaddr.AddressValueError,
-                          ipaddr.IPv4Network, '10/8')
-        self.assertRaises(ipaddr.AddressValueError,
-                          ipaddr.IPv6Network, '10/8')
-
-
-    def testBadNetMask(self):
-        self.assertRaises(ipaddr.NetmaskValueError,
-                          ipaddr.IPv4Network, '1.2.3.4/')
-        self.assertRaises(ipaddr.NetmaskValueError,
-                          ipaddr.IPv4Network, '1.2.3.4/33')
-        self.assertRaises(ipaddr.NetmaskValueError,
-                          ipaddr.IPv4Network, '1.2.3.4/254.254.255.256')
-        self.assertRaises(ipaddr.NetmaskValueError,
-                          ipaddr.IPv4Network, '1.1.1.1/240.255.0.0')
-        self.assertRaises(ipaddr.NetmaskValueError,
-                          ipaddr.IPv6Network, '::1/')
-        self.assertRaises(ipaddr.NetmaskValueError,
-                          ipaddr.IPv6Network, '::1/129')
-
-    def testNth(self):
-        self.assertEqual(str(self.ipv4[5]), '1.2.3.5')
-        self.assertRaises(IndexError, self.ipv4.__getitem__, 256)
-
-        self.assertEqual(str(self.ipv6[5]),
-                         '2001:658:22a:cafe::5')
-
-    def testGetitem(self):
-        # http://code.google.com/p/ipaddr-py/issues/detail?id=15
-        addr = ipaddr.IPv4Network('172.31.255.128/255.255.255.240')
-        self.assertEqual(28, addr.prefixlen)
-        addr_list = list(addr)
-        self.assertEqual('172.31.255.128', str(addr_list[0]))
-        self.assertEqual('172.31.255.128', str(addr[0]))
-        self.assertEqual('172.31.255.143', str(addr_list[-1]))
-        self.assertEqual('172.31.255.143', str(addr[-1]))
-        self.assertEqual(addr_list[-1], addr[-1])
-
-    def testEqual(self):
-        self.assertTrue(self.ipv4 == ipaddr.IPv4Network('1.2.3.4/24'))
-        self.assertFalse(self.ipv4 == ipaddr.IPv4Network('1.2.3.4/23'))
-        self.assertFalse(self.ipv4 == ipaddr.IPv6Network('::1.2.3.4/24'))
-        self.assertFalse(self.ipv4 == '')
-        self.assertFalse(self.ipv4 == [])
-        self.assertFalse(self.ipv4 == 2)
-        self.assertTrue(ipaddr.IPNetwork('1.1.1.1/32') ==
-                        ipaddr.IPAddress('1.1.1.1'))
-        self.assertTrue(ipaddr.IPNetwork('1.1.1.1/24') ==
-                        ipaddr.IPAddress('1.1.1.1'))
-        self.assertFalse(ipaddr.IPNetwork('1.1.1.0/24') ==
-                         ipaddr.IPAddress('1.1.1.1'))
-
-        self.assertTrue(self.ipv6 ==
-            ipaddr.IPv6Network('2001:658:22a:cafe:200::1/64'))
-        self.assertTrue(ipaddr.IPNetwork('::1/128') ==
-                        ipaddr.IPAddress('::1'))
-        self.assertTrue(ipaddr.IPNetwork('::1/127') ==
-                        ipaddr.IPAddress('::1'))
-        self.assertFalse(ipaddr.IPNetwork('::0/127') ==
-                         ipaddr.IPAddress('::1'))
-        self.assertFalse(self.ipv6 ==
-            ipaddr.IPv6Network('2001:658:22a:cafe:200::1/63'))
-        self.assertFalse(self.ipv6 == ipaddr.IPv4Network('1.2.3.4/23'))
-        self.assertFalse(self.ipv6 == '')
-        self.assertFalse(self.ipv6 == [])
-        self.assertFalse(self.ipv6 == 2)
-
-    def testNotEqual(self):
-        self.assertFalse(self.ipv4 != ipaddr.IPv4Network('1.2.3.4/24'))
-        self.assertTrue(self.ipv4 != ipaddr.IPv4Network('1.2.3.4/23'))
-        self.assertTrue(self.ipv4 != ipaddr.IPv6Network('::1.2.3.4/24'))
-        self.assertTrue(self.ipv4 != '')
-        self.assertTrue(self.ipv4 != [])
-        self.assertTrue(self.ipv4 != 2)
-
-        addr2 = ipaddr.IPAddress('2001:658:22a:cafe:200::1')
-        self.assertFalse(self.ipv6 !=
-            ipaddr.IPv6Network('2001:658:22a:cafe:200::1/64'))
-        self.assertTrue(self.ipv6 !=
-            ipaddr.IPv6Network('2001:658:22a:cafe:200::1/63'))
-        self.assertTrue(self.ipv6 != ipaddr.IPv4Network('1.2.3.4/23'))
-        self.assertTrue(self.ipv6 != '')
-        self.assertTrue(self.ipv6 != [])
-        self.assertTrue(self.ipv6 != 2)
-
-    def testSlash32Constructor(self):
-        self.assertEqual(str(ipaddr.IPv4Network('1.2.3.4/255.255.255.255')),
-                          '1.2.3.4/32')
-
-    def testSlash128Constructor(self):
-        self.assertEqual(str(ipaddr.IPv6Network('::1/128')),
-                                  '::1/128')
-
-    def testSlash0Constructor(self):
-        self.assertEqual(str(ipaddr.IPv4Network('1.2.3.4/0.0.0.0')),
-                          '1.2.3.4/0')
-
-    def testCollapsing(self):
-        # test only IP addresses including some duplicates
-        ip1 = ipaddr.IPv4Address('1.1.1.0')
-        ip2 = ipaddr.IPv4Address('1.1.1.1')
-        ip3 = ipaddr.IPv4Address('1.1.1.2')
-        ip4 = ipaddr.IPv4Address('1.1.1.3')
-        ip5 = ipaddr.IPv4Address('1.1.1.4')
-        ip6 = ipaddr.IPv4Address('1.1.1.0')
-        # check that addreses are subsumed properly.
-        collapsed = ipaddr.collapse_address_list([ip1, ip2, ip3, ip4, ip5, ip6])
-        self.assertEqual(collapsed, [ipaddr.IPv4Network('1.1.1.0/30'),
-                                     ipaddr.IPv4Network('1.1.1.4/32')])
-
-        # test a mix of IP addresses and networks including some duplicates
-        ip1 = ipaddr.IPv4Address('1.1.1.0')
-        ip2 = ipaddr.IPv4Address('1.1.1.1')
-        ip3 = ipaddr.IPv4Address('1.1.1.2')
-        ip4 = ipaddr.IPv4Address('1.1.1.3')
-        ip5 = ipaddr.IPv4Network('1.1.1.4/30')
-        ip6 = ipaddr.IPv4Network('1.1.1.4/30')
-        # check that addreses are subsumed properly.
-        collapsed = ipaddr.collapse_address_list([ip5, ip1, ip2, ip3, ip4, ip6])
-        self.assertEqual(collapsed, [ipaddr.IPv4Network('1.1.1.0/29')])
-
-        # test only IP networks
-        ip1 = ipaddr.IPv4Network('1.1.0.0/24')
-        ip2 = ipaddr.IPv4Network('1.1.1.0/24')
-        ip3 = ipaddr.IPv4Network('1.1.2.0/24')
-        ip4 = ipaddr.IPv4Network('1.1.3.0/24')
-        ip5 = ipaddr.IPv4Network('1.1.4.0/24')
-        # stored in no particular order b/c we want CollapseAddr to call [].sort
-        ip6 = ipaddr.IPv4Network('1.1.0.0/22')
-        # check that addreses are subsumed properly.
-        collapsed = ipaddr.collapse_address_list([ip1, ip2, ip3, ip4, ip5, ip6])
-        self.assertEqual(collapsed, [ipaddr.IPv4Network('1.1.0.0/22'),
-                                     ipaddr.IPv4Network('1.1.4.0/24')])
-
-        # test that two addresses are supernet'ed properly
-        collapsed = ipaddr.collapse_address_list([ip1, ip2])
-        self.assertEqual(collapsed, [ipaddr.IPv4Network('1.1.0.0/23')])
-
-        # test same IP networks
-        ip_same1 = ip_same2 = ipaddr.IPv4Network('1.1.1.1/32')
-        self.assertEqual(ipaddr.collapse_address_list([ip_same1, ip_same2]),
-                         [ip_same1])
-
-        # test same IP addresses
-        ip_same1 = ip_same2 = ipaddr.IPv4Address('1.1.1.1')
-        self.assertEqual(ipaddr.collapse_address_list([ip_same1, ip_same2]),
-                         [ipaddr.IPNetwork('1.1.1.1/32')])
-        ip1 = ipaddr.IPv6Network('::2001:1/100')
-        ip2 = ipaddr.IPv6Network('::2002:1/120')
-        ip3 = ipaddr.IPv6Network('::2001:1/96')
-        # test that ipv6 addresses are subsumed properly.
-        collapsed = ipaddr.collapse_address_list([ip1, ip2, ip3])
-        self.assertEqual(collapsed, [ip3])
-
-        # the toejam test
-        ip1 = ipaddr.IPAddress('1.1.1.1')
-        ip2 = ipaddr.IPAddress('::1')
-        self.assertRaises(TypeError, ipaddr.collapse_address_list,
-                          [ip1, ip2])
-
-    def testSummarizing(self):
-        #ip = ipaddr.IPAddress
-        #ipnet = ipaddr.IPNetwork
-        summarize = ipaddr.summarize_address_range
-        ip1 = ipaddr.IPAddress('1.1.1.0')
-        ip2 = ipaddr.IPAddress('1.1.1.255')
-        # test a /24 is sumamrized properly
-        self.assertEqual(summarize(ip1, ip2)[0], ipaddr.IPNetwork('1.1.1.0/24'))
-        # test an  IPv4 range that isn't on a network byte boundary
-        ip2 = ipaddr.IPAddress('1.1.1.8')
-        self.assertEqual(summarize(ip1, ip2), [ipaddr.IPNetwork('1.1.1.0/29'),
-                                               ipaddr.IPNetwork('1.1.1.8')])
-
-        ip1 = ipaddr.IPAddress('1::')
-        ip2 = ipaddr.IPAddress('1:ffff:ffff:ffff:ffff:ffff:ffff:ffff')
-        # test a IPv6 is sumamrized properly
-        self.assertEqual(summarize(ip1, ip2)[0], ipaddr.IPNetwork('1::/16'))
-        # test an IPv6 range that isn't on a network byte boundary
-        ip2 = ipaddr.IPAddress('2::')
-        self.assertEqual(summarize(ip1, ip2), [ipaddr.IPNetwork('1::/16'),
-                                               ipaddr.IPNetwork('2::/128')])
-
-        # test exception raised when first is greater than last
-        self.assertRaises(ValueError, summarize, ipaddr.IPAddress('1.1.1.0'),
-            ipaddr.IPAddress('1.1.0.0'))
-        # test exception raised when first and last aren't IP addresses
-        self.assertRaises(TypeError, summarize,
-                          ipaddr.IPNetwork('1.1.1.0'),
-                          ipaddr.IPNetwork('1.1.0.0'))
-        self.assertRaises(TypeError, summarize,
-            ipaddr.IPNetwork('1.1.1.0'), ipaddr.IPNetwork('1.1.0.0'))
-        # test exception raised when first and last are not same version
-        self.assertRaises(TypeError, summarize, ipaddr.IPAddress('::'),
-            ipaddr.IPNetwork('1.1.0.0'))
-
-    def testAddressComparison(self):
-        self.assertTrue(ipaddr.IPAddress('1.1.1.1') <=
-                        ipaddr.IPAddress('1.1.1.1'))
-        self.assertTrue(ipaddr.IPAddress('1.1.1.1') <=
-                        ipaddr.IPAddress('1.1.1.2'))
-        self.assertTrue(ipaddr.IPAddress('::1') <= ipaddr.IPAddress('::1'))
-        self.assertTrue(ipaddr.IPAddress('::1') <= ipaddr.IPAddress('::2'))
-
-    def testNetworkComparison(self):
-        # ip1 and ip2 have the same network address
-        ip1 = ipaddr.IPv4Network('1.1.1.0/24')
-        ip2 = ipaddr.IPv4Network('1.1.1.1/24')
-        ip3 = ipaddr.IPv4Network('1.1.2.0/24')
-
-        self.assertTrue(ip1 < ip3)
-        self.assertTrue(ip3 > ip2)
-
-        self.assertEqual(ip1.compare_networks(ip2), 0)
-        self.assertTrue(ip1._get_networks_key() == ip2._get_networks_key())
-        self.assertEqual(ip1.compare_networks(ip3), -1)
-        self.assertTrue(ip1._get_networks_key() < ip3._get_networks_key())
-
-        ip1 = ipaddr.IPv6Network('2001::2000/96')
-        ip2 = ipaddr.IPv6Network('2001::2001/96')
-        ip3 = ipaddr.IPv6Network('2001:ffff::2000/96')
-
-        self.assertTrue(ip1 < ip3)
-        self.assertTrue(ip3 > ip2)
-        self.assertEqual(ip1.compare_networks(ip2), 0)
-        self.assertTrue(ip1._get_networks_key() == ip2._get_networks_key())
-        self.assertEqual(ip1.compare_networks(ip3), -1)
-        self.assertTrue(ip1._get_networks_key() < ip3._get_networks_key())
-
-        # Test comparing different protocols.
-        # Should always raise a TypeError.
-        ipv6 = ipaddr.IPv6Network('::/0')
-        ipv4 = ipaddr.IPv4Network('0.0.0.0/0')
-        self.assertRaises(TypeError, ipv4.__lt__, ipv6)
-        self.assertRaises(TypeError, ipv4.__gt__, ipv6)
-        self.assertRaises(TypeError, ipv6.__lt__, ipv4)
-        self.assertRaises(TypeError, ipv6.__gt__, ipv4)
-
-        # Regression test for issue 19.
-        ip1 = ipaddr.IPNetwork('10.1.2.128/25')
-        self.assertFalse(ip1 < ip1)
-        self.assertFalse(ip1 > ip1)
-        ip2 = ipaddr.IPNetwork('10.1.3.0/24')
-        self.assertTrue(ip1 < ip2)
-        self.assertFalse(ip2 < ip1)
-        self.assertFalse(ip1 > ip2)
-        self.assertTrue(ip2 > ip1)
-        ip3 = ipaddr.IPNetwork('10.1.3.0/25')
-        self.assertTrue(ip2 < ip3)
-        self.assertFalse(ip3 < ip2)
-        self.assertFalse(ip2 > ip3)
-        self.assertTrue(ip3 > ip2)
-
-        # Regression test for issue 28.
-        ip1 = ipaddr.IPNetwork('10.10.10.0/31')
-        ip2 = ipaddr.IPNetwork('10.10.10.0')
-        ip3 = ipaddr.IPNetwork('10.10.10.2/31')
-        ip4 = ipaddr.IPNetwork('10.10.10.2')
-        sorted = [ip1, ip2, ip3, ip4]
-        unsorted = [ip2, ip4, ip1, ip3]
-        unsorted.sort()
-        self.assertEqual(sorted, unsorted)
-        unsorted = [ip4, ip1, ip3, ip2]
-        unsorted.sort()
-        self.assertEqual(sorted, unsorted)
-        self.assertRaises(TypeError, ip1.__lt__, ipaddr.IPAddress('10.10.10.0'))
-        self.assertRaises(TypeError, ip2.__lt__, ipaddr.IPAddress('10.10.10.0'))
-
-        # <=, >=
-        self.assertTrue(ipaddr.IPNetwork('1.1.1.1') <=
-                        ipaddr.IPNetwork('1.1.1.1'))
-        self.assertTrue(ipaddr.IPNetwork('1.1.1.1') <=
-                        ipaddr.IPNetwork('1.1.1.2'))
-        self.assertFalse(ipaddr.IPNetwork('1.1.1.2') <=
-                        ipaddr.IPNetwork('1.1.1.1'))
-        self.assertTrue(ipaddr.IPNetwork('::1') <= ipaddr.IPNetwork('::1'))
-        self.assertTrue(ipaddr.IPNetwork('::1') <= ipaddr.IPNetwork('::2'))
-        self.assertFalse(ipaddr.IPNetwork('::2') <= ipaddr.IPNetwork('::1'))
-
-    def testStrictNetworks(self):
-        self.assertRaises(ValueError, ipaddr.IPNetwork, '192.168.1.1/24',
-                          strict=True)
-        self.assertRaises(ValueError, ipaddr.IPNetwork, '::1/120', strict=True)
-
-    def testOverlaps(self):
-        other = ipaddr.IPv4Network('1.2.3.0/30')
-        other2 = ipaddr.IPv4Network('1.2.2.0/24')
-        other3 = ipaddr.IPv4Network('1.2.2.64/26')
-        self.assertTrue(self.ipv4.overlaps(other))
-        self.assertFalse(self.ipv4.overlaps(other2))
-        self.assertTrue(other2.overlaps(other3))
-
-    def testEmbeddedIpv4(self):
-        ipv4_string = '192.168.0.1'
-        ipv4 = ipaddr.IPv4Network(ipv4_string)
-        v4compat_ipv6 = ipaddr.IPv6Network('::%s' % ipv4_string)
-        self.assertEqual(int(v4compat_ipv6.ip), int(ipv4.ip))
-        v4mapped_ipv6 = ipaddr.IPv6Network('::ffff:%s' % ipv4_string)
-        self.assertNotEqual(v4mapped_ipv6.ip, ipv4.ip)
-        self.assertRaises(ipaddr.AddressValueError, ipaddr.IPv6Network,
-                          '2001:1.1.1.1:1.1.1.1')
-
-    # Issue 67: IPv6 with embedded IPv4 address not recognized.
-    def testIPv6AddressTooLarge(self):
-        # RFC4291 2.5.5.2
-        self.assertEqual(ipaddr.IPAddress('::FFFF:192.0.2.1'),
-                          ipaddr.IPAddress('::FFFF:c000:201'))
-        # RFC4291 2.2 (part 3) x::d.d.d.d 
-        self.assertEqual(ipaddr.IPAddress('FFFF::192.0.2.1'),
-                          ipaddr.IPAddress('FFFF::c000:201'))
-
-    def testIPVersion(self):
-        self.assertEqual(self.ipv4.version, 4)
-        self.assertEqual(self.ipv6.version, 6)
-
-    def testMaxPrefixLength(self):
-        self.assertEqual(self.ipv4.max_prefixlen, 32)
-        self.assertEqual(self.ipv6.max_prefixlen, 128)
-
-    def testPacked(self):
-        self.assertEqual(self.ipv4.packed,
-                         _cb('\x01\x02\x03\x04'))
-        self.assertEqual(ipaddr.IPv4Network('255.254.253.252').packed,
-                         _cb('\xff\xfe\xfd\xfc'))
-        self.assertEqual(self.ipv6.packed,
-                         _cb('\x20\x01\x06\x58\x02\x2a\xca\xfe'
-                             '\x02\x00\x00\x00\x00\x00\x00\x01'))
-        self.assertEqual(ipaddr.IPv6Network('ffff:2:3:4:ffff::').packed,
-                         _cb('\xff\xff\x00\x02\x00\x03\x00\x04\xff\xff'
-                            + '\x00' * 6))
-        self.assertEqual(ipaddr.IPv6Network('::1:0:0:0:0').packed,
-                         _cb('\x00' * 6 + '\x00\x01' + '\x00' * 8))
-
-    def testIpStrFromPrefixlen(self):
-        ipv4 = ipaddr.IPv4Network('1.2.3.4/24')
-        self.assertEqual(ipv4._ip_string_from_prefix(), '255.255.255.0')
-        self.assertEqual(ipv4._ip_string_from_prefix(28), '255.255.255.240')
-
-    def testIpType(self):
-        ipv4net = ipaddr.IPNetwork('1.2.3.4')
-        ipv4addr = ipaddr.IPAddress('1.2.3.4')
-        ipv6net = ipaddr.IPNetwork('::1.2.3.4')
-        ipv6addr = ipaddr.IPAddress('::1.2.3.4')
-        self.assertEqual(ipaddr.IPv4Network, type(ipv4net))
-        self.assertEqual(ipaddr.IPv4Address, type(ipv4addr))
-        self.assertEqual(ipaddr.IPv6Network, type(ipv6net))
-        self.assertEqual(ipaddr.IPv6Address, type(ipv6addr))
-
-    def testReservedIpv4(self):
-        # test networks
-        self.assertEqual(True, ipaddr.IPNetwork('224.1.1.1/31').is_multicast)
-        self.assertEqual(False, ipaddr.IPNetwork('240.0.0.0').is_multicast)
-
-        self.assertEqual(True, ipaddr.IPNetwork('192.168.1.1/17').is_private)
-        self.assertEqual(False, ipaddr.IPNetwork('192.169.0.0').is_private)
-        self.assertEqual(True, ipaddr.IPNetwork('10.255.255.255').is_private)
-        self.assertEqual(False, ipaddr.IPNetwork('11.0.0.0').is_private)
-        self.assertEqual(True, ipaddr.IPNetwork('172.31.255.255').is_private)
-        self.assertEqual(False, ipaddr.IPNetwork('172.32.0.0').is_private)
-
-        self.assertEqual(True,
-                          ipaddr.IPNetwork('169.254.100.200/24').is_link_local)
-        self.assertEqual(False,
-                          ipaddr.IPNetwork('169.255.100.200/24').is_link_local)
-
-        self.assertEqual(True,
-                          ipaddr.IPNetwork('127.100.200.254/32').is_loopback)
-        self.assertEqual(True, ipaddr.IPNetwork('127.42.0.0/16').is_loopback)
-        self.assertEqual(False, ipaddr.IPNetwork('128.0.0.0').is_loopback)
-
-        # test addresses
-        self.assertEqual(True, ipaddr.IPAddress('224.1.1.1').is_multicast)
-        self.assertEqual(False, ipaddr.IPAddress('240.0.0.0').is_multicast)
-
-        self.assertEqual(True, ipaddr.IPAddress('192.168.1.1').is_private)
-        self.assertEqual(False, ipaddr.IPAddress('192.169.0.0').is_private)
-        self.assertEqual(True, ipaddr.IPAddress('10.255.255.255').is_private)
-        self.assertEqual(False, ipaddr.IPAddress('11.0.0.0').is_private)
-        self.assertEqual(True, ipaddr.IPAddress('172.31.255.255').is_private)
-        self.assertEqual(False, ipaddr.IPAddress('172.32.0.0').is_private)
-
-        self.assertEqual(True,
-                          ipaddr.IPAddress('169.254.100.200').is_link_local)
-        self.assertEqual(False,
-                          ipaddr.IPAddress('169.255.100.200').is_link_local)
-
-        self.assertEqual(True,
-                          ipaddr.IPAddress('127.100.200.254').is_loopback)
-        self.assertEqual(True, ipaddr.IPAddress('127.42.0.0').is_loopback)
-        self.assertEqual(False, ipaddr.IPAddress('128.0.0.0').is_loopback)
-        self.assertEqual(True, ipaddr.IPNetwork('0.0.0.0').is_unspecified)
-
-    def testReservedIpv6(self):
-
-        self.assertEqual(True, ipaddr.IPNetwork('ffff::').is_multicast)
-        self.assertEqual(True, ipaddr.IPNetwork(2**128-1).is_multicast)
-        self.assertEqual(True, ipaddr.IPNetwork('ff00::').is_multicast)
-        self.assertEqual(False, ipaddr.IPNetwork('fdff::').is_multicast)
-
-        self.assertEqual(True, ipaddr.IPNetwork('fecf::').is_site_local)
-        self.assertEqual(True, ipaddr.IPNetwork(
-                'feff:ffff:ffff:ffff::').is_site_local)
-        self.assertEqual(False, ipaddr.IPNetwork('fbf:ffff::').is_site_local)
-        self.assertEqual(False, ipaddr.IPNetwork('ff00::').is_site_local)
-
-        self.assertEqual(True, ipaddr.IPNetwork('fc00::').is_private)
-        self.assertEqual(True, ipaddr.IPNetwork(
-                'fc00:ffff:ffff:ffff::').is_private)
-        self.assertEqual(False, ipaddr.IPNetwork('fbff:ffff::').is_private)
-        self.assertEqual(False, ipaddr.IPNetwork('fe00::').is_private)
-
-        self.assertEqual(True, ipaddr.IPNetwork('fea0::').is_link_local)
-        self.assertEqual(True, ipaddr.IPNetwork('febf:ffff::').is_link_local)
-        self.assertEqual(False, ipaddr.IPNetwork('fe7f:ffff::').is_link_local)
-        self.assertEqual(False, ipaddr.IPNetwork('fec0::').is_link_local)
-
-        self.assertEqual(True, ipaddr.IPNetwork('0:0::0:01').is_loopback)
-        self.assertEqual(False, ipaddr.IPNetwork('::1/127').is_loopback)
-        self.assertEqual(False, ipaddr.IPNetwork('::').is_loopback)
-        self.assertEqual(False, ipaddr.IPNetwork('::2').is_loopback)
-
-        self.assertEqual(True, ipaddr.IPNetwork('0::0').is_unspecified)
-        self.assertEqual(False, ipaddr.IPNetwork('::1').is_unspecified)
-        self.assertEqual(False, ipaddr.IPNetwork('::/127').is_unspecified)
-
-        # test addresses
-        self.assertEqual(True, ipaddr.IPAddress('ffff::').is_multicast)
-        self.assertEqual(True, ipaddr.IPAddress(2**128-1).is_multicast)
-        self.assertEqual(True, ipaddr.IPAddress('ff00::').is_multicast)
-        self.assertEqual(False, ipaddr.IPAddress('fdff::').is_multicast)
-
-        self.assertEqual(True, ipaddr.IPAddress('fecf::').is_site_local)
-        self.assertEqual(True, ipaddr.IPAddress(
-                'feff:ffff:ffff:ffff::').is_site_local)
-        self.assertEqual(False, ipaddr.IPAddress('fbf:ffff::').is_site_local)
-        self.assertEqual(False, ipaddr.IPAddress('ff00::').is_site_local)
-
-        self.assertEqual(True, ipaddr.IPAddress('fc00::').is_private)
-        self.assertEqual(True, ipaddr.IPAddress(
-                'fc00:ffff:ffff:ffff::').is_private)
-        self.assertEqual(False, ipaddr.IPAddress('fbff:ffff::').is_private)
-        self.assertEqual(False, ipaddr.IPAddress('fe00::').is_private)
-
-        self.assertEqual(True, ipaddr.IPAddress('fea0::').is_link_local)
-        self.assertEqual(True, ipaddr.IPAddress('febf:ffff::').is_link_local)
-        self.assertEqual(False, ipaddr.IPAddress('fe7f:ffff::').is_link_local)
-        self.assertEqual(False, ipaddr.IPAddress('fec0::').is_link_local)
-
-        self.assertEqual(True, ipaddr.IPAddress('0:0::0:01').is_loopback)
-        self.assertEqual(True, ipaddr.IPAddress('::1').is_loopback)
-        self.assertEqual(False, ipaddr.IPAddress('::2').is_loopback)
-
-        self.assertEqual(True, ipaddr.IPAddress('0::0').is_unspecified)
-        self.assertEqual(False, ipaddr.IPAddress('::1').is_unspecified)
-
-        # some generic IETF reserved addresses
-        self.assertEqual(True, ipaddr.IPAddress('100::').is_reserved)
-        self.assertEqual(True, ipaddr.IPNetwork('4000::1/128').is_reserved)
-
-    def testIpv4Mapped(self):
-        self.assertEqual(ipaddr.IPAddress('::ffff:192.168.1.1').ipv4_mapped,
-                         ipaddr.IPAddress('192.168.1.1'))
-        self.assertEqual(ipaddr.IPAddress('::c0a8:101').ipv4_mapped, None)
-        self.assertEqual(ipaddr.IPAddress('::ffff:c0a8:101').ipv4_mapped,
-                         ipaddr.IPAddress('192.168.1.1'))
-
-    def testAddrExclude(self):
-        addr1 = ipaddr.IPNetwork('10.1.1.0/24')
-        addr2 = ipaddr.IPNetwork('10.1.1.0/26')
-        addr3 = ipaddr.IPNetwork('10.2.1.0/24')
-        addr4 = ipaddr.IPAddress('10.1.1.0')
-        self.assertEqual(addr1.address_exclude(addr2),
-                         [ipaddr.IPNetwork('10.1.1.64/26'),
-                          ipaddr.IPNetwork('10.1.1.128/25')])
-        self.assertRaises(ValueError, addr1.address_exclude, addr3)
-        self.assertRaises(TypeError, addr1.address_exclude, addr4)
-        self.assertEqual(addr1.address_exclude(addr1), [])
-
-    def testHash(self):
-        self.assertEqual(hash(ipaddr.IPNetwork('10.1.1.0/24')),
-                          hash(ipaddr.IPNetwork('10.1.1.0/24')))
-        self.assertEqual(hash(ipaddr.IPAddress('10.1.1.0')),
-                          hash(ipaddr.IPAddress('10.1.1.0')))
-        # i70
-        self.assertEqual(hash(ipaddr.IPAddress('1.2.3.4')),
-                          hash(ipaddr.IPAddress(
-                    long(ipaddr.IPAddress('1.2.3.4')._ip))))
-        ip1 = ipaddr.IPAddress('10.1.1.0')
-        ip2 = ipaddr.IPAddress('1::')
-        dummy = {}
-        dummy[self.ipv4] = None
-        dummy[self.ipv6] = None
-        dummy[ip1] = None
-        dummy[ip2] = None
-        self.assertTrue(self.ipv4 in dummy)
-        self.assertTrue(ip2 in dummy)
-
-    def testCopyConstructor(self):
-        addr1 = ipaddr.IPNetwork('10.1.1.0/24')
-        addr2 = ipaddr.IPNetwork(addr1)
-        addr3 = ipaddr.IPNetwork('2001:658:22a:cafe:200::1/64')
-        addr4 = ipaddr.IPNetwork(addr3)
-        addr5 = ipaddr.IPv4Address('1.1.1.1')
-        addr6 = ipaddr.IPv6Address('2001:658:22a:cafe:200::1')
-
-        self.assertEqual(addr1, addr2)
-        self.assertEqual(addr3, addr4)
-        self.assertEqual(addr5, ipaddr.IPv4Address(addr5))
-        self.assertEqual(addr6, ipaddr.IPv6Address(addr6))
-
-    def testCompressIPv6Address(self):
-        test_addresses = {
-            '1:2:3:4:5:6:7:8': '1:2:3:4:5:6:7:8/128',
-            '2001:0:0:4:0:0:0:8': '2001:0:0:4::8/128',
-            '2001:0:0:4:5:6:7:8': '2001::4:5:6:7:8/128',
-            '2001:0:3:4:5:6:7:8': '2001:0:3:4:5:6:7:8/128',
-            '2001:0:3:4:5:6:7:8': '2001:0:3:4:5:6:7:8/128',
-            '0:0:3:0:0:0:0:ffff': '0:0:3::ffff/128',
-            '0:0:0:4:0:0:0:ffff': '::4:0:0:0:ffff/128',
-            '0:0:0:0:5:0:0:ffff': '::5:0:0:ffff/128',
-            '1:0:0:4:0:0:7:8': '1::4:0:0:7:8/128',
-            '0:0:0:0:0:0:0:0': '::/128',
-            '0:0:0:0:0:0:0:0/0': '::/0',
-            '0:0:0:0:0:0:0:1': '::1/128',
-            '2001:0658:022a:cafe:0000:0000:0000:0000/66':
-            '2001:658:22a:cafe::/66',
-            '::1.2.3.4': '::102:304/128',
-            '1:2:3:4:5:ffff:1.2.3.4': '1:2:3:4:5:ffff:102:304/128',
-            '::7:6:5:4:3:2:1': '0:7:6:5:4:3:2:1/128',
-            '::7:6:5:4:3:2:0': '0:7:6:5:4:3:2:0/128',
-            '7:6:5:4:3:2:1::': '7:6:5:4:3:2:1:0/128',
-            '0:6:5:4:3:2:1::': '0:6:5:4:3:2:1:0/128',
-            }
-        for uncompressed, compressed in test_addresses.items():
-            self.assertEqual(compressed, str(ipaddr.IPv6Network(uncompressed)))
-
-    def testExplodeShortHandIpStr(self):
-        addr1 = ipaddr.IPv6Network('2001::1')
-        addr2 = ipaddr.IPv6Address('2001:0:5ef5:79fd:0:59d:a0e5:ba1')
-        self.assertEqual('2001:0000:0000:0000:0000:0000:0000:0001',
-                         addr1._explode_shorthand_ip_string(str(addr1.ip)))
-        self.assertEqual('0000:0000:0000:0000:0000:0000:0000:0001',
-                         ipaddr.IPv6Network('::1/128').exploded)
-        # issue 77
-        self.assertEqual('2001:0000:5ef5:79fd:0000:059d:a0e5:0ba1',
-                         addr2.exploded)
-
-    def testIntRepresentation(self):
-        self.assertEqual(16909060, int(self.ipv4))
-        self.assertEqual(42540616829182469433547762482097946625, int(self.ipv6))
-
-    def testHexRepresentation(self):
-        self.assertEqual(hex(0x1020304),
-                         hex(self.ipv4))
-
-        self.assertEqual(hex(0x20010658022ACAFE0200000000000001),
-                         hex(self.ipv6))
-
-    # backwards compatibility
-    def testBackwardsCompability(self):
-        self.assertEqual(ipaddr.CollapseAddrList(
-            [ipaddr.IPNetwork('1.1.0.0/24'), ipaddr.IPNetwork('1.1.1.0/24')]),
-                         [ipaddr.IPNetwork('1.1.0.0/23')])
-
-        self.assertEqual(ipaddr.IPNetwork('::42:0/112').AddressExclude(
-            ipaddr.IPNetwork('::42:8000/113')),
-                         [ipaddr.IPNetwork('::42:0/113')])
-
-        self.assertTrue(ipaddr.IPNetwork('1::/8').CompareNetworks(
-            ipaddr.IPNetwork('2::/9')) < 0)
-
-        self.assertEqual(ipaddr.IPNetwork('1::/16').Contains(
-            ipaddr.IPNetwork('2::/16')), False)
-
-        self.assertEqual(ipaddr.IPNetwork('0.0.0.0/0').Subnet(),
-                         [ipaddr.IPNetwork('0.0.0.0/1'),
-                          ipaddr.IPNetwork('128.0.0.0/1')])
-        self.assertEqual(ipaddr.IPNetwork('::/127').Subnet(),
-                         [ipaddr.IPNetwork('::/128'),
-                          ipaddr.IPNetwork('::1/128')])
-
-        self.assertEqual(ipaddr.IPNetwork('1.0.0.0/32').Supernet(),
-                         ipaddr.IPNetwork('1.0.0.0/31'))
-        self.assertEqual(ipaddr.IPNetwork('::/121').Supernet(),
-                         ipaddr.IPNetwork('::/120'))
-
-        self.assertEqual(ipaddr.IPNetwork('10.0.0.2').IsRFC1918(), True)
-        self.assertEqual(ipaddr.IPNetwork('10.0.0.0').IsMulticast(), False)
-        self.assertEqual(ipaddr.IPNetwork('127.255.255.255').IsLoopback(), True)
-        self.assertEqual(ipaddr.IPNetwork('169.255.255.255').IsLinkLocal(),
-                         False)
-
-    def testForceVersion(self):
-        self.assertEqual(ipaddr.IPNetwork(1).version, 4)
-        self.assertEqual(ipaddr.IPNetwork(1, version=6).version, 6)
-
-    def testWithStar(self):
-        self.assertEqual(str(self.ipv4.with_prefixlen), "1.2.3.4/24")
-        self.assertEqual(str(self.ipv4.with_netmask), "1.2.3.4/255.255.255.0")
-        self.assertEqual(str(self.ipv4.with_hostmask), "1.2.3.4/0.0.0.255")
-
-        self.assertEqual(str(self.ipv6.with_prefixlen),
-                         '2001:658:22a:cafe:200::1/64')
-        # rfc3513 sec 2.3 says that ipv6 only uses cidr notation for
-        # subnets
-        self.assertEqual(str(self.ipv6.with_netmask),
-                         '2001:658:22a:cafe:200::1/64')
-        # this probably don't make much sense, but it's included for
-        # compatibility with ipv4
-        self.assertEqual(str(self.ipv6.with_hostmask),
-                         '2001:658:22a:cafe:200::1/::ffff:ffff:ffff:ffff')
-
-    def testNetworkElementCaching(self):
-        # V4 - make sure we're empty
-        self.assertFalse(self.ipv4._cache.has_key('network'))
-        self.assertFalse(self.ipv4._cache.has_key('broadcast'))
-        self.assertFalse(self.ipv4._cache.has_key('hostmask'))
-
-        # V4 - populate and test
-        self.assertEqual(self.ipv4.network, ipaddr.IPv4Address('1.2.3.0'))
-        self.assertEqual(self.ipv4.broadcast, ipaddr.IPv4Address('1.2.3.255'))
-        self.assertEqual(self.ipv4.hostmask, ipaddr.IPv4Address('0.0.0.255'))
-
-        # V4 - check we're cached
-        self.assertTrue(self.ipv4._cache.has_key('network'))
-        self.assertTrue(self.ipv4._cache.has_key('broadcast'))
-        self.assertTrue(self.ipv4._cache.has_key('hostmask'))
-
-        # V6 - make sure we're empty
-        self.assertFalse(self.ipv6._cache.has_key('network'))
-        self.assertFalse(self.ipv6._cache.has_key('broadcast'))
-        self.assertFalse(self.ipv6._cache.has_key('hostmask'))
-
-        # V6 - populate and test
-        self.assertEqual(self.ipv6.network,
-                         ipaddr.IPv6Address('2001:658:22a:cafe::'))
-        self.assertEqual(self.ipv6.broadcast, ipaddr.IPv6Address(
-            '2001:658:22a:cafe:ffff:ffff:ffff:ffff'))
-        self.assertEqual(self.ipv6.hostmask,
-                         ipaddr.IPv6Address('::ffff:ffff:ffff:ffff'))
-
-        # V6 - check we're cached
-        self.assertTrue(self.ipv6._cache.has_key('network'))
-        self.assertTrue(self.ipv6._cache.has_key('broadcast'))
-        self.assertTrue(self.ipv6._cache.has_key('hostmask'))
-
-    def testTeredo(self):
-        # stolen from wikipedia
-        server = ipaddr.IPv4Address('65.54.227.120')
-        client = ipaddr.IPv4Address('192.0.2.45')
-        teredo_addr = '2001:0000:4136:e378:8000:63bf:3fff:fdd2'
-        self.assertEqual((server, client),
-                         ipaddr.IPAddress(teredo_addr).teredo)
-        bad_addr = '2000::4136:e378:8000:63bf:3fff:fdd2'
-        self.assertFalse(ipaddr.IPAddress(bad_addr).teredo)
-        bad_addr = '2001:0001:4136:e378:8000:63bf:3fff:fdd2'
-        self.assertFalse(ipaddr.IPAddress(bad_addr).teredo)
-
-        # i77
-        teredo_addr = ipaddr.IPv6Address('2001:0:5ef5:79fd:0:59d:a0e5:ba1')
-        self.assertEqual((ipaddr.IPv4Address('94.245.121.253'),
-                          ipaddr.IPv4Address('95.26.244.94')),
-                         teredo_addr.teredo)
-
-
-    def testsixtofour(self):
-        sixtofouraddr = ipaddr.IPAddress('2002:ac1d:2d64::1')
-        bad_addr = ipaddr.IPAddress('2000:ac1d:2d64::1')
-        self.assertEqual(ipaddr.IPv4Address('172.29.45.100'),
-                         sixtofouraddr.sixtofour)
-        self.assertFalse(bad_addr.sixtofour)
-
-
-if __name__ == '__main__':
-    unittest.main()
diff --git a/contrib/ipaddr-py/setup.py b/contrib/ipaddr-py/setup.py
deleted file mode 100755
index 33564320e45..00000000000
--- a/contrib/ipaddr-py/setup.py
+++ /dev/null
@@ -1,36 +0,0 @@
-#!/usr/bin/python
-#
-# Copyright 2008 Google Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from distutils.core import setup
-
-import ipaddr
-
-
-setup(name='ipaddr',
-      maintainer='Google',
-      maintainer_email='ipaddr-py-dev@googlegroups.com',
-      version=ipaddr.__version__,
-      url='http://code.google.com/p/ipaddr-py/',
-      license='Apache License, Version 2.0',
-      classifiers=[
-          'Development Status :: 5 - Production/Stable',
-          'Intended Audience :: Developers',
-          'License :: OSI Approved :: Apache Software License',
-          'Operating System :: OS Independent',
-          'Topic :: Internet',
-          'Topic :: Software Development :: Libraries',
-          'Topic :: System :: Networking'],
-      py_modules=['ipaddr'])
diff --git a/contrib/ipaddr-py/test-2to3.sh b/contrib/ipaddr-py/test-2to3.sh
deleted file mode 100755
index 408d665bcc2..00000000000
--- a/contrib/ipaddr-py/test-2to3.sh
+++ /dev/null
@@ -1,15 +0,0 @@
-#!/bin/sh
-
-# Converts the python2 ipaddr files to python3 and runs the unit tests
-# with both python versions.
-
-mkdir -p 2to3output && \
-cp -f *.py 2to3output && \
-( cd 2to3output && 2to3 . | patch -p0 ) && \
-py3version=$(python3 --version 2>&1) && \
-echo -e "\nTesting with ${py3version}" && \
-python3 2to3output/ipaddr_test.py && \
-rm -r 2to3output && \
-pyversion=$(python --version 2>&1) && \
-echo -e "\nTesting with ${pyversion}" && \
-./ipaddr_test.py
diff --git a/contrib/macfuse/mount_darwin.c b/contrib/macfuse/mount_darwin.c
index 10eff204bc6..d1d1c34e761 100644
--- a/contrib/macfuse/mount_darwin.c
+++ b/contrib/macfuse/mount_darwin.c
@@ -34,16 +34,15 @@
 #include "fuse_param.h"
 #include "fuse_ioctl.h"
 
-#include "glusterfs.h"
-#include "logging.h"
-#include "common-utils.h"
+#include "glusterfs/glusterfs.h"
+#include "glusterfs/logging.h"
+#include "glusterfs/common-utils.h"
 
 #define GFFUSE_LOGERR(...) \
         gf_log ("glusterfs-fuse", GF_LOG_ERROR, ## __VA_ARGS__)
 
 int
-gf_fuse_mount (const char *mountpoint, char *fsname,
-               unsigned long mountflags, char *mnt_param,
+gf_fuse_mount (const char *mountpoint, char *fsname, char *mnt_param,
                pid_t *mnt_pid, int status_fd) /* Not used on OS X */
 {
         int fd       = 0;
@@ -95,7 +94,7 @@ gf_fuse_mount (const char *mountpoint, char *fsname,
         }
 
         /* sysctlbyname() includes the trailing '\0' in version_len */
-        version_len_desired = strlen("2.x.y") + 1;
+        version_len_desired = sizeof ("2.x.y");
 
         if (version_len != version_len_desired) {
                 gf_log ("glusterfs-fuse", GF_LOG_ERROR,
diff --git a/contrib/mount/mntent.c b/contrib/mount/mntent.c
index e9b448845a7..9a7e5f39bdb 100644
--- a/contrib/mount/mntent.c
+++ b/contrib/mount/mntent.c
@@ -36,6 +36,7 @@
  */
 
 #if !defined(GF_LINUX_HOST_OS)
+
 #include <stdlib.h>
 #include <string.h>
 #include <sys/param.h>
@@ -49,166 +50,209 @@ typedef struct statvfs gf_statfs_t;
 typedef struct statfs gf_statfs_t;
 #endif
 
-static int pos = -1;
-static int mntsize = -1;
-static struct mntent _mntent;
+typedef struct _mntent_state {
+        struct mntent  mntent;
+        gf_statfs_t   *statfs;
+        int            count;
+        int            pos;
+        /* A buffer big enough to store all defined flags as a string.
+         * Increase it if necessary when more flags are defined. */
+        char           buf[256];
+} mntent_state_t;
+
+typedef struct _mntflag {
+        unsigned long  value;
+        const char    *on;
+        const char    *off;
+} mntflag_t;
+
+static mntflag_t mntflags[] = {
+        { MNT_RDONLY,      "ro",          "rw" },
+        { MNT_SYNCHRONOUS, "sync",        NULL },
+        { MNT_NOEXEC,      "noexec",      NULL },
+        { MNT_NOSUID,      "nosuid",      NULL },
+#if !defined(__FreeBSD__)
+        { MNT_NODEV,       "nodev",       NULL },
+#endif /* __FreeBSD__ */
+        { MNT_UNION,       "union",       NULL },
+        { MNT_ASYNC,       "async",       NULL },
+#if !defined(GF_DARWIN_HOST_OS)
+        { MNT_NOATIME,     "noatime",     NULL },
+#if !defined(__NetBSD__)
+        { MNT_NOCLUSTERR,  "noclusterr",  NULL },
+        { MNT_NOCLUSTERW,  "noclusterw",  NULL },
+        { MNT_NOSYMFOLLOW, "nosymfollow", NULL },
+        { MNT_SUIDDIR,     "suiddir",     NULL },
+#endif /* !__NetBSD__ */
+#endif /* !GF_DARWIN_HOST_OS */
+        { 0,               NULL,          NULL }
+};
 
 char *
 hasmntopt (const struct mntent *mnt, const char *option)
 {
-        int found;
         char *opt, *optbuf;
+        int len;
 
         optbuf = strdup(mnt->mnt_opts);
-        found = 0;
-        for (opt = optbuf; (opt = strtok(opt, " ")) != NULL; opt = NULL) {
-                if (!strcasecmp(opt, option)) {
-                        opt = opt - optbuf + mnt->mnt_opts;
-                        free (optbuf);
-                        return (opt);
+        if (optbuf == NULL) {
+                return NULL;
+        }
+
+        opt = optbuf;
+        len = 0;
+        while (*opt) {
+                while (opt[len] != 0) {
+                        if (opt[len] == ' ') {
+                                opt[len++] = 0;
+                                break;
+                        }
+                        len++;
+                }
+                if ((*opt != 0) && (strcasecmp(opt, option) == 0)) {
+                        break;
                 }
+                opt += len;
+                len = 0;
         }
-        free (optbuf);
-        return (NULL);
+        free(optbuf);
+        if (len == 0) {
+                return NULL;
+        }
+
+        return opt - optbuf + mnt->mnt_opts;
 }
 
-static char *
-concatopt (char *s0, const char *s1)
+static int
+writeopt(const char *text, char *buf, int buflen, int pos)
 {
-        size_t i;
-        char *cp;
-
-        if (s1 == NULL || *s1 == '\0')
-                return s0;
-        if (s0 && *s0) {
-                i = strlen(s0) + strlen(s1) + 1 + 1;
-                if ((cp = (char *)malloc(i)) == NULL)
-                        return (NULL);
-                (void)snprintf(cp, i, "%s %s", s0, s1);
-        } else
-                cp = strdup(s1);
-
-        if (s0)
-                free(s0);
-        return (cp);
-}
+        int len;
+
+        /* buflen must be > 0 */
+
+        if (text == NULL) {
+                return pos;
+        }
+
+        buf += pos;
+        if (pos > 0) {
+                /* We are sure we have at least one byte to store the space.
+                 * We don't need to check buflen here. */
+                *buf++ = ' ';
+                pos++;
+        }
+        len = strlen(text) + 1;
+        pos += len;
+        if (pos >= buflen) {
+                /* There won't be enough space for the text and the
+                 * terminating null character. We copy as much as we can
+                 * of the text and mark the end of the string with '...' */
+                memcpy(buf, text, buflen - pos + len);
+                if (buflen > 3) {
+                        strcpy(buf + buflen - 4, "...");
+                } else {
+                        strncpy(buf, "...", buflen - 1);
+                        buf[buflen - 1] = 0;
+                }
+                pos = buflen;
+        } else {
+                memcpy(buf, text, len);
+        }
 
+        return pos;
+}
 
 static char *
-flags2opts (int flags)
+flags2opts (int flags, char *buf, int buflen)
 {
-        char *res;
-        res = NULL;
-        res = concatopt(res, (flags & MNT_RDONLY) ? "ro" : "rw");
-        if (flags & MNT_SYNCHRONOUS)    res = concatopt(res, "sync");
-        if (flags & MNT_NOEXEC)         res = concatopt(res, "noexec");
-        if (flags & MNT_NOSUID)         res = concatopt(res, "nosuid");
-#if !defined(__FreeBSD__)
-        if (flags & MNT_NODEV)          res = concatopt(res, "nodev");
-#endif /* __FreeBSD__ */
-        if (flags & MNT_UNION)          res = concatopt(res, "union");
-        if (flags & MNT_ASYNC)          res = concatopt(res, "async");
-#if !defined(GF_DARWIN_HOST_OS)
-        if (flags & MNT_NOATIME)        res = concatopt(res, "noatime");
-#if !defined(__NetBSD__)
-        if (flags & MNT_NOCLUSTERR)     res = concatopt(res, "noclusterr");
-        if (flags & MNT_NOCLUSTERW)     res = concatopt(res, "noclusterw");
-        if (flags & MNT_NOSYMFOLLOW)    res = concatopt(res, "nosymfollow");
-        if (flags & MNT_SUIDDIR)        res = concatopt(res, "suiddir");
-#endif /* !__NetBSD__ */
-#endif /* !GF_DARWIN_HOS_OS */
-        return res;
+        char other[16];
+        mntflag_t *flg;
+        int pos;
+
+        if (buflen == 0) {
+                return NULL;
+        }
+
+        pos = 0;
+        for (flg = mntflags; flg->value != 0; flg++) {
+                pos = writeopt((flags & flg->value) == 0 ? flg->off : flg->on,
+                               buf, buflen, pos);
+                flags &= ~flg->value;
+        }
+
+        if (flags != 0) {
+                sprintf(other, "[0x%x]", flags);
+                writeopt(other, buf, buflen, pos);
+        }
+
+        return buf;
 }
 
-static struct mntent *
-statfs_to_mntent (gf_statfs_t *mntbuf)
+static void
+statfs_to_mntent (struct mntent *mntent, gf_statfs_t *mntbuf, char *buf,
+                  int buflen)
 {
-        static char opts_buf[40], *tmp;
         int f_flags;
 
-        _mntent.mnt_fsname = mntbuf->f_mntfromname;
-        _mntent.mnt_dir = mntbuf->f_mntonname;
-        _mntent.mnt_type = mntbuf->f_fstypename;
+        mntent->mnt_fsname = mntbuf->f_mntfromname;
+        mntent->mnt_dir = mntbuf->f_mntonname;
+        mntent->mnt_type = mntbuf->f_fstypename;
 
 #ifdef __NetBSD__
         f_flags = mntbuf->f_flag;
 #else
         f_flags = mntbuf->f_flags;
 #endif
-        tmp = flags2opts (f_flags);
-        if (tmp) {
-                opts_buf[sizeof(opts_buf)-1] = '\0';
-                strncpy (opts_buf, tmp, sizeof(opts_buf)-1);
-                free (tmp);
-        } else {
-                *opts_buf = '\0';
-        }
-        _mntent.mnt_opts = opts_buf;
-        _mntent.mnt_freq = _mntent.mnt_passno = 0;
-        return (&_mntent);
+        mntent->mnt_opts = flags2opts (f_flags, buf, buflen);
+
+        mntent->mnt_freq = mntent->mnt_passno = 0;
 }
 
 struct mntent *
-getmntent (FILE *fp)
+getmntent_r (FILE *fp, struct mntent *mntent, char *buf, int buflen)
 {
-        gf_statfs_t *mntbuf;
+        mntent_state_t *state = (mntent_state_t *)fp;
 
-        if (!fp)
+        if (state->pos >= state->count) {
                 return NULL;
-
-        if (pos == -1 || mntsize == -1)
-                mntsize = getmntinfo (&mntbuf, MNT_NOWAIT);
-
-        ++pos;
-        if (pos == mntsize) {
-                pos = mntsize = -1;
-                return (NULL);
         }
 
-        return (statfs_to_mntent (&mntbuf[pos]));
+        statfs_to_mntent(mntent, &state->statfs[state->pos++], buf, buflen);
+
+        return mntent;
 }
 
-/*
-  Careful using this function ``buffer`` and ``bufsize`` are
-  ignored since there is no stream with strings to populate
-  them on OSX or NetBSD, if one wishes to populate them then
-  perhaps a new function should be written in this source file
-  which uses 'getmntinfo()' to stringify the mntent's
-*/
-
-struct mntent *getmntent_r (FILE *fp, struct mntent *result,
-                            char *buffer, int bufsize)
+struct mntent *
+getmntent (FILE *fp)
 {
-        struct mntent *ment = NULL;
-
-        if (!fp)
-                return NULL;
+        mntent_state_t *state = (mntent_state_t *)fp;
 
-        flockfile (fp);
-        ment = getmntent (fp);
-        memcpy (result, ment, sizeof(*ment));
-        funlockfile (fp);
-
-        return result;
+        return getmntent_r(fp, &state->mntent, state->buf,
+                           sizeof(state->buf));
 }
 
 FILE *
 setmntent (const char *filename, const char *type)
 {
-        FILE *fp = NULL;
-#ifdef GF_DARWIN_HOST_OS
-        fp = fopen (filename, "w");
-#else
-        fp = fopen (filename, type);
-#endif
-        return fp;
+        mntent_state_t *state;
+
+        /* We don't really need to access any file so we'll use the FILE* as
+         * a fake file to store state information.
+         */
+
+        state = malloc(sizeof(mntent_state_t));
+        if (state != NULL) {
+                state->pos = 0;
+                state->count = getmntinfo(&state->statfs, MNT_NOWAIT);
+        }
+
+        return (FILE *)state;
 }
 
 int
 endmntent (FILE *fp)
 {
-        if (fp)
-                fclose (fp);
+        free(fp);
 
         return 1; /* endmntent() always returns 1 */
 }
diff --git a/contrib/qemu/block.c b/contrib/qemu/block.c
deleted file mode 100644
index b56024113b8..00000000000
--- a/contrib/qemu/block.c
+++ /dev/null
@@ -1,4604 +0,0 @@
-/*
- * QEMU System Emulator block driver
- *
- * Copyright (c) 2003 Fabrice Bellard
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-#include "config-host.h"
-#include "qemu-common.h"
-#include "trace.h"
-#include "monitor/monitor.h"
-#include "block/block_int.h"
-#include "block/blockjob.h"
-#include "qemu/module.h"
-#include "qapi/qmp/qjson.h"
-#include "sysemu/sysemu.h"
-#include "qemu/notify.h"
-#include "block/coroutine.h"
-#include "qmp-commands.h"
-#include "qemu/timer.h"
-
-#ifdef CONFIG_BSD
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <sys/ioctl.h>
-#include <sys/queue.h>
-#ifndef __DragonFly__
-#include <sys/disk.h>
-#endif
-#endif
-
-#ifdef _WIN32
-#include <windows.h>
-#endif
-
-#define NOT_DONE 0x7fffffff /* used while emulated sync operation in progress */
-
-typedef enum {
-    BDRV_REQ_COPY_ON_READ = 0x1,
-    BDRV_REQ_ZERO_WRITE   = 0x2,
-} BdrvRequestFlags;
-
-static void bdrv_dev_change_media_cb(BlockDriverState *bs, bool load);
-static BlockDriverAIOCB *bdrv_aio_readv_em(BlockDriverState *bs,
-        int64_t sector_num, QEMUIOVector *qiov, int nb_sectors,
-        BlockDriverCompletionFunc *cb, void *opaque);
-static BlockDriverAIOCB *bdrv_aio_writev_em(BlockDriverState *bs,
-        int64_t sector_num, QEMUIOVector *qiov, int nb_sectors,
-        BlockDriverCompletionFunc *cb, void *opaque);
-static int coroutine_fn bdrv_co_readv_em(BlockDriverState *bs,
-                                         int64_t sector_num, int nb_sectors,
-                                         QEMUIOVector *iov);
-static int coroutine_fn bdrv_co_writev_em(BlockDriverState *bs,
-                                         int64_t sector_num, int nb_sectors,
-                                         QEMUIOVector *iov);
-static int coroutine_fn bdrv_co_do_readv(BlockDriverState *bs,
-    int64_t sector_num, int nb_sectors, QEMUIOVector *qiov,
-    BdrvRequestFlags flags);
-static int coroutine_fn bdrv_co_do_writev(BlockDriverState *bs,
-    int64_t sector_num, int nb_sectors, QEMUIOVector *qiov,
-    BdrvRequestFlags flags);
-static BlockDriverAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
-                                               int64_t sector_num,
-                                               QEMUIOVector *qiov,
-                                               int nb_sectors,
-                                               BlockDriverCompletionFunc *cb,
-                                               void *opaque,
-                                               bool is_write);
-static void coroutine_fn bdrv_co_do_rw(void *opaque);
-static int coroutine_fn bdrv_co_do_write_zeroes(BlockDriverState *bs,
-    int64_t sector_num, int nb_sectors);
-
-static bool bdrv_exceed_bps_limits(BlockDriverState *bs, int nb_sectors,
-        bool is_write, double elapsed_time, uint64_t *wait);
-static bool bdrv_exceed_iops_limits(BlockDriverState *bs, bool is_write,
-        double elapsed_time, uint64_t *wait);
-static bool bdrv_exceed_io_limits(BlockDriverState *bs, int nb_sectors,
-        bool is_write, int64_t *wait);
-
-static QTAILQ_HEAD(, BlockDriverState) bdrv_states =
-    QTAILQ_HEAD_INITIALIZER(bdrv_states);
-
-static QLIST_HEAD(, BlockDriver) bdrv_drivers =
-    QLIST_HEAD_INITIALIZER(bdrv_drivers);
-
-/* If non-zero, use only whitelisted block drivers */
-static int use_bdrv_whitelist;
-
-#ifdef _WIN32
-static int is_windows_drive_prefix(const char *filename)
-{
-    return (((filename[0] >= 'a' && filename[0] <= 'z') ||
-             (filename[0] >= 'A' && filename[0] <= 'Z')) &&
-            filename[1] == ':');
-}
-
-int is_windows_drive(const char *filename)
-{
-    if (is_windows_drive_prefix(filename) &&
-        filename[2] == '\0')
-        return 1;
-    if (strstart(filename, "\\\\.\\", NULL) ||
-        strstart(filename, "//./", NULL))
-        return 1;
-    return 0;
-}
-#endif
-
-/* throttling disk I/O limits */
-void bdrv_io_limits_disable(BlockDriverState *bs)
-{
-    bs->io_limits_enabled = false;
-
-    while (qemu_co_queue_next(&bs->throttled_reqs));
-
-    if (bs->block_timer) {
-        qemu_del_timer(bs->block_timer);
-        qemu_free_timer(bs->block_timer);
-        bs->block_timer = NULL;
-    }
-
-    bs->slice_start = 0;
-    bs->slice_end   = 0;
-}
-
-static void bdrv_block_timer(void *opaque)
-{
-    BlockDriverState *bs = opaque;
-
-    qemu_co_queue_next(&bs->throttled_reqs);
-}
-
-void bdrv_io_limits_enable(BlockDriverState *bs)
-{
-    qemu_co_queue_init(&bs->throttled_reqs);
-    bs->block_timer = qemu_new_timer_ns(vm_clock, bdrv_block_timer, bs);
-    bs->io_limits_enabled = true;
-}
-
-bool bdrv_io_limits_enabled(BlockDriverState *bs)
-{
-    BlockIOLimit *io_limits = &bs->io_limits;
-    return io_limits->bps[BLOCK_IO_LIMIT_READ]
-         || io_limits->bps[BLOCK_IO_LIMIT_WRITE]
-         || io_limits->bps[BLOCK_IO_LIMIT_TOTAL]
-         || io_limits->iops[BLOCK_IO_LIMIT_READ]
-         || io_limits->iops[BLOCK_IO_LIMIT_WRITE]
-         || io_limits->iops[BLOCK_IO_LIMIT_TOTAL];
-}
-
-static void bdrv_io_limits_intercept(BlockDriverState *bs,
-                                     bool is_write, int nb_sectors)
-{
-    int64_t wait_time = -1;
-
-    if (!qemu_co_queue_empty(&bs->throttled_reqs)) {
-        qemu_co_queue_wait(&bs->throttled_reqs);
-    }
-
-    /* In fact, we hope to keep each request's timing, in FIFO mode. The next
-     * throttled requests will not be dequeued until the current request is
-     * allowed to be serviced. So if the current request still exceeds the
-     * limits, it will be inserted to the head. All requests followed it will
-     * be still in throttled_reqs queue.
-     */
-
-    while (bdrv_exceed_io_limits(bs, nb_sectors, is_write, &wait_time)) {
-        qemu_mod_timer(bs->block_timer,
-                       wait_time + qemu_get_clock_ns(vm_clock));
-        qemu_co_queue_wait_insert_head(&bs->throttled_reqs);
-    }
-
-    qemu_co_queue_next(&bs->throttled_reqs);
-}
-
-/* check if the path starts with "<protocol>:" */
-static int path_has_protocol(const char *path)
-{
-    const char *p;
-
-#ifdef _WIN32
-    if (is_windows_drive(path) ||
-        is_windows_drive_prefix(path)) {
-        return 0;
-    }
-    p = path + strcspn(path, ":/\\");
-#else
-    p = path + strcspn(path, ":/");
-#endif
-
-    return *p == ':';
-}
-
-int path_is_absolute(const char *path)
-{
-#ifdef _WIN32
-    /* specific case for names like: "\\.\d:" */
-    if (is_windows_drive(path) || is_windows_drive_prefix(path)) {
-        return 1;
-    }
-    return (*path == '/' || *path == '\\');
-#else
-    return (*path == '/');
-#endif
-}
-
-/* if filename is absolute, just copy it to dest. Otherwise, build a
-   path to it by considering it is relative to base_path. URL are
-   supported. */
-void path_combine(char *dest, int dest_size,
-                  const char *base_path,
-                  const char *filename)
-{
-    const char *p, *p1;
-    int len;
-
-    if (dest_size <= 0)
-        return;
-    if (path_is_absolute(filename)) {
-        pstrcpy(dest, dest_size, filename);
-    } else {
-        p = strchr(base_path, ':');
-        if (p)
-            p++;
-        else
-            p = base_path;
-        p1 = strrchr(base_path, '/');
-#ifdef _WIN32
-        {
-            const char *p2;
-            p2 = strrchr(base_path, '\\');
-            if (!p1 || p2 > p1)
-                p1 = p2;
-        }
-#endif
-        if (p1)
-            p1++;
-        else
-            p1 = base_path;
-        if (p1 > p)
-            p = p1;
-        len = p - base_path;
-        if (len > dest_size - 1)
-            len = dest_size - 1;
-        memcpy(dest, base_path, len);
-        dest[len] = '\0';
-        pstrcat(dest, dest_size, filename);
-    }
-}
-
-void bdrv_get_full_backing_filename(BlockDriverState *bs, char *dest, size_t sz)
-{
-    if (bs->backing_file[0] == '\0' || path_has_protocol(bs->backing_file)) {
-        pstrcpy(dest, sz, bs->backing_file);
-    } else {
-        path_combine(dest, sz, bs->filename, bs->backing_file);
-    }
-}
-
-void bdrv_register(BlockDriver *bdrv)
-{
-    /* Block drivers without coroutine functions need emulation */
-    if (!bdrv->bdrv_co_readv) {
-        bdrv->bdrv_co_readv = bdrv_co_readv_em;
-        bdrv->bdrv_co_writev = bdrv_co_writev_em;
-
-        /* bdrv_co_readv_em()/brdv_co_writev_em() work in terms of aio, so if
-         * the block driver lacks aio we need to emulate that too.
-         */
-        if (!bdrv->bdrv_aio_readv) {
-            /* add AIO emulation layer */
-            bdrv->bdrv_aio_readv = bdrv_aio_readv_em;
-            bdrv->bdrv_aio_writev = bdrv_aio_writev_em;
-        }
-    }
-
-    QLIST_INSERT_HEAD(&bdrv_drivers, bdrv, list);
-}
-
-/* create a new block device (by default it is empty) */
-BlockDriverState *bdrv_new(const char *device_name)
-{
-    BlockDriverState *bs;
-
-    bs = g_malloc0(sizeof(BlockDriverState));
-    pstrcpy(bs->device_name, sizeof(bs->device_name), device_name);
-    if (device_name[0] != '\0') {
-        QTAILQ_INSERT_TAIL(&bdrv_states, bs, list);
-    }
-    bdrv_iostatus_disable(bs);
-    notifier_list_init(&bs->close_notifiers);
-    notifier_with_return_list_init(&bs->before_write_notifiers);
-
-    return bs;
-}
-
-void bdrv_add_close_notifier(BlockDriverState *bs, Notifier *notify)
-{
-    notifier_list_add(&bs->close_notifiers, notify);
-}
-
-BlockDriver *bdrv_find_format(const char *format_name)
-{
-    BlockDriver *drv1;
-    QLIST_FOREACH(drv1, &bdrv_drivers, list) {
-        if (!strcmp(drv1->format_name, format_name)) {
-            return drv1;
-        }
-    }
-    return NULL;
-}
-
-static int bdrv_is_whitelisted(BlockDriver *drv, bool read_only)
-{
-    static const char *whitelist_rw[] = {
-        CONFIG_BDRV_RW_WHITELIST
-    };
-    static const char *whitelist_ro[] = {
-        CONFIG_BDRV_RO_WHITELIST
-    };
-    const char **p;
-
-    if (!whitelist_rw[0] && !whitelist_ro[0]) {
-        return 1;               /* no whitelist, anything goes */
-    }
-
-    for (p = whitelist_rw; *p; p++) {
-        if (!strcmp(drv->format_name, *p)) {
-            return 1;
-        }
-    }
-    if (read_only) {
-        for (p = whitelist_ro; *p; p++) {
-            if (!strcmp(drv->format_name, *p)) {
-                return 1;
-            }
-        }
-    }
-    return 0;
-}
-
-BlockDriver *bdrv_find_whitelisted_format(const char *format_name,
-                                          bool read_only)
-{
-    BlockDriver *drv = bdrv_find_format(format_name);
-    return drv && bdrv_is_whitelisted(drv, read_only) ? drv : NULL;
-}
-
-typedef struct CreateCo {
-    BlockDriver *drv;
-    char *filename;
-    QEMUOptionParameter *options;
-    int ret;
-} CreateCo;
-
-static void coroutine_fn bdrv_create_co_entry(void *opaque)
-{
-    CreateCo *cco = opaque;
-    assert(cco->drv);
-
-    cco->ret = cco->drv->bdrv_create(cco->filename, cco->options);
-}
-
-int bdrv_create(BlockDriver *drv, const char* filename,
-    QEMUOptionParameter *options)
-{
-    int ret;
-
-    Coroutine *co;
-    CreateCo cco = {
-        .drv = drv,
-        .filename = g_strdup(filename),
-        .options = options,
-        .ret = NOT_DONE,
-    };
-
-    if (!drv->bdrv_create) {
-        ret = -ENOTSUP;
-        goto out;
-    }
-
-    if (qemu_in_coroutine()) {
-        /* Fast-path if already in coroutine context */
-        bdrv_create_co_entry(&cco);
-    } else {
-        co = qemu_coroutine_create(bdrv_create_co_entry);
-        qemu_coroutine_enter(co, &cco);
-        while (cco.ret == NOT_DONE) {
-            qemu_aio_wait();
-        }
-    }
-
-    ret = cco.ret;
-
-out:
-    g_free(cco.filename);
-    return ret;
-}
-
-int bdrv_create_file(const char* filename, QEMUOptionParameter *options)
-{
-    BlockDriver *drv;
-
-    drv = bdrv_find_protocol(filename, true);
-    if (drv == NULL) {
-        return -ENOENT;
-    }
-
-    return bdrv_create(drv, filename, options);
-}
-
-/*
- * Create a uniquely-named empty temporary file.
- * Return 0 upon success, otherwise a negative errno value.
- */
-int get_tmp_filename(char *filename, int size)
-{
-#ifdef _WIN32
-    char temp_dir[MAX_PATH];
-    /* GetTempFileName requires that its output buffer (4th param)
-       have length MAX_PATH or greater.  */
-    assert(size >= MAX_PATH);
-    return (GetTempPath(MAX_PATH, temp_dir)
-            && GetTempFileName(temp_dir, "qem", 0, filename)
-            ? 0 : -GetLastError());
-#else
-    int fd;
-    const char *tmpdir;
-    tmpdir = getenv("TMPDIR");
-    if (!tmpdir)
-        tmpdir = "/tmp";
-    if (snprintf(filename, size, "%s/vl.XXXXXX", tmpdir) >= size) {
-        return -EOVERFLOW;
-    }
-    fd = mkstemp(filename);
-    if (fd < 0) {
-        return -errno;
-    }
-    if (close(fd) != 0) {
-        unlink(filename);
-        return -errno;
-    }
-    return 0;
-#endif
-}
-
-/*
- * Detect host devices. By convention, /dev/cdrom[N] is always
- * recognized as a host CDROM.
- */
-static BlockDriver *find_hdev_driver(const char *filename)
-{
-    int score_max = 0, score;
-    BlockDriver *drv = NULL, *d;
-
-    QLIST_FOREACH(d, &bdrv_drivers, list) {
-        if (d->bdrv_probe_device) {
-            score = d->bdrv_probe_device(filename);
-            if (score > score_max) {
-                score_max = score;
-                drv = d;
-            }
-        }
-    }
-
-    return drv;
-}
-
-BlockDriver *bdrv_find_protocol(const char *filename,
-                                bool allow_protocol_prefix)
-{
-    BlockDriver *drv1;
-    char protocol[128];
-    int len;
-    const char *p;
-
-    /* TODO Drivers without bdrv_file_open must be specified explicitly */
-
-    /*
-     * XXX(hch): we really should not let host device detection
-     * override an explicit protocol specification, but moving this
-     * later breaks access to device names with colons in them.
-     * Thanks to the brain-dead persistent naming schemes on udev-
-     * based Linux systems those actually are quite common.
-     */
-    drv1 = find_hdev_driver(filename);
-    if (drv1) {
-        return drv1;
-    }
-
-    if (!path_has_protocol(filename) || !allow_protocol_prefix) {
-        return bdrv_find_format("file");
-    }
-
-    p = strchr(filename, ':');
-    assert(p != NULL);
-    len = p - filename;
-    if (len > sizeof(protocol) - 1)
-        len = sizeof(protocol) - 1;
-    memcpy(protocol, filename, len);
-    protocol[len] = '\0';
-    QLIST_FOREACH(drv1, &bdrv_drivers, list) {
-        if (drv1->protocol_name &&
-            !strcmp(drv1->protocol_name, protocol)) {
-            return drv1;
-        }
-    }
-    return NULL;
-}
-
-static int find_image_format(BlockDriverState *bs, const char *filename,
-                             BlockDriver **pdrv)
-{
-    int score, score_max;
-    BlockDriver *drv1, *drv;
-    uint8_t buf[2048];
-    int ret = 0;
-
-    /* Return the raw BlockDriver * to scsi-generic devices or empty drives */
-    if (bs->sg || !bdrv_is_inserted(bs) || bdrv_getlength(bs) == 0) {
-        drv = bdrv_find_format("raw");
-        if (!drv) {
-            ret = -ENOENT;
-        }
-        *pdrv = drv;
-        return ret;
-    }
-
-    ret = bdrv_pread(bs, 0, buf, sizeof(buf));
-    if (ret < 0) {
-        *pdrv = NULL;
-        return ret;
-    }
-
-    score_max = 0;
-    drv = NULL;
-    QLIST_FOREACH(drv1, &bdrv_drivers, list) {
-        if (drv1->bdrv_probe) {
-            score = drv1->bdrv_probe(buf, ret, filename);
-            if (score > score_max) {
-                score_max = score;
-                drv = drv1;
-            }
-        }
-    }
-    if (!drv) {
-        ret = -ENOENT;
-    }
-    *pdrv = drv;
-    return ret;
-}
-
-/**
- * Set the current 'total_sectors' value
- */
-static int refresh_total_sectors(BlockDriverState *bs, int64_t hint)
-{
-    BlockDriver *drv = bs->drv;
-
-    /* Do not attempt drv->bdrv_getlength() on scsi-generic devices */
-    if (bs->sg)
-        return 0;
-
-    /* query actual device if possible, otherwise just trust the hint */
-    if (drv->bdrv_getlength) {
-        int64_t length = drv->bdrv_getlength(bs);
-        if (length < 0) {
-            return length;
-        }
-        hint = length >> BDRV_SECTOR_BITS;
-    }
-
-    bs->total_sectors = hint;
-    return 0;
-}
-
-/**
- * Set open flags for a given discard mode
- *
- * Return 0 on success, -1 if the discard mode was invalid.
- */
-int bdrv_parse_discard_flags(const char *mode, int *flags)
-{
-    *flags &= ~BDRV_O_UNMAP;
-
-    if (!strcmp(mode, "off") || !strcmp(mode, "ignore")) {
-        /* do nothing */
-    } else if (!strcmp(mode, "on") || !strcmp(mode, "unmap")) {
-        *flags |= BDRV_O_UNMAP;
-    } else {
-        return -1;
-    }
-
-    return 0;
-}
-
-/**
- * Set open flags for a given cache mode
- *
- * Return 0 on success, -1 if the cache mode was invalid.
- */
-int bdrv_parse_cache_flags(const char *mode, int *flags)
-{
-    *flags &= ~BDRV_O_CACHE_MASK;
-
-    if (!strcmp(mode, "off") || !strcmp(mode, "none")) {
-        *flags |= BDRV_O_NOCACHE | BDRV_O_CACHE_WB;
-    } else if (!strcmp(mode, "directsync")) {
-        *flags |= BDRV_O_NOCACHE;
-    } else if (!strcmp(mode, "writeback")) {
-        *flags |= BDRV_O_CACHE_WB;
-    } else if (!strcmp(mode, "unsafe")) {
-        *flags |= BDRV_O_CACHE_WB;
-        *flags |= BDRV_O_NO_FLUSH;
-    } else if (!strcmp(mode, "writethrough")) {
-        /* this is the default */
-    } else {
-        return -1;
-    }
-
-    return 0;
-}
-
-/**
- * The copy-on-read flag is actually a reference count so multiple users may
- * use the feature without worrying about clobbering its previous state.
- * Copy-on-read stays enabled until all users have called to disable it.
- */
-void bdrv_enable_copy_on_read(BlockDriverState *bs)
-{
-    bs->copy_on_read++;
-}
-
-void bdrv_disable_copy_on_read(BlockDriverState *bs)
-{
-    assert(bs->copy_on_read > 0);
-    bs->copy_on_read--;
-}
-
-static int bdrv_open_flags(BlockDriverState *bs, int flags)
-{
-    int open_flags = flags | BDRV_O_CACHE_WB;
-
-    /*
-     * Clear flags that are internal to the block layer before opening the
-     * image.
-     */
-    open_flags &= ~(BDRV_O_SNAPSHOT | BDRV_O_NO_BACKING);
-
-    /*
-     * Snapshots should be writable.
-     */
-    if (bs->is_temporary) {
-        open_flags |= BDRV_O_RDWR;
-    }
-
-    return open_flags;
-}
-
-/*
- * Common part for opening disk images and files
- *
- * Removes all processed options from *options.
- */
-static int bdrv_open_common(BlockDriverState *bs, BlockDriverState *file,
-    QDict *options, int flags, BlockDriver *drv)
-{
-    int ret, open_flags;
-    const char *filename;
-
-    assert(drv != NULL);
-    assert(bs->file == NULL);
-    assert(options != NULL && bs->options != options);
-
-    if (file != NULL) {
-        filename = file->filename;
-    } else {
-        filename = qdict_get_try_str(options, "filename");
-    }
-
-    trace_bdrv_open_common(bs, filename ?: "", flags, drv->format_name);
-
-    /* bdrv_open() with directly using a protocol as drv. This layer is already
-     * opened, so assign it to bs (while file becomes a closed BlockDriverState)
-     * and return immediately. */
-    if (file != NULL && drv->bdrv_file_open) {
-        bdrv_swap(file, bs);
-        return 0;
-    }
-
-    bs->open_flags = flags;
-    bs->buffer_alignment = 512;
-    open_flags = bdrv_open_flags(bs, flags);
-    bs->read_only = !(open_flags & BDRV_O_RDWR);
-
-    if (use_bdrv_whitelist && !bdrv_is_whitelisted(drv, bs->read_only)) {
-        return -ENOTSUP;
-    }
-
-    assert(bs->copy_on_read == 0); /* bdrv_new() and bdrv_close() make it so */
-    if (!bs->read_only && (flags & BDRV_O_COPY_ON_READ)) {
-        bdrv_enable_copy_on_read(bs);
-    }
-
-    if (filename != NULL) {
-        pstrcpy(bs->filename, sizeof(bs->filename), filename);
-    } else {
-        bs->filename[0] = '\0';
-    }
-
-    bs->drv = drv;
-    bs->opaque = g_malloc0(drv->instance_size);
-
-    bs->enable_write_cache = !!(flags & BDRV_O_CACHE_WB);
-
-    /* Open the image, either directly or using a protocol */
-    if (drv->bdrv_file_open) {
-        assert(file == NULL);
-        assert(drv->bdrv_parse_filename || filename != NULL);
-        ret = drv->bdrv_file_open(bs, options, open_flags);
-    } else {
-        if (file == NULL) {
-            qerror_report(ERROR_CLASS_GENERIC_ERROR, "Can't use '%s' as a "
-                          "block driver for the protocol level",
-                          drv->format_name);
-            ret = -EINVAL;
-            goto free_and_fail;
-        }
-        assert(file != NULL);
-        bs->file = file;
-        ret = drv->bdrv_open(bs, options, open_flags);
-    }
-
-    if (ret < 0) {
-        goto free_and_fail;
-    }
-
-    ret = refresh_total_sectors(bs, bs->total_sectors);
-    if (ret < 0) {
-        goto free_and_fail;
-    }
-
-#ifndef _WIN32
-    if (bs->is_temporary) {
-        assert(filename != NULL);
-        unlink(filename);
-    }
-#endif
-    return 0;
-
-free_and_fail:
-    bs->file = NULL;
-    g_free(bs->opaque);
-    bs->opaque = NULL;
-    bs->drv = NULL;
-    return ret;
-}
-
-/*
- * Opens a file using a protocol (file, host_device, nbd, ...)
- *
- * options is a QDict of options to pass to the block drivers, or NULL for an
- * empty set of options. The reference to the QDict belongs to the block layer
- * after the call (even on failure), so if the caller intends to reuse the
- * dictionary, it needs to use QINCREF() before calling bdrv_file_open.
- */
-int bdrv_file_open(BlockDriverState **pbs, const char *filename,
-                   QDict *options, int flags)
-{
-    BlockDriverState *bs;
-    BlockDriver *drv;
-    const char *drvname;
-    bool allow_protocol_prefix = false;
-    int ret;
-
-    /* NULL means an empty set of options */
-    if (options == NULL) {
-        options = qdict_new();
-    }
-
-    bs = bdrv_new("");
-    bs->options = options;
-    options = qdict_clone_shallow(options);
-
-    /* Fetch the file name from the options QDict if necessary */
-    if (!filename) {
-        filename = qdict_get_try_str(options, "filename");
-    } else if (filename && !qdict_haskey(options, "filename")) {
-        qdict_put(options, "filename", qstring_from_str(filename));
-        allow_protocol_prefix = true;
-    } else {
-        qerror_report(ERROR_CLASS_GENERIC_ERROR, "Can't specify 'file' and "
-                      "'filename' options at the same time");
-        ret = -EINVAL;
-        goto fail;
-    }
-
-    /* Find the right block driver */
-    drvname = qdict_get_try_str(options, "driver");
-    if (drvname) {
-        drv = bdrv_find_whitelisted_format(drvname, !(flags & BDRV_O_RDWR));
-        qdict_del(options, "driver");
-    } else if (filename) {
-        drv = bdrv_find_protocol(filename, allow_protocol_prefix);
-        if (!drv) {
-            qerror_report(ERROR_CLASS_GENERIC_ERROR, "Unknown protocol");
-        }
-    } else {
-        qerror_report(ERROR_CLASS_GENERIC_ERROR,
-                      "Must specify either driver or file");
-        drv = NULL;
-    }
-
-    if (!drv) {
-        ret = -ENOENT;
-        goto fail;
-    }
-
-    /* Parse the filename and open it */
-    if (drv->bdrv_parse_filename && filename) {
-        Error *local_err = NULL;
-        drv->bdrv_parse_filename(filename, options, &local_err);
-        if (error_is_set(&local_err)) {
-            qerror_report_err(local_err);
-            error_free(local_err);
-            ret = -EINVAL;
-            goto fail;
-        }
-        qdict_del(options, "filename");
-    } else if (!drv->bdrv_parse_filename && !filename) {
-        qerror_report(ERROR_CLASS_GENERIC_ERROR,
-                      "The '%s' block driver requires a file name",
-                      drv->format_name);
-        ret = -EINVAL;
-        goto fail;
-    }
-
-    ret = bdrv_open_common(bs, NULL, options, flags, drv);
-    if (ret < 0) {
-        goto fail;
-    }
-
-    /* Check if any unknown options were used */
-    if (qdict_size(options) != 0) {
-        const QDictEntry *entry = qdict_first(options);
-        qerror_report(ERROR_CLASS_GENERIC_ERROR, "Block protocol '%s' doesn't "
-                      "support the option '%s'",
-                      drv->format_name, entry->key);
-        ret = -EINVAL;
-        goto fail;
-    }
-    QDECREF(options);
-
-    bs->growable = 1;
-    *pbs = bs;
-    return 0;
-
-fail:
-    QDECREF(options);
-    if (!bs->drv) {
-        QDECREF(bs->options);
-    }
-    bdrv_delete(bs);
-    return ret;
-}
-
-/*
- * Opens the backing file for a BlockDriverState if not yet open
- *
- * options is a QDict of options to pass to the block drivers, or NULL for an
- * empty set of options. The reference to the QDict is transferred to this
- * function (even on failure), so if the caller intends to reuse the dictionary,
- * it needs to use QINCREF() before calling bdrv_file_open.
- */
-int bdrv_open_backing_file(BlockDriverState *bs, QDict *options)
-{
-    char backing_filename[PATH_MAX];
-    int back_flags, ret;
-    BlockDriver *back_drv = NULL;
-
-    if (bs->backing_hd != NULL) {
-        QDECREF(options);
-        return 0;
-    }
-
-    /* NULL means an empty set of options */
-    if (options == NULL) {
-        options = qdict_new();
-    }
-
-    bs->open_flags &= ~BDRV_O_NO_BACKING;
-    if (qdict_haskey(options, "file.filename")) {
-        backing_filename[0] = '\0';
-    } else if (bs->backing_file[0] == '\0' && qdict_size(options) == 0) {
-        QDECREF(options);
-        return 0;
-    }
-
-    bs->backing_hd = bdrv_new("");
-    bdrv_get_full_backing_filename(bs, backing_filename,
-                                   sizeof(backing_filename));
-
-    if (bs->backing_format[0] != '\0') {
-        back_drv = bdrv_find_format(bs->backing_format);
-    }
-
-    /* backing files always opened read-only */
-    back_flags = bs->open_flags & ~(BDRV_O_RDWR | BDRV_O_SNAPSHOT);
-
-    ret = bdrv_open(bs->backing_hd,
-                    *backing_filename ? backing_filename : NULL, options,
-                    back_flags, back_drv);
-    if (ret < 0) {
-        bdrv_delete(bs->backing_hd);
-        bs->backing_hd = NULL;
-        bs->open_flags |= BDRV_O_NO_BACKING;
-        return ret;
-    }
-    return 0;
-}
-
-static void extract_subqdict(QDict *src, QDict **dst, const char *start)
-{
-    const QDictEntry *entry, *next;
-    const char *p;
-
-    *dst = qdict_new();
-    entry = qdict_first(src);
-
-    while (entry != NULL) {
-        next = qdict_next(src, entry);
-        if (strstart(entry->key, start, &p)) {
-            qobject_incref(entry->value);
-            qdict_put_obj(*dst, p, entry->value);
-            qdict_del(src, entry->key);
-        }
-        entry = next;
-    }
-}
-
-/*
- * Opens a disk image (raw, qcow2, vmdk, ...)
- *
- * options is a QDict of options to pass to the block drivers, or NULL for an
- * empty set of options. The reference to the QDict belongs to the block layer
- * after the call (even on failure), so if the caller intends to reuse the
- * dictionary, it needs to use QINCREF() before calling bdrv_open.
- */
-int bdrv_open(BlockDriverState *bs, const char *filename, QDict *options,
-              int flags, BlockDriver *drv)
-{
-    int ret;
-    /* TODO: extra byte is a hack to ensure MAX_PATH space on Windows. */
-    char tmp_filename[PATH_MAX + 1];
-    BlockDriverState *file = NULL;
-    QDict *file_options = NULL;
-
-    /* NULL means an empty set of options */
-    if (options == NULL) {
-        options = qdict_new();
-    }
-
-    bs->options = options;
-    options = qdict_clone_shallow(options);
-
-    /* For snapshot=on, create a temporary qcow2 overlay */
-    if (flags & BDRV_O_SNAPSHOT) {
-        BlockDriverState *bs1;
-        int64_t total_size;
-        BlockDriver *bdrv_qcow2;
-        QEMUOptionParameter *create_options;
-        char backing_filename[PATH_MAX];
-
-        if (qdict_size(options) != 0) {
-            error_report("Can't use snapshot=on with driver-specific options");
-            ret = -EINVAL;
-            goto fail;
-        }
-        assert(filename != NULL);
-
-        /* if snapshot, we create a temporary backing file and open it
-           instead of opening 'filename' directly */
-
-        /* if there is a backing file, use it */
-        bs1 = bdrv_new("");
-        ret = bdrv_open(bs1, filename, NULL, 0, drv);
-        if (ret < 0) {
-            bdrv_delete(bs1);
-            goto fail;
-        }
-        total_size = bdrv_getlength(bs1) & BDRV_SECTOR_MASK;
-
-        bdrv_delete(bs1);
-
-        ret = get_tmp_filename(tmp_filename, sizeof(tmp_filename));
-        if (ret < 0) {
-            goto fail;
-        }
-
-        /* Real path is meaningless for protocols */
-        if (path_has_protocol(filename)) {
-            snprintf(backing_filename, sizeof(backing_filename),
-                     "%s", filename);
-        } else if (!realpath(filename, backing_filename)) {
-            ret = -errno;
-            goto fail;
-        }
-
-        bdrv_qcow2 = bdrv_find_format("qcow2");
-        create_options = parse_option_parameters("", bdrv_qcow2->create_options,
-                                                 NULL);
-
-        set_option_parameter_int(create_options, BLOCK_OPT_SIZE, total_size);
-        set_option_parameter(create_options, BLOCK_OPT_BACKING_FILE,
-                             backing_filename);
-        if (drv) {
-            set_option_parameter(create_options, BLOCK_OPT_BACKING_FMT,
-                drv->format_name);
-        }
-
-        ret = bdrv_create(bdrv_qcow2, tmp_filename, create_options);
-        free_option_parameters(create_options);
-        if (ret < 0) {
-            goto fail;
-        }
-
-        filename = tmp_filename;
-        drv = bdrv_qcow2;
-        bs->is_temporary = 1;
-    }
-
-    /* Open image file without format layer */
-    if (flags & BDRV_O_RDWR) {
-        flags |= BDRV_O_ALLOW_RDWR;
-    }
-
-    extract_subqdict(options, &file_options, "file.");
-
-    ret = bdrv_file_open(&file, filename, file_options,
-                         bdrv_open_flags(bs, flags | BDRV_O_UNMAP));
-    if (ret < 0) {
-        goto fail;
-    }
-
-    /* Find the right image format driver */
-    if (!drv) {
-        ret = find_image_format(file, filename, &drv);
-    }
-
-    if (!drv) {
-        goto unlink_and_fail;
-    }
-
-    /* Open the image */
-    ret = bdrv_open_common(bs, file, options, flags, drv);
-    if (ret < 0) {
-        goto unlink_and_fail;
-    }
-
-    if (bs->file != file) {
-        bdrv_delete(file);
-        file = NULL;
-    }
-
-    /* If there is a backing file, use it */
-    if ((flags & BDRV_O_NO_BACKING) == 0) {
-        QDict *backing_options;
-
-        extract_subqdict(options, &backing_options, "backing.");
-        ret = bdrv_open_backing_file(bs, backing_options);
-        if (ret < 0) {
-            goto close_and_fail;
-        }
-    }
-
-    /* Check if any unknown options were used */
-    if (qdict_size(options) != 0) {
-        const QDictEntry *entry = qdict_first(options);
-        qerror_report(ERROR_CLASS_GENERIC_ERROR, "Block format '%s' used by "
-            "device '%s' doesn't support the option '%s'",
-            drv->format_name, bs->device_name, entry->key);
-
-        ret = -EINVAL;
-        goto close_and_fail;
-    }
-    QDECREF(options);
-
-    if (!bdrv_key_required(bs)) {
-        bdrv_dev_change_media_cb(bs, true);
-    }
-
-    /* throttling disk I/O limits */
-    if (bs->io_limits_enabled) {
-        bdrv_io_limits_enable(bs);
-    }
-
-    return 0;
-
-unlink_and_fail:
-    if (file != NULL) {
-        bdrv_delete(file);
-    }
-    if (bs->is_temporary) {
-        unlink(filename);
-    }
-fail:
-    QDECREF(bs->options);
-    QDECREF(options);
-    bs->options = NULL;
-    return ret;
-
-close_and_fail:
-    bdrv_close(bs);
-    QDECREF(options);
-    return ret;
-}
-
-typedef struct BlockReopenQueueEntry {
-     bool prepared;
-     BDRVReopenState state;
-     QSIMPLEQ_ENTRY(BlockReopenQueueEntry) entry;
-} BlockReopenQueueEntry;
-
-/*
- * Adds a BlockDriverState to a simple queue for an atomic, transactional
- * reopen of multiple devices.
- *
- * bs_queue can either be an existing BlockReopenQueue that has had QSIMPLE_INIT
- * already performed, or alternatively may be NULL a new BlockReopenQueue will
- * be created and initialized. This newly created BlockReopenQueue should be
- * passed back in for subsequent calls that are intended to be of the same
- * atomic 'set'.
- *
- * bs is the BlockDriverState to add to the reopen queue.
- *
- * flags contains the open flags for the associated bs
- *
- * returns a pointer to bs_queue, which is either the newly allocated
- * bs_queue, or the existing bs_queue being used.
- *
- */
-BlockReopenQueue *bdrv_reopen_queue(BlockReopenQueue *bs_queue,
-                                    BlockDriverState *bs, int flags)
-{
-    assert(bs != NULL);
-
-    BlockReopenQueueEntry *bs_entry;
-    if (bs_queue == NULL) {
-        bs_queue = g_new0(BlockReopenQueue, 1);
-        QSIMPLEQ_INIT(bs_queue);
-    }
-
-    if (bs->file) {
-        bdrv_reopen_queue(bs_queue, bs->file, flags);
-    }
-
-    bs_entry = g_new0(BlockReopenQueueEntry, 1);
-    QSIMPLEQ_INSERT_TAIL(bs_queue, bs_entry, entry);
-
-    bs_entry->state.bs = bs;
-    bs_entry->state.flags = flags;
-
-    return bs_queue;
-}
-
-/*
- * Reopen multiple BlockDriverStates atomically & transactionally.
- *
- * The queue passed in (bs_queue) must have been built up previous
- * via bdrv_reopen_queue().
- *
- * Reopens all BDS specified in the queue, with the appropriate
- * flags.  All devices are prepared for reopen, and failure of any
- * device will cause all device changes to be abandonded, and intermediate
- * data cleaned up.
- *
- * If all devices prepare successfully, then the changes are committed
- * to all devices.
- *
- */
-int bdrv_reopen_multiple(BlockReopenQueue *bs_queue, Error **errp)
-{
-    int ret = -1;
-    BlockReopenQueueEntry *bs_entry, *next;
-    Error *local_err = NULL;
-
-    assert(bs_queue != NULL);
-
-    bdrv_drain_all();
-
-    QSIMPLEQ_FOREACH(bs_entry, bs_queue, entry) {
-        if (bdrv_reopen_prepare(&bs_entry->state, bs_queue, &local_err)) {
-            error_propagate(errp, local_err);
-            goto cleanup;
-        }
-        bs_entry->prepared = true;
-    }
-
-    /* If we reach this point, we have success and just need to apply the
-     * changes
-     */
-    QSIMPLEQ_FOREACH(bs_entry, bs_queue, entry) {
-        bdrv_reopen_commit(&bs_entry->state);
-    }
-
-    ret = 0;
-
-cleanup:
-    QSIMPLEQ_FOREACH_SAFE(bs_entry, bs_queue, entry, next) {
-        if (ret && bs_entry->prepared) {
-            bdrv_reopen_abort(&bs_entry->state);
-        }
-        g_free(bs_entry);
-    }
-    g_free(bs_queue);
-    return ret;
-}
-
-
-/* Reopen a single BlockDriverState with the specified flags. */
-int bdrv_reopen(BlockDriverState *bs, int bdrv_flags, Error **errp)
-{
-    int ret = -1;
-    Error *local_err = NULL;
-    BlockReopenQueue *queue = bdrv_reopen_queue(NULL, bs, bdrv_flags);
-
-    ret = bdrv_reopen_multiple(queue, &local_err);
-    if (local_err != NULL) {
-        error_propagate(errp, local_err);
-    }
-    return ret;
-}
-
-
-/*
- * Prepares a BlockDriverState for reopen. All changes are staged in the
- * 'opaque' field of the BDRVReopenState, which is used and allocated by
- * the block driver layer .bdrv_reopen_prepare()
- *
- * bs is the BlockDriverState to reopen
- * flags are the new open flags
- * queue is the reopen queue
- *
- * Returns 0 on success, non-zero on error.  On error errp will be set
- * as well.
- *
- * On failure, bdrv_reopen_abort() will be called to clean up any data.
- * It is the responsibility of the caller to then call the abort() or
- * commit() for any other BDS that have been left in a prepare() state
- *
- */
-int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue,
-                        Error **errp)
-{
-    int ret = -1;
-    Error *local_err = NULL;
-    BlockDriver *drv;
-
-    assert(reopen_state != NULL);
-    assert(reopen_state->bs->drv != NULL);
-    drv = reopen_state->bs->drv;
-
-    /* if we are to stay read-only, do not allow permission change
-     * to r/w */
-    if (!(reopen_state->bs->open_flags & BDRV_O_ALLOW_RDWR) &&
-        reopen_state->flags & BDRV_O_RDWR) {
-        error_set(errp, QERR_DEVICE_IS_READ_ONLY,
-                  reopen_state->bs->device_name);
-        goto error;
-    }
-
-
-    ret = bdrv_flush(reopen_state->bs);
-    if (ret) {
-        error_set(errp, ERROR_CLASS_GENERIC_ERROR, "Error (%s) flushing drive",
-                  strerror(-ret));
-        goto error;
-    }
-
-    if (drv->bdrv_reopen_prepare) {
-        ret = drv->bdrv_reopen_prepare(reopen_state, queue, &local_err);
-        if (ret) {
-            if (local_err != NULL) {
-                error_propagate(errp, local_err);
-            } else {
-                error_setg(errp, "failed while preparing to reopen image '%s'",
-                           reopen_state->bs->filename);
-            }
-            goto error;
-        }
-    } else {
-        /* It is currently mandatory to have a bdrv_reopen_prepare()
-         * handler for each supported drv. */
-        error_set(errp, QERR_BLOCK_FORMAT_FEATURE_NOT_SUPPORTED,
-                  drv->format_name, reopen_state->bs->device_name,
-                 "reopening of file");
-        ret = -1;
-        goto error;
-    }
-
-    ret = 0;
-
-error:
-    return ret;
-}
-
-/*
- * Takes the staged changes for the reopen from bdrv_reopen_prepare(), and
- * makes them final by swapping the staging BlockDriverState contents into
- * the active BlockDriverState contents.
- */
-void bdrv_reopen_commit(BDRVReopenState *reopen_state)
-{
-    BlockDriver *drv;
-
-    assert(reopen_state != NULL);
-    drv = reopen_state->bs->drv;
-    assert(drv != NULL);
-
-    /* If there are any driver level actions to take */
-    if (drv->bdrv_reopen_commit) {
-        drv->bdrv_reopen_commit(reopen_state);
-    }
-
-    /* set BDS specific flags now */
-    reopen_state->bs->open_flags         = reopen_state->flags;
-    reopen_state->bs->enable_write_cache = !!(reopen_state->flags &
-                                              BDRV_O_CACHE_WB);
-    reopen_state->bs->read_only = !(reopen_state->flags & BDRV_O_RDWR);
-}
-
-/*
- * Abort the reopen, and delete and free the staged changes in
- * reopen_state
- */
-void bdrv_reopen_abort(BDRVReopenState *reopen_state)
-{
-    BlockDriver *drv;
-
-    assert(reopen_state != NULL);
-    drv = reopen_state->bs->drv;
-    assert(drv != NULL);
-
-    if (drv->bdrv_reopen_abort) {
-        drv->bdrv_reopen_abort(reopen_state);
-    }
-}
-
-
-void bdrv_close(BlockDriverState *bs)
-{
-    if (bs->job) {
-        block_job_cancel_sync(bs->job);
-    }
-    bdrv_drain_all(); /* complete I/O */
-    bdrv_flush(bs);
-    bdrv_drain_all(); /* in case flush left pending I/O */
-    notifier_list_notify(&bs->close_notifiers, bs);
-
-    if (bs->drv) {
-        if (bs->backing_hd) {
-            bdrv_delete(bs->backing_hd);
-            bs->backing_hd = NULL;
-        }
-        bs->drv->bdrv_close(bs);
-        g_free(bs->opaque);
-#ifdef _WIN32
-        if (bs->is_temporary) {
-            unlink(bs->filename);
-        }
-#endif
-        bs->opaque = NULL;
-        bs->drv = NULL;
-        bs->copy_on_read = 0;
-        bs->backing_file[0] = '\0';
-        bs->backing_format[0] = '\0';
-        bs->total_sectors = 0;
-        bs->encrypted = 0;
-        bs->valid_key = 0;
-        bs->sg = 0;
-        bs->growable = 0;
-        QDECREF(bs->options);
-        bs->options = NULL;
-
-        if (bs->file != NULL) {
-            bdrv_delete(bs->file);
-            bs->file = NULL;
-        }
-    }
-
-    bdrv_dev_change_media_cb(bs, false);
-
-    /*throttling disk I/O limits*/
-    if (bs->io_limits_enabled) {
-        bdrv_io_limits_disable(bs);
-    }
-}
-
-void bdrv_close_all(void)
-{
-    BlockDriverState *bs;
-
-    QTAILQ_FOREACH(bs, &bdrv_states, list) {
-        bdrv_close(bs);
-    }
-}
-
-/*
- * Wait for pending requests to complete across all BlockDriverStates
- *
- * This function does not flush data to disk, use bdrv_flush_all() for that
- * after calling this function.
- *
- * Note that completion of an asynchronous I/O operation can trigger any
- * number of other I/O operations on other devices---for example a coroutine
- * can be arbitrarily complex and a constant flow of I/O can come until the
- * coroutine is complete.  Because of this, it is not possible to have a
- * function to drain a single device's I/O queue.
- */
-void bdrv_drain_all(void)
-{
-    BlockDriverState *bs;
-    bool busy;
-
-    do {
-        busy = qemu_aio_wait();
-
-        /* FIXME: We do not have timer support here, so this is effectively
-         * a busy wait.
-         */
-        QTAILQ_FOREACH(bs, &bdrv_states, list) {
-            if (!qemu_co_queue_empty(&bs->throttled_reqs)) {
-                qemu_co_queue_restart_all(&bs->throttled_reqs);
-                busy = true;
-            }
-        }
-    } while (busy);
-
-    /* If requests are still pending there is a bug somewhere */
-    QTAILQ_FOREACH(bs, &bdrv_states, list) {
-        assert(QLIST_EMPTY(&bs->tracked_requests));
-        assert(qemu_co_queue_empty(&bs->throttled_reqs));
-    }
-}
-
-/* make a BlockDriverState anonymous by removing from bdrv_state list.
-   Also, NULL terminate the device_name to prevent double remove */
-void bdrv_make_anon(BlockDriverState *bs)
-{
-    if (bs->device_name[0] != '\0') {
-        QTAILQ_REMOVE(&bdrv_states, bs, list);
-    }
-    bs->device_name[0] = '\0';
-}
-
-static void bdrv_rebind(BlockDriverState *bs)
-{
-    if (bs->drv && bs->drv->bdrv_rebind) {
-        bs->drv->bdrv_rebind(bs);
-    }
-}
-
-static void bdrv_move_feature_fields(BlockDriverState *bs_dest,
-                                     BlockDriverState *bs_src)
-{
-    /* move some fields that need to stay attached to the device */
-    bs_dest->open_flags         = bs_src->open_flags;
-
-    /* dev info */
-    bs_dest->dev_ops            = bs_src->dev_ops;
-    bs_dest->dev_opaque         = bs_src->dev_opaque;
-    bs_dest->dev                = bs_src->dev;
-    bs_dest->buffer_alignment   = bs_src->buffer_alignment;
-    bs_dest->copy_on_read       = bs_src->copy_on_read;
-
-    bs_dest->enable_write_cache = bs_src->enable_write_cache;
-
-    /* i/o timing parameters */
-    bs_dest->slice_start        = bs_src->slice_start;
-    bs_dest->slice_end          = bs_src->slice_end;
-    bs_dest->slice_submitted    = bs_src->slice_submitted;
-    bs_dest->io_limits          = bs_src->io_limits;
-    bs_dest->throttled_reqs     = bs_src->throttled_reqs;
-    bs_dest->block_timer        = bs_src->block_timer;
-    bs_dest->io_limits_enabled  = bs_src->io_limits_enabled;
-
-    /* r/w error */
-    bs_dest->on_read_error      = bs_src->on_read_error;
-    bs_dest->on_write_error     = bs_src->on_write_error;
-
-    /* i/o status */
-    bs_dest->iostatus_enabled   = bs_src->iostatus_enabled;
-    bs_dest->iostatus           = bs_src->iostatus;
-
-    /* dirty bitmap */
-    bs_dest->dirty_bitmap       = bs_src->dirty_bitmap;
-
-    /* job */
-    bs_dest->in_use             = bs_src->in_use;
-    bs_dest->job                = bs_src->job;
-
-    /* keep the same entry in bdrv_states */
-    pstrcpy(bs_dest->device_name, sizeof(bs_dest->device_name),
-            bs_src->device_name);
-    bs_dest->list = bs_src->list;
-}
-
-/*
- * Swap bs contents for two image chains while they are live,
- * while keeping required fields on the BlockDriverState that is
- * actually attached to a device.
- *
- * This will modify the BlockDriverState fields, and swap contents
- * between bs_new and bs_old. Both bs_new and bs_old are modified.
- *
- * bs_new is required to be anonymous.
- *
- * This function does not create any image files.
- */
-void bdrv_swap(BlockDriverState *bs_new, BlockDriverState *bs_old)
-{
-    BlockDriverState tmp;
-
-    /* bs_new must be anonymous and shouldn't have anything fancy enabled */
-    assert(bs_new->device_name[0] == '\0');
-    assert(bs_new->dirty_bitmap == NULL);
-    assert(bs_new->job == NULL);
-    assert(bs_new->dev == NULL);
-    assert(bs_new->in_use == 0);
-    assert(bs_new->io_limits_enabled == false);
-    assert(bs_new->block_timer == NULL);
-
-    tmp = *bs_new;
-    *bs_new = *bs_old;
-    *bs_old = tmp;
-
-    /* there are some fields that should not be swapped, move them back */
-    bdrv_move_feature_fields(&tmp, bs_old);
-    bdrv_move_feature_fields(bs_old, bs_new);
-    bdrv_move_feature_fields(bs_new, &tmp);
-
-    /* bs_new shouldn't be in bdrv_states even after the swap!  */
-    assert(bs_new->device_name[0] == '\0');
-
-    /* Check a few fields that should remain attached to the device */
-    assert(bs_new->dev == NULL);
-    assert(bs_new->job == NULL);
-    assert(bs_new->in_use == 0);
-    assert(bs_new->io_limits_enabled == false);
-    assert(bs_new->block_timer == NULL);
-
-    bdrv_rebind(bs_new);
-    bdrv_rebind(bs_old);
-}
-
-/*
- * Add new bs contents at the top of an image chain while the chain is
- * live, while keeping required fields on the top layer.
- *
- * This will modify the BlockDriverState fields, and swap contents
- * between bs_new and bs_top. Both bs_new and bs_top are modified.
- *
- * bs_new is required to be anonymous.
- *
- * This function does not create any image files.
- */
-void bdrv_append(BlockDriverState *bs_new, BlockDriverState *bs_top)
-{
-    bdrv_swap(bs_new, bs_top);
-
-    /* The contents of 'tmp' will become bs_top, as we are
-     * swapping bs_new and bs_top contents. */
-    bs_top->backing_hd = bs_new;
-    bs_top->open_flags &= ~BDRV_O_NO_BACKING;
-    pstrcpy(bs_top->backing_file, sizeof(bs_top->backing_file),
-            bs_new->filename);
-    pstrcpy(bs_top->backing_format, sizeof(bs_top->backing_format),
-            bs_new->drv ? bs_new->drv->format_name : "");
-}
-
-void bdrv_delete(BlockDriverState *bs)
-{
-    assert(!bs->dev);
-    assert(!bs->job);
-    assert(!bs->in_use);
-
-    /* remove from list, if necessary */
-    bdrv_make_anon(bs);
-
-    bdrv_close(bs);
-
-    g_free(bs);
-}
-
-int bdrv_attach_dev(BlockDriverState *bs, void *dev)
-/* TODO change to DeviceState *dev when all users are qdevified */
-{
-    if (bs->dev) {
-        return -EBUSY;
-    }
-    bs->dev = dev;
-    bdrv_iostatus_reset(bs);
-    return 0;
-}
-
-/* TODO qdevified devices don't use this, remove when devices are qdevified */
-void bdrv_attach_dev_nofail(BlockDriverState *bs, void *dev)
-{
-    if (bdrv_attach_dev(bs, dev) < 0) {
-        abort();
-    }
-}
-
-void bdrv_detach_dev(BlockDriverState *bs, void *dev)
-/* TODO change to DeviceState *dev when all users are qdevified */
-{
-    assert(bs->dev == dev);
-    bs->dev = NULL;
-    bs->dev_ops = NULL;
-    bs->dev_opaque = NULL;
-    bs->buffer_alignment = 512;
-}
-
-/* TODO change to return DeviceState * when all users are qdevified */
-void *bdrv_get_attached_dev(BlockDriverState *bs)
-{
-    return bs->dev;
-}
-
-void bdrv_set_dev_ops(BlockDriverState *bs, const BlockDevOps *ops,
-                      void *opaque)
-{
-    bs->dev_ops = ops;
-    bs->dev_opaque = opaque;
-}
-
-void bdrv_emit_qmp_error_event(const BlockDriverState *bdrv,
-                               enum MonitorEvent ev,
-                               BlockErrorAction action, bool is_read)
-{
-    QObject *data;
-    const char *action_str;
-
-    switch (action) {
-    case BDRV_ACTION_REPORT:
-        action_str = "report";
-        break;
-    case BDRV_ACTION_IGNORE:
-        action_str = "ignore";
-        break;
-    case BDRV_ACTION_STOP:
-        action_str = "stop";
-        break;
-    default:
-        abort();
-    }
-
-    data = qobject_from_jsonf("{ 'device': %s, 'action': %s, 'operation': %s }",
-                              bdrv->device_name,
-                              action_str,
-                              is_read ? "read" : "write");
-    monitor_protocol_event(ev, data);
-
-    qobject_decref(data);
-}
-
-static void bdrv_emit_qmp_eject_event(BlockDriverState *bs, bool ejected)
-{
-    QObject *data;
-
-    data = qobject_from_jsonf("{ 'device': %s, 'tray-open': %i }",
-                              bdrv_get_device_name(bs), ejected);
-    monitor_protocol_event(QEVENT_DEVICE_TRAY_MOVED, data);
-
-    qobject_decref(data);
-}
-
-static void bdrv_dev_change_media_cb(BlockDriverState *bs, bool load)
-{
-    if (bs->dev_ops && bs->dev_ops->change_media_cb) {
-        bool tray_was_closed = !bdrv_dev_is_tray_open(bs);
-        bs->dev_ops->change_media_cb(bs->dev_opaque, load);
-        if (tray_was_closed) {
-            /* tray open */
-            bdrv_emit_qmp_eject_event(bs, true);
-        }
-        if (load) {
-            /* tray close */
-            bdrv_emit_qmp_eject_event(bs, false);
-        }
-    }
-}
-
-bool bdrv_dev_has_removable_media(BlockDriverState *bs)
-{
-    return !bs->dev || (bs->dev_ops && bs->dev_ops->change_media_cb);
-}
-
-void bdrv_dev_eject_request(BlockDriverState *bs, bool force)
-{
-    if (bs->dev_ops && bs->dev_ops->eject_request_cb) {
-        bs->dev_ops->eject_request_cb(bs->dev_opaque, force);
-    }
-}
-
-bool bdrv_dev_is_tray_open(BlockDriverState *bs)
-{
-    if (bs->dev_ops && bs->dev_ops->is_tray_open) {
-        return bs->dev_ops->is_tray_open(bs->dev_opaque);
-    }
-    return false;
-}
-
-static void bdrv_dev_resize_cb(BlockDriverState *bs)
-{
-    if (bs->dev_ops && bs->dev_ops->resize_cb) {
-        bs->dev_ops->resize_cb(bs->dev_opaque);
-    }
-}
-
-bool bdrv_dev_is_medium_locked(BlockDriverState *bs)
-{
-    if (bs->dev_ops && bs->dev_ops->is_medium_locked) {
-        return bs->dev_ops->is_medium_locked(bs->dev_opaque);
-    }
-    return false;
-}
-
-/*
- * Run consistency checks on an image
- *
- * Returns 0 if the check could be completed (it doesn't mean that the image is
- * free of errors) or -errno when an internal error occurred. The results of the
- * check are stored in res.
- */
-int bdrv_check(BlockDriverState *bs, BdrvCheckResult *res, BdrvCheckMode fix)
-{
-    if (bs->drv->bdrv_check == NULL) {
-        return -ENOTSUP;
-    }
-
-    memset(res, 0, sizeof(*res));
-    return bs->drv->bdrv_check(bs, res, fix);
-}
-
-#define COMMIT_BUF_SECTORS 2048
-
-/* commit COW file into the raw image */
-int bdrv_commit(BlockDriverState *bs)
-{
-    BlockDriver *drv = bs->drv;
-    int64_t sector, total_sectors;
-    int n, ro, open_flags;
-    int ret = 0;
-    uint8_t *buf;
-    char filename[PATH_MAX];
-
-    if (!drv)
-        return -ENOMEDIUM;
-    
-    if (!bs->backing_hd) {
-        return -ENOTSUP;
-    }
-
-    if (bdrv_in_use(bs) || bdrv_in_use(bs->backing_hd)) {
-        return -EBUSY;
-    }
-
-    ro = bs->backing_hd->read_only;
-    /* Use pstrcpy (not strncpy): filename must be NUL-terminated. */
-    pstrcpy(filename, sizeof(filename), bs->backing_hd->filename);
-    open_flags =  bs->backing_hd->open_flags;
-
-    if (ro) {
-        if (bdrv_reopen(bs->backing_hd, open_flags | BDRV_O_RDWR, NULL)) {
-            return -EACCES;
-        }
-    }
-
-    total_sectors = bdrv_getlength(bs) >> BDRV_SECTOR_BITS;
-    buf = g_malloc(COMMIT_BUF_SECTORS * BDRV_SECTOR_SIZE);
-
-    for (sector = 0; sector < total_sectors; sector += n) {
-        if (bdrv_is_allocated(bs, sector, COMMIT_BUF_SECTORS, &n)) {
-
-            if (bdrv_read(bs, sector, buf, n) != 0) {
-                ret = -EIO;
-                goto ro_cleanup;
-            }
-
-            if (bdrv_write(bs->backing_hd, sector, buf, n) != 0) {
-                ret = -EIO;
-                goto ro_cleanup;
-            }
-        }
-    }
-
-    if (drv->bdrv_make_empty) {
-        ret = drv->bdrv_make_empty(bs);
-        bdrv_flush(bs);
-    }
-
-    /*
-     * Make sure all data we wrote to the backing device is actually
-     * stable on disk.
-     */
-    if (bs->backing_hd)
-        bdrv_flush(bs->backing_hd);
-
-ro_cleanup:
-    g_free(buf);
-
-    if (ro) {
-        /* ignoring error return here */
-        bdrv_reopen(bs->backing_hd, open_flags & ~BDRV_O_RDWR, NULL);
-    }
-
-    return ret;
-}
-
-int bdrv_commit_all(void)
-{
-    BlockDriverState *bs;
-
-    QTAILQ_FOREACH(bs, &bdrv_states, list) {
-        if (bs->drv && bs->backing_hd) {
-            int ret = bdrv_commit(bs);
-            if (ret < 0) {
-                return ret;
-            }
-        }
-    }
-    return 0;
-}
-
-/**
- * Remove an active request from the tracked requests list
- *
- * This function should be called when a tracked request is completing.
- */
-static void tracked_request_end(BdrvTrackedRequest *req)
-{
-    QLIST_REMOVE(req, list);
-    qemu_co_queue_restart_all(&req->wait_queue);
-}
-
-/**
- * Add an active request to the tracked requests list
- */
-static void tracked_request_begin(BdrvTrackedRequest *req,
-                                  BlockDriverState *bs,
-                                  int64_t sector_num,
-                                  int nb_sectors, bool is_write)
-{
-    *req = (BdrvTrackedRequest){
-        .bs = bs,
-        .sector_num = sector_num,
-        .nb_sectors = nb_sectors,
-        .is_write = is_write,
-        .co = qemu_coroutine_self(),
-    };
-
-    qemu_co_queue_init(&req->wait_queue);
-
-    QLIST_INSERT_HEAD(&bs->tracked_requests, req, list);
-}
-
-/**
- * Round a region to cluster boundaries
- */
-void bdrv_round_to_clusters(BlockDriverState *bs,
-                            int64_t sector_num, int nb_sectors,
-                            int64_t *cluster_sector_num,
-                            int *cluster_nb_sectors)
-{
-    BlockDriverInfo bdi;
-
-    if (bdrv_get_info(bs, &bdi) < 0 || bdi.cluster_size == 0) {
-        *cluster_sector_num = sector_num;
-        *cluster_nb_sectors = nb_sectors;
-    } else {
-        int64_t c = bdi.cluster_size / BDRV_SECTOR_SIZE;
-        *cluster_sector_num = QEMU_ALIGN_DOWN(sector_num, c);
-        *cluster_nb_sectors = QEMU_ALIGN_UP(sector_num - *cluster_sector_num +
-                                            nb_sectors, c);
-    }
-}
-
-static bool tracked_request_overlaps(BdrvTrackedRequest *req,
-                                     int64_t sector_num, int nb_sectors) {
-    /*        aaaa   bbbb */
-    if (sector_num >= req->sector_num + req->nb_sectors) {
-        return false;
-    }
-    /* bbbb   aaaa        */
-    if (req->sector_num >= sector_num + nb_sectors) {
-        return false;
-    }
-    return true;
-}
-
-static void coroutine_fn wait_for_overlapping_requests(BlockDriverState *bs,
-        int64_t sector_num, int nb_sectors)
-{
-    BdrvTrackedRequest *req;
-    int64_t cluster_sector_num;
-    int cluster_nb_sectors;
-    bool retry;
-
-    /* If we touch the same cluster it counts as an overlap.  This guarantees
-     * that allocating writes will be serialized and not race with each other
-     * for the same cluster.  For example, in copy-on-read it ensures that the
-     * CoR read and write operations are atomic and guest writes cannot
-     * interleave between them.
-     */
-    bdrv_round_to_clusters(bs, sector_num, nb_sectors,
-                           &cluster_sector_num, &cluster_nb_sectors);
-
-    do {
-        retry = false;
-        QLIST_FOREACH(req, &bs->tracked_requests, list) {
-            if (tracked_request_overlaps(req, cluster_sector_num,
-                                         cluster_nb_sectors)) {
-                /* Hitting this means there was a reentrant request, for
-                 * example, a block driver issuing nested requests.  This must
-                 * never happen since it means deadlock.
-                 */
-                assert(qemu_coroutine_self() != req->co);
-
-                qemu_co_queue_wait(&req->wait_queue);
-                retry = true;
-                break;
-            }
-        }
-    } while (retry);
-}
-
-/*
- * Return values:
- * 0        - success
- * -EINVAL  - backing format specified, but no file
- * -ENOSPC  - can't update the backing file because no space is left in the
- *            image file header
- * -ENOTSUP - format driver doesn't support changing the backing file
- */
-int bdrv_change_backing_file(BlockDriverState *bs,
-    const char *backing_file, const char *backing_fmt)
-{
-    BlockDriver *drv = bs->drv;
-    int ret;
-
-    /* Backing file format doesn't make sense without a backing file */
-    if (backing_fmt && !backing_file) {
-        return -EINVAL;
-    }
-
-    if (drv->bdrv_change_backing_file != NULL) {
-        ret = drv->bdrv_change_backing_file(bs, backing_file, backing_fmt);
-    } else {
-        ret = -ENOTSUP;
-    }
-
-    if (ret == 0) {
-        pstrcpy(bs->backing_file, sizeof(bs->backing_file), backing_file ?: "");
-        pstrcpy(bs->backing_format, sizeof(bs->backing_format), backing_fmt ?: "");
-    }
-    return ret;
-}
-
-/*
- * Finds the image layer in the chain that has 'bs' as its backing file.
- *
- * active is the current topmost image.
- *
- * Returns NULL if bs is not found in active's image chain,
- * or if active == bs.
- */
-BlockDriverState *bdrv_find_overlay(BlockDriverState *active,
-                                    BlockDriverState *bs)
-{
-    BlockDriverState *overlay = NULL;
-    BlockDriverState *intermediate;
-
-    assert(active != NULL);
-    assert(bs != NULL);
-
-    /* if bs is the same as active, then by definition it has no overlay
-     */
-    if (active == bs) {
-        return NULL;
-    }
-
-    intermediate = active;
-    while (intermediate->backing_hd) {
-        if (intermediate->backing_hd == bs) {
-            overlay = intermediate;
-            break;
-        }
-        intermediate = intermediate->backing_hd;
-    }
-
-    return overlay;
-}
-
-typedef struct BlkIntermediateStates {
-    BlockDriverState *bs;
-    QSIMPLEQ_ENTRY(BlkIntermediateStates) entry;
-} BlkIntermediateStates;
-
-
-/*
- * Drops images above 'base' up to and including 'top', and sets the image
- * above 'top' to have base as its backing file.
- *
- * Requires that the overlay to 'top' is opened r/w, so that the backing file
- * information in 'bs' can be properly updated.
- *
- * E.g., this will convert the following chain:
- * bottom <- base <- intermediate <- top <- active
- *
- * to
- *
- * bottom <- base <- active
- *
- * It is allowed for bottom==base, in which case it converts:
- *
- * base <- intermediate <- top <- active
- *
- * to
- *
- * base <- active
- *
- * Error conditions:
- *  if active == top, that is considered an error
- *
- */
-int bdrv_drop_intermediate(BlockDriverState *active, BlockDriverState *top,
-                           BlockDriverState *base)
-{
-    BlockDriverState *intermediate;
-    BlockDriverState *base_bs = NULL;
-    BlockDriverState *new_top_bs = NULL;
-    BlkIntermediateStates *intermediate_state, *next;
-    int ret = -EIO;
-
-    QSIMPLEQ_HEAD(states_to_delete, BlkIntermediateStates) states_to_delete;
-    QSIMPLEQ_INIT(&states_to_delete);
-
-    if (!top->drv || !base->drv) {
-        goto exit;
-    }
-
-    new_top_bs = bdrv_find_overlay(active, top);
-
-    if (new_top_bs == NULL) {
-        /* we could not find the image above 'top', this is an error */
-        goto exit;
-    }
-
-    /* special case of new_top_bs->backing_hd already pointing to base - nothing
-     * to do, no intermediate images */
-    if (new_top_bs->backing_hd == base) {
-        ret = 0;
-        goto exit;
-    }
-
-    intermediate = top;
-
-    /* now we will go down through the list, and add each BDS we find
-     * into our deletion queue, until we hit the 'base'
-     */
-    while (intermediate) {
-        intermediate_state = g_malloc0(sizeof(BlkIntermediateStates));
-        intermediate_state->bs = intermediate;
-        QSIMPLEQ_INSERT_TAIL(&states_to_delete, intermediate_state, entry);
-
-        if (intermediate->backing_hd == base) {
-            base_bs = intermediate->backing_hd;
-            break;
-        }
-        intermediate = intermediate->backing_hd;
-    }
-    if (base_bs == NULL) {
-        /* something went wrong, we did not end at the base. safely
-         * unravel everything, and exit with error */
-        goto exit;
-    }
-
-    /* success - we can delete the intermediate states, and link top->base */
-    ret = bdrv_change_backing_file(new_top_bs, base_bs->filename,
-                                   base_bs->drv ? base_bs->drv->format_name : "");
-    if (ret) {
-        goto exit;
-    }
-    new_top_bs->backing_hd = base_bs;
-
-
-    QSIMPLEQ_FOREACH_SAFE(intermediate_state, &states_to_delete, entry, next) {
-        /* so that bdrv_close() does not recursively close the chain */
-        intermediate_state->bs->backing_hd = NULL;
-        bdrv_delete(intermediate_state->bs);
-    }
-    ret = 0;
-
-exit:
-    QSIMPLEQ_FOREACH_SAFE(intermediate_state, &states_to_delete, entry, next) {
-        g_free(intermediate_state);
-    }
-    return ret;
-}
-
-
-static int bdrv_check_byte_request(BlockDriverState *bs, int64_t offset,
-                                   size_t size)
-{
-    int64_t len;
-
-    if (!bdrv_is_inserted(bs))
-        return -ENOMEDIUM;
-
-    if (bs->growable)
-        return 0;
-
-    len = bdrv_getlength(bs);
-
-    if (offset < 0)
-        return -EIO;
-
-    if ((offset > len) || (len - offset < size))
-        return -EIO;
-
-    return 0;
-}
-
-static int bdrv_check_request(BlockDriverState *bs, int64_t sector_num,
-                              int nb_sectors)
-{
-    return bdrv_check_byte_request(bs, sector_num * BDRV_SECTOR_SIZE,
-                                   nb_sectors * BDRV_SECTOR_SIZE);
-}
-
-typedef struct RwCo {
-    BlockDriverState *bs;
-    int64_t sector_num;
-    int nb_sectors;
-    QEMUIOVector *qiov;
-    bool is_write;
-    int ret;
-} RwCo;
-
-static void coroutine_fn bdrv_rw_co_entry(void *opaque)
-{
-    RwCo *rwco = opaque;
-
-    if (!rwco->is_write) {
-        rwco->ret = bdrv_co_do_readv(rwco->bs, rwco->sector_num,
-                                     rwco->nb_sectors, rwco->qiov, 0);
-    } else {
-        rwco->ret = bdrv_co_do_writev(rwco->bs, rwco->sector_num,
-                                      rwco->nb_sectors, rwco->qiov, 0);
-    }
-}
-
-/*
- * Process a vectored synchronous request using coroutines
- */
-static int bdrv_rwv_co(BlockDriverState *bs, int64_t sector_num,
-                       QEMUIOVector *qiov, bool is_write)
-{
-    Coroutine *co;
-    RwCo rwco = {
-        .bs = bs,
-        .sector_num = sector_num,
-        .nb_sectors = qiov->size >> BDRV_SECTOR_BITS,
-        .qiov = qiov,
-        .is_write = is_write,
-        .ret = NOT_DONE,
-    };
-    assert((qiov->size & (BDRV_SECTOR_SIZE - 1)) == 0);
-
-    /**
-     * In sync call context, when the vcpu is blocked, this throttling timer
-     * will not fire; so the I/O throttling function has to be disabled here
-     * if it has been enabled.
-     */
-    if (bs->io_limits_enabled) {
-        fprintf(stderr, "Disabling I/O throttling on '%s' due "
-                        "to synchronous I/O.\n", bdrv_get_device_name(bs));
-        bdrv_io_limits_disable(bs);
-    }
-
-    if (qemu_in_coroutine()) {
-        /* Fast-path if already in coroutine context */
-        bdrv_rw_co_entry(&rwco);
-    } else {
-        co = qemu_coroutine_create(bdrv_rw_co_entry);
-        qemu_coroutine_enter(co, &rwco);
-        while (rwco.ret == NOT_DONE) {
-            qemu_aio_wait();
-        }
-    }
-    return rwco.ret;
-}
-
-/*
- * Process a synchronous request using coroutines
- */
-static int bdrv_rw_co(BlockDriverState *bs, int64_t sector_num, uint8_t *buf,
-                      int nb_sectors, bool is_write)
-{
-    QEMUIOVector qiov;
-    struct iovec iov = {
-        .iov_base = (void *)buf,
-        .iov_len = nb_sectors * BDRV_SECTOR_SIZE,
-    };
-
-    qemu_iovec_init_external(&qiov, &iov, 1);
-    return bdrv_rwv_co(bs, sector_num, &qiov, is_write);
-}
-
-/* return < 0 if error. See bdrv_write() for the return codes */
-int bdrv_read(BlockDriverState *bs, int64_t sector_num,
-              uint8_t *buf, int nb_sectors)
-{
-    return bdrv_rw_co(bs, sector_num, buf, nb_sectors, false);
-}
-
-/* Just like bdrv_read(), but with I/O throttling temporarily disabled */
-int bdrv_read_unthrottled(BlockDriverState *bs, int64_t sector_num,
-                          uint8_t *buf, int nb_sectors)
-{
-    bool enabled;
-    int ret;
-
-    enabled = bs->io_limits_enabled;
-    bs->io_limits_enabled = false;
-    ret = bdrv_read(bs, 0, buf, 1);
-    bs->io_limits_enabled = enabled;
-    return ret;
-}
-
-/* Return < 0 if error. Important errors are:
-  -EIO         generic I/O error (may happen for all errors)
-  -ENOMEDIUM   No media inserted.
-  -EINVAL      Invalid sector number or nb_sectors
-  -EACCES      Trying to write a read-only device
-*/
-int bdrv_write(BlockDriverState *bs, int64_t sector_num,
-               const uint8_t *buf, int nb_sectors)
-{
-    return bdrv_rw_co(bs, sector_num, (uint8_t *)buf, nb_sectors, true);
-}
-
-int bdrv_writev(BlockDriverState *bs, int64_t sector_num, QEMUIOVector *qiov)
-{
-    return bdrv_rwv_co(bs, sector_num, qiov, true);
-}
-
-int bdrv_pread(BlockDriverState *bs, int64_t offset,
-               void *buf, int count1)
-{
-    uint8_t tmp_buf[BDRV_SECTOR_SIZE];
-    int len, nb_sectors, count;
-    int64_t sector_num;
-    int ret;
-
-    count = count1;
-    /* first read to align to sector start */
-    len = (BDRV_SECTOR_SIZE - offset) & (BDRV_SECTOR_SIZE - 1);
-    if (len > count)
-        len = count;
-    sector_num = offset >> BDRV_SECTOR_BITS;
-    if (len > 0) {
-        if ((ret = bdrv_read(bs, sector_num, tmp_buf, 1)) < 0)
-            return ret;
-        memcpy(buf, tmp_buf + (offset & (BDRV_SECTOR_SIZE - 1)), len);
-        count -= len;
-        if (count == 0)
-            return count1;
-        sector_num++;
-        buf += len;
-    }
-
-    /* read the sectors "in place" */
-    nb_sectors = count >> BDRV_SECTOR_BITS;
-    if (nb_sectors > 0) {
-        if ((ret = bdrv_read(bs, sector_num, buf, nb_sectors)) < 0)
-            return ret;
-        sector_num += nb_sectors;
-        len = nb_sectors << BDRV_SECTOR_BITS;
-        buf += len;
-        count -= len;
-    }
-
-    /* add data from the last sector */
-    if (count > 0) {
-        if ((ret = bdrv_read(bs, sector_num, tmp_buf, 1)) < 0)
-            return ret;
-        memcpy(buf, tmp_buf, count);
-    }
-    return count1;
-}
-
-int bdrv_pwritev(BlockDriverState *bs, int64_t offset, QEMUIOVector *qiov)
-{
-    uint8_t tmp_buf[BDRV_SECTOR_SIZE];
-    int len, nb_sectors, count;
-    int64_t sector_num;
-    int ret;
-
-    count = qiov->size;
-
-    /* first write to align to sector start */
-    len = (BDRV_SECTOR_SIZE - offset) & (BDRV_SECTOR_SIZE - 1);
-    if (len > count)
-        len = count;
-    sector_num = offset >> BDRV_SECTOR_BITS;
-    if (len > 0) {
-        if ((ret = bdrv_read(bs, sector_num, tmp_buf, 1)) < 0)
-            return ret;
-        qemu_iovec_to_buf(qiov, 0, tmp_buf + (offset & (BDRV_SECTOR_SIZE - 1)),
-                          len);
-        if ((ret = bdrv_write(bs, sector_num, tmp_buf, 1)) < 0)
-            return ret;
-        count -= len;
-        if (count == 0)
-            return qiov->size;
-        sector_num++;
-    }
-
-    /* write the sectors "in place" */
-    nb_sectors = count >> BDRV_SECTOR_BITS;
-    if (nb_sectors > 0) {
-        QEMUIOVector qiov_inplace;
-
-        qemu_iovec_init(&qiov_inplace, qiov->niov);
-        qemu_iovec_concat(&qiov_inplace, qiov, len,
-                          nb_sectors << BDRV_SECTOR_BITS);
-        ret = bdrv_writev(bs, sector_num, &qiov_inplace);
-        qemu_iovec_destroy(&qiov_inplace);
-        if (ret < 0) {
-            return ret;
-        }
-
-        sector_num += nb_sectors;
-        len = nb_sectors << BDRV_SECTOR_BITS;
-        count -= len;
-    }
-
-    /* add data from the last sector */
-    if (count > 0) {
-        if ((ret = bdrv_read(bs, sector_num, tmp_buf, 1)) < 0)
-            return ret;
-        qemu_iovec_to_buf(qiov, qiov->size - count, tmp_buf, count);
-        if ((ret = bdrv_write(bs, sector_num, tmp_buf, 1)) < 0)
-            return ret;
-    }
-    return qiov->size;
-}
-
-int bdrv_pwrite(BlockDriverState *bs, int64_t offset,
-                const void *buf, int count1)
-{
-    QEMUIOVector qiov;
-    struct iovec iov = {
-        .iov_base   = (void *) buf,
-        .iov_len    = count1,
-    };
-
-    qemu_iovec_init_external(&qiov, &iov, 1);
-    return bdrv_pwritev(bs, offset, &qiov);
-}
-
-/*
- * Writes to the file and ensures that no writes are reordered across this
- * request (acts as a barrier)
- *
- * Returns 0 on success, -errno in error cases.
- */
-int bdrv_pwrite_sync(BlockDriverState *bs, int64_t offset,
-    const void *buf, int count)
-{
-    int ret;
-
-    ret = bdrv_pwrite(bs, offset, buf, count);
-    if (ret < 0) {
-        return ret;
-    }
-
-    /* No flush needed for cache modes that already do it */
-    if (bs->enable_write_cache) {
-        bdrv_flush(bs);
-    }
-
-    return 0;
-}
-
-static int coroutine_fn bdrv_co_do_copy_on_readv(BlockDriverState *bs,
-        int64_t sector_num, int nb_sectors, QEMUIOVector *qiov)
-{
-    /* Perform I/O through a temporary buffer so that users who scribble over
-     * their read buffer while the operation is in progress do not end up
-     * modifying the image file.  This is critical for zero-copy guest I/O
-     * where anything might happen inside guest memory.
-     */
-    void *bounce_buffer;
-
-    BlockDriver *drv = bs->drv;
-    struct iovec iov;
-    QEMUIOVector bounce_qiov;
-    int64_t cluster_sector_num;
-    int cluster_nb_sectors;
-    size_t skip_bytes;
-    int ret;
-
-    /* Cover entire cluster so no additional backing file I/O is required when
-     * allocating cluster in the image file.
-     */
-    bdrv_round_to_clusters(bs, sector_num, nb_sectors,
-                           &cluster_sector_num, &cluster_nb_sectors);
-
-    trace_bdrv_co_do_copy_on_readv(bs, sector_num, nb_sectors,
-                                   cluster_sector_num, cluster_nb_sectors);
-
-    iov.iov_len = cluster_nb_sectors * BDRV_SECTOR_SIZE;
-    iov.iov_base = bounce_buffer = qemu_blockalign(bs, iov.iov_len);
-    qemu_iovec_init_external(&bounce_qiov, &iov, 1);
-
-    ret = drv->bdrv_co_readv(bs, cluster_sector_num, cluster_nb_sectors,
-                             &bounce_qiov);
-    if (ret < 0) {
-        goto err;
-    }
-
-    if (drv->bdrv_co_write_zeroes &&
-        buffer_is_zero(bounce_buffer, iov.iov_len)) {
-        ret = bdrv_co_do_write_zeroes(bs, cluster_sector_num,
-                                      cluster_nb_sectors);
-    } else {
-        /* This does not change the data on the disk, it is not necessary
-         * to flush even in cache=writethrough mode.
-         */
-        ret = drv->bdrv_co_writev(bs, cluster_sector_num, cluster_nb_sectors,
-                                  &bounce_qiov);
-    }
-
-    if (ret < 0) {
-        /* It might be okay to ignore write errors for guest requests.  If this
-         * is a deliberate copy-on-read then we don't want to ignore the error.
-         * Simply report it in all cases.
-         */
-        goto err;
-    }
-
-    skip_bytes = (sector_num - cluster_sector_num) * BDRV_SECTOR_SIZE;
-    qemu_iovec_from_buf(qiov, 0, bounce_buffer + skip_bytes,
-                        nb_sectors * BDRV_SECTOR_SIZE);
-
-err:
-    qemu_vfree(bounce_buffer);
-    return ret;
-}
-
-/*
- * Handle a read request in coroutine context
- */
-static int coroutine_fn bdrv_co_do_readv(BlockDriverState *bs,
-    int64_t sector_num, int nb_sectors, QEMUIOVector *qiov,
-    BdrvRequestFlags flags)
-{
-    BlockDriver *drv = bs->drv;
-    BdrvTrackedRequest req;
-    int ret;
-
-    if (!drv) {
-        return -ENOMEDIUM;
-    }
-    if (bdrv_check_request(bs, sector_num, nb_sectors)) {
-        return -EIO;
-    }
-
-    /* throttling disk read I/O */
-    if (bs->io_limits_enabled) {
-        bdrv_io_limits_intercept(bs, false, nb_sectors);
-    }
-
-    if (bs->copy_on_read) {
-        flags |= BDRV_REQ_COPY_ON_READ;
-    }
-    if (flags & BDRV_REQ_COPY_ON_READ) {
-        bs->copy_on_read_in_flight++;
-    }
-
-    if (bs->copy_on_read_in_flight) {
-        wait_for_overlapping_requests(bs, sector_num, nb_sectors);
-    }
-
-    tracked_request_begin(&req, bs, sector_num, nb_sectors, false);
-
-    if (flags & BDRV_REQ_COPY_ON_READ) {
-        int pnum;
-
-        ret = bdrv_co_is_allocated(bs, sector_num, nb_sectors, &pnum);
-        if (ret < 0) {
-            goto out;
-        }
-
-        if (!ret || pnum != nb_sectors) {
-            ret = bdrv_co_do_copy_on_readv(bs, sector_num, nb_sectors, qiov);
-            goto out;
-        }
-    }
-
-    ret = drv->bdrv_co_readv(bs, sector_num, nb_sectors, qiov);
-
-out:
-    tracked_request_end(&req);
-
-    if (flags & BDRV_REQ_COPY_ON_READ) {
-        bs->copy_on_read_in_flight--;
-    }
-
-    return ret;
-}
-
-int coroutine_fn bdrv_co_readv(BlockDriverState *bs, int64_t sector_num,
-    int nb_sectors, QEMUIOVector *qiov)
-{
-    trace_bdrv_co_readv(bs, sector_num, nb_sectors);
-
-    return bdrv_co_do_readv(bs, sector_num, nb_sectors, qiov, 0);
-}
-
-int coroutine_fn bdrv_co_copy_on_readv(BlockDriverState *bs,
-    int64_t sector_num, int nb_sectors, QEMUIOVector *qiov)
-{
-    trace_bdrv_co_copy_on_readv(bs, sector_num, nb_sectors);
-
-    return bdrv_co_do_readv(bs, sector_num, nb_sectors, qiov,
-                            BDRV_REQ_COPY_ON_READ);
-}
-
-static int coroutine_fn bdrv_co_do_write_zeroes(BlockDriverState *bs,
-    int64_t sector_num, int nb_sectors)
-{
-    BlockDriver *drv = bs->drv;
-    QEMUIOVector qiov;
-    struct iovec iov;
-    int ret;
-
-    /* TODO Emulate only part of misaligned requests instead of letting block
-     * drivers return -ENOTSUP and emulate everything */
-
-    /* First try the efficient write zeroes operation */
-    if (drv->bdrv_co_write_zeroes) {
-        ret = drv->bdrv_co_write_zeroes(bs, sector_num, nb_sectors);
-        if (ret != -ENOTSUP) {
-            return ret;
-        }
-    }
-
-    /* Fall back to bounce buffer if write zeroes is unsupported */
-    iov.iov_len  = nb_sectors * BDRV_SECTOR_SIZE;
-    iov.iov_base = qemu_blockalign(bs, iov.iov_len);
-    memset(iov.iov_base, 0, iov.iov_len);
-    qemu_iovec_init_external(&qiov, &iov, 1);
-
-    ret = drv->bdrv_co_writev(bs, sector_num, nb_sectors, &qiov);
-
-    qemu_vfree(iov.iov_base);
-    return ret;
-}
-
-/*
- * Handle a write request in coroutine context
- */
-static int coroutine_fn bdrv_co_do_writev(BlockDriverState *bs,
-    int64_t sector_num, int nb_sectors, QEMUIOVector *qiov,
-    BdrvRequestFlags flags)
-{
-    BlockDriver *drv = bs->drv;
-    BdrvTrackedRequest req;
-    int ret;
-
-    if (!bs->drv) {
-        return -ENOMEDIUM;
-    }
-    if (bs->read_only) {
-        return -EACCES;
-    }
-    if (bdrv_check_request(bs, sector_num, nb_sectors)) {
-        return -EIO;
-    }
-
-    /* throttling disk write I/O */
-    if (bs->io_limits_enabled) {
-        bdrv_io_limits_intercept(bs, true, nb_sectors);
-    }
-
-    if (bs->copy_on_read_in_flight) {
-        wait_for_overlapping_requests(bs, sector_num, nb_sectors);
-    }
-
-    tracked_request_begin(&req, bs, sector_num, nb_sectors, true);
-
-    ret = notifier_with_return_list_notify(&bs->before_write_notifiers, &req);
-
-    if (ret < 0) {
-        /* Do nothing, write notifier decided to fail this request */
-    } else if (flags & BDRV_REQ_ZERO_WRITE) {
-        ret = bdrv_co_do_write_zeroes(bs, sector_num, nb_sectors);
-    } else {
-        ret = drv->bdrv_co_writev(bs, sector_num, nb_sectors, qiov);
-    }
-
-    if (ret == 0 && !bs->enable_write_cache) {
-        ret = bdrv_co_flush(bs);
-    }
-
-    if (bs->dirty_bitmap) {
-        bdrv_set_dirty(bs, sector_num, nb_sectors);
-    }
-
-    if (bs->wr_highest_sector < sector_num + nb_sectors - 1) {
-        bs->wr_highest_sector = sector_num + nb_sectors - 1;
-    }
-
-    tracked_request_end(&req);
-
-    return ret;
-}
-
-int coroutine_fn bdrv_co_writev(BlockDriverState *bs, int64_t sector_num,
-    int nb_sectors, QEMUIOVector *qiov)
-{
-    trace_bdrv_co_writev(bs, sector_num, nb_sectors);
-
-    return bdrv_co_do_writev(bs, sector_num, nb_sectors, qiov, 0);
-}
-
-int coroutine_fn bdrv_co_write_zeroes(BlockDriverState *bs,
-                                      int64_t sector_num, int nb_sectors)
-{
-    trace_bdrv_co_write_zeroes(bs, sector_num, nb_sectors);
-
-    return bdrv_co_do_writev(bs, sector_num, nb_sectors, NULL,
-                             BDRV_REQ_ZERO_WRITE);
-}
-
-/**
- * Truncate file to 'offset' bytes (needed only for file protocols)
- */
-int bdrv_truncate(BlockDriverState *bs, int64_t offset)
-{
-    BlockDriver *drv = bs->drv;
-    int ret;
-    if (!drv)
-        return -ENOMEDIUM;
-    if (!drv->bdrv_truncate)
-        return -ENOTSUP;
-    if (bs->read_only)
-        return -EACCES;
-    if (bdrv_in_use(bs))
-        return -EBUSY;
-    ret = drv->bdrv_truncate(bs, offset);
-    if (ret == 0) {
-        ret = refresh_total_sectors(bs, offset >> BDRV_SECTOR_BITS);
-        bdrv_dev_resize_cb(bs);
-    }
-    return ret;
-}
-
-/**
- * Length of a allocated file in bytes. Sparse files are counted by actual
- * allocated space. Return < 0 if error or unknown.
- */
-int64_t bdrv_get_allocated_file_size(BlockDriverState *bs)
-{
-    BlockDriver *drv = bs->drv;
-    if (!drv) {
-        return -ENOMEDIUM;
-    }
-    if (drv->bdrv_get_allocated_file_size) {
-        return drv->bdrv_get_allocated_file_size(bs);
-    }
-    if (bs->file) {
-        return bdrv_get_allocated_file_size(bs->file);
-    }
-    return -ENOTSUP;
-}
-
-/**
- * Length of a file in bytes. Return < 0 if error or unknown.
- */
-int64_t bdrv_getlength(BlockDriverState *bs)
-{
-    BlockDriver *drv = bs->drv;
-    if (!drv)
-        return -ENOMEDIUM;
-
-    if (bs->growable || bdrv_dev_has_removable_media(bs)) {
-        if (drv->bdrv_getlength) {
-            return drv->bdrv_getlength(bs);
-        }
-    }
-    return bs->total_sectors * BDRV_SECTOR_SIZE;
-}
-
-/* return 0 as number of sectors if no device present or error */
-void bdrv_get_geometry(BlockDriverState *bs, uint64_t *nb_sectors_ptr)
-{
-    int64_t length;
-    length = bdrv_getlength(bs);
-    if (length < 0)
-        length = 0;
-    else
-        length = length >> BDRV_SECTOR_BITS;
-    *nb_sectors_ptr = length;
-}
-
-/* throttling disk io limits */
-void bdrv_set_io_limits(BlockDriverState *bs,
-                        BlockIOLimit *io_limits)
-{
-    bs->io_limits = *io_limits;
-    bs->io_limits_enabled = bdrv_io_limits_enabled(bs);
-}
-
-void bdrv_set_on_error(BlockDriverState *bs, BlockdevOnError on_read_error,
-                       BlockdevOnError on_write_error)
-{
-    bs->on_read_error = on_read_error;
-    bs->on_write_error = on_write_error;
-}
-
-BlockdevOnError bdrv_get_on_error(BlockDriverState *bs, bool is_read)
-{
-    return is_read ? bs->on_read_error : bs->on_write_error;
-}
-
-BlockErrorAction bdrv_get_error_action(BlockDriverState *bs, bool is_read, int error)
-{
-    BlockdevOnError on_err = is_read ? bs->on_read_error : bs->on_write_error;
-
-    switch (on_err) {
-    case BLOCKDEV_ON_ERROR_ENOSPC:
-        return (error == ENOSPC) ? BDRV_ACTION_STOP : BDRV_ACTION_REPORT;
-    case BLOCKDEV_ON_ERROR_STOP:
-        return BDRV_ACTION_STOP;
-    case BLOCKDEV_ON_ERROR_REPORT:
-        return BDRV_ACTION_REPORT;
-    case BLOCKDEV_ON_ERROR_IGNORE:
-        return BDRV_ACTION_IGNORE;
-    default:
-        abort();
-    }
-}
-
-/* This is done by device models because, while the block layer knows
- * about the error, it does not know whether an operation comes from
- * the device or the block layer (from a job, for example).
- */
-void bdrv_error_action(BlockDriverState *bs, BlockErrorAction action,
-                       bool is_read, int error)
-{
-    assert(error >= 0);
-    bdrv_emit_qmp_error_event(bs, QEVENT_BLOCK_IO_ERROR, action, is_read);
-    if (action == BDRV_ACTION_STOP) {
-        vm_stop(RUN_STATE_IO_ERROR);
-        bdrv_iostatus_set_err(bs, error);
-    }
-}
-
-int bdrv_is_read_only(BlockDriverState *bs)
-{
-    return bs->read_only;
-}
-
-int bdrv_is_sg(BlockDriverState *bs)
-{
-    return bs->sg;
-}
-
-int bdrv_enable_write_cache(BlockDriverState *bs)
-{
-    return bs->enable_write_cache;
-}
-
-void bdrv_set_enable_write_cache(BlockDriverState *bs, bool wce)
-{
-    bs->enable_write_cache = wce;
-
-    /* so a reopen() will preserve wce */
-    if (wce) {
-        bs->open_flags |= BDRV_O_CACHE_WB;
-    } else {
-        bs->open_flags &= ~BDRV_O_CACHE_WB;
-    }
-}
-
-int bdrv_is_encrypted(BlockDriverState *bs)
-{
-    if (bs->backing_hd && bs->backing_hd->encrypted)
-        return 1;
-    return bs->encrypted;
-}
-
-int bdrv_key_required(BlockDriverState *bs)
-{
-    BlockDriverState *backing_hd = bs->backing_hd;
-
-    if (backing_hd && backing_hd->encrypted && !backing_hd->valid_key)
-        return 1;
-    return (bs->encrypted && !bs->valid_key);
-}
-
-int bdrv_set_key(BlockDriverState *bs, const char *key)
-{
-    int ret;
-    if (bs->backing_hd && bs->backing_hd->encrypted) {
-        ret = bdrv_set_key(bs->backing_hd, key);
-        if (ret < 0)
-            return ret;
-        if (!bs->encrypted)
-            return 0;
-    }
-    if (!bs->encrypted) {
-        return -EINVAL;
-    } else if (!bs->drv || !bs->drv->bdrv_set_key) {
-        return -ENOMEDIUM;
-    }
-    ret = bs->drv->bdrv_set_key(bs, key);
-    if (ret < 0) {
-        bs->valid_key = 0;
-    } else if (!bs->valid_key) {
-        bs->valid_key = 1;
-        /* call the change callback now, we skipped it on open */
-        bdrv_dev_change_media_cb(bs, true);
-    }
-    return ret;
-}
-
-const char *bdrv_get_format_name(BlockDriverState *bs)
-{
-    return bs->drv ? bs->drv->format_name : NULL;
-}
-
-void bdrv_iterate_format(void (*it)(void *opaque, const char *name),
-                         void *opaque)
-{
-    BlockDriver *drv;
-
-    QLIST_FOREACH(drv, &bdrv_drivers, list) {
-        it(opaque, drv->format_name);
-    }
-}
-
-BlockDriverState *bdrv_find(const char *name)
-{
-    BlockDriverState *bs;
-
-    QTAILQ_FOREACH(bs, &bdrv_states, list) {
-        if (!strcmp(name, bs->device_name)) {
-            return bs;
-        }
-    }
-    return NULL;
-}
-
-BlockDriverState *bdrv_next(BlockDriverState *bs)
-{
-    if (!bs) {
-        return QTAILQ_FIRST(&bdrv_states);
-    }
-    return QTAILQ_NEXT(bs, list);
-}
-
-void bdrv_iterate(void (*it)(void *opaque, BlockDriverState *bs), void *opaque)
-{
-    BlockDriverState *bs;
-
-    QTAILQ_FOREACH(bs, &bdrv_states, list) {
-        it(opaque, bs);
-    }
-}
-
-const char *bdrv_get_device_name(BlockDriverState *bs)
-{
-    return bs->device_name;
-}
-
-int bdrv_get_flags(BlockDriverState *bs)
-{
-    return bs->open_flags;
-}
-
-int bdrv_flush_all(void)
-{
-    BlockDriverState *bs;
-    int result = 0;
-
-    QTAILQ_FOREACH(bs, &bdrv_states, list) {
-        int ret = bdrv_flush(bs);
-        if (ret < 0 && !result) {
-            result = ret;
-        }
-    }
-
-    return result;
-}
-
-int bdrv_has_zero_init_1(BlockDriverState *bs)
-{
-    return 1;
-}
-
-int bdrv_has_zero_init(BlockDriverState *bs)
-{
-    assert(bs->drv);
-
-    if (bs->drv->bdrv_has_zero_init) {
-        return bs->drv->bdrv_has_zero_init(bs);
-    }
-
-    /* safe default */
-    return 0;
-}
-
-typedef struct BdrvCoIsAllocatedData {
-    BlockDriverState *bs;
-    BlockDriverState *base;
-    int64_t sector_num;
-    int nb_sectors;
-    int *pnum;
-    int ret;
-    bool done;
-} BdrvCoIsAllocatedData;
-
-/*
- * Returns true iff the specified sector is present in the disk image. Drivers
- * not implementing the functionality are assumed to not support backing files,
- * hence all their sectors are reported as allocated.
- *
- * If 'sector_num' is beyond the end of the disk image the return value is 0
- * and 'pnum' is set to 0.
- *
- * 'pnum' is set to the number of sectors (including and immediately following
- * the specified sector) that are known to be in the same
- * allocated/unallocated state.
- *
- * 'nb_sectors' is the max value 'pnum' should be set to.  If nb_sectors goes
- * beyond the end of the disk image it will be clamped.
- */
-int coroutine_fn bdrv_co_is_allocated(BlockDriverState *bs, int64_t sector_num,
-                                      int nb_sectors, int *pnum)
-{
-    int64_t n;
-
-    if (sector_num >= bs->total_sectors) {
-        *pnum = 0;
-        return 0;
-    }
-
-    n = bs->total_sectors - sector_num;
-    if (n < nb_sectors) {
-        nb_sectors = n;
-    }
-
-    if (!bs->drv->bdrv_co_is_allocated) {
-        *pnum = nb_sectors;
-        return 1;
-    }
-
-    return bs->drv->bdrv_co_is_allocated(bs, sector_num, nb_sectors, pnum);
-}
-
-/* Coroutine wrapper for bdrv_is_allocated() */
-static void coroutine_fn bdrv_is_allocated_co_entry(void *opaque)
-{
-    BdrvCoIsAllocatedData *data = opaque;
-    BlockDriverState *bs = data->bs;
-
-    data->ret = bdrv_co_is_allocated(bs, data->sector_num, data->nb_sectors,
-                                     data->pnum);
-    data->done = true;
-}
-
-/*
- * Synchronous wrapper around bdrv_co_is_allocated().
- *
- * See bdrv_co_is_allocated() for details.
- */
-int bdrv_is_allocated(BlockDriverState *bs, int64_t sector_num, int nb_sectors,
-                      int *pnum)
-{
-    Coroutine *co;
-    BdrvCoIsAllocatedData data = {
-        .bs = bs,
-        .sector_num = sector_num,
-        .nb_sectors = nb_sectors,
-        .pnum = pnum,
-        .done = false,
-    };
-
-    co = qemu_coroutine_create(bdrv_is_allocated_co_entry);
-    qemu_coroutine_enter(co, &data);
-    while (!data.done) {
-        qemu_aio_wait();
-    }
-    return data.ret;
-}
-
-/*
- * Given an image chain: ... -> [BASE] -> [INTER1] -> [INTER2] -> [TOP]
- *
- * Return true if the given sector is allocated in any image between
- * BASE and TOP (inclusive).  BASE can be NULL to check if the given
- * sector is allocated in any image of the chain.  Return false otherwise.
- *
- * 'pnum' is set to the number of sectors (including and immediately following
- *  the specified sector) that are known to be in the same
- *  allocated/unallocated state.
- *
- */
-int coroutine_fn bdrv_co_is_allocated_above(BlockDriverState *top,
-                                            BlockDriverState *base,
-                                            int64_t sector_num,
-                                            int nb_sectors, int *pnum)
-{
-    BlockDriverState *intermediate;
-    int ret, n = nb_sectors;
-
-    intermediate = top;
-    while (intermediate && intermediate != base) {
-        int pnum_inter;
-        ret = bdrv_co_is_allocated(intermediate, sector_num, nb_sectors,
-                                   &pnum_inter);
-        if (ret < 0) {
-            return ret;
-        } else if (ret) {
-            *pnum = pnum_inter;
-            return 1;
-        }
-
-        /*
-         * [sector_num, nb_sectors] is unallocated on top but intermediate
-         * might have
-         *
-         * [sector_num+x, nr_sectors] allocated.
-         */
-        if (n > pnum_inter &&
-            (intermediate == top ||
-             sector_num + pnum_inter < intermediate->total_sectors)) {
-            n = pnum_inter;
-        }
-
-        intermediate = intermediate->backing_hd;
-    }
-
-    *pnum = n;
-    return 0;
-}
-
-/* Coroutine wrapper for bdrv_is_allocated_above() */
-static void coroutine_fn bdrv_is_allocated_above_co_entry(void *opaque)
-{
-    BdrvCoIsAllocatedData *data = opaque;
-    BlockDriverState *top = data->bs;
-    BlockDriverState *base = data->base;
-
-    data->ret = bdrv_co_is_allocated_above(top, base, data->sector_num,
-                                           data->nb_sectors, data->pnum);
-    data->done = true;
-}
-
-/*
- * Synchronous wrapper around bdrv_co_is_allocated_above().
- *
- * See bdrv_co_is_allocated_above() for details.
- */
-int bdrv_is_allocated_above(BlockDriverState *top, BlockDriverState *base,
-                            int64_t sector_num, int nb_sectors, int *pnum)
-{
-    Coroutine *co;
-    BdrvCoIsAllocatedData data = {
-        .bs = top,
-        .base = base,
-        .sector_num = sector_num,
-        .nb_sectors = nb_sectors,
-        .pnum = pnum,
-        .done = false,
-    };
-
-    co = qemu_coroutine_create(bdrv_is_allocated_above_co_entry);
-    qemu_coroutine_enter(co, &data);
-    while (!data.done) {
-        qemu_aio_wait();
-    }
-    return data.ret;
-}
-
-const char *bdrv_get_encrypted_filename(BlockDriverState *bs)
-{
-    if (bs->backing_hd && bs->backing_hd->encrypted)
-        return bs->backing_file;
-    else if (bs->encrypted)
-        return bs->filename;
-    else
-        return NULL;
-}
-
-void bdrv_get_backing_filename(BlockDriverState *bs,
-                               char *filename, int filename_size)
-{
-    pstrcpy(filename, filename_size, bs->backing_file);
-}
-
-int bdrv_write_compressed(BlockDriverState *bs, int64_t sector_num,
-                          const uint8_t *buf, int nb_sectors)
-{
-    BlockDriver *drv = bs->drv;
-    if (!drv)
-        return -ENOMEDIUM;
-    if (!drv->bdrv_write_compressed)
-        return -ENOTSUP;
-    if (bdrv_check_request(bs, sector_num, nb_sectors))
-        return -EIO;
-
-    assert(!bs->dirty_bitmap);
-
-    return drv->bdrv_write_compressed(bs, sector_num, buf, nb_sectors);
-}
-
-int bdrv_get_info(BlockDriverState *bs, BlockDriverInfo *bdi)
-{
-    BlockDriver *drv = bs->drv;
-    if (!drv)
-        return -ENOMEDIUM;
-    if (!drv->bdrv_get_info)
-        return -ENOTSUP;
-    memset(bdi, 0, sizeof(*bdi));
-    return drv->bdrv_get_info(bs, bdi);
-}
-
-int bdrv_save_vmstate(BlockDriverState *bs, const uint8_t *buf,
-                      int64_t pos, int size)
-{
-    QEMUIOVector qiov;
-    struct iovec iov = {
-        .iov_base   = (void *) buf,
-        .iov_len    = size,
-    };
-
-    qemu_iovec_init_external(&qiov, &iov, 1);
-    return bdrv_writev_vmstate(bs, &qiov, pos);
-}
-
-int bdrv_writev_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos)
-{
-    BlockDriver *drv = bs->drv;
-
-    if (!drv) {
-        return -ENOMEDIUM;
-    } else if (drv->bdrv_save_vmstate) {
-        return drv->bdrv_save_vmstate(bs, qiov, pos);
-    } else if (bs->file) {
-        return bdrv_writev_vmstate(bs->file, qiov, pos);
-    }
-
-    return -ENOTSUP;
-}
-
-int bdrv_load_vmstate(BlockDriverState *bs, uint8_t *buf,
-                      int64_t pos, int size)
-{
-    BlockDriver *drv = bs->drv;
-    if (!drv)
-        return -ENOMEDIUM;
-    if (drv->bdrv_load_vmstate)
-        return drv->bdrv_load_vmstate(bs, buf, pos, size);
-    if (bs->file)
-        return bdrv_load_vmstate(bs->file, buf, pos, size);
-    return -ENOTSUP;
-}
-
-void bdrv_debug_event(BlockDriverState *bs, BlkDebugEvent event)
-{
-    if (!bs || !bs->drv || !bs->drv->bdrv_debug_event) {
-        return;
-    }
-
-    bs->drv->bdrv_debug_event(bs, event);
-}
-
-int bdrv_debug_breakpoint(BlockDriverState *bs, const char *event,
-                          const char *tag)
-{
-    while (bs && bs->drv && !bs->drv->bdrv_debug_breakpoint) {
-        bs = bs->file;
-    }
-
-    if (bs && bs->drv && bs->drv->bdrv_debug_breakpoint) {
-        return bs->drv->bdrv_debug_breakpoint(bs, event, tag);
-    }
-
-    return -ENOTSUP;
-}
-
-int bdrv_debug_resume(BlockDriverState *bs, const char *tag)
-{
-    while (bs && bs->drv && !bs->drv->bdrv_debug_resume) {
-        bs = bs->file;
-    }
-
-    if (bs && bs->drv && bs->drv->bdrv_debug_resume) {
-        return bs->drv->bdrv_debug_resume(bs, tag);
-    }
-
-    return -ENOTSUP;
-}
-
-bool bdrv_debug_is_suspended(BlockDriverState *bs, const char *tag)
-{
-    while (bs && bs->drv && !bs->drv->bdrv_debug_is_suspended) {
-        bs = bs->file;
-    }
-
-    if (bs && bs->drv && bs->drv->bdrv_debug_is_suspended) {
-        return bs->drv->bdrv_debug_is_suspended(bs, tag);
-    }
-
-    return false;
-}
-
-int bdrv_is_snapshot(BlockDriverState *bs)
-{
-    return !!(bs->open_flags & BDRV_O_SNAPSHOT);
-}
-
-/* backing_file can either be relative, or absolute, or a protocol.  If it is
- * relative, it must be relative to the chain.  So, passing in bs->filename
- * from a BDS as backing_file should not be done, as that may be relative to
- * the CWD rather than the chain. */
-BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
-        const char *backing_file)
-{
-    char *filename_full = NULL;
-    char *backing_file_full = NULL;
-    char *filename_tmp = NULL;
-    int is_protocol = 0;
-    BlockDriverState *curr_bs = NULL;
-    BlockDriverState *retval = NULL;
-
-    if (!bs || !bs->drv || !backing_file) {
-        return NULL;
-    }
-
-    filename_full     = g_malloc(PATH_MAX);
-    backing_file_full = g_malloc(PATH_MAX);
-    filename_tmp      = g_malloc(PATH_MAX);
-
-    is_protocol = path_has_protocol(backing_file);
-
-    for (curr_bs = bs; curr_bs->backing_hd; curr_bs = curr_bs->backing_hd) {
-
-        /* If either of the filename paths is actually a protocol, then
-         * compare unmodified paths; otherwise make paths relative */
-        if (is_protocol || path_has_protocol(curr_bs->backing_file)) {
-            if (strcmp(backing_file, curr_bs->backing_file) == 0) {
-                retval = curr_bs->backing_hd;
-                break;
-            }
-        } else {
-            /* If not an absolute filename path, make it relative to the current
-             * image's filename path */
-            path_combine(filename_tmp, PATH_MAX, curr_bs->filename,
-                         backing_file);
-
-            /* We are going to compare absolute pathnames */
-            if (!realpath(filename_tmp, filename_full)) {
-                continue;
-            }
-
-            /* We need to make sure the backing filename we are comparing against
-             * is relative to the current image filename (or absolute) */
-            path_combine(filename_tmp, PATH_MAX, curr_bs->filename,
-                         curr_bs->backing_file);
-
-            if (!realpath(filename_tmp, backing_file_full)) {
-                continue;
-            }
-
-            if (strcmp(backing_file_full, filename_full) == 0) {
-                retval = curr_bs->backing_hd;
-                break;
-            }
-        }
-    }
-
-    g_free(filename_full);
-    g_free(backing_file_full);
-    g_free(filename_tmp);
-    return retval;
-}
-
-int bdrv_get_backing_file_depth(BlockDriverState *bs)
-{
-    if (!bs->drv) {
-        return 0;
-    }
-
-    if (!bs->backing_hd) {
-        return 0;
-    }
-
-    return 1 + bdrv_get_backing_file_depth(bs->backing_hd);
-}
-
-BlockDriverState *bdrv_find_base(BlockDriverState *bs)
-{
-    BlockDriverState *curr_bs = NULL;
-
-    if (!bs) {
-        return NULL;
-    }
-
-    curr_bs = bs;
-
-    while (curr_bs->backing_hd) {
-        curr_bs = curr_bs->backing_hd;
-    }
-    return curr_bs;
-}
-
-/**************************************************************/
-/* async I/Os */
-
-BlockDriverAIOCB *bdrv_aio_readv(BlockDriverState *bs, int64_t sector_num,
-                                 QEMUIOVector *qiov, int nb_sectors,
-                                 BlockDriverCompletionFunc *cb, void *opaque)
-{
-    trace_bdrv_aio_readv(bs, sector_num, nb_sectors, opaque);
-
-    return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors,
-                                 cb, opaque, false);
-}
-
-BlockDriverAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
-                                  QEMUIOVector *qiov, int nb_sectors,
-                                  BlockDriverCompletionFunc *cb, void *opaque)
-{
-    trace_bdrv_aio_writev(bs, sector_num, nb_sectors, opaque);
-
-    return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors,
-                                 cb, opaque, true);
-}
-
-
-typedef struct MultiwriteCB {
-    int error;
-    int num_requests;
-    int num_callbacks;
-    struct {
-        BlockDriverCompletionFunc *cb;
-        void *opaque;
-        QEMUIOVector *free_qiov;
-    } callbacks[];
-} MultiwriteCB;
-
-static void multiwrite_user_cb(MultiwriteCB *mcb)
-{
-    int i;
-
-    for (i = 0; i < mcb->num_callbacks; i++) {
-        mcb->callbacks[i].cb(mcb->callbacks[i].opaque, mcb->error);
-        if (mcb->callbacks[i].free_qiov) {
-            qemu_iovec_destroy(mcb->callbacks[i].free_qiov);
-        }
-        g_free(mcb->callbacks[i].free_qiov);
-    }
-}
-
-static void multiwrite_cb(void *opaque, int ret)
-{
-    MultiwriteCB *mcb = opaque;
-
-    trace_multiwrite_cb(mcb, ret);
-
-    if (ret < 0 && !mcb->error) {
-        mcb->error = ret;
-    }
-
-    mcb->num_requests--;
-    if (mcb->num_requests == 0) {
-        multiwrite_user_cb(mcb);
-        g_free(mcb);
-    }
-}
-
-static int multiwrite_req_compare(const void *a, const void *b)
-{
-    const BlockRequest *req1 = a, *req2 = b;
-
-    /*
-     * Note that we can't simply subtract req2->sector from req1->sector
-     * here as that could overflow the return value.
-     */
-    if (req1->sector > req2->sector) {
-        return 1;
-    } else if (req1->sector < req2->sector) {
-        return -1;
-    } else {
-        return 0;
-    }
-}
-
-/*
- * Takes a bunch of requests and tries to merge them. Returns the number of
- * requests that remain after merging.
- */
-static int multiwrite_merge(BlockDriverState *bs, BlockRequest *reqs,
-    int num_reqs, MultiwriteCB *mcb)
-{
-    int i, outidx;
-
-    // Sort requests by start sector
-    qsort(reqs, num_reqs, sizeof(*reqs), &multiwrite_req_compare);
-
-    // Check if adjacent requests touch the same clusters. If so, combine them,
-    // filling up gaps with zero sectors.
-    outidx = 0;
-    for (i = 1; i < num_reqs; i++) {
-        int merge = 0;
-        int64_t oldreq_last = reqs[outidx].sector + reqs[outidx].nb_sectors;
-
-        // Handle exactly sequential writes and overlapping writes.
-        if (reqs[i].sector <= oldreq_last) {
-            merge = 1;
-        }
-
-        if (reqs[outidx].qiov->niov + reqs[i].qiov->niov + 1 > IOV_MAX) {
-            merge = 0;
-        }
-
-        if (merge) {
-            size_t size;
-            QEMUIOVector *qiov = g_malloc0(sizeof(*qiov));
-            qemu_iovec_init(qiov,
-                reqs[outidx].qiov->niov + reqs[i].qiov->niov + 1);
-
-            // Add the first request to the merged one. If the requests are
-            // overlapping, drop the last sectors of the first request.
-            size = (reqs[i].sector - reqs[outidx].sector) << 9;
-            qemu_iovec_concat(qiov, reqs[outidx].qiov, 0, size);
-
-            // We should need to add any zeros between the two requests
-            assert (reqs[i].sector <= oldreq_last);
-
-            // Add the second request
-            qemu_iovec_concat(qiov, reqs[i].qiov, 0, reqs[i].qiov->size);
-
-            reqs[outidx].nb_sectors = qiov->size >> 9;
-            reqs[outidx].qiov = qiov;
-
-            mcb->callbacks[i].free_qiov = reqs[outidx].qiov;
-        } else {
-            outidx++;
-            reqs[outidx].sector     = reqs[i].sector;
-            reqs[outidx].nb_sectors = reqs[i].nb_sectors;
-            reqs[outidx].qiov       = reqs[i].qiov;
-        }
-    }
-
-    return outidx + 1;
-}
-
-/*
- * Submit multiple AIO write requests at once.
- *
- * On success, the function returns 0 and all requests in the reqs array have
- * been submitted. In error case this function returns -1, and any of the
- * requests may or may not be submitted yet. In particular, this means that the
- * callback will be called for some of the requests, for others it won't. The
- * caller must check the error field of the BlockRequest to wait for the right
- * callbacks (if error != 0, no callback will be called).
- *
- * The implementation may modify the contents of the reqs array, e.g. to merge
- * requests. However, the fields opaque and error are left unmodified as they
- * are used to signal failure for a single request to the caller.
- */
-int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs)
-{
-    MultiwriteCB *mcb;
-    int i;
-
-    /* don't submit writes if we don't have a medium */
-    if (bs->drv == NULL) {
-        for (i = 0; i < num_reqs; i++) {
-            reqs[i].error = -ENOMEDIUM;
-        }
-        return -1;
-    }
-
-    if (num_reqs == 0) {
-        return 0;
-    }
-
-    // Create MultiwriteCB structure
-    mcb = g_malloc0(sizeof(*mcb) + num_reqs * sizeof(*mcb->callbacks));
-    mcb->num_requests = 0;
-    mcb->num_callbacks = num_reqs;
-
-    for (i = 0; i < num_reqs; i++) {
-        mcb->callbacks[i].cb = reqs[i].cb;
-        mcb->callbacks[i].opaque = reqs[i].opaque;
-    }
-
-    // Check for mergable requests
-    num_reqs = multiwrite_merge(bs, reqs, num_reqs, mcb);
-
-    trace_bdrv_aio_multiwrite(mcb, mcb->num_callbacks, num_reqs);
-
-    /* Run the aio requests. */
-    mcb->num_requests = num_reqs;
-    for (i = 0; i < num_reqs; i++) {
-        bdrv_aio_writev(bs, reqs[i].sector, reqs[i].qiov,
-            reqs[i].nb_sectors, multiwrite_cb, mcb);
-    }
-
-    return 0;
-}
-
-void bdrv_aio_cancel(BlockDriverAIOCB *acb)
-{
-    acb->aiocb_info->cancel(acb);
-}
-
-/* block I/O throttling */
-static bool bdrv_exceed_bps_limits(BlockDriverState *bs, int nb_sectors,
-                 bool is_write, double elapsed_time, uint64_t *wait)
-{
-    uint64_t bps_limit = 0;
-    uint64_t extension;
-    double   bytes_limit, bytes_base, bytes_res;
-    double   slice_time, wait_time;
-
-    if (bs->io_limits.bps[BLOCK_IO_LIMIT_TOTAL]) {
-        bps_limit = bs->io_limits.bps[BLOCK_IO_LIMIT_TOTAL];
-    } else if (bs->io_limits.bps[is_write]) {
-        bps_limit = bs->io_limits.bps[is_write];
-    } else {
-        if (wait) {
-            *wait = 0;
-        }
-
-        return false;
-    }
-
-    slice_time = bs->slice_end - bs->slice_start;
-    slice_time /= (NANOSECONDS_PER_SECOND);
-    bytes_limit = bps_limit * slice_time;
-    bytes_base  = bs->slice_submitted.bytes[is_write];
-    if (bs->io_limits.bps[BLOCK_IO_LIMIT_TOTAL]) {
-        bytes_base += bs->slice_submitted.bytes[!is_write];
-    }
-
-    /* bytes_base: the bytes of data which have been read/written; and
-     *             it is obtained from the history statistic info.
-     * bytes_res: the remaining bytes of data which need to be read/written.
-     * (bytes_base + bytes_res) / bps_limit: used to calcuate
-     *             the total time for completing reading/writting all data.
-     */
-    bytes_res   = (unsigned) nb_sectors * BDRV_SECTOR_SIZE;
-
-    if (bytes_base + bytes_res <= bytes_limit) {
-        if (wait) {
-            *wait = 0;
-        }
-
-        return false;
-    }
-
-    /* Calc approx time to dispatch */
-    wait_time = (bytes_base + bytes_res) / bps_limit - elapsed_time;
-
-    /* When the I/O rate at runtime exceeds the limits,
-     * bs->slice_end need to be extended in order that the current statistic
-     * info can be kept until the timer fire, so it is increased and tuned
-     * based on the result of experiment.
-     */
-    extension = wait_time * NANOSECONDS_PER_SECOND;
-    extension = DIV_ROUND_UP(extension, BLOCK_IO_SLICE_TIME) *
-                BLOCK_IO_SLICE_TIME;
-    bs->slice_end += extension;
-    if (wait) {
-        *wait = wait_time * NANOSECONDS_PER_SECOND;
-    }
-
-    return true;
-}
-
-static bool bdrv_exceed_iops_limits(BlockDriverState *bs, bool is_write,
-                             double elapsed_time, uint64_t *wait)
-{
-    uint64_t iops_limit = 0;
-    double   ios_limit, ios_base;
-    double   slice_time, wait_time;
-
-    if (bs->io_limits.iops[BLOCK_IO_LIMIT_TOTAL]) {
-        iops_limit = bs->io_limits.iops[BLOCK_IO_LIMIT_TOTAL];
-    } else if (bs->io_limits.iops[is_write]) {
-        iops_limit = bs->io_limits.iops[is_write];
-    } else {
-        if (wait) {
-            *wait = 0;
-        }
-
-        return false;
-    }
-
-    slice_time = bs->slice_end - bs->slice_start;
-    slice_time /= (NANOSECONDS_PER_SECOND);
-    ios_limit  = iops_limit * slice_time;
-    ios_base   = bs->slice_submitted.ios[is_write];
-    if (bs->io_limits.iops[BLOCK_IO_LIMIT_TOTAL]) {
-        ios_base += bs->slice_submitted.ios[!is_write];
-    }
-
-    if (ios_base + 1 <= ios_limit) {
-        if (wait) {
-            *wait = 0;
-        }
-
-        return false;
-    }
-
-    /* Calc approx time to dispatch, in seconds */
-    wait_time = (ios_base + 1) / iops_limit;
-    if (wait_time > elapsed_time) {
-        wait_time = wait_time - elapsed_time;
-    } else {
-        wait_time = 0;
-    }
-
-    /* Exceeded current slice, extend it by another slice time */
-    bs->slice_end += BLOCK_IO_SLICE_TIME;
-    if (wait) {
-        *wait = wait_time * NANOSECONDS_PER_SECOND;
-    }
-
-    return true;
-}
-
-static bool bdrv_exceed_io_limits(BlockDriverState *bs, int nb_sectors,
-                           bool is_write, int64_t *wait)
-{
-    int64_t  now, max_wait;
-    uint64_t bps_wait = 0, iops_wait = 0;
-    double   elapsed_time;
-    int      bps_ret, iops_ret;
-
-    now = qemu_get_clock_ns(vm_clock);
-    if (now > bs->slice_end) {
-        bs->slice_start = now;
-        bs->slice_end   = now + BLOCK_IO_SLICE_TIME;
-        memset(&bs->slice_submitted, 0, sizeof(bs->slice_submitted));
-    }
-
-    elapsed_time  = now - bs->slice_start;
-    elapsed_time  /= (NANOSECONDS_PER_SECOND);
-
-    bps_ret  = bdrv_exceed_bps_limits(bs, nb_sectors,
-                                      is_write, elapsed_time, &bps_wait);
-    iops_ret = bdrv_exceed_iops_limits(bs, is_write,
-                                      elapsed_time, &iops_wait);
-    if (bps_ret || iops_ret) {
-        max_wait = bps_wait > iops_wait ? bps_wait : iops_wait;
-        if (wait) {
-            *wait = max_wait;
-        }
-
-        now = qemu_get_clock_ns(vm_clock);
-        if (bs->slice_end < now + max_wait) {
-            bs->slice_end = now + max_wait;
-        }
-
-        return true;
-    }
-
-    if (wait) {
-        *wait = 0;
-    }
-
-    bs->slice_submitted.bytes[is_write] += (int64_t)nb_sectors *
-                                           BDRV_SECTOR_SIZE;
-    bs->slice_submitted.ios[is_write]++;
-
-    return false;
-}
-
-/**************************************************************/
-/* async block device emulation */
-
-typedef struct BlockDriverAIOCBSync {
-    BlockDriverAIOCB common;
-    QEMUBH *bh;
-    int ret;
-    /* vector translation state */
-    QEMUIOVector *qiov;
-    uint8_t *bounce;
-    int is_write;
-} BlockDriverAIOCBSync;
-
-static void bdrv_aio_cancel_em(BlockDriverAIOCB *blockacb)
-{
-    BlockDriverAIOCBSync *acb =
-        container_of(blockacb, BlockDriverAIOCBSync, common);
-    qemu_bh_delete(acb->bh);
-    acb->bh = NULL;
-    qemu_aio_release(acb);
-}
-
-static const AIOCBInfo bdrv_em_aiocb_info = {
-    .aiocb_size         = sizeof(BlockDriverAIOCBSync),
-    .cancel             = bdrv_aio_cancel_em,
-};
-
-static void bdrv_aio_bh_cb(void *opaque)
-{
-    BlockDriverAIOCBSync *acb = opaque;
-
-    if (!acb->is_write)
-        qemu_iovec_from_buf(acb->qiov, 0, acb->bounce, acb->qiov->size);
-    qemu_vfree(acb->bounce);
-    acb->common.cb(acb->common.opaque, acb->ret);
-    qemu_bh_delete(acb->bh);
-    acb->bh = NULL;
-    qemu_aio_release(acb);
-}
-
-static BlockDriverAIOCB *bdrv_aio_rw_vector(BlockDriverState *bs,
-                                            int64_t sector_num,
-                                            QEMUIOVector *qiov,
-                                            int nb_sectors,
-                                            BlockDriverCompletionFunc *cb,
-                                            void *opaque,
-                                            int is_write)
-
-{
-    BlockDriverAIOCBSync *acb;
-
-    acb = qemu_aio_get(&bdrv_em_aiocb_info, bs, cb, opaque);
-    acb->is_write = is_write;
-    acb->qiov = qiov;
-    acb->bounce = qemu_blockalign(bs, qiov->size);
-    acb->bh = qemu_bh_new(bdrv_aio_bh_cb, acb);
-
-    if (is_write) {
-        qemu_iovec_to_buf(acb->qiov, 0, acb->bounce, qiov->size);
-        acb->ret = bs->drv->bdrv_write(bs, sector_num, acb->bounce, nb_sectors);
-    } else {
-        acb->ret = bs->drv->bdrv_read(bs, sector_num, acb->bounce, nb_sectors);
-    }
-
-    qemu_bh_schedule(acb->bh);
-
-    return &acb->common;
-}
-
-static BlockDriverAIOCB *bdrv_aio_readv_em(BlockDriverState *bs,
-        int64_t sector_num, QEMUIOVector *qiov, int nb_sectors,
-        BlockDriverCompletionFunc *cb, void *opaque)
-{
-    return bdrv_aio_rw_vector(bs, sector_num, qiov, nb_sectors, cb, opaque, 0);
-}
-
-static BlockDriverAIOCB *bdrv_aio_writev_em(BlockDriverState *bs,
-        int64_t sector_num, QEMUIOVector *qiov, int nb_sectors,
-        BlockDriverCompletionFunc *cb, void *opaque)
-{
-    return bdrv_aio_rw_vector(bs, sector_num, qiov, nb_sectors, cb, opaque, 1);
-}
-
-
-typedef struct BlockDriverAIOCBCoroutine {
-    BlockDriverAIOCB common;
-    BlockRequest req;
-    bool is_write;
-    bool *done;
-    QEMUBH* bh;
-} BlockDriverAIOCBCoroutine;
-
-static void bdrv_aio_co_cancel_em(BlockDriverAIOCB *blockacb)
-{
-    BlockDriverAIOCBCoroutine *acb =
-        container_of(blockacb, BlockDriverAIOCBCoroutine, common);
-    bool done = false;
-
-    acb->done = &done;
-    while (!done) {
-        qemu_aio_wait();
-    }
-}
-
-static const AIOCBInfo bdrv_em_co_aiocb_info = {
-    .aiocb_size         = sizeof(BlockDriverAIOCBCoroutine),
-    .cancel             = bdrv_aio_co_cancel_em,
-};
-
-static void bdrv_co_em_bh(void *opaque)
-{
-    BlockDriverAIOCBCoroutine *acb = opaque;
-
-    acb->common.cb(acb->common.opaque, acb->req.error);
-
-    if (acb->done) {
-        *acb->done = true;
-    }
-
-    qemu_bh_delete(acb->bh);
-    qemu_aio_release(acb);
-}
-
-/* Invoke bdrv_co_do_readv/bdrv_co_do_writev */
-static void coroutine_fn bdrv_co_do_rw(void *opaque)
-{
-    BlockDriverAIOCBCoroutine *acb = opaque;
-    BlockDriverState *bs = acb->common.bs;
-
-    if (!acb->is_write) {
-        acb->req.error = bdrv_co_do_readv(bs, acb->req.sector,
-            acb->req.nb_sectors, acb->req.qiov, 0);
-    } else {
-        acb->req.error = bdrv_co_do_writev(bs, acb->req.sector,
-            acb->req.nb_sectors, acb->req.qiov, 0);
-    }
-
-    acb->bh = qemu_bh_new(bdrv_co_em_bh, acb);
-    qemu_bh_schedule(acb->bh);
-}
-
-static BlockDriverAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
-                                               int64_t sector_num,
-                                               QEMUIOVector *qiov,
-                                               int nb_sectors,
-                                               BlockDriverCompletionFunc *cb,
-                                               void *opaque,
-                                               bool is_write)
-{
-    Coroutine *co;
-    BlockDriverAIOCBCoroutine *acb;
-
-    acb = qemu_aio_get(&bdrv_em_co_aiocb_info, bs, cb, opaque);
-    acb->req.sector = sector_num;
-    acb->req.nb_sectors = nb_sectors;
-    acb->req.qiov = qiov;
-    acb->is_write = is_write;
-    acb->done = NULL;
-
-    co = qemu_coroutine_create(bdrv_co_do_rw);
-    qemu_coroutine_enter(co, acb);
-
-    return &acb->common;
-}
-
-static void coroutine_fn bdrv_aio_flush_co_entry(void *opaque)
-{
-    BlockDriverAIOCBCoroutine *acb = opaque;
-    BlockDriverState *bs = acb->common.bs;
-
-    acb->req.error = bdrv_co_flush(bs);
-    acb->bh = qemu_bh_new(bdrv_co_em_bh, acb);
-    qemu_bh_schedule(acb->bh);
-}
-
-BlockDriverAIOCB *bdrv_aio_flush(BlockDriverState *bs,
-        BlockDriverCompletionFunc *cb, void *opaque)
-{
-    trace_bdrv_aio_flush(bs, opaque);
-
-    Coroutine *co;
-    BlockDriverAIOCBCoroutine *acb;
-
-    acb = qemu_aio_get(&bdrv_em_co_aiocb_info, bs, cb, opaque);
-    acb->done = NULL;
-
-    co = qemu_coroutine_create(bdrv_aio_flush_co_entry);
-    qemu_coroutine_enter(co, acb);
-
-    return &acb->common;
-}
-
-static void coroutine_fn bdrv_aio_discard_co_entry(void *opaque)
-{
-    BlockDriverAIOCBCoroutine *acb = opaque;
-    BlockDriverState *bs = acb->common.bs;
-
-    acb->req.error = bdrv_co_discard(bs, acb->req.sector, acb->req.nb_sectors);
-    acb->bh = qemu_bh_new(bdrv_co_em_bh, acb);
-    qemu_bh_schedule(acb->bh);
-}
-
-BlockDriverAIOCB *bdrv_aio_discard(BlockDriverState *bs,
-        int64_t sector_num, int nb_sectors,
-        BlockDriverCompletionFunc *cb, void *opaque)
-{
-    Coroutine *co;
-    BlockDriverAIOCBCoroutine *acb;
-
-    trace_bdrv_aio_discard(bs, sector_num, nb_sectors, opaque);
-
-    acb = qemu_aio_get(&bdrv_em_co_aiocb_info, bs, cb, opaque);
-    acb->req.sector = sector_num;
-    acb->req.nb_sectors = nb_sectors;
-    acb->done = NULL;
-    co = qemu_coroutine_create(bdrv_aio_discard_co_entry);
-    qemu_coroutine_enter(co, acb);
-
-    return &acb->common;
-}
-
-void bdrv_init(void)
-{
-    module_call_init(MODULE_INIT_BLOCK);
-}
-
-void bdrv_init_with_whitelist(void)
-{
-    use_bdrv_whitelist = 1;
-    bdrv_init();
-}
-
-void *qemu_aio_get(const AIOCBInfo *aiocb_info, BlockDriverState *bs,
-                   BlockDriverCompletionFunc *cb, void *opaque)
-{
-    BlockDriverAIOCB *acb;
-
-    acb = g_slice_alloc(aiocb_info->aiocb_size);
-    acb->aiocb_info = aiocb_info;
-    acb->bs = bs;
-    acb->cb = cb;
-    acb->opaque = opaque;
-    return acb;
-}
-
-void qemu_aio_release(void *p)
-{
-    BlockDriverAIOCB *acb = p;
-    g_slice_free1(acb->aiocb_info->aiocb_size, acb);
-}
-
-/**************************************************************/
-/* Coroutine block device emulation */
-
-typedef struct CoroutineIOCompletion {
-    Coroutine *coroutine;
-    int ret;
-} CoroutineIOCompletion;
-
-static void bdrv_co_io_em_complete(void *opaque, int ret)
-{
-    CoroutineIOCompletion *co = opaque;
-
-    co->ret = ret;
-    qemu_coroutine_enter(co->coroutine, NULL);
-}
-
-static int coroutine_fn bdrv_co_io_em(BlockDriverState *bs, int64_t sector_num,
-                                      int nb_sectors, QEMUIOVector *iov,
-                                      bool is_write)
-{
-    CoroutineIOCompletion co = {
-        .coroutine = qemu_coroutine_self(),
-    };
-    BlockDriverAIOCB *acb;
-
-    if (is_write) {
-        acb = bs->drv->bdrv_aio_writev(bs, sector_num, iov, nb_sectors,
-                                       bdrv_co_io_em_complete, &co);
-    } else {
-        acb = bs->drv->bdrv_aio_readv(bs, sector_num, iov, nb_sectors,
-                                      bdrv_co_io_em_complete, &co);
-    }
-
-    trace_bdrv_co_io_em(bs, sector_num, nb_sectors, is_write, acb);
-    if (!acb) {
-        return -EIO;
-    }
-    qemu_coroutine_yield();
-
-    return co.ret;
-}
-
-static int coroutine_fn bdrv_co_readv_em(BlockDriverState *bs,
-                                         int64_t sector_num, int nb_sectors,
-                                         QEMUIOVector *iov)
-{
-    return bdrv_co_io_em(bs, sector_num, nb_sectors, iov, false);
-}
-
-static int coroutine_fn bdrv_co_writev_em(BlockDriverState *bs,
-                                         int64_t sector_num, int nb_sectors,
-                                         QEMUIOVector *iov)
-{
-    return bdrv_co_io_em(bs, sector_num, nb_sectors, iov, true);
-}
-
-static void coroutine_fn bdrv_flush_co_entry(void *opaque)
-{
-    RwCo *rwco = opaque;
-
-    rwco->ret = bdrv_co_flush(rwco->bs);
-}
-
-int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
-{
-    int ret;
-
-    if (!bs || !bdrv_is_inserted(bs) || bdrv_is_read_only(bs)) {
-        return 0;
-    }
-
-    /* Write back cached data to the OS even with cache=unsafe */
-    BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_OS);
-    if (bs->drv->bdrv_co_flush_to_os) {
-        ret = bs->drv->bdrv_co_flush_to_os(bs);
-        if (ret < 0) {
-            return ret;
-        }
-    }
-
-    /* But don't actually force it to the disk with cache=unsafe */
-    if (bs->open_flags & BDRV_O_NO_FLUSH) {
-        goto flush_parent;
-    }
-
-    BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_DISK);
-    if (bs->drv->bdrv_co_flush_to_disk) {
-        ret = bs->drv->bdrv_co_flush_to_disk(bs);
-    } else if (bs->drv->bdrv_aio_flush) {
-        BlockDriverAIOCB *acb;
-        CoroutineIOCompletion co = {
-            .coroutine = qemu_coroutine_self(),
-        };
-
-        acb = bs->drv->bdrv_aio_flush(bs, bdrv_co_io_em_complete, &co);
-        if (acb == NULL) {
-            ret = -EIO;
-        } else {
-            qemu_coroutine_yield();
-            ret = co.ret;
-        }
-    } else {
-        /*
-         * Some block drivers always operate in either writethrough or unsafe
-         * mode and don't support bdrv_flush therefore. Usually qemu doesn't
-         * know how the server works (because the behaviour is hardcoded or
-         * depends on server-side configuration), so we can't ensure that
-         * everything is safe on disk. Returning an error doesn't work because
-         * that would break guests even if the server operates in writethrough
-         * mode.
-         *
-         * Let's hope the user knows what he's doing.
-         */
-        ret = 0;
-    }
-    if (ret < 0) {
-        return ret;
-    }
-
-    /* Now flush the underlying protocol.  It will also have BDRV_O_NO_FLUSH
-     * in the case of cache=unsafe, so there are no useless flushes.
-     */
-flush_parent:
-    return bdrv_co_flush(bs->file);
-}
-
-void bdrv_invalidate_cache(BlockDriverState *bs)
-{
-    if (bs->drv && bs->drv->bdrv_invalidate_cache) {
-        bs->drv->bdrv_invalidate_cache(bs);
-    }
-}
-
-void bdrv_invalidate_cache_all(void)
-{
-    BlockDriverState *bs;
-
-    QTAILQ_FOREACH(bs, &bdrv_states, list) {
-        bdrv_invalidate_cache(bs);
-    }
-}
-
-void bdrv_clear_incoming_migration_all(void)
-{
-    BlockDriverState *bs;
-
-    QTAILQ_FOREACH(bs, &bdrv_states, list) {
-        bs->open_flags = bs->open_flags & ~(BDRV_O_INCOMING);
-    }
-}
-
-int bdrv_flush(BlockDriverState *bs)
-{
-    Coroutine *co;
-    RwCo rwco = {
-        .bs = bs,
-        .ret = NOT_DONE,
-    };
-
-    if (qemu_in_coroutine()) {
-        /* Fast-path if already in coroutine context */
-        bdrv_flush_co_entry(&rwco);
-    } else {
-        co = qemu_coroutine_create(bdrv_flush_co_entry);
-        qemu_coroutine_enter(co, &rwco);
-        while (rwco.ret == NOT_DONE) {
-            qemu_aio_wait();
-        }
-    }
-
-    return rwco.ret;
-}
-
-static void coroutine_fn bdrv_discard_co_entry(void *opaque)
-{
-    RwCo *rwco = opaque;
-
-    rwco->ret = bdrv_co_discard(rwco->bs, rwco->sector_num, rwco->nb_sectors);
-}
-
-int coroutine_fn bdrv_co_discard(BlockDriverState *bs, int64_t sector_num,
-                                 int nb_sectors)
-{
-    if (!bs->drv) {
-        return -ENOMEDIUM;
-    } else if (bdrv_check_request(bs, sector_num, nb_sectors)) {
-        return -EIO;
-    } else if (bs->read_only) {
-        return -EROFS;
-    }
-
-    if (bs->dirty_bitmap) {
-        bdrv_reset_dirty(bs, sector_num, nb_sectors);
-    }
-
-    /* Do nothing if disabled.  */
-    if (!(bs->open_flags & BDRV_O_UNMAP)) {
-        return 0;
-    }
-
-    if (bs->drv->bdrv_co_discard) {
-        return bs->drv->bdrv_co_discard(bs, sector_num, nb_sectors);
-    } else if (bs->drv->bdrv_aio_discard) {
-        BlockDriverAIOCB *acb;
-        CoroutineIOCompletion co = {
-            .coroutine = qemu_coroutine_self(),
-        };
-
-        acb = bs->drv->bdrv_aio_discard(bs, sector_num, nb_sectors,
-                                        bdrv_co_io_em_complete, &co);
-        if (acb == NULL) {
-            return -EIO;
-        } else {
-            qemu_coroutine_yield();
-            return co.ret;
-        }
-    } else {
-        return 0;
-    }
-}
-
-int bdrv_discard(BlockDriverState *bs, int64_t sector_num, int nb_sectors)
-{
-    Coroutine *co;
-    RwCo rwco = {
-        .bs = bs,
-        .sector_num = sector_num,
-        .nb_sectors = nb_sectors,
-        .ret = NOT_DONE,
-    };
-
-    if (qemu_in_coroutine()) {
-        /* Fast-path if already in coroutine context */
-        bdrv_discard_co_entry(&rwco);
-    } else {
-        co = qemu_coroutine_create(bdrv_discard_co_entry);
-        qemu_coroutine_enter(co, &rwco);
-        while (rwco.ret == NOT_DONE) {
-            qemu_aio_wait();
-        }
-    }
-
-    return rwco.ret;
-}
-
-/**************************************************************/
-/* removable device support */
-
-/**
- * Return TRUE if the media is present
- */
-int bdrv_is_inserted(BlockDriverState *bs)
-{
-    BlockDriver *drv = bs->drv;
-
-    if (!drv)
-        return 0;
-    if (!drv->bdrv_is_inserted)
-        return 1;
-    return drv->bdrv_is_inserted(bs);
-}
-
-/**
- * Return whether the media changed since the last call to this
- * function, or -ENOTSUP if we don't know.  Most drivers don't know.
- */
-int bdrv_media_changed(BlockDriverState *bs)
-{
-    BlockDriver *drv = bs->drv;
-
-    if (drv && drv->bdrv_media_changed) {
-        return drv->bdrv_media_changed(bs);
-    }
-    return -ENOTSUP;
-}
-
-/**
- * If eject_flag is TRUE, eject the media. Otherwise, close the tray
- */
-void bdrv_eject(BlockDriverState *bs, bool eject_flag)
-{
-    BlockDriver *drv = bs->drv;
-
-    if (drv && drv->bdrv_eject) {
-        drv->bdrv_eject(bs, eject_flag);
-    }
-
-    if (bs->device_name[0] != '\0') {
-        bdrv_emit_qmp_eject_event(bs, eject_flag);
-    }
-}
-
-/**
- * Lock or unlock the media (if it is locked, the user won't be able
- * to eject it manually).
- */
-void bdrv_lock_medium(BlockDriverState *bs, bool locked)
-{
-    BlockDriver *drv = bs->drv;
-
-    trace_bdrv_lock_medium(bs, locked);
-
-    if (drv && drv->bdrv_lock_medium) {
-        drv->bdrv_lock_medium(bs, locked);
-    }
-}
-
-/* needed for generic scsi interface */
-
-int bdrv_ioctl(BlockDriverState *bs, unsigned long int req, void *buf)
-{
-    BlockDriver *drv = bs->drv;
-
-    if (drv && drv->bdrv_ioctl)
-        return drv->bdrv_ioctl(bs, req, buf);
-    return -ENOTSUP;
-}
-
-BlockDriverAIOCB *bdrv_aio_ioctl(BlockDriverState *bs,
-        unsigned long int req, void *buf,
-        BlockDriverCompletionFunc *cb, void *opaque)
-{
-    BlockDriver *drv = bs->drv;
-
-    if (drv && drv->bdrv_aio_ioctl)
-        return drv->bdrv_aio_ioctl(bs, req, buf, cb, opaque);
-    return NULL;
-}
-
-void bdrv_set_buffer_alignment(BlockDriverState *bs, int align)
-{
-    bs->buffer_alignment = align;
-}
-
-void *qemu_blockalign(BlockDriverState *bs, size_t size)
-{
-    return qemu_memalign((bs && bs->buffer_alignment) ? bs->buffer_alignment : 512, size);
-}
-
-/*
- * Check if all memory in this vector is sector aligned.
- */
-bool bdrv_qiov_is_aligned(BlockDriverState *bs, QEMUIOVector *qiov)
-{
-    int i;
-
-    for (i = 0; i < qiov->niov; i++) {
-        if ((uintptr_t) qiov->iov[i].iov_base % bs->buffer_alignment) {
-            return false;
-        }
-    }
-
-    return true;
-}
-
-void bdrv_set_dirty_tracking(BlockDriverState *bs, int granularity)
-{
-    int64_t bitmap_size;
-
-    assert((granularity & (granularity - 1)) == 0);
-
-    if (granularity) {
-        granularity >>= BDRV_SECTOR_BITS;
-        assert(!bs->dirty_bitmap);
-        bitmap_size = (bdrv_getlength(bs) >> BDRV_SECTOR_BITS);
-        bs->dirty_bitmap = hbitmap_alloc(bitmap_size, ffs(granularity) - 1);
-    } else {
-        if (bs->dirty_bitmap) {
-            hbitmap_free(bs->dirty_bitmap);
-            bs->dirty_bitmap = NULL;
-        }
-    }
-}
-
-int bdrv_get_dirty(BlockDriverState *bs, int64_t sector)
-{
-    if (bs->dirty_bitmap) {
-        return hbitmap_get(bs->dirty_bitmap, sector);
-    } else {
-        return 0;
-    }
-}
-
-void bdrv_dirty_iter_init(BlockDriverState *bs, HBitmapIter *hbi)
-{
-    hbitmap_iter_init(hbi, bs->dirty_bitmap, 0);
-}
-
-void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector,
-                    int nr_sectors)
-{
-    hbitmap_set(bs->dirty_bitmap, cur_sector, nr_sectors);
-}
-
-void bdrv_reset_dirty(BlockDriverState *bs, int64_t cur_sector,
-                      int nr_sectors)
-{
-    hbitmap_reset(bs->dirty_bitmap, cur_sector, nr_sectors);
-}
-
-int64_t bdrv_get_dirty_count(BlockDriverState *bs)
-{
-    if (bs->dirty_bitmap) {
-        return hbitmap_count(bs->dirty_bitmap);
-    } else {
-        return 0;
-    }
-}
-
-void bdrv_set_in_use(BlockDriverState *bs, int in_use)
-{
-    assert(bs->in_use != in_use);
-    bs->in_use = in_use;
-}
-
-int bdrv_in_use(BlockDriverState *bs)
-{
-    return bs->in_use;
-}
-
-void bdrv_iostatus_enable(BlockDriverState *bs)
-{
-    bs->iostatus_enabled = true;
-    bs->iostatus = BLOCK_DEVICE_IO_STATUS_OK;
-}
-
-/* The I/O status is only enabled if the drive explicitly
- * enables it _and_ the VM is configured to stop on errors */
-bool bdrv_iostatus_is_enabled(const BlockDriverState *bs)
-{
-    return (bs->iostatus_enabled &&
-           (bs->on_write_error == BLOCKDEV_ON_ERROR_ENOSPC ||
-            bs->on_write_error == BLOCKDEV_ON_ERROR_STOP   ||
-            bs->on_read_error == BLOCKDEV_ON_ERROR_STOP));
-}
-
-void bdrv_iostatus_disable(BlockDriverState *bs)
-{
-    bs->iostatus_enabled = false;
-}
-
-void bdrv_iostatus_reset(BlockDriverState *bs)
-{
-    if (bdrv_iostatus_is_enabled(bs)) {
-        bs->iostatus = BLOCK_DEVICE_IO_STATUS_OK;
-        if (bs->job) {
-            block_job_iostatus_reset(bs->job);
-        }
-    }
-}
-
-void bdrv_iostatus_set_err(BlockDriverState *bs, int error)
-{
-    assert(bdrv_iostatus_is_enabled(bs));
-    if (bs->iostatus == BLOCK_DEVICE_IO_STATUS_OK) {
-        bs->iostatus = error == ENOSPC ? BLOCK_DEVICE_IO_STATUS_NOSPACE :
-                                         BLOCK_DEVICE_IO_STATUS_FAILED;
-    }
-}
-
-void
-bdrv_acct_start(BlockDriverState *bs, BlockAcctCookie *cookie, int64_t bytes,
-        enum BlockAcctType type)
-{
-    assert(type < BDRV_MAX_IOTYPE);
-
-    cookie->bytes = bytes;
-    cookie->start_time_ns = get_clock();
-    cookie->type = type;
-}
-
-void
-bdrv_acct_done(BlockDriverState *bs, BlockAcctCookie *cookie)
-{
-    assert(cookie->type < BDRV_MAX_IOTYPE);
-
-    bs->nr_bytes[cookie->type] += cookie->bytes;
-    bs->nr_ops[cookie->type]++;
-    bs->total_time_ns[cookie->type] += get_clock() - cookie->start_time_ns;
-}
-
-void bdrv_img_create(const char *filename, const char *fmt,
-                     const char *base_filename, const char *base_fmt,
-                     char *options, uint64_t img_size, int flags,
-                     Error **errp, bool quiet)
-{
-    QEMUOptionParameter *param = NULL, *create_options = NULL;
-    QEMUOptionParameter *backing_fmt, *backing_file, *size;
-    BlockDriverState *bs = NULL;
-    BlockDriver *drv, *proto_drv;
-    BlockDriver *backing_drv = NULL;
-    int ret = 0;
-
-    /* Find driver and parse its options */
-    drv = bdrv_find_format(fmt);
-    if (!drv) {
-        error_setg(errp, "Unknown file format '%s'", fmt);
-        return;
-    }
-
-    proto_drv = bdrv_find_protocol(filename, true);
-    if (!proto_drv) {
-        error_setg(errp, "Unknown protocol '%s'", filename);
-        return;
-    }
-
-    create_options = append_option_parameters(create_options,
-                                              drv->create_options);
-    create_options = append_option_parameters(create_options,
-                                              proto_drv->create_options);
-
-    /* Create parameter list with default values */
-    param = parse_option_parameters("", create_options, param);
-
-    set_option_parameter_int(param, BLOCK_OPT_SIZE, img_size);
-
-    /* Parse -o options */
-    if (options) {
-        param = parse_option_parameters(options, create_options, param);
-        if (param == NULL) {
-            error_setg(errp, "Invalid options for file format '%s'.", fmt);
-            goto out;
-        }
-    }
-
-    if (base_filename) {
-        if (set_option_parameter(param, BLOCK_OPT_BACKING_FILE,
-                                 base_filename)) {
-            error_setg(errp, "Backing file not supported for file format '%s'",
-                       fmt);
-            goto out;
-        }
-    }
-
-    if (base_fmt) {
-        if (set_option_parameter(param, BLOCK_OPT_BACKING_FMT, base_fmt)) {
-            error_setg(errp, "Backing file format not supported for file "
-                             "format '%s'", fmt);
-            goto out;
-        }
-    }
-
-    backing_file = get_option_parameter(param, BLOCK_OPT_BACKING_FILE);
-    if (backing_file && backing_file->value.s) {
-        if (!strcmp(filename, backing_file->value.s)) {
-            error_setg(errp, "Error: Trying to create an image with the "
-                             "same filename as the backing file");
-            goto out;
-        }
-    }
-
-    backing_fmt = get_option_parameter(param, BLOCK_OPT_BACKING_FMT);
-    if (backing_fmt && backing_fmt->value.s) {
-        backing_drv = bdrv_find_format(backing_fmt->value.s);
-        if (!backing_drv) {
-            error_setg(errp, "Unknown backing file format '%s'",
-                       backing_fmt->value.s);
-            goto out;
-        }
-    }
-
-    // The size for the image must always be specified, with one exception:
-    // If we are using a backing file, we can obtain the size from there
-    size = get_option_parameter(param, BLOCK_OPT_SIZE);
-    if (size && size->value.n == -1) {
-        if (backing_file && backing_file->value.s) {
-            uint64_t size;
-            char buf[32];
-            int back_flags;
-
-            /* backing files always opened read-only */
-            back_flags =
-                flags & ~(BDRV_O_RDWR | BDRV_O_SNAPSHOT | BDRV_O_NO_BACKING);
-
-            bs = bdrv_new("");
-
-            ret = bdrv_open(bs, backing_file->value.s, NULL, back_flags,
-                            backing_drv);
-            if (ret < 0) {
-                error_setg_errno(errp, -ret, "Could not open '%s'",
-                                 backing_file->value.s);
-                goto out;
-            }
-            bdrv_get_geometry(bs, &size);
-            size *= 512;
-
-            snprintf(buf, sizeof(buf), "%" PRId64, size);
-            set_option_parameter(param, BLOCK_OPT_SIZE, buf);
-        } else {
-            error_setg(errp, "Image creation needs a size parameter");
-            goto out;
-        }
-    }
-
-    if (!quiet) {
-        printf("Formatting '%s', fmt=%s ", filename, fmt);
-        print_option_parameters(param);
-        puts("");
-    }
-    ret = bdrv_create(drv, filename, param);
-    if (ret < 0) {
-        if (ret == -ENOTSUP) {
-            error_setg(errp,"Formatting or formatting option not supported for "
-                            "file format '%s'", fmt);
-        } else if (ret == -EFBIG) {
-            const char *cluster_size_hint = "";
-            if (get_option_parameter(create_options, BLOCK_OPT_CLUSTER_SIZE)) {
-                cluster_size_hint = " (try using a larger cluster size)";
-            }
-            error_setg(errp, "The image size is too large for file format '%s'%s",
-                       fmt, cluster_size_hint);
-        } else {
-            error_setg(errp, "%s: error while creating %s: %s", filename, fmt,
-                       strerror(-ret));
-        }
-    }
-
-out:
-    free_option_parameters(create_options);
-    free_option_parameters(param);
-
-    if (bs) {
-        bdrv_delete(bs);
-    }
-}
-
-AioContext *bdrv_get_aio_context(BlockDriverState *bs)
-{
-    /* Currently BlockDriverState always uses the main loop AioContext */
-    return qemu_get_aio_context();
-}
-
-void bdrv_add_before_write_notifier(BlockDriverState *bs,
-                                    NotifierWithReturn *notifier)
-{
-    notifier_with_return_list_add(&bs->before_write_notifiers, notifier);
-}
diff --git a/contrib/qemu/block/qcow.c b/contrib/qemu/block/qcow.c
deleted file mode 100644
index 5239bd68f1c..00000000000
--- a/contrib/qemu/block/qcow.c
+++ /dev/null
@@ -1,914 +0,0 @@
-/*
- * Block driver for the QCOW format
- *
- * Copyright (c) 2004-2006 Fabrice Bellard
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-#include "qemu-common.h"
-#include "block/block_int.h"
-#include "qemu/module.h"
-#include <zlib.h>
-#include "qemu/aes.h"
-#include "migration/migration.h"
-
-/**************************************************************/
-/* QEMU COW block driver with compression and encryption support */
-
-#define QCOW_MAGIC (('Q' << 24) | ('F' << 16) | ('I' << 8) | 0xfb)
-#define QCOW_VERSION 1
-
-#define QCOW_CRYPT_NONE 0
-#define QCOW_CRYPT_AES  1
-
-#define QCOW_OFLAG_COMPRESSED (1LL << 63)
-
-typedef struct QCowHeader {
-    uint32_t magic;
-    uint32_t version;
-    uint64_t backing_file_offset;
-    uint32_t backing_file_size;
-    uint32_t mtime;
-    uint64_t size; /* in bytes */
-    uint8_t cluster_bits;
-    uint8_t l2_bits;
-    uint32_t crypt_method;
-    uint64_t l1_table_offset;
-} QCowHeader;
-
-#define L2_CACHE_SIZE 16
-
-typedef struct BDRVQcowState {
-    int cluster_bits;
-    int cluster_size;
-    int cluster_sectors;
-    int l2_bits;
-    int l2_size;
-    int l1_size;
-    uint64_t cluster_offset_mask;
-    uint64_t l1_table_offset;
-    uint64_t *l1_table;
-    uint64_t *l2_cache;
-    uint64_t l2_cache_offsets[L2_CACHE_SIZE];
-    uint32_t l2_cache_counts[L2_CACHE_SIZE];
-    uint8_t *cluster_cache;
-    uint8_t *cluster_data;
-    uint64_t cluster_cache_offset;
-    uint32_t crypt_method; /* current crypt method, 0 if no key yet */
-    uint32_t crypt_method_header;
-    AES_KEY aes_encrypt_key;
-    AES_KEY aes_decrypt_key;
-    CoMutex lock;
-    Error *migration_blocker;
-} BDRVQcowState;
-
-static int decompress_cluster(BlockDriverState *bs, uint64_t cluster_offset);
-
-static int qcow_probe(const uint8_t *buf, int buf_size, const char *filename)
-{
-    const QCowHeader *cow_header = (const void *)buf;
-
-    if (buf_size >= sizeof(QCowHeader) &&
-        be32_to_cpu(cow_header->magic) == QCOW_MAGIC &&
-        be32_to_cpu(cow_header->version) == QCOW_VERSION)
-        return 100;
-    else
-        return 0;
-}
-
-static int qcow_open(BlockDriverState *bs, QDict *options, int flags)
-{
-    BDRVQcowState *s = bs->opaque;
-    int len, i, shift, ret;
-    QCowHeader header;
-
-    ret = bdrv_pread(bs->file, 0, &header, sizeof(header));
-    if (ret < 0) {
-        goto fail;
-    }
-    be32_to_cpus(&header.magic);
-    be32_to_cpus(&header.version);
-    be64_to_cpus(&header.backing_file_offset);
-    be32_to_cpus(&header.backing_file_size);
-    be32_to_cpus(&header.mtime);
-    be64_to_cpus(&header.size);
-    be32_to_cpus(&header.crypt_method);
-    be64_to_cpus(&header.l1_table_offset);
-
-    if (header.magic != QCOW_MAGIC) {
-        ret = -EMEDIUMTYPE;
-        goto fail;
-    }
-    if (header.version != QCOW_VERSION) {
-        char version[64];
-        snprintf(version, sizeof(version), "QCOW version %d", header.version);
-        qerror_report(QERR_UNKNOWN_BLOCK_FORMAT_FEATURE,
-            bs->device_name, "qcow", version);
-        ret = -ENOTSUP;
-        goto fail;
-    }
-
-    if (header.size <= 1 || header.cluster_bits < 9) {
-        ret = -EINVAL;
-        goto fail;
-    }
-    if (header.crypt_method > QCOW_CRYPT_AES) {
-        ret = -EINVAL;
-        goto fail;
-    }
-    s->crypt_method_header = header.crypt_method;
-    if (s->crypt_method_header) {
-        bs->encrypted = 1;
-    }
-    s->cluster_bits = header.cluster_bits;
-    s->cluster_size = 1 << s->cluster_bits;
-    s->cluster_sectors = 1 << (s->cluster_bits - 9);
-    s->l2_bits = header.l2_bits;
-    s->l2_size = 1 << s->l2_bits;
-    bs->total_sectors = header.size / 512;
-    s->cluster_offset_mask = (1LL << (63 - s->cluster_bits)) - 1;
-
-    /* read the level 1 table */
-    shift = s->cluster_bits + s->l2_bits;
-    s->l1_size = (header.size + (1LL << shift) - 1) >> shift;
-
-    s->l1_table_offset = header.l1_table_offset;
-    s->l1_table = g_malloc(s->l1_size * sizeof(uint64_t));
-
-    ret = bdrv_pread(bs->file, s->l1_table_offset, s->l1_table,
-               s->l1_size * sizeof(uint64_t));
-    if (ret < 0) {
-        goto fail;
-    }
-
-    for(i = 0;i < s->l1_size; i++) {
-        be64_to_cpus(&s->l1_table[i]);
-    }
-    /* alloc L2 cache */
-    s->l2_cache = g_malloc(s->l2_size * L2_CACHE_SIZE * sizeof(uint64_t));
-    s->cluster_cache = g_malloc(s->cluster_size);
-    s->cluster_data = g_malloc(s->cluster_size);
-    s->cluster_cache_offset = -1;
-
-    /* read the backing file name */
-    if (header.backing_file_offset != 0) {
-        len = header.backing_file_size;
-        if (len > 1023) {
-            len = 1023;
-        }
-        ret = bdrv_pread(bs->file, header.backing_file_offset,
-                   bs->backing_file, len);
-        if (ret < 0) {
-            goto fail;
-        }
-        bs->backing_file[len] = '\0';
-    }
-
-    /* Disable migration when qcow images are used */
-    error_set(&s->migration_blocker,
-              QERR_BLOCK_FORMAT_FEATURE_NOT_SUPPORTED,
-              "qcow", bs->device_name, "live migration");
-    migrate_add_blocker(s->migration_blocker);
-
-    qemu_co_mutex_init(&s->lock);
-    return 0;
-
- fail:
-    g_free(s->l1_table);
-    g_free(s->l2_cache);
-    g_free(s->cluster_cache);
-    g_free(s->cluster_data);
-    return ret;
-}
-
-
-/* We have nothing to do for QCOW reopen, stubs just return
- * success */
-static int qcow_reopen_prepare(BDRVReopenState *state,
-                               BlockReopenQueue *queue, Error **errp)
-{
-    return 0;
-}
-
-static int qcow_set_key(BlockDriverState *bs, const char *key)
-{
-    BDRVQcowState *s = bs->opaque;
-    uint8_t keybuf[16];
-    int len, i;
-
-    memset(keybuf, 0, 16);
-    len = strlen(key);
-    if (len > 16)
-        len = 16;
-    /* XXX: we could compress the chars to 7 bits to increase
-       entropy */
-    for(i = 0;i < len;i++) {
-        keybuf[i] = key[i];
-    }
-    s->crypt_method = s->crypt_method_header;
-
-    if (AES_set_encrypt_key(keybuf, 128, &s->aes_encrypt_key) != 0)
-        return -1;
-    if (AES_set_decrypt_key(keybuf, 128, &s->aes_decrypt_key) != 0)
-        return -1;
-    return 0;
-}
-
-/* The crypt function is compatible with the linux cryptoloop
-   algorithm for < 4 GB images. NOTE: out_buf == in_buf is
-   supported */
-static void encrypt_sectors(BDRVQcowState *s, int64_t sector_num,
-                            uint8_t *out_buf, const uint8_t *in_buf,
-                            int nb_sectors, int enc,
-                            const AES_KEY *key)
-{
-    union {
-        uint64_t ll[2];
-        uint8_t b[16];
-    } ivec;
-    int i;
-
-    for(i = 0; i < nb_sectors; i++) {
-        ivec.ll[0] = cpu_to_le64(sector_num);
-        ivec.ll[1] = 0;
-        AES_cbc_encrypt(in_buf, out_buf, 512, key,
-                        ivec.b, enc);
-        sector_num++;
-        in_buf += 512;
-        out_buf += 512;
-    }
-}
-
-/* 'allocate' is:
- *
- * 0 to not allocate.
- *
- * 1 to allocate a normal cluster (for sector indexes 'n_start' to
- * 'n_end')
- *
- * 2 to allocate a compressed cluster of size
- * 'compressed_size'. 'compressed_size' must be > 0 and <
- * cluster_size
- *
- * return 0 if not allocated.
- */
-static uint64_t get_cluster_offset(BlockDriverState *bs,
-                                   uint64_t offset, int allocate,
-                                   int compressed_size,
-                                   int n_start, int n_end)
-{
-    BDRVQcowState *s = bs->opaque;
-    int min_index, i, j, l1_index, l2_index;
-    uint64_t l2_offset, *l2_table, cluster_offset, tmp;
-    uint32_t min_count;
-    int new_l2_table;
-
-    l1_index = offset >> (s->l2_bits + s->cluster_bits);
-    l2_offset = s->l1_table[l1_index];
-    new_l2_table = 0;
-    if (!l2_offset) {
-        if (!allocate)
-            return 0;
-        /* allocate a new l2 entry */
-        l2_offset = bdrv_getlength(bs->file);
-        /* round to cluster size */
-        l2_offset = (l2_offset + s->cluster_size - 1) & ~(s->cluster_size - 1);
-        /* update the L1 entry */
-        s->l1_table[l1_index] = l2_offset;
-        tmp = cpu_to_be64(l2_offset);
-        if (bdrv_pwrite_sync(bs->file,
-                s->l1_table_offset + l1_index * sizeof(tmp),
-                &tmp, sizeof(tmp)) < 0)
-            return 0;
-        new_l2_table = 1;
-    }
-    for(i = 0; i < L2_CACHE_SIZE; i++) {
-        if (l2_offset == s->l2_cache_offsets[i]) {
-            /* increment the hit count */
-            if (++s->l2_cache_counts[i] == 0xffffffff) {
-                for(j = 0; j < L2_CACHE_SIZE; j++) {
-                    s->l2_cache_counts[j] >>= 1;
-                }
-            }
-            l2_table = s->l2_cache + (i << s->l2_bits);
-            goto found;
-        }
-    }
-    /* not found: load a new entry in the least used one */
-    min_index = 0;
-    min_count = 0xffffffff;
-    for(i = 0; i < L2_CACHE_SIZE; i++) {
-        if (s->l2_cache_counts[i] < min_count) {
-            min_count = s->l2_cache_counts[i];
-            min_index = i;
-        }
-    }
-    l2_table = s->l2_cache + (min_index << s->l2_bits);
-    if (new_l2_table) {
-        memset(l2_table, 0, s->l2_size * sizeof(uint64_t));
-        if (bdrv_pwrite_sync(bs->file, l2_offset, l2_table,
-                s->l2_size * sizeof(uint64_t)) < 0)
-            return 0;
-    } else {
-        if (bdrv_pread(bs->file, l2_offset, l2_table, s->l2_size * sizeof(uint64_t)) !=
-            s->l2_size * sizeof(uint64_t))
-            return 0;
-    }
-    s->l2_cache_offsets[min_index] = l2_offset;
-    s->l2_cache_counts[min_index] = 1;
- found:
-    l2_index = (offset >> s->cluster_bits) & (s->l2_size - 1);
-    cluster_offset = be64_to_cpu(l2_table[l2_index]);
-    if (!cluster_offset ||
-        ((cluster_offset & QCOW_OFLAG_COMPRESSED) && allocate == 1)) {
-        if (!allocate)
-            return 0;
-        /* allocate a new cluster */
-        if ((cluster_offset & QCOW_OFLAG_COMPRESSED) &&
-            (n_end - n_start) < s->cluster_sectors) {
-            /* if the cluster is already compressed, we must
-               decompress it in the case it is not completely
-               overwritten */
-            if (decompress_cluster(bs, cluster_offset) < 0)
-                return 0;
-            cluster_offset = bdrv_getlength(bs->file);
-            cluster_offset = (cluster_offset + s->cluster_size - 1) &
-                ~(s->cluster_size - 1);
-            /* write the cluster content */
-            if (bdrv_pwrite(bs->file, cluster_offset, s->cluster_cache, s->cluster_size) !=
-                s->cluster_size)
-                return -1;
-        } else {
-            cluster_offset = bdrv_getlength(bs->file);
-            if (allocate == 1) {
-                /* round to cluster size */
-                cluster_offset = (cluster_offset + s->cluster_size - 1) &
-                    ~(s->cluster_size - 1);
-                bdrv_truncate(bs->file, cluster_offset + s->cluster_size);
-                /* if encrypted, we must initialize the cluster
-                   content which won't be written */
-                if (s->crypt_method &&
-                    (n_end - n_start) < s->cluster_sectors) {
-                    uint64_t start_sect;
-                    start_sect = (offset & ~(s->cluster_size - 1)) >> 9;
-                    memset(s->cluster_data + 512, 0x00, 512);
-                    for(i = 0; i < s->cluster_sectors; i++) {
-                        if (i < n_start || i >= n_end) {
-                            encrypt_sectors(s, start_sect + i,
-                                            s->cluster_data,
-                                            s->cluster_data + 512, 1, 1,
-                                            &s->aes_encrypt_key);
-                            if (bdrv_pwrite(bs->file, cluster_offset + i * 512,
-                                            s->cluster_data, 512) != 512)
-                                return -1;
-                        }
-                    }
-                }
-            } else if (allocate == 2) {
-                cluster_offset |= QCOW_OFLAG_COMPRESSED |
-                    (uint64_t)compressed_size << (63 - s->cluster_bits);
-            }
-        }
-        /* update L2 table */
-        tmp = cpu_to_be64(cluster_offset);
-        l2_table[l2_index] = tmp;
-        if (bdrv_pwrite_sync(bs->file, l2_offset + l2_index * sizeof(tmp),
-                &tmp, sizeof(tmp)) < 0)
-            return 0;
-    }
-    return cluster_offset;
-}
-
-static int coroutine_fn qcow_co_is_allocated(BlockDriverState *bs,
-        int64_t sector_num, int nb_sectors, int *pnum)
-{
-    BDRVQcowState *s = bs->opaque;
-    int index_in_cluster, n;
-    uint64_t cluster_offset;
-
-    qemu_co_mutex_lock(&s->lock);
-    cluster_offset = get_cluster_offset(bs, sector_num << 9, 0, 0, 0, 0);
-    qemu_co_mutex_unlock(&s->lock);
-    index_in_cluster = sector_num & (s->cluster_sectors - 1);
-    n = s->cluster_sectors - index_in_cluster;
-    if (n > nb_sectors)
-        n = nb_sectors;
-    *pnum = n;
-    return (cluster_offset != 0);
-}
-
-static int decompress_buffer(uint8_t *out_buf, int out_buf_size,
-                             const uint8_t *buf, int buf_size)
-{
-    z_stream strm1, *strm = &strm1;
-    int ret, out_len;
-
-    memset(strm, 0, sizeof(*strm));
-
-    strm->next_in = (uint8_t *)buf;
-    strm->avail_in = buf_size;
-    strm->next_out = out_buf;
-    strm->avail_out = out_buf_size;
-
-    ret = inflateInit2(strm, -12);
-    if (ret != Z_OK)
-        return -1;
-    ret = inflate(strm, Z_FINISH);
-    out_len = strm->next_out - out_buf;
-    if ((ret != Z_STREAM_END && ret != Z_BUF_ERROR) ||
-        out_len != out_buf_size) {
-        inflateEnd(strm);
-        return -1;
-    }
-    inflateEnd(strm);
-    return 0;
-}
-
-static int decompress_cluster(BlockDriverState *bs, uint64_t cluster_offset)
-{
-    BDRVQcowState *s = bs->opaque;
-    int ret, csize;
-    uint64_t coffset;
-
-    coffset = cluster_offset & s->cluster_offset_mask;
-    if (s->cluster_cache_offset != coffset) {
-        csize = cluster_offset >> (63 - s->cluster_bits);
-        csize &= (s->cluster_size - 1);
-        ret = bdrv_pread(bs->file, coffset, s->cluster_data, csize);
-        if (ret != csize)
-            return -1;
-        if (decompress_buffer(s->cluster_cache, s->cluster_size,
-                              s->cluster_data, csize) < 0) {
-            return -1;
-        }
-        s->cluster_cache_offset = coffset;
-    }
-    return 0;
-}
-
-static coroutine_fn int qcow_co_readv(BlockDriverState *bs, int64_t sector_num,
-                         int nb_sectors, QEMUIOVector *qiov)
-{
-    BDRVQcowState *s = bs->opaque;
-    int index_in_cluster;
-    int ret = 0, n;
-    uint64_t cluster_offset;
-    struct iovec hd_iov;
-    QEMUIOVector hd_qiov;
-    uint8_t *buf;
-    void *orig_buf;
-
-    if (qiov->niov > 1) {
-        buf = orig_buf = qemu_blockalign(bs, qiov->size);
-    } else {
-        orig_buf = NULL;
-        buf = (uint8_t *)qiov->iov->iov_base;
-    }
-
-    qemu_co_mutex_lock(&s->lock);
-
-    while (nb_sectors != 0) {
-        /* prepare next request */
-        cluster_offset = get_cluster_offset(bs, sector_num << 9,
-                                                 0, 0, 0, 0);
-        index_in_cluster = sector_num & (s->cluster_sectors - 1);
-        n = s->cluster_sectors - index_in_cluster;
-        if (n > nb_sectors) {
-            n = nb_sectors;
-        }
-
-        if (!cluster_offset) {
-            if (bs->backing_hd) {
-                /* read from the base image */
-                hd_iov.iov_base = (void *)buf;
-                hd_iov.iov_len = n * 512;
-                qemu_iovec_init_external(&hd_qiov, &hd_iov, 1);
-                qemu_co_mutex_unlock(&s->lock);
-                ret = bdrv_co_readv(bs->backing_hd, sector_num,
-                                    n, &hd_qiov);
-                qemu_co_mutex_lock(&s->lock);
-                if (ret < 0) {
-                    goto fail;
-                }
-            } else {
-                /* Note: in this case, no need to wait */
-                memset(buf, 0, 512 * n);
-            }
-        } else if (cluster_offset & QCOW_OFLAG_COMPRESSED) {
-            /* add AIO support for compressed blocks ? */
-            if (decompress_cluster(bs, cluster_offset) < 0) {
-                goto fail;
-            }
-            memcpy(buf,
-                   s->cluster_cache + index_in_cluster * 512, 512 * n);
-        } else {
-            if ((cluster_offset & 511) != 0) {
-                goto fail;
-            }
-            hd_iov.iov_base = (void *)buf;
-            hd_iov.iov_len = n * 512;
-            qemu_iovec_init_external(&hd_qiov, &hd_iov, 1);
-            qemu_co_mutex_unlock(&s->lock);
-            ret = bdrv_co_readv(bs->file,
-                                (cluster_offset >> 9) + index_in_cluster,
-                                n, &hd_qiov);
-            qemu_co_mutex_lock(&s->lock);
-            if (ret < 0) {
-                break;
-            }
-            if (s->crypt_method) {
-                encrypt_sectors(s, sector_num, buf, buf,
-                                n, 0,
-                                &s->aes_decrypt_key);
-            }
-        }
-        ret = 0;
-
-        nb_sectors -= n;
-        sector_num += n;
-        buf += n * 512;
-    }
-
-done:
-    qemu_co_mutex_unlock(&s->lock);
-
-    if (qiov->niov > 1) {
-        qemu_iovec_from_buf(qiov, 0, orig_buf, qiov->size);
-        qemu_vfree(orig_buf);
-    }
-
-    return ret;
-
-fail:
-    ret = -EIO;
-    goto done;
-}
-
-static coroutine_fn int qcow_co_writev(BlockDriverState *bs, int64_t sector_num,
-                          int nb_sectors, QEMUIOVector *qiov)
-{
-    BDRVQcowState *s = bs->opaque;
-    int index_in_cluster;
-    uint64_t cluster_offset;
-    const uint8_t *src_buf;
-    int ret = 0, n;
-    uint8_t *cluster_data = NULL;
-    struct iovec hd_iov;
-    QEMUIOVector hd_qiov;
-    uint8_t *buf;
-    void *orig_buf;
-
-    s->cluster_cache_offset = -1; /* disable compressed cache */
-
-    if (qiov->niov > 1) {
-        buf = orig_buf = qemu_blockalign(bs, qiov->size);
-        qemu_iovec_to_buf(qiov, 0, buf, qiov->size);
-    } else {
-        orig_buf = NULL;
-        buf = (uint8_t *)qiov->iov->iov_base;
-    }
-
-    qemu_co_mutex_lock(&s->lock);
-
-    while (nb_sectors != 0) {
-
-        index_in_cluster = sector_num & (s->cluster_sectors - 1);
-        n = s->cluster_sectors - index_in_cluster;
-        if (n > nb_sectors) {
-            n = nb_sectors;
-        }
-        cluster_offset = get_cluster_offset(bs, sector_num << 9, 1, 0,
-                                            index_in_cluster,
-                                            index_in_cluster + n);
-        if (!cluster_offset || (cluster_offset & 511) != 0) {
-            ret = -EIO;
-            break;
-        }
-        if (s->crypt_method) {
-            if (!cluster_data) {
-                cluster_data = g_malloc0(s->cluster_size);
-            }
-            encrypt_sectors(s, sector_num, cluster_data, buf,
-                            n, 1, &s->aes_encrypt_key);
-            src_buf = cluster_data;
-        } else {
-            src_buf = buf;
-        }
-
-        hd_iov.iov_base = (void *)src_buf;
-        hd_iov.iov_len = n * 512;
-        qemu_iovec_init_external(&hd_qiov, &hd_iov, 1);
-        qemu_co_mutex_unlock(&s->lock);
-        ret = bdrv_co_writev(bs->file,
-                             (cluster_offset >> 9) + index_in_cluster,
-                             n, &hd_qiov);
-        qemu_co_mutex_lock(&s->lock);
-        if (ret < 0) {
-            break;
-        }
-        ret = 0;
-
-        nb_sectors -= n;
-        sector_num += n;
-        buf += n * 512;
-    }
-    qemu_co_mutex_unlock(&s->lock);
-
-    if (qiov->niov > 1) {
-        qemu_vfree(orig_buf);
-    }
-    g_free(cluster_data);
-
-    return ret;
-}
-
-static void qcow_close(BlockDriverState *bs)
-{
-    BDRVQcowState *s = bs->opaque;
-
-    g_free(s->l1_table);
-    g_free(s->l2_cache);
-    g_free(s->cluster_cache);
-    g_free(s->cluster_data);
-
-    migrate_del_blocker(s->migration_blocker);
-    error_free(s->migration_blocker);
-}
-
-static int qcow_create(const char *filename, QEMUOptionParameter *options)
-{
-    int header_size, backing_filename_len, l1_size, shift, i;
-    QCowHeader header;
-    uint8_t *tmp;
-    int64_t total_size = 0;
-    const char *backing_file = NULL;
-    int flags = 0;
-    int ret;
-    BlockDriverState *qcow_bs;
-
-    /* Read out options */
-    while (options && options->name) {
-        if (!strcmp(options->name, BLOCK_OPT_SIZE)) {
-            total_size = options->value.n / 512;
-        } else if (!strcmp(options->name, BLOCK_OPT_BACKING_FILE)) {
-            backing_file = options->value.s;
-        } else if (!strcmp(options->name, BLOCK_OPT_ENCRYPT)) {
-            flags |= options->value.n ? BLOCK_FLAG_ENCRYPT : 0;
-        }
-        options++;
-    }
-
-    ret = bdrv_create_file(filename, options);
-    if (ret < 0) {
-        return ret;
-    }
-
-    ret = bdrv_file_open(&qcow_bs, filename, NULL, BDRV_O_RDWR);
-    if (ret < 0) {
-        return ret;
-    }
-
-    ret = bdrv_truncate(qcow_bs, 0);
-    if (ret < 0) {
-        goto exit;
-    }
-
-    memset(&header, 0, sizeof(header));
-    header.magic = cpu_to_be32(QCOW_MAGIC);
-    header.version = cpu_to_be32(QCOW_VERSION);
-    header.size = cpu_to_be64(total_size * 512);
-    header_size = sizeof(header);
-    backing_filename_len = 0;
-    if (backing_file) {
-        if (strcmp(backing_file, "fat:")) {
-            header.backing_file_offset = cpu_to_be64(header_size);
-            backing_filename_len = strlen(backing_file);
-            header.backing_file_size = cpu_to_be32(backing_filename_len);
-            header_size += backing_filename_len;
-        } else {
-            /* special backing file for vvfat */
-            backing_file = NULL;
-        }
-        header.cluster_bits = 9; /* 512 byte cluster to avoid copying
-                                    unmodifyed sectors */
-        header.l2_bits = 12; /* 32 KB L2 tables */
-    } else {
-        header.cluster_bits = 12; /* 4 KB clusters */
-        header.l2_bits = 9; /* 4 KB L2 tables */
-    }
-    header_size = (header_size + 7) & ~7;
-    shift = header.cluster_bits + header.l2_bits;
-    l1_size = ((total_size * 512) + (1LL << shift) - 1) >> shift;
-
-    header.l1_table_offset = cpu_to_be64(header_size);
-    if (flags & BLOCK_FLAG_ENCRYPT) {
-        header.crypt_method = cpu_to_be32(QCOW_CRYPT_AES);
-    } else {
-        header.crypt_method = cpu_to_be32(QCOW_CRYPT_NONE);
-    }
-
-    /* write all the data */
-    ret = bdrv_pwrite(qcow_bs, 0, &header, sizeof(header));
-    if (ret != sizeof(header)) {
-        goto exit;
-    }
-
-    if (backing_file) {
-        ret = bdrv_pwrite(qcow_bs, sizeof(header),
-            backing_file, backing_filename_len);
-        if (ret != backing_filename_len) {
-            goto exit;
-        }
-    }
-
-    tmp = g_malloc0(BDRV_SECTOR_SIZE);
-    for (i = 0; i < ((sizeof(uint64_t)*l1_size + BDRV_SECTOR_SIZE - 1)/
-        BDRV_SECTOR_SIZE); i++) {
-        ret = bdrv_pwrite(qcow_bs, header_size +
-            BDRV_SECTOR_SIZE*i, tmp, BDRV_SECTOR_SIZE);
-        if (ret != BDRV_SECTOR_SIZE) {
-            g_free(tmp);
-            goto exit;
-        }
-    }
-
-    g_free(tmp);
-    ret = 0;
-exit:
-    bdrv_delete(qcow_bs);
-    return ret;
-}
-
-static int qcow_make_empty(BlockDriverState *bs)
-{
-    BDRVQcowState *s = bs->opaque;
-    uint32_t l1_length = s->l1_size * sizeof(uint64_t);
-    int ret;
-
-    memset(s->l1_table, 0, l1_length);
-    if (bdrv_pwrite_sync(bs->file, s->l1_table_offset, s->l1_table,
-            l1_length) < 0)
-        return -1;
-    ret = bdrv_truncate(bs->file, s->l1_table_offset + l1_length);
-    if (ret < 0)
-        return ret;
-
-    memset(s->l2_cache, 0, s->l2_size * L2_CACHE_SIZE * sizeof(uint64_t));
-    memset(s->l2_cache_offsets, 0, L2_CACHE_SIZE * sizeof(uint64_t));
-    memset(s->l2_cache_counts, 0, L2_CACHE_SIZE * sizeof(uint32_t));
-
-    return 0;
-}
-
-/* XXX: put compressed sectors first, then all the cluster aligned
-   tables to avoid losing bytes in alignment */
-static int qcow_write_compressed(BlockDriverState *bs, int64_t sector_num,
-                                 const uint8_t *buf, int nb_sectors)
-{
-    BDRVQcowState *s = bs->opaque;
-    z_stream strm;
-    int ret, out_len;
-    uint8_t *out_buf;
-    uint64_t cluster_offset;
-
-    if (nb_sectors != s->cluster_sectors) {
-        ret = -EINVAL;
-
-        /* Zero-pad last write if image size is not cluster aligned */
-        if (sector_num + nb_sectors == bs->total_sectors &&
-            nb_sectors < s->cluster_sectors) {
-            uint8_t *pad_buf = qemu_blockalign(bs, s->cluster_size);
-            memset(pad_buf, 0, s->cluster_size);
-            memcpy(pad_buf, buf, nb_sectors * BDRV_SECTOR_SIZE);
-            ret = qcow_write_compressed(bs, sector_num,
-                                        pad_buf, s->cluster_sectors);
-            qemu_vfree(pad_buf);
-        }
-        return ret;
-    }
-
-    out_buf = g_malloc(s->cluster_size + (s->cluster_size / 1000) + 128);
-
-    /* best compression, small window, no zlib header */
-    memset(&strm, 0, sizeof(strm));
-    ret = deflateInit2(&strm, Z_DEFAULT_COMPRESSION,
-                       Z_DEFLATED, -12,
-                       9, Z_DEFAULT_STRATEGY);
-    if (ret != 0) {
-        ret = -EINVAL;
-        goto fail;
-    }
-
-    strm.avail_in = s->cluster_size;
-    strm.next_in = (uint8_t *)buf;
-    strm.avail_out = s->cluster_size;
-    strm.next_out = out_buf;
-
-    ret = deflate(&strm, Z_FINISH);
-    if (ret != Z_STREAM_END && ret != Z_OK) {
-        deflateEnd(&strm);
-        ret = -EINVAL;
-        goto fail;
-    }
-    out_len = strm.next_out - out_buf;
-
-    deflateEnd(&strm);
-
-    if (ret != Z_STREAM_END || out_len >= s->cluster_size) {
-        /* could not compress: write normal cluster */
-        ret = bdrv_write(bs, sector_num, buf, s->cluster_sectors);
-        if (ret < 0) {
-            goto fail;
-        }
-    } else {
-        cluster_offset = get_cluster_offset(bs, sector_num << 9, 2,
-                                            out_len, 0, 0);
-        if (cluster_offset == 0) {
-            ret = -EIO;
-            goto fail;
-        }
-
-        cluster_offset &= s->cluster_offset_mask;
-        ret = bdrv_pwrite(bs->file, cluster_offset, out_buf, out_len);
-        if (ret < 0) {
-            goto fail;
-        }
-    }
-
-    ret = 0;
-fail:
-    g_free(out_buf);
-    return ret;
-}
-
-static int qcow_get_info(BlockDriverState *bs, BlockDriverInfo *bdi)
-{
-    BDRVQcowState *s = bs->opaque;
-    bdi->cluster_size = s->cluster_size;
-    return 0;
-}
-
-
-static QEMUOptionParameter qcow_create_options[] = {
-    {
-        .name = BLOCK_OPT_SIZE,
-        .type = OPT_SIZE,
-        .help = "Virtual disk size"
-    },
-    {
-        .name = BLOCK_OPT_BACKING_FILE,
-        .type = OPT_STRING,
-        .help = "File name of a base image"
-    },
-    {
-        .name = BLOCK_OPT_ENCRYPT,
-        .type = OPT_FLAG,
-        .help = "Encrypt the image"
-    },
-    { NULL }
-};
-
-static BlockDriver bdrv_qcow = {
-    .format_name	= "qcow",
-    .instance_size	= sizeof(BDRVQcowState),
-    .bdrv_probe		= qcow_probe,
-    .bdrv_open		= qcow_open,
-    .bdrv_close		= qcow_close,
-    .bdrv_reopen_prepare = qcow_reopen_prepare,
-    .bdrv_create	= qcow_create,
-    .bdrv_has_zero_init     = bdrv_has_zero_init_1,
-
-    .bdrv_co_readv          = qcow_co_readv,
-    .bdrv_co_writev         = qcow_co_writev,
-    .bdrv_co_is_allocated   = qcow_co_is_allocated,
-
-    .bdrv_set_key           = qcow_set_key,
-    .bdrv_make_empty        = qcow_make_empty,
-    .bdrv_write_compressed  = qcow_write_compressed,
-    .bdrv_get_info          = qcow_get_info,
-
-    .create_options = qcow_create_options,
-};
-
-static void bdrv_qcow_init(void)
-{
-    bdrv_register(&bdrv_qcow);
-}
-
-block_init(bdrv_qcow_init);
diff --git a/contrib/qemu/block/qcow2-cache.c b/contrib/qemu/block/qcow2-cache.c
deleted file mode 100644
index 2f3114ecc24..00000000000
--- a/contrib/qemu/block/qcow2-cache.c
+++ /dev/null
@@ -1,323 +0,0 @@
-/*
- * L2/refcount table cache for the QCOW2 format
- *
- * Copyright (c) 2010 Kevin Wolf <kwolf@redhat.com>
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-
-#include "block/block_int.h"
-#include "qemu-common.h"
-#include "qcow2.h"
-#include "trace.h"
-
-typedef struct Qcow2CachedTable {
-    void*   table;
-    int64_t offset;
-    bool    dirty;
-    int     cache_hits;
-    int     ref;
-} Qcow2CachedTable;
-
-struct Qcow2Cache {
-    Qcow2CachedTable*       entries;
-    struct Qcow2Cache*      depends;
-    int                     size;
-    bool                    depends_on_flush;
-};
-
-Qcow2Cache *qcow2_cache_create(BlockDriverState *bs, int num_tables)
-{
-    BDRVQcowState *s = bs->opaque;
-    Qcow2Cache *c;
-    int i;
-
-    c = g_malloc0(sizeof(*c));
-    c->size = num_tables;
-    c->entries = g_malloc0(sizeof(*c->entries) * num_tables);
-
-    for (i = 0; i < c->size; i++) {
-        c->entries[i].table = qemu_blockalign(bs, s->cluster_size);
-    }
-
-    return c;
-}
-
-int qcow2_cache_destroy(BlockDriverState* bs, Qcow2Cache *c)
-{
-    int i;
-
-    for (i = 0; i < c->size; i++) {
-        assert(c->entries[i].ref == 0);
-        qemu_vfree(c->entries[i].table);
-    }
-
-    g_free(c->entries);
-    g_free(c);
-
-    return 0;
-}
-
-static int qcow2_cache_flush_dependency(BlockDriverState *bs, Qcow2Cache *c)
-{
-    int ret;
-
-    ret = qcow2_cache_flush(bs, c->depends);
-    if (ret < 0) {
-        return ret;
-    }
-
-    c->depends = NULL;
-    c->depends_on_flush = false;
-
-    return 0;
-}
-
-static int qcow2_cache_entry_flush(BlockDriverState *bs, Qcow2Cache *c, int i)
-{
-    BDRVQcowState *s = bs->opaque;
-    int ret = 0;
-
-    if (!c->entries[i].dirty || !c->entries[i].offset) {
-        return 0;
-    }
-
-    trace_qcow2_cache_entry_flush(qemu_coroutine_self(),
-                                  c == s->l2_table_cache, i);
-
-    if (c->depends) {
-        ret = qcow2_cache_flush_dependency(bs, c);
-    } else if (c->depends_on_flush) {
-        ret = bdrv_flush(bs->file);
-        if (ret >= 0) {
-            c->depends_on_flush = false;
-        }
-    }
-
-    if (ret < 0) {
-        return ret;
-    }
-
-    if (c == s->refcount_block_cache) {
-        BLKDBG_EVENT(bs->file, BLKDBG_REFBLOCK_UPDATE_PART);
-    } else if (c == s->l2_table_cache) {
-        BLKDBG_EVENT(bs->file, BLKDBG_L2_UPDATE);
-    }
-
-    ret = bdrv_pwrite(bs->file, c->entries[i].offset, c->entries[i].table,
-        s->cluster_size);
-    if (ret < 0) {
-        return ret;
-    }
-
-    c->entries[i].dirty = false;
-
-    return 0;
-}
-
-int qcow2_cache_flush(BlockDriverState *bs, Qcow2Cache *c)
-{
-    BDRVQcowState *s = bs->opaque;
-    int result = 0;
-    int ret;
-    int i;
-
-    trace_qcow2_cache_flush(qemu_coroutine_self(), c == s->l2_table_cache);
-
-    for (i = 0; i < c->size; i++) {
-        ret = qcow2_cache_entry_flush(bs, c, i);
-        if (ret < 0 && result != -ENOSPC) {
-            result = ret;
-        }
-    }
-
-    if (result == 0) {
-        ret = bdrv_flush(bs->file);
-        if (ret < 0) {
-            result = ret;
-        }
-    }
-
-    return result;
-}
-
-int qcow2_cache_set_dependency(BlockDriverState *bs, Qcow2Cache *c,
-    Qcow2Cache *dependency)
-{
-    int ret;
-
-    if (dependency->depends) {
-        ret = qcow2_cache_flush_dependency(bs, dependency);
-        if (ret < 0) {
-            return ret;
-        }
-    }
-
-    if (c->depends && (c->depends != dependency)) {
-        ret = qcow2_cache_flush_dependency(bs, c);
-        if (ret < 0) {
-            return ret;
-        }
-    }
-
-    c->depends = dependency;
-    return 0;
-}
-
-void qcow2_cache_depends_on_flush(Qcow2Cache *c)
-{
-    c->depends_on_flush = true;
-}
-
-static int qcow2_cache_find_entry_to_replace(Qcow2Cache *c)
-{
-    int i;
-    int min_count = INT_MAX;
-    int min_index = -1;
-
-
-    for (i = 0; i < c->size; i++) {
-        if (c->entries[i].ref) {
-            continue;
-        }
-
-        if (c->entries[i].cache_hits < min_count) {
-            min_index = i;
-            min_count = c->entries[i].cache_hits;
-        }
-
-        /* Give newer hits priority */
-        /* TODO Check how to optimize the replacement strategy */
-        c->entries[i].cache_hits /= 2;
-    }
-
-    if (min_index == -1) {
-        /* This can't happen in current synchronous code, but leave the check
-         * here as a reminder for whoever starts using AIO with the cache */
-        abort();
-    }
-    return min_index;
-}
-
-static int qcow2_cache_do_get(BlockDriverState *bs, Qcow2Cache *c,
-    uint64_t offset, void **table, bool read_from_disk)
-{
-    BDRVQcowState *s = bs->opaque;
-    int i;
-    int ret;
-
-    trace_qcow2_cache_get(qemu_coroutine_self(), c == s->l2_table_cache,
-                          offset, read_from_disk);
-
-    /* Check if the table is already cached */
-    for (i = 0; i < c->size; i++) {
-        if (c->entries[i].offset == offset) {
-            goto found;
-        }
-    }
-
-    /* If not, write a table back and replace it */
-    i = qcow2_cache_find_entry_to_replace(c);
-    trace_qcow2_cache_get_replace_entry(qemu_coroutine_self(),
-                                        c == s->l2_table_cache, i);
-    if (i < 0) {
-        return i;
-    }
-
-    ret = qcow2_cache_entry_flush(bs, c, i);
-    if (ret < 0) {
-        return ret;
-    }
-
-    trace_qcow2_cache_get_read(qemu_coroutine_self(),
-                               c == s->l2_table_cache, i);
-    c->entries[i].offset = 0;
-    if (read_from_disk) {
-        if (c == s->l2_table_cache) {
-            BLKDBG_EVENT(bs->file, BLKDBG_L2_LOAD);
-        }
-
-        ret = bdrv_pread(bs->file, offset, c->entries[i].table, s->cluster_size);
-        if (ret < 0) {
-            return ret;
-        }
-    }
-
-    /* Give the table some hits for the start so that it won't be replaced
-     * immediately. The number 32 is completely arbitrary. */
-    c->entries[i].cache_hits = 32;
-    c->entries[i].offset = offset;
-
-    /* And return the right table */
-found:
-    c->entries[i].cache_hits++;
-    c->entries[i].ref++;
-    *table = c->entries[i].table;
-
-    trace_qcow2_cache_get_done(qemu_coroutine_self(),
-                               c == s->l2_table_cache, i);
-
-    return 0;
-}
-
-int qcow2_cache_get(BlockDriverState *bs, Qcow2Cache *c, uint64_t offset,
-    void **table)
-{
-    return qcow2_cache_do_get(bs, c, offset, table, true);
-}
-
-int qcow2_cache_get_empty(BlockDriverState *bs, Qcow2Cache *c, uint64_t offset,
-    void **table)
-{
-    return qcow2_cache_do_get(bs, c, offset, table, false);
-}
-
-int qcow2_cache_put(BlockDriverState *bs, Qcow2Cache *c, void **table)
-{
-    int i;
-
-    for (i = 0; i < c->size; i++) {
-        if (c->entries[i].table == *table) {
-            goto found;
-        }
-    }
-    return -ENOENT;
-
-found:
-    c->entries[i].ref--;
-    *table = NULL;
-
-    assert(c->entries[i].ref >= 0);
-    return 0;
-}
-
-void qcow2_cache_entry_mark_dirty(Qcow2Cache *c, void *table)
-{
-    int i;
-
-    for (i = 0; i < c->size; i++) {
-        if (c->entries[i].table == table) {
-            goto found;
-        }
-    }
-    abort();
-
-found:
-    c->entries[i].dirty = true;
-}
diff --git a/contrib/qemu/block/qcow2-cluster.c b/contrib/qemu/block/qcow2-cluster.c
deleted file mode 100644
index cca76d4fcdd..00000000000
--- a/contrib/qemu/block/qcow2-cluster.c
+++ /dev/null
@@ -1,1478 +0,0 @@
-/*
- * Block driver for the QCOW version 2 format
- *
- * Copyright (c) 2004-2006 Fabrice Bellard
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-
-#include <zlib.h>
-
-#include "qemu-common.h"
-#include "block/block_int.h"
-#include "block/qcow2.h"
-#include "trace.h"
-
-int qcow2_grow_l1_table(BlockDriverState *bs, uint64_t min_size,
-                        bool exact_size)
-{
-    BDRVQcowState *s = bs->opaque;
-    int new_l1_size2, ret, i;
-    uint64_t *new_l1_table;
-    int64_t new_l1_table_offset, new_l1_size;
-    uint8_t data[12];
-
-    if (min_size <= s->l1_size)
-        return 0;
-
-    if (exact_size) {
-        new_l1_size = min_size;
-    } else {
-        /* Bump size up to reduce the number of times we have to grow */
-        new_l1_size = s->l1_size;
-        if (new_l1_size == 0) {
-            new_l1_size = 1;
-        }
-        while (min_size > new_l1_size) {
-            new_l1_size = (new_l1_size * 3 + 1) / 2;
-        }
-    }
-
-    if (new_l1_size > INT_MAX) {
-        return -EFBIG;
-    }
-
-#ifdef DEBUG_ALLOC2
-    fprintf(stderr, "grow l1_table from %d to %" PRId64 "\n",
-            s->l1_size, new_l1_size);
-#endif
-
-    new_l1_size2 = sizeof(uint64_t) * new_l1_size;
-    new_l1_table = g_malloc0(align_offset(new_l1_size2, 512));
-    memcpy(new_l1_table, s->l1_table, s->l1_size * sizeof(uint64_t));
-
-    /* write new table (align to cluster) */
-    BLKDBG_EVENT(bs->file, BLKDBG_L1_GROW_ALLOC_TABLE);
-    new_l1_table_offset = qcow2_alloc_clusters(bs, new_l1_size2);
-    if (new_l1_table_offset < 0) {
-        g_free(new_l1_table);
-        return new_l1_table_offset;
-    }
-
-    ret = qcow2_cache_flush(bs, s->refcount_block_cache);
-    if (ret < 0) {
-        goto fail;
-    }
-
-    BLKDBG_EVENT(bs->file, BLKDBG_L1_GROW_WRITE_TABLE);
-    for(i = 0; i < s->l1_size; i++)
-        new_l1_table[i] = cpu_to_be64(new_l1_table[i]);
-    ret = bdrv_pwrite_sync(bs->file, new_l1_table_offset, new_l1_table, new_l1_size2);
-    if (ret < 0)
-        goto fail;
-    for(i = 0; i < s->l1_size; i++)
-        new_l1_table[i] = be64_to_cpu(new_l1_table[i]);
-
-    /* set new table */
-    BLKDBG_EVENT(bs->file, BLKDBG_L1_GROW_ACTIVATE_TABLE);
-    cpu_to_be32w((uint32_t*)data, new_l1_size);
-    cpu_to_be64wu((uint64_t*)(data + 4), new_l1_table_offset);
-    ret = bdrv_pwrite_sync(bs->file, offsetof(QCowHeader, l1_size), data,sizeof(data));
-    if (ret < 0) {
-        goto fail;
-    }
-    g_free(s->l1_table);
-    qcow2_free_clusters(bs, s->l1_table_offset, s->l1_size * sizeof(uint64_t),
-                        QCOW2_DISCARD_OTHER);
-    s->l1_table_offset = new_l1_table_offset;
-    s->l1_table = new_l1_table;
-    s->l1_size = new_l1_size;
-    return 0;
- fail:
-    g_free(new_l1_table);
-    qcow2_free_clusters(bs, new_l1_table_offset, new_l1_size2,
-                        QCOW2_DISCARD_OTHER);
-    return ret;
-}
-
-/*
- * l2_load
- *
- * Loads a L2 table into memory. If the table is in the cache, the cache
- * is used; otherwise the L2 table is loaded from the image file.
- *
- * Returns a pointer to the L2 table on success, or NULL if the read from
- * the image file failed.
- */
-
-static int l2_load(BlockDriverState *bs, uint64_t l2_offset,
-    uint64_t **l2_table)
-{
-    BDRVQcowState *s = bs->opaque;
-    int ret;
-
-    ret = qcow2_cache_get(bs, s->l2_table_cache, l2_offset, (void**) l2_table);
-
-    return ret;
-}
-
-/*
- * Writes one sector of the L1 table to the disk (can't update single entries
- * and we really don't want bdrv_pread to perform a read-modify-write)
- */
-#define L1_ENTRIES_PER_SECTOR (512 / 8)
-static int write_l1_entry(BlockDriverState *bs, int l1_index)
-{
-    BDRVQcowState *s = bs->opaque;
-    uint64_t buf[L1_ENTRIES_PER_SECTOR];
-    int l1_start_index;
-    int i, ret;
-
-    l1_start_index = l1_index & ~(L1_ENTRIES_PER_SECTOR - 1);
-    for (i = 0; i < L1_ENTRIES_PER_SECTOR; i++) {
-        buf[i] = cpu_to_be64(s->l1_table[l1_start_index + i]);
-    }
-
-    BLKDBG_EVENT(bs->file, BLKDBG_L1_UPDATE);
-    ret = bdrv_pwrite_sync(bs->file, s->l1_table_offset + 8 * l1_start_index,
-        buf, sizeof(buf));
-    if (ret < 0) {
-        return ret;
-    }
-
-    return 0;
-}
-
-/*
- * l2_allocate
- *
- * Allocate a new l2 entry in the file. If l1_index points to an already
- * used entry in the L2 table (i.e. we are doing a copy on write for the L2
- * table) copy the contents of the old L2 table into the newly allocated one.
- * Otherwise the new table is initialized with zeros.
- *
- */
-
-static int l2_allocate(BlockDriverState *bs, int l1_index, uint64_t **table)
-{
-    BDRVQcowState *s = bs->opaque;
-    uint64_t old_l2_offset;
-    uint64_t *l2_table;
-    int64_t l2_offset;
-    int ret;
-
-    old_l2_offset = s->l1_table[l1_index];
-
-    trace_qcow2_l2_allocate(bs, l1_index);
-
-    /* allocate a new l2 entry */
-
-    l2_offset = qcow2_alloc_clusters(bs, s->l2_size * sizeof(uint64_t));
-    if (l2_offset < 0) {
-        return l2_offset;
-    }
-
-    ret = qcow2_cache_flush(bs, s->refcount_block_cache);
-    if (ret < 0) {
-        goto fail;
-    }
-
-    /* allocate a new entry in the l2 cache */
-
-    trace_qcow2_l2_allocate_get_empty(bs, l1_index);
-    ret = qcow2_cache_get_empty(bs, s->l2_table_cache, l2_offset, (void**) table);
-    if (ret < 0) {
-        return ret;
-    }
-
-    l2_table = *table;
-
-    if ((old_l2_offset & L1E_OFFSET_MASK) == 0) {
-        /* if there was no old l2 table, clear the new table */
-        memset(l2_table, 0, s->l2_size * sizeof(uint64_t));
-    } else {
-        uint64_t* old_table;
-
-        /* if there was an old l2 table, read it from the disk */
-        BLKDBG_EVENT(bs->file, BLKDBG_L2_ALLOC_COW_READ);
-        ret = qcow2_cache_get(bs, s->l2_table_cache,
-            old_l2_offset & L1E_OFFSET_MASK,
-            (void**) &old_table);
-        if (ret < 0) {
-            goto fail;
-        }
-
-        memcpy(l2_table, old_table, s->cluster_size);
-
-        ret = qcow2_cache_put(bs, s->l2_table_cache, (void**) &old_table);
-        if (ret < 0) {
-            goto fail;
-        }
-    }
-
-    /* write the l2 table to the file */
-    BLKDBG_EVENT(bs->file, BLKDBG_L2_ALLOC_WRITE);
-
-    trace_qcow2_l2_allocate_write_l2(bs, l1_index);
-    qcow2_cache_entry_mark_dirty(s->l2_table_cache, l2_table);
-    ret = qcow2_cache_flush(bs, s->l2_table_cache);
-    if (ret < 0) {
-        goto fail;
-    }
-
-    /* update the L1 entry */
-    trace_qcow2_l2_allocate_write_l1(bs, l1_index);
-    s->l1_table[l1_index] = l2_offset | QCOW_OFLAG_COPIED;
-    ret = write_l1_entry(bs, l1_index);
-    if (ret < 0) {
-        goto fail;
-    }
-
-    *table = l2_table;
-    trace_qcow2_l2_allocate_done(bs, l1_index, 0);
-    return 0;
-
-fail:
-    trace_qcow2_l2_allocate_done(bs, l1_index, ret);
-    qcow2_cache_put(bs, s->l2_table_cache, (void**) table);
-    s->l1_table[l1_index] = old_l2_offset;
-    return ret;
-}
-
-/*
- * Checks how many clusters in a given L2 table are contiguous in the image
- * file. As soon as one of the flags in the bitmask stop_flags changes compared
- * to the first cluster, the search is stopped and the cluster is not counted
- * as contiguous. (This allows it, for example, to stop at the first compressed
- * cluster which may require a different handling)
- */
-static int count_contiguous_clusters(uint64_t nb_clusters, int cluster_size,
-        uint64_t *l2_table, uint64_t start, uint64_t stop_flags)
-{
-    int i;
-    uint64_t mask = stop_flags | L2E_OFFSET_MASK;
-    uint64_t offset = be64_to_cpu(l2_table[0]) & mask;
-
-    if (!offset)
-        return 0;
-
-    for (i = start; i < start + nb_clusters; i++) {
-        uint64_t l2_entry = be64_to_cpu(l2_table[i]) & mask;
-        if (offset + (uint64_t) i * cluster_size != l2_entry) {
-            break;
-        }
-    }
-
-	return (i - start);
-}
-
-static int count_contiguous_free_clusters(uint64_t nb_clusters, uint64_t *l2_table)
-{
-    int i;
-
-    for (i = 0; i < nb_clusters; i++) {
-        int type = qcow2_get_cluster_type(be64_to_cpu(l2_table[i]));
-
-        if (type != QCOW2_CLUSTER_UNALLOCATED) {
-            break;
-        }
-    }
-
-    return i;
-}
-
-/* The crypt function is compatible with the linux cryptoloop
-   algorithm for < 4 GB images. NOTE: out_buf == in_buf is
-   supported */
-void qcow2_encrypt_sectors(BDRVQcowState *s, int64_t sector_num,
-                           uint8_t *out_buf, const uint8_t *in_buf,
-                           int nb_sectors, int enc,
-                           const AES_KEY *key)
-{
-    union {
-        uint64_t ll[2];
-        uint8_t b[16];
-    } ivec;
-    int i;
-
-    for(i = 0; i < nb_sectors; i++) {
-        ivec.ll[0] = cpu_to_le64(sector_num);
-        ivec.ll[1] = 0;
-        AES_cbc_encrypt(in_buf, out_buf, 512, key,
-                        ivec.b, enc);
-        sector_num++;
-        in_buf += 512;
-        out_buf += 512;
-    }
-}
-
-static int coroutine_fn copy_sectors(BlockDriverState *bs,
-                                     uint64_t start_sect,
-                                     uint64_t cluster_offset,
-                                     int n_start, int n_end)
-{
-    BDRVQcowState *s = bs->opaque;
-    QEMUIOVector qiov;
-    struct iovec iov;
-    int n, ret;
-
-    /*
-     * If this is the last cluster and it is only partially used, we must only
-     * copy until the end of the image, or bdrv_check_request will fail for the
-     * bdrv_read/write calls below.
-     */
-    if (start_sect + n_end > bs->total_sectors) {
-        n_end = bs->total_sectors - start_sect;
-    }
-
-    n = n_end - n_start;
-    if (n <= 0) {
-        return 0;
-    }
-
-    iov.iov_len = n * BDRV_SECTOR_SIZE;
-    iov.iov_base = qemu_blockalign(bs, iov.iov_len);
-
-    qemu_iovec_init_external(&qiov, &iov, 1);
-
-    BLKDBG_EVENT(bs->file, BLKDBG_COW_READ);
-
-    /* Call .bdrv_co_readv() directly instead of using the public block-layer
-     * interface.  This avoids double I/O throttling and request tracking,
-     * which can lead to deadlock when block layer copy-on-read is enabled.
-     */
-    ret = bs->drv->bdrv_co_readv(bs, start_sect + n_start, n, &qiov);
-    if (ret < 0) {
-        goto out;
-    }
-
-    if (s->crypt_method) {
-        qcow2_encrypt_sectors(s, start_sect + n_start,
-                        iov.iov_base, iov.iov_base, n, 1,
-                        &s->aes_encrypt_key);
-    }
-
-    BLKDBG_EVENT(bs->file, BLKDBG_COW_WRITE);
-    ret = bdrv_co_writev(bs->file, (cluster_offset >> 9) + n_start, n, &qiov);
-    if (ret < 0) {
-        goto out;
-    }
-
-    ret = 0;
-out:
-    qemu_vfree(iov.iov_base);
-    return ret;
-}
-
-
-/*
- * get_cluster_offset
- *
- * For a given offset of the disk image, find the cluster offset in
- * qcow2 file. The offset is stored in *cluster_offset.
- *
- * on entry, *num is the number of contiguous sectors we'd like to
- * access following offset.
- *
- * on exit, *num is the number of contiguous sectors we can read.
- *
- * Returns the cluster type (QCOW2_CLUSTER_*) on success, -errno in error
- * cases.
- */
-int qcow2_get_cluster_offset(BlockDriverState *bs, uint64_t offset,
-    int *num, uint64_t *cluster_offset)
-{
-    BDRVQcowState *s = bs->opaque;
-    unsigned int l2_index;
-    uint64_t l1_index, l2_offset, *l2_table;
-    int l1_bits, c;
-    unsigned int index_in_cluster, nb_clusters;
-    uint64_t nb_available, nb_needed;
-    int ret;
-
-    index_in_cluster = (offset >> 9) & (s->cluster_sectors - 1);
-    nb_needed = *num + index_in_cluster;
-
-    l1_bits = s->l2_bits + s->cluster_bits;
-
-    /* compute how many bytes there are between the offset and
-     * the end of the l1 entry
-     */
-
-    nb_available = (1ULL << l1_bits) - (offset & ((1ULL << l1_bits) - 1));
-
-    /* compute the number of available sectors */
-
-    nb_available = (nb_available >> 9) + index_in_cluster;
-
-    if (nb_needed > nb_available) {
-        nb_needed = nb_available;
-    }
-
-    *cluster_offset = 0;
-
-    /* seek the the l2 offset in the l1 table */
-
-    l1_index = offset >> l1_bits;
-    if (l1_index >= s->l1_size) {
-        ret = QCOW2_CLUSTER_UNALLOCATED;
-        goto out;
-    }
-
-    l2_offset = s->l1_table[l1_index] & L1E_OFFSET_MASK;
-    if (!l2_offset) {
-        ret = QCOW2_CLUSTER_UNALLOCATED;
-        goto out;
-    }
-
-    /* load the l2 table in memory */
-
-    ret = l2_load(bs, l2_offset, &l2_table);
-    if (ret < 0) {
-        return ret;
-    }
-
-    /* find the cluster offset for the given disk offset */
-
-    l2_index = (offset >> s->cluster_bits) & (s->l2_size - 1);
-    *cluster_offset = be64_to_cpu(l2_table[l2_index]);
-    nb_clusters = size_to_clusters(s, nb_needed << 9);
-
-    ret = qcow2_get_cluster_type(*cluster_offset);
-    switch (ret) {
-    case QCOW2_CLUSTER_COMPRESSED:
-        /* Compressed clusters can only be processed one by one */
-        c = 1;
-        *cluster_offset &= L2E_COMPRESSED_OFFSET_SIZE_MASK;
-        break;
-    case QCOW2_CLUSTER_ZERO:
-        if (s->qcow_version < 3) {
-            return -EIO;
-        }
-        c = count_contiguous_clusters(nb_clusters, s->cluster_size,
-                &l2_table[l2_index], 0,
-                QCOW_OFLAG_COMPRESSED | QCOW_OFLAG_ZERO);
-        *cluster_offset = 0;
-        break;
-    case QCOW2_CLUSTER_UNALLOCATED:
-        /* how many empty clusters ? */
-        c = count_contiguous_free_clusters(nb_clusters, &l2_table[l2_index]);
-        *cluster_offset = 0;
-        break;
-    case QCOW2_CLUSTER_NORMAL:
-        /* how many allocated clusters ? */
-        c = count_contiguous_clusters(nb_clusters, s->cluster_size,
-                &l2_table[l2_index], 0,
-                QCOW_OFLAG_COMPRESSED | QCOW_OFLAG_ZERO);
-        *cluster_offset &= L2E_OFFSET_MASK;
-        break;
-    default:
-        abort();
-    }
-
-    qcow2_cache_put(bs, s->l2_table_cache, (void**) &l2_table);
-
-    nb_available = (c * s->cluster_sectors);
-
-out:
-    if (nb_available > nb_needed)
-        nb_available = nb_needed;
-
-    *num = nb_available - index_in_cluster;
-
-    return ret;
-}
-
-/*
- * get_cluster_table
- *
- * for a given disk offset, load (and allocate if needed)
- * the l2 table.
- *
- * the l2 table offset in the qcow2 file and the cluster index
- * in the l2 table are given to the caller.
- *
- * Returns 0 on success, -errno in failure case
- */
-static int get_cluster_table(BlockDriverState *bs, uint64_t offset,
-                             uint64_t **new_l2_table,
-                             int *new_l2_index)
-{
-    BDRVQcowState *s = bs->opaque;
-    unsigned int l2_index;
-    uint64_t l1_index, l2_offset;
-    uint64_t *l2_table = NULL;
-    int ret;
-
-    /* seek the the l2 offset in the l1 table */
-
-    l1_index = offset >> (s->l2_bits + s->cluster_bits);
-    if (l1_index >= s->l1_size) {
-        ret = qcow2_grow_l1_table(bs, l1_index + 1, false);
-        if (ret < 0) {
-            return ret;
-        }
-    }
-
-    assert(l1_index < s->l1_size);
-    l2_offset = s->l1_table[l1_index] & L1E_OFFSET_MASK;
-
-    /* seek the l2 table of the given l2 offset */
-
-    if (s->l1_table[l1_index] & QCOW_OFLAG_COPIED) {
-        /* load the l2 table in memory */
-        ret = l2_load(bs, l2_offset, &l2_table);
-        if (ret < 0) {
-            return ret;
-        }
-    } else {
-        /* First allocate a new L2 table (and do COW if needed) */
-        ret = l2_allocate(bs, l1_index, &l2_table);
-        if (ret < 0) {
-            return ret;
-        }
-
-        /* Then decrease the refcount of the old table */
-        if (l2_offset) {
-            qcow2_free_clusters(bs, l2_offset, s->l2_size * sizeof(uint64_t),
-                                QCOW2_DISCARD_OTHER);
-        }
-    }
-
-    /* find the cluster offset for the given disk offset */
-
-    l2_index = (offset >> s->cluster_bits) & (s->l2_size - 1);
-
-    *new_l2_table = l2_table;
-    *new_l2_index = l2_index;
-
-    return 0;
-}
-
-/*
- * alloc_compressed_cluster_offset
- *
- * For a given offset of the disk image, return cluster offset in
- * qcow2 file.
- *
- * If the offset is not found, allocate a new compressed cluster.
- *
- * Return the cluster offset if successful,
- * Return 0, otherwise.
- *
- */
-
-uint64_t qcow2_alloc_compressed_cluster_offset(BlockDriverState *bs,
-                                               uint64_t offset,
-                                               int compressed_size)
-{
-    BDRVQcowState *s = bs->opaque;
-    int l2_index, ret;
-    uint64_t *l2_table;
-    int64_t cluster_offset;
-    int nb_csectors;
-
-    ret = get_cluster_table(bs, offset, &l2_table, &l2_index);
-    if (ret < 0) {
-        return 0;
-    }
-
-    /* Compression can't overwrite anything. Fail if the cluster was already
-     * allocated. */
-    cluster_offset = be64_to_cpu(l2_table[l2_index]);
-    if (cluster_offset & L2E_OFFSET_MASK) {
-        qcow2_cache_put(bs, s->l2_table_cache, (void**) &l2_table);
-        return 0;
-    }
-
-    cluster_offset = qcow2_alloc_bytes(bs, compressed_size);
-    if (cluster_offset < 0) {
-        qcow2_cache_put(bs, s->l2_table_cache, (void**) &l2_table);
-        return 0;
-    }
-
-    nb_csectors = ((cluster_offset + compressed_size - 1) >> 9) -
-                  (cluster_offset >> 9);
-
-    cluster_offset |= QCOW_OFLAG_COMPRESSED |
-                      ((uint64_t)nb_csectors << s->csize_shift);
-
-    /* update L2 table */
-
-    /* compressed clusters never have the copied flag */
-
-    BLKDBG_EVENT(bs->file, BLKDBG_L2_UPDATE_COMPRESSED);
-    qcow2_cache_entry_mark_dirty(s->l2_table_cache, l2_table);
-    l2_table[l2_index] = cpu_to_be64(cluster_offset);
-    ret = qcow2_cache_put(bs, s->l2_table_cache, (void**) &l2_table);
-    if (ret < 0) {
-        return 0;
-    }
-
-    return cluster_offset;
-}
-
-static int perform_cow(BlockDriverState *bs, QCowL2Meta *m, Qcow2COWRegion *r)
-{
-    BDRVQcowState *s = bs->opaque;
-    int ret;
-
-    if (r->nb_sectors == 0) {
-        return 0;
-    }
-
-    qemu_co_mutex_unlock(&s->lock);
-    ret = copy_sectors(bs, m->offset / BDRV_SECTOR_SIZE, m->alloc_offset,
-                       r->offset / BDRV_SECTOR_SIZE,
-                       r->offset / BDRV_SECTOR_SIZE + r->nb_sectors);
-    qemu_co_mutex_lock(&s->lock);
-
-    if (ret < 0) {
-        return ret;
-    }
-
-    /*
-     * Before we update the L2 table to actually point to the new cluster, we
-     * need to be sure that the refcounts have been increased and COW was
-     * handled.
-     */
-    qcow2_cache_depends_on_flush(s->l2_table_cache);
-
-    return 0;
-}
-
-int qcow2_alloc_cluster_link_l2(BlockDriverState *bs, QCowL2Meta *m)
-{
-    BDRVQcowState *s = bs->opaque;
-    int i, j = 0, l2_index, ret;
-    uint64_t *old_cluster, *l2_table;
-    uint64_t cluster_offset = m->alloc_offset;
-
-    trace_qcow2_cluster_link_l2(qemu_coroutine_self(), m->nb_clusters);
-    assert(m->nb_clusters > 0);
-
-    old_cluster = g_malloc(m->nb_clusters * sizeof(uint64_t));
-
-    /* copy content of unmodified sectors */
-    ret = perform_cow(bs, m, &m->cow_start);
-    if (ret < 0) {
-        goto err;
-    }
-
-    ret = perform_cow(bs, m, &m->cow_end);
-    if (ret < 0) {
-        goto err;
-    }
-
-    /* Update L2 table. */
-    if (s->use_lazy_refcounts) {
-        qcow2_mark_dirty(bs);
-    }
-    if (qcow2_need_accurate_refcounts(s)) {
-        qcow2_cache_set_dependency(bs, s->l2_table_cache,
-                                   s->refcount_block_cache);
-    }
-
-    ret = get_cluster_table(bs, m->offset, &l2_table, &l2_index);
-    if (ret < 0) {
-        goto err;
-    }
-    qcow2_cache_entry_mark_dirty(s->l2_table_cache, l2_table);
-
-    for (i = 0; i < m->nb_clusters; i++) {
-        /* if two concurrent writes happen to the same unallocated cluster
-	 * each write allocates separate cluster and writes data concurrently.
-	 * The first one to complete updates l2 table with pointer to its
-	 * cluster the second one has to do RMW (which is done above by
-	 * copy_sectors()), update l2 table with its cluster pointer and free
-	 * old cluster. This is what this loop does */
-        if(l2_table[l2_index + i] != 0)
-            old_cluster[j++] = l2_table[l2_index + i];
-
-        l2_table[l2_index + i] = cpu_to_be64((cluster_offset +
-                    (i << s->cluster_bits)) | QCOW_OFLAG_COPIED);
-     }
-
-
-    ret = qcow2_cache_put(bs, s->l2_table_cache, (void**) &l2_table);
-    if (ret < 0) {
-        goto err;
-    }
-
-    /*
-     * If this was a COW, we need to decrease the refcount of the old cluster.
-     * Also flush bs->file to get the right order for L2 and refcount update.
-     *
-     * Don't discard clusters that reach a refcount of 0 (e.g. compressed
-     * clusters), the next write will reuse them anyway.
-     */
-    if (j != 0) {
-        for (i = 0; i < j; i++) {
-            qcow2_free_any_clusters(bs, be64_to_cpu(old_cluster[i]), 1,
-                                    QCOW2_DISCARD_NEVER);
-        }
-    }
-
-    ret = 0;
-err:
-    g_free(old_cluster);
-    return ret;
- }
-
-/*
- * Returns the number of contiguous clusters that can be used for an allocating
- * write, but require COW to be performed (this includes yet unallocated space,
- * which must copy from the backing file)
- */
-static int count_cow_clusters(BDRVQcowState *s, int nb_clusters,
-    uint64_t *l2_table, int l2_index)
-{
-    int i;
-
-    for (i = 0; i < nb_clusters; i++) {
-        uint64_t l2_entry = be64_to_cpu(l2_table[l2_index + i]);
-        int cluster_type = qcow2_get_cluster_type(l2_entry);
-
-        switch(cluster_type) {
-        case QCOW2_CLUSTER_NORMAL:
-            if (l2_entry & QCOW_OFLAG_COPIED) {
-                goto out;
-            }
-            break;
-        case QCOW2_CLUSTER_UNALLOCATED:
-        case QCOW2_CLUSTER_COMPRESSED:
-        case QCOW2_CLUSTER_ZERO:
-            break;
-        default:
-            abort();
-        }
-    }
-
-out:
-    assert(i <= nb_clusters);
-    return i;
-}
-
-/*
- * Check if there already is an AIO write request in flight which allocates
- * the same cluster. In this case we need to wait until the previous
- * request has completed and updated the L2 table accordingly.
- *
- * Returns:
- *   0       if there was no dependency. *cur_bytes indicates the number of
- *           bytes from guest_offset that can be read before the next
- *           dependency must be processed (or the request is complete)
- *
- *   -EAGAIN if we had to wait for another request, previously gathered
- *           information on cluster allocation may be invalid now. The caller
- *           must start over anyway, so consider *cur_bytes undefined.
- */
-static int handle_dependencies(BlockDriverState *bs, uint64_t guest_offset,
-    uint64_t *cur_bytes, QCowL2Meta **m)
-{
-    BDRVQcowState *s = bs->opaque;
-    QCowL2Meta *old_alloc;
-    uint64_t bytes = *cur_bytes;
-
-    QLIST_FOREACH(old_alloc, &s->cluster_allocs, next_in_flight) {
-
-        uint64_t start = guest_offset;
-        uint64_t end = start + bytes;
-        uint64_t old_start = l2meta_cow_start(old_alloc);
-        uint64_t old_end = l2meta_cow_end(old_alloc);
-
-        if (end <= old_start || start >= old_end) {
-            /* No intersection */
-        } else {
-            if (start < old_start) {
-                /* Stop at the start of a running allocation */
-                bytes = old_start - start;
-            } else {
-                bytes = 0;
-            }
-
-            /* Stop if already an l2meta exists. After yielding, it wouldn't
-             * be valid any more, so we'd have to clean up the old L2Metas
-             * and deal with requests depending on them before starting to
-             * gather new ones. Not worth the trouble. */
-            if (bytes == 0 && *m) {
-                *cur_bytes = 0;
-                return 0;
-            }
-
-            if (bytes == 0) {
-                /* Wait for the dependency to complete. We need to recheck
-                 * the free/allocated clusters when we continue. */
-                qemu_co_mutex_unlock(&s->lock);
-                qemu_co_queue_wait(&old_alloc->dependent_requests);
-                qemu_co_mutex_lock(&s->lock);
-                return -EAGAIN;
-            }
-        }
-    }
-
-    /* Make sure that existing clusters and new allocations are only used up to
-     * the next dependency if we shortened the request above */
-    *cur_bytes = bytes;
-
-    return 0;
-}
-
-/*
- * Checks how many already allocated clusters that don't require a copy on
- * write there are at the given guest_offset (up to *bytes). If
- * *host_offset is not zero, only physically contiguous clusters beginning at
- * this host offset are counted.
- *
- * Note that guest_offset may not be cluster aligned. In this case, the
- * returned *host_offset points to exact byte referenced by guest_offset and
- * therefore isn't cluster aligned as well.
- *
- * Returns:
- *   0:     if no allocated clusters are available at the given offset.
- *          *bytes is normally unchanged. It is set to 0 if the cluster
- *          is allocated and doesn't need COW, but doesn't have the right
- *          physical offset.
- *
- *   1:     if allocated clusters that don't require a COW are available at
- *          the requested offset. *bytes may have decreased and describes
- *          the length of the area that can be written to.
- *
- *  -errno: in error cases
- */
-static int handle_copied(BlockDriverState *bs, uint64_t guest_offset,
-    uint64_t *host_offset, uint64_t *bytes, QCowL2Meta **m)
-{
-    BDRVQcowState *s = bs->opaque;
-    int l2_index;
-    uint64_t cluster_offset;
-    uint64_t *l2_table;
-    unsigned int nb_clusters;
-    unsigned int keep_clusters;
-    int ret, pret;
-
-    trace_qcow2_handle_copied(qemu_coroutine_self(), guest_offset, *host_offset,
-                              *bytes);
-
-    assert(*host_offset == 0 ||    offset_into_cluster(s, guest_offset)
-                                == offset_into_cluster(s, *host_offset));
-
-    /*
-     * Calculate the number of clusters to look for. We stop at L2 table
-     * boundaries to keep things simple.
-     */
-    nb_clusters =
-        size_to_clusters(s, offset_into_cluster(s, guest_offset) + *bytes);
-
-    l2_index = offset_to_l2_index(s, guest_offset);
-    nb_clusters = MIN(nb_clusters, s->l2_size - l2_index);
-
-    /* Find L2 entry for the first involved cluster */
-    ret = get_cluster_table(bs, guest_offset, &l2_table, &l2_index);
-    if (ret < 0) {
-        return ret;
-    }
-
-    cluster_offset = be64_to_cpu(l2_table[l2_index]);
-
-    /* Check how many clusters are already allocated and don't need COW */
-    if (qcow2_get_cluster_type(cluster_offset) == QCOW2_CLUSTER_NORMAL
-        && (cluster_offset & QCOW_OFLAG_COPIED))
-    {
-        /* If a specific host_offset is required, check it */
-        bool offset_matches =
-            (cluster_offset & L2E_OFFSET_MASK) == *host_offset;
-
-        if (*host_offset != 0 && !offset_matches) {
-            *bytes = 0;
-            ret = 0;
-            goto out;
-        }
-
-        /* We keep all QCOW_OFLAG_COPIED clusters */
-        keep_clusters =
-            count_contiguous_clusters(nb_clusters, s->cluster_size,
-                                      &l2_table[l2_index], 0,
-                                      QCOW_OFLAG_COPIED | QCOW_OFLAG_ZERO);
-        assert(keep_clusters <= nb_clusters);
-
-        *bytes = MIN(*bytes,
-                 keep_clusters * s->cluster_size
-                 - offset_into_cluster(s, guest_offset));
-
-        ret = 1;
-    } else {
-        ret = 0;
-    }
-
-    /* Cleanup */
-out:
-    pret = qcow2_cache_put(bs, s->l2_table_cache, (void**) &l2_table);
-    if (pret < 0) {
-        return pret;
-    }
-
-    /* Only return a host offset if we actually made progress. Otherwise we
-     * would make requirements for handle_alloc() that it can't fulfill */
-    if (ret) {
-        *host_offset = (cluster_offset & L2E_OFFSET_MASK)
-                     + offset_into_cluster(s, guest_offset);
-    }
-
-    return ret;
-}
-
-/*
- * Allocates new clusters for the given guest_offset.
- *
- * At most *nb_clusters are allocated, and on return *nb_clusters is updated to
- * contain the number of clusters that have been allocated and are contiguous
- * in the image file.
- *
- * If *host_offset is non-zero, it specifies the offset in the image file at
- * which the new clusters must start. *nb_clusters can be 0 on return in this
- * case if the cluster at host_offset is already in use. If *host_offset is
- * zero, the clusters can be allocated anywhere in the image file.
- *
- * *host_offset is updated to contain the offset into the image file at which
- * the first allocated cluster starts.
- *
- * Return 0 on success and -errno in error cases. -EAGAIN means that the
- * function has been waiting for another request and the allocation must be
- * restarted, but the whole request should not be failed.
- */
-static int do_alloc_cluster_offset(BlockDriverState *bs, uint64_t guest_offset,
-    uint64_t *host_offset, unsigned int *nb_clusters)
-{
-    BDRVQcowState *s = bs->opaque;
-
-    trace_qcow2_do_alloc_clusters_offset(qemu_coroutine_self(), guest_offset,
-                                         *host_offset, *nb_clusters);
-
-    /* Allocate new clusters */
-    trace_qcow2_cluster_alloc_phys(qemu_coroutine_self());
-    if (*host_offset == 0) {
-        int64_t cluster_offset =
-            qcow2_alloc_clusters(bs, *nb_clusters * s->cluster_size);
-        if (cluster_offset < 0) {
-            return cluster_offset;
-        }
-        *host_offset = cluster_offset;
-        return 0;
-    } else {
-        int ret = qcow2_alloc_clusters_at(bs, *host_offset, *nb_clusters);
-        if (ret < 0) {
-            return ret;
-        }
-        *nb_clusters = ret;
-        return 0;
-    }
-}
-
-/*
- * Allocates new clusters for an area that either is yet unallocated or needs a
- * copy on write. If *host_offset is non-zero, clusters are only allocated if
- * the new allocation can match the specified host offset.
- *
- * Note that guest_offset may not be cluster aligned. In this case, the
- * returned *host_offset points to exact byte referenced by guest_offset and
- * therefore isn't cluster aligned as well.
- *
- * Returns:
- *   0:     if no clusters could be allocated. *bytes is set to 0,
- *          *host_offset is left unchanged.
- *
- *   1:     if new clusters were allocated. *bytes may be decreased if the
- *          new allocation doesn't cover all of the requested area.
- *          *host_offset is updated to contain the host offset of the first
- *          newly allocated cluster.
- *
- *  -errno: in error cases
- */
-static int handle_alloc(BlockDriverState *bs, uint64_t guest_offset,
-    uint64_t *host_offset, uint64_t *bytes, QCowL2Meta **m)
-{
-    BDRVQcowState *s = bs->opaque;
-    int l2_index;
-    uint64_t *l2_table;
-    uint64_t entry;
-    unsigned int nb_clusters;
-    int ret;
-
-    uint64_t alloc_cluster_offset;
-
-    trace_qcow2_handle_alloc(qemu_coroutine_self(), guest_offset, *host_offset,
-                             *bytes);
-    assert(*bytes > 0);
-
-    /*
-     * Calculate the number of clusters to look for. We stop at L2 table
-     * boundaries to keep things simple.
-     */
-    nb_clusters =
-        size_to_clusters(s, offset_into_cluster(s, guest_offset) + *bytes);
-
-    l2_index = offset_to_l2_index(s, guest_offset);
-    nb_clusters = MIN(nb_clusters, s->l2_size - l2_index);
-
-    /* Find L2 entry for the first involved cluster */
-    ret = get_cluster_table(bs, guest_offset, &l2_table, &l2_index);
-    if (ret < 0) {
-        return ret;
-    }
-
-    entry = be64_to_cpu(l2_table[l2_index]);
-
-    /* For the moment, overwrite compressed clusters one by one */
-    if (entry & QCOW_OFLAG_COMPRESSED) {
-        nb_clusters = 1;
-    } else {
-        nb_clusters = count_cow_clusters(s, nb_clusters, l2_table, l2_index);
-    }
-
-    /* This function is only called when there were no non-COW clusters, so if
-     * we can't find any unallocated or COW clusters either, something is
-     * wrong with our code. */
-    assert(nb_clusters > 0);
-
-    ret = qcow2_cache_put(bs, s->l2_table_cache, (void**) &l2_table);
-    if (ret < 0) {
-        return ret;
-    }
-
-    /* Allocate, if necessary at a given offset in the image file */
-    alloc_cluster_offset = start_of_cluster(s, *host_offset);
-    ret = do_alloc_cluster_offset(bs, guest_offset, &alloc_cluster_offset,
-                                  &nb_clusters);
-    if (ret < 0) {
-        goto fail;
-    }
-
-    /* Can't extend contiguous allocation */
-    if (nb_clusters == 0) {
-        *bytes = 0;
-        return 0;
-    }
-
-    /*
-     * Save info needed for meta data update.
-     *
-     * requested_sectors: Number of sectors from the start of the first
-     * newly allocated cluster to the end of the (possibly shortened
-     * before) write request.
-     *
-     * avail_sectors: Number of sectors from the start of the first
-     * newly allocated to the end of the last newly allocated cluster.
-     *
-     * nb_sectors: The number of sectors from the start of the first
-     * newly allocated cluster to the end of the area that the write
-     * request actually writes to (excluding COW at the end)
-     */
-    int requested_sectors =
-        (*bytes + offset_into_cluster(s, guest_offset))
-        >> BDRV_SECTOR_BITS;
-    int avail_sectors = nb_clusters
-                        << (s->cluster_bits - BDRV_SECTOR_BITS);
-    int alloc_n_start = offset_into_cluster(s, guest_offset)
-                        >> BDRV_SECTOR_BITS;
-    int nb_sectors = MIN(requested_sectors, avail_sectors);
-    QCowL2Meta *old_m = *m;
-
-    *m = g_malloc0(sizeof(**m));
-
-    **m = (QCowL2Meta) {
-        .next           = old_m,
-
-        .alloc_offset   = alloc_cluster_offset,
-        .offset         = start_of_cluster(s, guest_offset),
-        .nb_clusters    = nb_clusters,
-        .nb_available   = nb_sectors,
-
-        .cow_start = {
-            .offset     = 0,
-            .nb_sectors = alloc_n_start,
-        },
-        .cow_end = {
-            .offset     = nb_sectors * BDRV_SECTOR_SIZE,
-            .nb_sectors = avail_sectors - nb_sectors,
-        },
-    };
-    qemu_co_queue_init(&(*m)->dependent_requests);
-    QLIST_INSERT_HEAD(&s->cluster_allocs, *m, next_in_flight);
-
-    *host_offset = alloc_cluster_offset + offset_into_cluster(s, guest_offset);
-    *bytes = MIN(*bytes, (nb_sectors * BDRV_SECTOR_SIZE)
-                         - offset_into_cluster(s, guest_offset));
-    assert(*bytes != 0);
-
-    return 1;
-
-fail:
-    if (*m && (*m)->nb_clusters > 0) {
-        QLIST_REMOVE(*m, next_in_flight);
-    }
-    return ret;
-}
-
-/*
- * alloc_cluster_offset
- *
- * For a given offset on the virtual disk, find the cluster offset in qcow2
- * file. If the offset is not found, allocate a new cluster.
- *
- * If the cluster was already allocated, m->nb_clusters is set to 0 and
- * other fields in m are meaningless.
- *
- * If the cluster is newly allocated, m->nb_clusters is set to the number of
- * contiguous clusters that have been allocated. In this case, the other
- * fields of m are valid and contain information about the first allocated
- * cluster.
- *
- * If the request conflicts with another write request in flight, the coroutine
- * is queued and will be reentered when the dependency has completed.
- *
- * Return 0 on success and -errno in error cases
- */
-int qcow2_alloc_cluster_offset(BlockDriverState *bs, uint64_t offset,
-    int n_start, int n_end, int *num, uint64_t *host_offset, QCowL2Meta **m)
-{
-    BDRVQcowState *s = bs->opaque;
-    uint64_t start, remaining;
-    uint64_t cluster_offset;
-    uint64_t cur_bytes;
-    int ret;
-
-    trace_qcow2_alloc_clusters_offset(qemu_coroutine_self(), offset,
-                                      n_start, n_end);
-
-    assert(n_start * BDRV_SECTOR_SIZE == offset_into_cluster(s, offset));
-    offset = start_of_cluster(s, offset);
-
-again:
-    start = offset + (n_start << BDRV_SECTOR_BITS);
-    remaining = (n_end - n_start) << BDRV_SECTOR_BITS;
-    cluster_offset = 0;
-    *host_offset = 0;
-    cur_bytes = 0;
-    *m = NULL;
-
-    while (true) {
-
-        if (!*host_offset) {
-            *host_offset = start_of_cluster(s, cluster_offset);
-        }
-
-        assert(remaining >= cur_bytes);
-
-        start           += cur_bytes;
-        remaining       -= cur_bytes;
-        cluster_offset  += cur_bytes;
-
-        if (remaining == 0) {
-            break;
-        }
-
-        cur_bytes = remaining;
-
-        /*
-         * Now start gathering as many contiguous clusters as possible:
-         *
-         * 1. Check for overlaps with in-flight allocations
-         *
-         *      a) Overlap not in the first cluster -> shorten this request and
-         *         let the caller handle the rest in its next loop iteration.
-         *
-         *      b) Real overlaps of two requests. Yield and restart the search
-         *         for contiguous clusters (the situation could have changed
-         *         while we were sleeping)
-         *
-         *      c) TODO: Request starts in the same cluster as the in-flight
-         *         allocation ends. Shorten the COW of the in-fight allocation,
-         *         set cluster_offset to write to the same cluster and set up
-         *         the right synchronisation between the in-flight request and
-         *         the new one.
-         */
-        ret = handle_dependencies(bs, start, &cur_bytes, m);
-        if (ret == -EAGAIN) {
-            /* Currently handle_dependencies() doesn't yield if we already had
-             * an allocation. If it did, we would have to clean up the L2Meta
-             * structs before starting over. */
-            assert(*m == NULL);
-            goto again;
-        } else if (ret < 0) {
-            return ret;
-        } else if (cur_bytes == 0) {
-            break;
-        } else {
-            /* handle_dependencies() may have decreased cur_bytes (shortened
-             * the allocations below) so that the next dependency is processed
-             * correctly during the next loop iteration. */
-        }
-
-        /*
-         * 2. Count contiguous COPIED clusters.
-         */
-        ret = handle_copied(bs, start, &cluster_offset, &cur_bytes, m);
-        if (ret < 0) {
-            return ret;
-        } else if (ret) {
-            continue;
-        } else if (cur_bytes == 0) {
-            break;
-        }
-
-        /*
-         * 3. If the request still hasn't completed, allocate new clusters,
-         *    considering any cluster_offset of steps 1c or 2.
-         */
-        ret = handle_alloc(bs, start, &cluster_offset, &cur_bytes, m);
-        if (ret < 0) {
-            return ret;
-        } else if (ret) {
-            continue;
-        } else {
-            assert(cur_bytes == 0);
-            break;
-        }
-    }
-
-    *num = (n_end - n_start) - (remaining >> BDRV_SECTOR_BITS);
-    assert(*num > 0);
-    assert(*host_offset != 0);
-
-    return 0;
-}
-
-static int decompress_buffer(uint8_t *out_buf, int out_buf_size,
-                             const uint8_t *buf, int buf_size)
-{
-    z_stream strm1, *strm = &strm1;
-    int ret, out_len;
-
-    memset(strm, 0, sizeof(*strm));
-
-    strm->next_in = (uint8_t *)buf;
-    strm->avail_in = buf_size;
-    strm->next_out = out_buf;
-    strm->avail_out = out_buf_size;
-
-    ret = inflateInit2(strm, -12);
-    if (ret != Z_OK)
-        return -1;
-    ret = inflate(strm, Z_FINISH);
-    out_len = strm->next_out - out_buf;
-    if ((ret != Z_STREAM_END && ret != Z_BUF_ERROR) ||
-        out_len != out_buf_size) {
-        inflateEnd(strm);
-        return -1;
-    }
-    inflateEnd(strm);
-    return 0;
-}
-
-int qcow2_decompress_cluster(BlockDriverState *bs, uint64_t cluster_offset)
-{
-    BDRVQcowState *s = bs->opaque;
-    int ret, csize, nb_csectors, sector_offset;
-    uint64_t coffset;
-
-    coffset = cluster_offset & s->cluster_offset_mask;
-    if (s->cluster_cache_offset != coffset) {
-        nb_csectors = ((cluster_offset >> s->csize_shift) & s->csize_mask) + 1;
-        sector_offset = coffset & 511;
-        csize = nb_csectors * 512 - sector_offset;
-        BLKDBG_EVENT(bs->file, BLKDBG_READ_COMPRESSED);
-        ret = bdrv_read(bs->file, coffset >> 9, s->cluster_data, nb_csectors);
-        if (ret < 0) {
-            return ret;
-        }
-        if (decompress_buffer(s->cluster_cache, s->cluster_size,
-                              s->cluster_data + sector_offset, csize) < 0) {
-            return -EIO;
-        }
-        s->cluster_cache_offset = coffset;
-    }
-    return 0;
-}
-
-/*
- * This discards as many clusters of nb_clusters as possible at once (i.e.
- * all clusters in the same L2 table) and returns the number of discarded
- * clusters.
- */
-static int discard_single_l2(BlockDriverState *bs, uint64_t offset,
-    unsigned int nb_clusters)
-{
-    BDRVQcowState *s = bs->opaque;
-    uint64_t *l2_table;
-    int l2_index;
-    int ret;
-    int i;
-
-    ret = get_cluster_table(bs, offset, &l2_table, &l2_index);
-    if (ret < 0) {
-        return ret;
-    }
-
-    /* Limit nb_clusters to one L2 table */
-    nb_clusters = MIN(nb_clusters, s->l2_size - l2_index);
-
-    for (i = 0; i < nb_clusters; i++) {
-        uint64_t old_offset;
-
-        old_offset = be64_to_cpu(l2_table[l2_index + i]);
-        if ((old_offset & L2E_OFFSET_MASK) == 0) {
-            continue;
-        }
-
-        /* First remove L2 entries */
-        qcow2_cache_entry_mark_dirty(s->l2_table_cache, l2_table);
-        l2_table[l2_index + i] = cpu_to_be64(0);
-
-        /* Then decrease the refcount */
-        qcow2_free_any_clusters(bs, old_offset, 1, QCOW2_DISCARD_REQUEST);
-    }
-
-    ret = qcow2_cache_put(bs, s->l2_table_cache, (void**) &l2_table);
-    if (ret < 0) {
-        return ret;
-    }
-
-    return nb_clusters;
-}
-
-int qcow2_discard_clusters(BlockDriverState *bs, uint64_t offset,
-    int nb_sectors)
-{
-    BDRVQcowState *s = bs->opaque;
-    uint64_t end_offset;
-    unsigned int nb_clusters;
-    int ret;
-
-    end_offset = offset + (nb_sectors << BDRV_SECTOR_BITS);
-
-    /* Round start up and end down */
-    offset = align_offset(offset, s->cluster_size);
-    end_offset &= ~(s->cluster_size - 1);
-
-    if (offset > end_offset) {
-        return 0;
-    }
-
-    nb_clusters = size_to_clusters(s, end_offset - offset);
-
-    s->cache_discards = true;
-
-    /* Each L2 table is handled by its own loop iteration */
-    while (nb_clusters > 0) {
-        ret = discard_single_l2(bs, offset, nb_clusters);
-        if (ret < 0) {
-            goto fail;
-        }
-
-        nb_clusters -= ret;
-        offset += (ret * s->cluster_size);
-    }
-
-    ret = 0;
-fail:
-    s->cache_discards = false;
-    qcow2_process_discards(bs, ret);
-
-    return ret;
-}
-
-/*
- * This zeroes as many clusters of nb_clusters as possible at once (i.e.
- * all clusters in the same L2 table) and returns the number of zeroed
- * clusters.
- */
-static int zero_single_l2(BlockDriverState *bs, uint64_t offset,
-    unsigned int nb_clusters)
-{
-    BDRVQcowState *s = bs->opaque;
-    uint64_t *l2_table;
-    int l2_index;
-    int ret;
-    int i;
-
-    ret = get_cluster_table(bs, offset, &l2_table, &l2_index);
-    if (ret < 0) {
-        return ret;
-    }
-
-    /* Limit nb_clusters to one L2 table */
-    nb_clusters = MIN(nb_clusters, s->l2_size - l2_index);
-
-    for (i = 0; i < nb_clusters; i++) {
-        uint64_t old_offset;
-
-        old_offset = be64_to_cpu(l2_table[l2_index + i]);
-
-        /* Update L2 entries */
-        qcow2_cache_entry_mark_dirty(s->l2_table_cache, l2_table);
-        if (old_offset & QCOW_OFLAG_COMPRESSED) {
-            l2_table[l2_index + i] = cpu_to_be64(QCOW_OFLAG_ZERO);
-            qcow2_free_any_clusters(bs, old_offset, 1, QCOW2_DISCARD_REQUEST);
-        } else {
-            l2_table[l2_index + i] |= cpu_to_be64(QCOW_OFLAG_ZERO);
-        }
-    }
-
-    ret = qcow2_cache_put(bs, s->l2_table_cache, (void**) &l2_table);
-    if (ret < 0) {
-        return ret;
-    }
-
-    return nb_clusters;
-}
-
-int qcow2_zero_clusters(BlockDriverState *bs, uint64_t offset, int nb_sectors)
-{
-    BDRVQcowState *s = bs->opaque;
-    unsigned int nb_clusters;
-    int ret;
-
-    /* The zero flag is only supported by version 3 and newer */
-    if (s->qcow_version < 3) {
-        return -ENOTSUP;
-    }
-
-    /* Each L2 table is handled by its own loop iteration */
-    nb_clusters = size_to_clusters(s, nb_sectors << BDRV_SECTOR_BITS);
-
-    s->cache_discards = true;
-
-    while (nb_clusters > 0) {
-        ret = zero_single_l2(bs, offset, nb_clusters);
-        if (ret < 0) {
-            goto fail;
-        }
-
-        nb_clusters -= ret;
-        offset += (ret * s->cluster_size);
-    }
-
-    ret = 0;
-fail:
-    s->cache_discards = false;
-    qcow2_process_discards(bs, ret);
-
-    return ret;
-}
diff --git a/contrib/qemu/block/qcow2-refcount.c b/contrib/qemu/block/qcow2-refcount.c
deleted file mode 100644
index 1244693f39e..00000000000
--- a/contrib/qemu/block/qcow2-refcount.c
+++ /dev/null
@@ -1,1374 +0,0 @@
-/*
- * Block driver for the QCOW version 2 format
- *
- * Copyright (c) 2004-2006 Fabrice Bellard
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-
-#include "qemu-common.h"
-#include "block/block_int.h"
-#include "block/qcow2.h"
-
-static int64_t alloc_clusters_noref(BlockDriverState *bs, int64_t size);
-static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs,
-                            int64_t offset, int64_t length,
-                            int addend, enum qcow2_discard_type type);
-
-
-/*********************************************************/
-/* refcount handling */
-
-int qcow2_refcount_init(BlockDriverState *bs)
-{
-    BDRVQcowState *s = bs->opaque;
-    int ret, refcount_table_size2, i;
-
-    refcount_table_size2 = s->refcount_table_size * sizeof(uint64_t);
-    s->refcount_table = g_malloc(refcount_table_size2);
-    if (s->refcount_table_size > 0) {
-        BLKDBG_EVENT(bs->file, BLKDBG_REFTABLE_LOAD);
-        ret = bdrv_pread(bs->file, s->refcount_table_offset,
-                         s->refcount_table, refcount_table_size2);
-        if (ret != refcount_table_size2)
-            goto fail;
-        for(i = 0; i < s->refcount_table_size; i++)
-            be64_to_cpus(&s->refcount_table[i]);
-    }
-    return 0;
- fail:
-    return -ENOMEM;
-}
-
-void qcow2_refcount_close(BlockDriverState *bs)
-{
-    BDRVQcowState *s = bs->opaque;
-    g_free(s->refcount_table);
-}
-
-
-static int load_refcount_block(BlockDriverState *bs,
-                               int64_t refcount_block_offset,
-                               void **refcount_block)
-{
-    BDRVQcowState *s = bs->opaque;
-    int ret;
-
-    BLKDBG_EVENT(bs->file, BLKDBG_REFBLOCK_LOAD);
-    ret = qcow2_cache_get(bs, s->refcount_block_cache, refcount_block_offset,
-        refcount_block);
-
-    return ret;
-}
-
-/*
- * Returns the refcount of the cluster given by its index. Any non-negative
- * return value is the refcount of the cluster, negative values are -errno
- * and indicate an error.
- */
-static int get_refcount(BlockDriverState *bs, int64_t cluster_index)
-{
-    BDRVQcowState *s = bs->opaque;
-    int refcount_table_index, block_index;
-    int64_t refcount_block_offset;
-    int ret;
-    uint16_t *refcount_block;
-    uint16_t refcount;
-
-    refcount_table_index = cluster_index >> (s->cluster_bits - REFCOUNT_SHIFT);
-    if (refcount_table_index >= s->refcount_table_size)
-        return 0;
-    refcount_block_offset = s->refcount_table[refcount_table_index];
-    if (!refcount_block_offset)
-        return 0;
-
-    ret = qcow2_cache_get(bs, s->refcount_block_cache, refcount_block_offset,
-        (void**) &refcount_block);
-    if (ret < 0) {
-        return ret;
-    }
-
-    block_index = cluster_index &
-        ((1 << (s->cluster_bits - REFCOUNT_SHIFT)) - 1);
-    refcount = be16_to_cpu(refcount_block[block_index]);
-
-    ret = qcow2_cache_put(bs, s->refcount_block_cache,
-        (void**) &refcount_block);
-    if (ret < 0) {
-        return ret;
-    }
-
-    return refcount;
-}
-
-/*
- * Rounds the refcount table size up to avoid growing the table for each single
- * refcount block that is allocated.
- */
-static unsigned int next_refcount_table_size(BDRVQcowState *s,
-    unsigned int min_size)
-{
-    unsigned int min_clusters = (min_size >> (s->cluster_bits - 3)) + 1;
-    unsigned int refcount_table_clusters =
-        MAX(1, s->refcount_table_size >> (s->cluster_bits - 3));
-
-    while (min_clusters > refcount_table_clusters) {
-        refcount_table_clusters = (refcount_table_clusters * 3 + 1) / 2;
-    }
-
-    return refcount_table_clusters << (s->cluster_bits - 3);
-}
-
-
-/* Checks if two offsets are described by the same refcount block */
-static int in_same_refcount_block(BDRVQcowState *s, uint64_t offset_a,
-    uint64_t offset_b)
-{
-    uint64_t block_a = offset_a >> (2 * s->cluster_bits - REFCOUNT_SHIFT);
-    uint64_t block_b = offset_b >> (2 * s->cluster_bits - REFCOUNT_SHIFT);
-
-    return (block_a == block_b);
-}
-
-/*
- * Loads a refcount block. If it doesn't exist yet, it is allocated first
- * (including growing the refcount table if needed).
- *
- * Returns 0 on success or -errno in error case
- */
-static int alloc_refcount_block(BlockDriverState *bs,
-    int64_t cluster_index, uint16_t **refcount_block)
-{
-    BDRVQcowState *s = bs->opaque;
-    unsigned int refcount_table_index;
-    int ret;
-
-    BLKDBG_EVENT(bs->file, BLKDBG_REFBLOCK_ALLOC);
-
-    /* Find the refcount block for the given cluster */
-    refcount_table_index = cluster_index >> (s->cluster_bits - REFCOUNT_SHIFT);
-
-    if (refcount_table_index < s->refcount_table_size) {
-
-        uint64_t refcount_block_offset =
-            s->refcount_table[refcount_table_index] & REFT_OFFSET_MASK;
-
-        /* If it's already there, we're done */
-        if (refcount_block_offset) {
-             return load_refcount_block(bs, refcount_block_offset,
-                 (void**) refcount_block);
-        }
-    }
-
-    /*
-     * If we came here, we need to allocate something. Something is at least
-     * a cluster for the new refcount block. It may also include a new refcount
-     * table if the old refcount table is too small.
-     *
-     * Note that allocating clusters here needs some special care:
-     *
-     * - We can't use the normal qcow2_alloc_clusters(), it would try to
-     *   increase the refcount and very likely we would end up with an endless
-     *   recursion. Instead we must place the refcount blocks in a way that
-     *   they can describe them themselves.
-     *
-     * - We need to consider that at this point we are inside update_refcounts
-     *   and doing the initial refcount increase. This means that some clusters
-     *   have already been allocated by the caller, but their refcount isn't
-     *   accurate yet. free_cluster_index tells us where this allocation ends
-     *   as long as we don't overwrite it by freeing clusters.
-     *
-     * - alloc_clusters_noref and qcow2_free_clusters may load a different
-     *   refcount block into the cache
-     */
-
-    *refcount_block = NULL;
-
-    /* We write to the refcount table, so we might depend on L2 tables */
-    ret = qcow2_cache_flush(bs, s->l2_table_cache);
-    if (ret < 0) {
-        return ret;
-    }
-
-    /* Allocate the refcount block itself and mark it as used */
-    int64_t new_block = alloc_clusters_noref(bs, s->cluster_size);
-    if (new_block < 0) {
-        return new_block;
-    }
-
-#ifdef DEBUG_ALLOC2
-    fprintf(stderr, "qcow2: Allocate refcount block %d for %" PRIx64
-        " at %" PRIx64 "\n",
-        refcount_table_index, cluster_index << s->cluster_bits, new_block);
-#endif
-
-    if (in_same_refcount_block(s, new_block, cluster_index << s->cluster_bits)) {
-        /* Zero the new refcount block before updating it */
-        ret = qcow2_cache_get_empty(bs, s->refcount_block_cache, new_block,
-            (void**) refcount_block);
-        if (ret < 0) {
-            goto fail_block;
-        }
-
-        memset(*refcount_block, 0, s->cluster_size);
-
-        /* The block describes itself, need to update the cache */
-        int block_index = (new_block >> s->cluster_bits) &
-            ((1 << (s->cluster_bits - REFCOUNT_SHIFT)) - 1);
-        (*refcount_block)[block_index] = cpu_to_be16(1);
-    } else {
-        /* Described somewhere else. This can recurse at most twice before we
-         * arrive at a block that describes itself. */
-        ret = update_refcount(bs, new_block, s->cluster_size, 1,
-                              QCOW2_DISCARD_NEVER);
-        if (ret < 0) {
-            goto fail_block;
-        }
-
-        ret = qcow2_cache_flush(bs, s->refcount_block_cache);
-        if (ret < 0) {
-            goto fail_block;
-        }
-
-        /* Initialize the new refcount block only after updating its refcount,
-         * update_refcount uses the refcount cache itself */
-        ret = qcow2_cache_get_empty(bs, s->refcount_block_cache, new_block,
-            (void**) refcount_block);
-        if (ret < 0) {
-            goto fail_block;
-        }
-
-        memset(*refcount_block, 0, s->cluster_size);
-    }
-
-    /* Now the new refcount block needs to be written to disk */
-    BLKDBG_EVENT(bs->file, BLKDBG_REFBLOCK_ALLOC_WRITE);
-    qcow2_cache_entry_mark_dirty(s->refcount_block_cache, *refcount_block);
-    ret = qcow2_cache_flush(bs, s->refcount_block_cache);
-    if (ret < 0) {
-        goto fail_block;
-    }
-
-    /* If the refcount table is big enough, just hook the block up there */
-    if (refcount_table_index < s->refcount_table_size) {
-        uint64_t data64 = cpu_to_be64(new_block);
-        BLKDBG_EVENT(bs->file, BLKDBG_REFBLOCK_ALLOC_HOOKUP);
-        ret = bdrv_pwrite_sync(bs->file,
-            s->refcount_table_offset + refcount_table_index * sizeof(uint64_t),
-            &data64, sizeof(data64));
-        if (ret < 0) {
-            goto fail_block;
-        }
-
-        s->refcount_table[refcount_table_index] = new_block;
-        return 0;
-    }
-
-    ret = qcow2_cache_put(bs, s->refcount_block_cache, (void**) refcount_block);
-    if (ret < 0) {
-        goto fail_block;
-    }
-
-    /*
-     * If we come here, we need to grow the refcount table. Again, a new
-     * refcount table needs some space and we can't simply allocate to avoid
-     * endless recursion.
-     *
-     * Therefore let's grab new refcount blocks at the end of the image, which
-     * will describe themselves and the new refcount table. This way we can
-     * reference them only in the new table and do the switch to the new
-     * refcount table at once without producing an inconsistent state in
-     * between.
-     */
-    BLKDBG_EVENT(bs->file, BLKDBG_REFTABLE_GROW);
-
-    /* Calculate the number of refcount blocks needed so far */
-    uint64_t refcount_block_clusters = 1 << (s->cluster_bits - REFCOUNT_SHIFT);
-    uint64_t blocks_used = (s->free_cluster_index +
-        refcount_block_clusters - 1) / refcount_block_clusters;
-
-    /* And now we need at least one block more for the new metadata */
-    uint64_t table_size = next_refcount_table_size(s, blocks_used + 1);
-    uint64_t last_table_size;
-    uint64_t blocks_clusters;
-    do {
-        uint64_t table_clusters =
-            size_to_clusters(s, table_size * sizeof(uint64_t));
-        blocks_clusters = 1 +
-            ((table_clusters + refcount_block_clusters - 1)
-            / refcount_block_clusters);
-        uint64_t meta_clusters = table_clusters + blocks_clusters;
-
-        last_table_size = table_size;
-        table_size = next_refcount_table_size(s, blocks_used +
-            ((meta_clusters + refcount_block_clusters - 1)
-            / refcount_block_clusters));
-
-    } while (last_table_size != table_size);
-
-#ifdef DEBUG_ALLOC2
-    fprintf(stderr, "qcow2: Grow refcount table %" PRId32 " => %" PRId64 "\n",
-        s->refcount_table_size, table_size);
-#endif
-
-    /* Create the new refcount table and blocks */
-    uint64_t meta_offset = (blocks_used * refcount_block_clusters) *
-        s->cluster_size;
-    uint64_t table_offset = meta_offset + blocks_clusters * s->cluster_size;
-    uint16_t *new_blocks = g_malloc0(blocks_clusters * s->cluster_size);
-    uint64_t *new_table = g_malloc0(table_size * sizeof(uint64_t));
-
-    assert(meta_offset >= (s->free_cluster_index * s->cluster_size));
-
-    /* Fill the new refcount table */
-    memcpy(new_table, s->refcount_table,
-        s->refcount_table_size * sizeof(uint64_t));
-    new_table[refcount_table_index] = new_block;
-
-    int i;
-    for (i = 0; i < blocks_clusters; i++) {
-        new_table[blocks_used + i] = meta_offset + (i * s->cluster_size);
-    }
-
-    /* Fill the refcount blocks */
-    uint64_t table_clusters = size_to_clusters(s, table_size * sizeof(uint64_t));
-    int block = 0;
-    for (i = 0; i < table_clusters + blocks_clusters; i++) {
-        new_blocks[block++] = cpu_to_be16(1);
-    }
-
-    /* Write refcount blocks to disk */
-    BLKDBG_EVENT(bs->file, BLKDBG_REFBLOCK_ALLOC_WRITE_BLOCKS);
-    ret = bdrv_pwrite_sync(bs->file, meta_offset, new_blocks,
-        blocks_clusters * s->cluster_size);
-    g_free(new_blocks);
-    if (ret < 0) {
-        goto fail_table;
-    }
-
-    /* Write refcount table to disk */
-    for(i = 0; i < table_size; i++) {
-        cpu_to_be64s(&new_table[i]);
-    }
-
-    BLKDBG_EVENT(bs->file, BLKDBG_REFBLOCK_ALLOC_WRITE_TABLE);
-    ret = bdrv_pwrite_sync(bs->file, table_offset, new_table,
-        table_size * sizeof(uint64_t));
-    if (ret < 0) {
-        goto fail_table;
-    }
-
-    for(i = 0; i < table_size; i++) {
-        be64_to_cpus(&new_table[i]);
-    }
-
-    /* Hook up the new refcount table in the qcow2 header */
-    uint8_t data[12];
-    cpu_to_be64w((uint64_t*)data, table_offset);
-    cpu_to_be32w((uint32_t*)(data + 8), table_clusters);
-    BLKDBG_EVENT(bs->file, BLKDBG_REFBLOCK_ALLOC_SWITCH_TABLE);
-    ret = bdrv_pwrite_sync(bs->file, offsetof(QCowHeader, refcount_table_offset),
-        data, sizeof(data));
-    if (ret < 0) {
-        goto fail_table;
-    }
-
-    /* And switch it in memory */
-    uint64_t old_table_offset = s->refcount_table_offset;
-    uint64_t old_table_size = s->refcount_table_size;
-
-    g_free(s->refcount_table);
-    s->refcount_table = new_table;
-    s->refcount_table_size = table_size;
-    s->refcount_table_offset = table_offset;
-
-    /* Free old table. Remember, we must not change free_cluster_index */
-    uint64_t old_free_cluster_index = s->free_cluster_index;
-    qcow2_free_clusters(bs, old_table_offset, old_table_size * sizeof(uint64_t),
-                        QCOW2_DISCARD_OTHER);
-    s->free_cluster_index = old_free_cluster_index;
-
-    ret = load_refcount_block(bs, new_block, (void**) refcount_block);
-    if (ret < 0) {
-        return ret;
-    }
-
-    return 0;
-
-fail_table:
-    g_free(new_table);
-fail_block:
-    if (*refcount_block != NULL) {
-        qcow2_cache_put(bs, s->refcount_block_cache, (void**) refcount_block);
-    }
-    return ret;
-}
-
-void qcow2_process_discards(BlockDriverState *bs, int ret)
-{
-    BDRVQcowState *s = bs->opaque;
-    Qcow2DiscardRegion *d, *next;
-
-    QTAILQ_FOREACH_SAFE(d, &s->discards, next, next) {
-        QTAILQ_REMOVE(&s->discards, d, next);
-
-        /* Discard is optional, ignore the return value */
-        if (ret >= 0) {
-            bdrv_discard(bs->file,
-                         d->offset >> BDRV_SECTOR_BITS,
-                         d->bytes >> BDRV_SECTOR_BITS);
-        }
-
-        g_free(d);
-    }
-}
-
-static void update_refcount_discard(BlockDriverState *bs,
-                                    uint64_t offset, uint64_t length)
-{
-    BDRVQcowState *s = bs->opaque;
-    Qcow2DiscardRegion *d, *p, *next;
-
-    QTAILQ_FOREACH(d, &s->discards, next) {
-        uint64_t new_start = MIN(offset, d->offset);
-        uint64_t new_end = MAX(offset + length, d->offset + d->bytes);
-
-        if (new_end - new_start <= length + d->bytes) {
-            /* There can't be any overlap, areas ending up here have no
-             * references any more and therefore shouldn't get freed another
-             * time. */
-            assert(d->bytes + length == new_end - new_start);
-            d->offset = new_start;
-            d->bytes = new_end - new_start;
-            goto found;
-        }
-    }
-
-    d = g_malloc(sizeof(*d));
-    *d = (Qcow2DiscardRegion) {
-        .bs     = bs,
-        .offset = offset,
-        .bytes  = length,
-    };
-    QTAILQ_INSERT_TAIL(&s->discards, d, next);
-
-found:
-    /* Merge discard requests if they are adjacent now */
-    QTAILQ_FOREACH_SAFE(p, &s->discards, next, next) {
-        if (p == d
-            || p->offset > d->offset + d->bytes
-            || d->offset > p->offset + p->bytes)
-        {
-            continue;
-        }
-
-        /* Still no overlap possible */
-        assert(p->offset == d->offset + d->bytes
-            || d->offset == p->offset + p->bytes);
-
-        QTAILQ_REMOVE(&s->discards, p, next);
-        d->offset = MIN(d->offset, p->offset);
-        d->bytes += p->bytes;
-    }
-}
-
-/* XXX: cache several refcount block clusters ? */
-static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs,
-    int64_t offset, int64_t length, int addend, enum qcow2_discard_type type)
-{
-    BDRVQcowState *s = bs->opaque;
-    int64_t start, last, cluster_offset;
-    uint16_t *refcount_block = NULL;
-    int64_t old_table_index = -1;
-    int ret;
-
-#ifdef DEBUG_ALLOC2
-    fprintf(stderr, "update_refcount: offset=%" PRId64 " size=%" PRId64 " addend=%d\n",
-           offset, length, addend);
-#endif
-    if (length < 0) {
-        return -EINVAL;
-    } else if (length == 0) {
-        return 0;
-    }
-
-    if (addend < 0) {
-        qcow2_cache_set_dependency(bs, s->refcount_block_cache,
-            s->l2_table_cache);
-    }
-
-    start = offset & ~(s->cluster_size - 1);
-    last = (offset + length - 1) & ~(s->cluster_size - 1);
-    for(cluster_offset = start; cluster_offset <= last;
-        cluster_offset += s->cluster_size)
-    {
-        int block_index, refcount;
-        int64_t cluster_index = cluster_offset >> s->cluster_bits;
-        int64_t table_index =
-            cluster_index >> (s->cluster_bits - REFCOUNT_SHIFT);
-
-        /* Load the refcount block and allocate it if needed */
-        if (table_index != old_table_index) {
-            if (refcount_block) {
-                ret = qcow2_cache_put(bs, s->refcount_block_cache,
-                    (void**) &refcount_block);
-                if (ret < 0) {
-                    goto fail;
-                }
-            }
-
-            ret = alloc_refcount_block(bs, cluster_index, &refcount_block);
-            if (ret < 0) {
-                goto fail;
-            }
-        }
-        old_table_index = table_index;
-
-        qcow2_cache_entry_mark_dirty(s->refcount_block_cache, refcount_block);
-
-        /* we can update the count and save it */
-        block_index = cluster_index &
-            ((1 << (s->cluster_bits - REFCOUNT_SHIFT)) - 1);
-
-        refcount = be16_to_cpu(refcount_block[block_index]);
-        refcount += addend;
-        if (refcount < 0 || refcount > 0xffff) {
-            ret = -EINVAL;
-            goto fail;
-        }
-        if (refcount == 0 && cluster_index < s->free_cluster_index) {
-            s->free_cluster_index = cluster_index;
-        }
-        refcount_block[block_index] = cpu_to_be16(refcount);
-
-        if (refcount == 0 && s->discard_passthrough[type]) {
-            update_refcount_discard(bs, cluster_offset, s->cluster_size);
-        }
-    }
-
-    ret = 0;
-fail:
-    if (!s->cache_discards) {
-        qcow2_process_discards(bs, ret);
-    }
-
-    /* Write last changed block to disk */
-    if (refcount_block) {
-        int wret;
-        wret = qcow2_cache_put(bs, s->refcount_block_cache,
-            (void**) &refcount_block);
-        if (wret < 0) {
-            return ret < 0 ? ret : wret;
-        }
-    }
-
-    /*
-     * Try do undo any updates if an error is returned (This may succeed in
-     * some cases like ENOSPC for allocating a new refcount block)
-     */
-    if (ret < 0) {
-        int dummy;
-        dummy = update_refcount(bs, offset, cluster_offset - offset, -addend,
-                                QCOW2_DISCARD_NEVER);
-        (void)dummy;
-    }
-
-    return ret;
-}
-
-/*
- * Increases or decreases the refcount of a given cluster by one.
- * addend must be 1 or -1.
- *
- * If the return value is non-negative, it is the new refcount of the cluster.
- * If it is negative, it is -errno and indicates an error.
- */
-static int update_cluster_refcount(BlockDriverState *bs,
-                                   int64_t cluster_index,
-                                   int addend,
-                                   enum qcow2_discard_type type)
-{
-    BDRVQcowState *s = bs->opaque;
-    int ret;
-
-    ret = update_refcount(bs, cluster_index << s->cluster_bits, 1, addend,
-                          type);
-    if (ret < 0) {
-        return ret;
-    }
-
-    return get_refcount(bs, cluster_index);
-}
-
-
-
-/*********************************************************/
-/* cluster allocation functions */
-
-
-
-/* return < 0 if error */
-static int64_t alloc_clusters_noref(BlockDriverState *bs, int64_t size)
-{
-    BDRVQcowState *s = bs->opaque;
-    int i, nb_clusters, refcount;
-
-    nb_clusters = size_to_clusters(s, size);
-retry:
-    for(i = 0; i < nb_clusters; i++) {
-        int64_t next_cluster_index = s->free_cluster_index++;
-        refcount = get_refcount(bs, next_cluster_index);
-
-        if (refcount < 0) {
-            return refcount;
-        } else if (refcount != 0) {
-            goto retry;
-        }
-    }
-#ifdef DEBUG_ALLOC2
-    fprintf(stderr, "alloc_clusters: size=%" PRId64 " -> %" PRId64 "\n",
-            size,
-            (s->free_cluster_index - nb_clusters) << s->cluster_bits);
-#endif
-    return (s->free_cluster_index - nb_clusters) << s->cluster_bits;
-}
-
-int64_t qcow2_alloc_clusters(BlockDriverState *bs, int64_t size)
-{
-    int64_t offset;
-    int ret;
-
-    BLKDBG_EVENT(bs->file, BLKDBG_CLUSTER_ALLOC);
-    offset = alloc_clusters_noref(bs, size);
-    if (offset < 0) {
-        return offset;
-    }
-
-    ret = update_refcount(bs, offset, size, 1, QCOW2_DISCARD_NEVER);
-    if (ret < 0) {
-        return ret;
-    }
-
-    return offset;
-}
-
-int qcow2_alloc_clusters_at(BlockDriverState *bs, uint64_t offset,
-    int nb_clusters)
-{
-    BDRVQcowState *s = bs->opaque;
-    uint64_t cluster_index;
-    uint64_t old_free_cluster_index;
-    int i, refcount, ret;
-
-    /* Check how many clusters there are free */
-    cluster_index = offset >> s->cluster_bits;
-    for(i = 0; i < nb_clusters; i++) {
-        refcount = get_refcount(bs, cluster_index++);
-
-        if (refcount < 0) {
-            return refcount;
-        } else if (refcount != 0) {
-            break;
-        }
-    }
-
-    /* And then allocate them */
-    old_free_cluster_index = s->free_cluster_index;
-    s->free_cluster_index = cluster_index + i;
-
-    ret = update_refcount(bs, offset, i << s->cluster_bits, 1,
-                          QCOW2_DISCARD_NEVER);
-    if (ret < 0) {
-        return ret;
-    }
-
-    s->free_cluster_index = old_free_cluster_index;
-
-    return i;
-}
-
-/* only used to allocate compressed sectors. We try to allocate
-   contiguous sectors. size must be <= cluster_size */
-int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)
-{
-    BDRVQcowState *s = bs->opaque;
-    int64_t offset, cluster_offset;
-    int free_in_cluster;
-
-    BLKDBG_EVENT(bs->file, BLKDBG_CLUSTER_ALLOC_BYTES);
-    assert(size > 0 && size <= s->cluster_size);
-    if (s->free_byte_offset == 0) {
-        offset = qcow2_alloc_clusters(bs, s->cluster_size);
-        if (offset < 0) {
-            return offset;
-        }
-        s->free_byte_offset = offset;
-    }
- redo:
-    free_in_cluster = s->cluster_size -
-        (s->free_byte_offset & (s->cluster_size - 1));
-    if (size <= free_in_cluster) {
-        /* enough space in current cluster */
-        offset = s->free_byte_offset;
-        s->free_byte_offset += size;
-        free_in_cluster -= size;
-        if (free_in_cluster == 0)
-            s->free_byte_offset = 0;
-        if ((offset & (s->cluster_size - 1)) != 0)
-            update_cluster_refcount(bs, offset >> s->cluster_bits, 1,
-                                    QCOW2_DISCARD_NEVER);
-    } else {
-        offset = qcow2_alloc_clusters(bs, s->cluster_size);
-        if (offset < 0) {
-            return offset;
-        }
-        cluster_offset = s->free_byte_offset & ~(s->cluster_size - 1);
-        if ((cluster_offset + s->cluster_size) == offset) {
-            /* we are lucky: contiguous data */
-            offset = s->free_byte_offset;
-            update_cluster_refcount(bs, offset >> s->cluster_bits, 1,
-                                    QCOW2_DISCARD_NEVER);
-            s->free_byte_offset += size;
-        } else {
-            s->free_byte_offset = offset;
-            goto redo;
-        }
-    }
-
-    /* The cluster refcount was incremented, either by qcow2_alloc_clusters()
-     * or explicitly by update_cluster_refcount().  Refcount blocks must be
-     * flushed before the caller's L2 table updates.
-     */
-    qcow2_cache_set_dependency(bs, s->l2_table_cache, s->refcount_block_cache);
-    return offset;
-}
-
-void qcow2_free_clusters(BlockDriverState *bs,
-                          int64_t offset, int64_t size,
-                          enum qcow2_discard_type type)
-{
-    int ret;
-
-    BLKDBG_EVENT(bs->file, BLKDBG_CLUSTER_FREE);
-    ret = update_refcount(bs, offset, size, -1, type);
-    if (ret < 0) {
-        fprintf(stderr, "qcow2_free_clusters failed: %s\n", strerror(-ret));
-        /* TODO Remember the clusters to free them later and avoid leaking */
-    }
-}
-
-/*
- * Free a cluster using its L2 entry (handles clusters of all types, e.g.
- * normal cluster, compressed cluster, etc.)
- */
-void qcow2_free_any_clusters(BlockDriverState *bs, uint64_t l2_entry,
-                             int nb_clusters, enum qcow2_discard_type type)
-{
-    BDRVQcowState *s = bs->opaque;
-
-    switch (qcow2_get_cluster_type(l2_entry)) {
-    case QCOW2_CLUSTER_COMPRESSED:
-        {
-            int nb_csectors;
-            nb_csectors = ((l2_entry >> s->csize_shift) &
-                           s->csize_mask) + 1;
-            qcow2_free_clusters(bs,
-                (l2_entry & s->cluster_offset_mask) & ~511,
-                nb_csectors * 512, type);
-        }
-        break;
-    case QCOW2_CLUSTER_NORMAL:
-        qcow2_free_clusters(bs, l2_entry & L2E_OFFSET_MASK,
-                            nb_clusters << s->cluster_bits, type);
-        break;
-    case QCOW2_CLUSTER_UNALLOCATED:
-    case QCOW2_CLUSTER_ZERO:
-        break;
-    default:
-        abort();
-    }
-}
-
-
-
-/*********************************************************/
-/* snapshots and image creation */
-
-
-
-/* update the refcounts of snapshots and the copied flag */
-int qcow2_update_snapshot_refcount(BlockDriverState *bs,
-    int64_t l1_table_offset, int l1_size, int addend)
-{
-    BDRVQcowState *s = bs->opaque;
-    uint64_t *l1_table, *l2_table, l2_offset, offset, l1_size2, l1_allocated;
-    int64_t old_offset, old_l2_offset;
-    int i, j, l1_modified = 0, nb_csectors, refcount;
-    int ret;
-
-    l2_table = NULL;
-    l1_table = NULL;
-    l1_size2 = l1_size * sizeof(uint64_t);
-
-    s->cache_discards = true;
-
-    /* WARNING: qcow2_snapshot_goto relies on this function not using the
-     * l1_table_offset when it is the current s->l1_table_offset! Be careful
-     * when changing this! */
-    if (l1_table_offset != s->l1_table_offset) {
-        l1_table = g_malloc0(align_offset(l1_size2, 512));
-        l1_allocated = 1;
-
-        ret = bdrv_pread(bs->file, l1_table_offset, l1_table, l1_size2);
-        if (ret < 0) {
-            goto fail;
-        }
-
-        for(i = 0;i < l1_size; i++)
-            be64_to_cpus(&l1_table[i]);
-    } else {
-        assert(l1_size == s->l1_size);
-        l1_table = s->l1_table;
-        l1_allocated = 0;
-    }
-
-    for(i = 0; i < l1_size; i++) {
-        l2_offset = l1_table[i];
-        if (l2_offset) {
-            old_l2_offset = l2_offset;
-            l2_offset &= L1E_OFFSET_MASK;
-
-            ret = qcow2_cache_get(bs, s->l2_table_cache, l2_offset,
-                (void**) &l2_table);
-            if (ret < 0) {
-                goto fail;
-            }
-
-            for(j = 0; j < s->l2_size; j++) {
-                offset = be64_to_cpu(l2_table[j]);
-                if (offset != 0) {
-                    old_offset = offset;
-                    offset &= ~QCOW_OFLAG_COPIED;
-                    if (offset & QCOW_OFLAG_COMPRESSED) {
-                        nb_csectors = ((offset >> s->csize_shift) &
-                                       s->csize_mask) + 1;
-                        if (addend != 0) {
-                            int ret;
-                            ret = update_refcount(bs,
-                                (offset & s->cluster_offset_mask) & ~511,
-                                nb_csectors * 512, addend,
-                                QCOW2_DISCARD_SNAPSHOT);
-                            if (ret < 0) {
-                                goto fail;
-                            }
-                        }
-                        /* compressed clusters are never modified */
-                        refcount = 2;
-                    } else {
-                        uint64_t cluster_index = (offset & L2E_OFFSET_MASK) >> s->cluster_bits;
-                        if (addend != 0) {
-                            refcount = update_cluster_refcount(bs, cluster_index, addend,
-                                                               QCOW2_DISCARD_SNAPSHOT);
-                        } else {
-                            refcount = get_refcount(bs, cluster_index);
-                        }
-
-                        if (refcount < 0) {
-                            ret = refcount;
-                            goto fail;
-                        }
-                    }
-
-                    if (refcount == 1) {
-                        offset |= QCOW_OFLAG_COPIED;
-                    }
-                    if (offset != old_offset) {
-                        if (addend > 0) {
-                            qcow2_cache_set_dependency(bs, s->l2_table_cache,
-                                s->refcount_block_cache);
-                        }
-                        l2_table[j] = cpu_to_be64(offset);
-                        qcow2_cache_entry_mark_dirty(s->l2_table_cache, l2_table);
-                    }
-                }
-            }
-
-            ret = qcow2_cache_put(bs, s->l2_table_cache, (void**) &l2_table);
-            if (ret < 0) {
-                goto fail;
-            }
-
-
-            if (addend != 0) {
-                refcount = update_cluster_refcount(bs, l2_offset >> s->cluster_bits, addend,
-                                                   QCOW2_DISCARD_SNAPSHOT);
-            } else {
-                refcount = get_refcount(bs, l2_offset >> s->cluster_bits);
-            }
-            if (refcount < 0) {
-                ret = refcount;
-                goto fail;
-            } else if (refcount == 1) {
-                l2_offset |= QCOW_OFLAG_COPIED;
-            }
-            if (l2_offset != old_l2_offset) {
-                l1_table[i] = l2_offset;
-                l1_modified = 1;
-            }
-        }
-    }
-
-    ret = bdrv_flush(bs);
-fail:
-    if (l2_table) {
-        qcow2_cache_put(bs, s->l2_table_cache, (void**) &l2_table);
-    }
-
-    s->cache_discards = false;
-    qcow2_process_discards(bs, ret);
-
-    /* Update L1 only if it isn't deleted anyway (addend = -1) */
-    if (ret == 0 && addend >= 0 && l1_modified) {
-        for (i = 0; i < l1_size; i++) {
-            cpu_to_be64s(&l1_table[i]);
-        }
-
-        ret = bdrv_pwrite_sync(bs->file, l1_table_offset, l1_table, l1_size2);
-
-        for (i = 0; i < l1_size; i++) {
-            be64_to_cpus(&l1_table[i]);
-        }
-    }
-    if (l1_allocated)
-        g_free(l1_table);
-    return ret;
-}
-
-
-
-
-/*********************************************************/
-/* refcount checking functions */
-
-
-
-/*
- * Increases the refcount for a range of clusters in a given refcount table.
- * This is used to construct a temporary refcount table out of L1 and L2 tables
- * which can be compared the the refcount table saved in the image.
- *
- * Modifies the number of errors in res.
- */
-static void inc_refcounts(BlockDriverState *bs,
-                          BdrvCheckResult *res,
-                          uint16_t *refcount_table,
-                          int refcount_table_size,
-                          int64_t offset, int64_t size)
-{
-    BDRVQcowState *s = bs->opaque;
-    int64_t start, last, cluster_offset;
-    int k;
-
-    if (size <= 0)
-        return;
-
-    start = offset & ~(s->cluster_size - 1);
-    last = (offset + size - 1) & ~(s->cluster_size - 1);
-    for(cluster_offset = start; cluster_offset <= last;
-        cluster_offset += s->cluster_size) {
-        k = cluster_offset >> s->cluster_bits;
-        if (k < 0) {
-            fprintf(stderr, "ERROR: invalid cluster offset=0x%" PRIx64 "\n",
-                cluster_offset);
-            res->corruptions++;
-        } else if (k >= refcount_table_size) {
-            fprintf(stderr, "Warning: cluster offset=0x%" PRIx64 " is after "
-                "the end of the image file, can't properly check refcounts.\n",
-                cluster_offset);
-            res->check_errors++;
-        } else {
-            if (++refcount_table[k] == 0) {
-                fprintf(stderr, "ERROR: overflow cluster offset=0x%" PRIx64
-                    "\n", cluster_offset);
-                res->corruptions++;
-            }
-        }
-    }
-}
-
-/* Flags for check_refcounts_l1() and check_refcounts_l2() */
-enum {
-    CHECK_OFLAG_COPIED = 0x1,   /* check QCOW_OFLAG_COPIED matches refcount */
-    CHECK_FRAG_INFO = 0x2,      /* update BlockFragInfo counters */
-};
-
-/*
- * Increases the refcount in the given refcount table for the all clusters
- * referenced in the L2 table. While doing so, performs some checks on L2
- * entries.
- *
- * Returns the number of errors found by the checks or -errno if an internal
- * error occurred.
- */
-static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res,
-    uint16_t *refcount_table, int refcount_table_size, int64_t l2_offset,
-    int flags)
-{
-    BDRVQcowState *s = bs->opaque;
-    uint64_t *l2_table, l2_entry;
-    uint64_t next_contiguous_offset = 0;
-    int i, l2_size, nb_csectors, refcount;
-
-    /* Read L2 table from disk */
-    l2_size = s->l2_size * sizeof(uint64_t);
-    l2_table = g_malloc(l2_size);
-
-    if (bdrv_pread(bs->file, l2_offset, l2_table, l2_size) != l2_size)
-        goto fail;
-
-    /* Do the actual checks */
-    for(i = 0; i < s->l2_size; i++) {
-        l2_entry = be64_to_cpu(l2_table[i]);
-
-        switch (qcow2_get_cluster_type(l2_entry)) {
-        case QCOW2_CLUSTER_COMPRESSED:
-            /* Compressed clusters don't have QCOW_OFLAG_COPIED */
-            if (l2_entry & QCOW_OFLAG_COPIED) {
-                fprintf(stderr, "ERROR: cluster %" PRId64 ": "
-                    "copied flag must never be set for compressed "
-                    "clusters\n", l2_entry >> s->cluster_bits);
-                l2_entry &= ~QCOW_OFLAG_COPIED;
-                res->corruptions++;
-            }
-
-            /* Mark cluster as used */
-            nb_csectors = ((l2_entry >> s->csize_shift) &
-                           s->csize_mask) + 1;
-            l2_entry &= s->cluster_offset_mask;
-            inc_refcounts(bs, res, refcount_table, refcount_table_size,
-                l2_entry & ~511, nb_csectors * 512);
-
-            if (flags & CHECK_FRAG_INFO) {
-                res->bfi.allocated_clusters++;
-                res->bfi.compressed_clusters++;
-
-                /* Compressed clusters are fragmented by nature.  Since they
-                 * take up sub-sector space but we only have sector granularity
-                 * I/O we need to re-read the same sectors even for adjacent
-                 * compressed clusters.
-                 */
-                res->bfi.fragmented_clusters++;
-            }
-            break;
-
-        case QCOW2_CLUSTER_ZERO:
-            if ((l2_entry & L2E_OFFSET_MASK) == 0) {
-                break;
-            }
-            /* fall through */
-
-        case QCOW2_CLUSTER_NORMAL:
-        {
-            /* QCOW_OFLAG_COPIED must be set iff refcount == 1 */
-            uint64_t offset = l2_entry & L2E_OFFSET_MASK;
-
-            if (flags & CHECK_OFLAG_COPIED) {
-                refcount = get_refcount(bs, offset >> s->cluster_bits);
-                if (refcount < 0) {
-                    fprintf(stderr, "Can't get refcount for offset %"
-                        PRIx64 ": %s\n", l2_entry, strerror(-refcount));
-                    goto fail;
-                }
-                if ((refcount == 1) != ((l2_entry & QCOW_OFLAG_COPIED) != 0)) {
-                    fprintf(stderr, "ERROR OFLAG_COPIED: offset=%"
-                        PRIx64 " refcount=%d\n", l2_entry, refcount);
-                    res->corruptions++;
-                }
-            }
-
-            if (flags & CHECK_FRAG_INFO) {
-                res->bfi.allocated_clusters++;
-                if (next_contiguous_offset &&
-                    offset != next_contiguous_offset) {
-                    res->bfi.fragmented_clusters++;
-                }
-                next_contiguous_offset = offset + s->cluster_size;
-            }
-
-            /* Mark cluster as used */
-            inc_refcounts(bs, res, refcount_table,refcount_table_size,
-                offset, s->cluster_size);
-
-            /* Correct offsets are cluster aligned */
-            if (offset & (s->cluster_size - 1)) {
-                fprintf(stderr, "ERROR offset=%" PRIx64 ": Cluster is not "
-                    "properly aligned; L2 entry corrupted.\n", offset);
-                res->corruptions++;
-            }
-            break;
-        }
-
-        case QCOW2_CLUSTER_UNALLOCATED:
-            break;
-
-        default:
-            abort();
-        }
-    }
-
-    g_free(l2_table);
-    return 0;
-
-fail:
-    fprintf(stderr, "ERROR: I/O error in check_refcounts_l2\n");
-    g_free(l2_table);
-    return -EIO;
-}
-
-/*
- * Increases the refcount for the L1 table, its L2 tables and all referenced
- * clusters in the given refcount table. While doing so, performs some checks
- * on L1 and L2 entries.
- *
- * Returns the number of errors found by the checks or -errno if an internal
- * error occurred.
- */
-static int check_refcounts_l1(BlockDriverState *bs,
-                              BdrvCheckResult *res,
-                              uint16_t *refcount_table,
-                              int refcount_table_size,
-                              int64_t l1_table_offset, int l1_size,
-                              int flags)
-{
-    BDRVQcowState *s = bs->opaque;
-    uint64_t *l1_table, l2_offset, l1_size2;
-    int i, refcount, ret;
-
-    l1_size2 = l1_size * sizeof(uint64_t);
-
-    /* Mark L1 table as used */
-    inc_refcounts(bs, res, refcount_table, refcount_table_size,
-        l1_table_offset, l1_size2);
-
-    /* Read L1 table entries from disk */
-    if (l1_size2 == 0) {
-        l1_table = NULL;
-    } else {
-        l1_table = g_malloc(l1_size2);
-        if (bdrv_pread(bs->file, l1_table_offset,
-                       l1_table, l1_size2) != l1_size2)
-            goto fail;
-        for(i = 0;i < l1_size; i++)
-            be64_to_cpus(&l1_table[i]);
-    }
-
-    /* Do the actual checks */
-    for(i = 0; i < l1_size; i++) {
-        l2_offset = l1_table[i];
-        if (l2_offset) {
-            /* QCOW_OFLAG_COPIED must be set iff refcount == 1 */
-            if (flags & CHECK_OFLAG_COPIED) {
-                refcount = get_refcount(bs, (l2_offset & ~QCOW_OFLAG_COPIED)
-                    >> s->cluster_bits);
-                if (refcount < 0) {
-                    fprintf(stderr, "Can't get refcount for l2_offset %"
-                        PRIx64 ": %s\n", l2_offset, strerror(-refcount));
-                    goto fail;
-                }
-                if ((refcount == 1) != ((l2_offset & QCOW_OFLAG_COPIED) != 0)) {
-                    fprintf(stderr, "ERROR OFLAG_COPIED: l2_offset=%" PRIx64
-                        " refcount=%d\n", l2_offset, refcount);
-                    res->corruptions++;
-                }
-            }
-
-            /* Mark L2 table as used */
-            l2_offset &= L1E_OFFSET_MASK;
-            inc_refcounts(bs, res, refcount_table, refcount_table_size,
-                l2_offset, s->cluster_size);
-
-            /* L2 tables are cluster aligned */
-            if (l2_offset & (s->cluster_size - 1)) {
-                fprintf(stderr, "ERROR l2_offset=%" PRIx64 ": Table is not "
-                    "cluster aligned; L1 entry corrupted\n", l2_offset);
-                res->corruptions++;
-            }
-
-            /* Process and check L2 entries */
-            ret = check_refcounts_l2(bs, res, refcount_table,
-                                     refcount_table_size, l2_offset, flags);
-            if (ret < 0) {
-                goto fail;
-            }
-        }
-    }
-    g_free(l1_table);
-    return 0;
-
-fail:
-    fprintf(stderr, "ERROR: I/O error in check_refcounts_l1\n");
-    res->check_errors++;
-    g_free(l1_table);
-    return -EIO;
-}
-
-/*
- * Checks an image for refcount consistency.
- *
- * Returns 0 if no errors are found, the number of errors in case the image is
- * detected as corrupted, and -errno when an internal error occurred.
- */
-int qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
-                          BdrvCheckMode fix)
-{
-    BDRVQcowState *s = bs->opaque;
-    int64_t size, i, highest_cluster;
-    int nb_clusters, refcount1, refcount2;
-    QCowSnapshot *sn;
-    uint16_t *refcount_table;
-    int ret;
-
-    size = bdrv_getlength(bs->file);
-    nb_clusters = size_to_clusters(s, size);
-    refcount_table = g_malloc0(nb_clusters * sizeof(uint16_t));
-
-    res->bfi.total_clusters =
-        size_to_clusters(s, bs->total_sectors * BDRV_SECTOR_SIZE);
-
-    /* header */
-    inc_refcounts(bs, res, refcount_table, nb_clusters,
-        0, s->cluster_size);
-
-    /* current L1 table */
-    ret = check_refcounts_l1(bs, res, refcount_table, nb_clusters,
-                             s->l1_table_offset, s->l1_size,
-                             CHECK_OFLAG_COPIED | CHECK_FRAG_INFO);
-    if (ret < 0) {
-        goto fail;
-    }
-
-    /* snapshots */
-    for(i = 0; i < s->nb_snapshots; i++) {
-        sn = s->snapshots + i;
-        ret = check_refcounts_l1(bs, res, refcount_table, nb_clusters,
-            sn->l1_table_offset, sn->l1_size, 0);
-        if (ret < 0) {
-            goto fail;
-        }
-    }
-    inc_refcounts(bs, res, refcount_table, nb_clusters,
-        s->snapshots_offset, s->snapshots_size);
-
-    /* refcount data */
-    inc_refcounts(bs, res, refcount_table, nb_clusters,
-        s->refcount_table_offset,
-        s->refcount_table_size * sizeof(uint64_t));
-
-    for(i = 0; i < s->refcount_table_size; i++) {
-        uint64_t offset, cluster;
-        offset = s->refcount_table[i];
-        cluster = offset >> s->cluster_bits;
-
-        /* Refcount blocks are cluster aligned */
-        if (offset & (s->cluster_size - 1)) {
-            fprintf(stderr, "ERROR refcount block %" PRId64 " is not "
-                "cluster aligned; refcount table entry corrupted\n", i);
-            res->corruptions++;
-            continue;
-        }
-
-        if (cluster >= nb_clusters) {
-            fprintf(stderr, "ERROR refcount block %" PRId64
-                    " is outside image\n", i);
-            res->corruptions++;
-            continue;
-        }
-
-        if (offset != 0) {
-            inc_refcounts(bs, res, refcount_table, nb_clusters,
-                offset, s->cluster_size);
-            if (refcount_table[cluster] != 1) {
-                fprintf(stderr, "ERROR refcount block %" PRId64
-                    " refcount=%d\n",
-                    i, refcount_table[cluster]);
-                res->corruptions++;
-            }
-        }
-    }
-
-    /* compare ref counts */
-    for (i = 0, highest_cluster = 0; i < nb_clusters; i++) {
-        refcount1 = get_refcount(bs, i);
-        if (refcount1 < 0) {
-            fprintf(stderr, "Can't get refcount for cluster %" PRId64 ": %s\n",
-                i, strerror(-refcount1));
-            res->check_errors++;
-            continue;
-        }
-
-        refcount2 = refcount_table[i];
-
-        if (refcount1 > 0 || refcount2 > 0) {
-            highest_cluster = i;
-        }
-
-        if (refcount1 != refcount2) {
-
-            /* Check if we're allowed to fix the mismatch */
-            int *num_fixed = NULL;
-            if (refcount1 > refcount2 && (fix & BDRV_FIX_LEAKS)) {
-                num_fixed = &res->leaks_fixed;
-            } else if (refcount1 < refcount2 && (fix & BDRV_FIX_ERRORS)) {
-                num_fixed = &res->corruptions_fixed;
-            }
-
-            fprintf(stderr, "%s cluster %" PRId64 " refcount=%d reference=%d\n",
-                   num_fixed != NULL     ? "Repairing" :
-                   refcount1 < refcount2 ? "ERROR" :
-                                           "Leaked",
-                   i, refcount1, refcount2);
-
-            if (num_fixed) {
-                ret = update_refcount(bs, i << s->cluster_bits, 1,
-                                      refcount2 - refcount1,
-                                      QCOW2_DISCARD_ALWAYS);
-                if (ret >= 0) {
-                    (*num_fixed)++;
-                    continue;
-                }
-            }
-
-            /* And if we couldn't, print an error */
-            if (refcount1 < refcount2) {
-                res->corruptions++;
-            } else {
-                res->leaks++;
-            }
-        }
-    }
-
-    res->image_end_offset = (highest_cluster + 1) * s->cluster_size;
-    ret = 0;
-
-fail:
-    g_free(refcount_table);
-
-    return ret;
-}
-
diff --git a/contrib/qemu/block/qcow2-snapshot.c b/contrib/qemu/block/qcow2-snapshot.c
deleted file mode 100644
index 0caac9055f8..00000000000
--- a/contrib/qemu/block/qcow2-snapshot.c
+++ /dev/null
@@ -1,660 +0,0 @@
-/*
- * Block driver for the QCOW version 2 format
- *
- * Copyright (c) 2004-2006 Fabrice Bellard
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-
-#include "qemu-common.h"
-#include "block/block_int.h"
-#include "block/qcow2.h"
-
-typedef struct QEMU_PACKED QCowSnapshotHeader {
-    /* header is 8 byte aligned */
-    uint64_t l1_table_offset;
-
-    uint32_t l1_size;
-    uint16_t id_str_size;
-    uint16_t name_size;
-
-    uint32_t date_sec;
-    uint32_t date_nsec;
-
-    uint64_t vm_clock_nsec;
-
-    uint32_t vm_state_size;
-    uint32_t extra_data_size; /* for extension */
-    /* extra data follows */
-    /* id_str follows */
-    /* name follows  */
-} QCowSnapshotHeader;
-
-typedef struct QEMU_PACKED QCowSnapshotExtraData {
-    uint64_t vm_state_size_large;
-    uint64_t disk_size;
-} QCowSnapshotExtraData;
-
-void qcow2_free_snapshots(BlockDriverState *bs)
-{
-    BDRVQcowState *s = bs->opaque;
-    int i;
-
-    for(i = 0; i < s->nb_snapshots; i++) {
-        g_free(s->snapshots[i].name);
-        g_free(s->snapshots[i].id_str);
-    }
-    g_free(s->snapshots);
-    s->snapshots = NULL;
-    s->nb_snapshots = 0;
-}
-
-int qcow2_read_snapshots(BlockDriverState *bs)
-{
-    BDRVQcowState *s = bs->opaque;
-    QCowSnapshotHeader h;
-    QCowSnapshotExtraData extra;
-    QCowSnapshot *sn;
-    int i, id_str_size, name_size;
-    int64_t offset;
-    uint32_t extra_data_size;
-    int ret;
-
-    if (!s->nb_snapshots) {
-        s->snapshots = NULL;
-        s->snapshots_size = 0;
-        return 0;
-    }
-
-    offset = s->snapshots_offset;
-    s->snapshots = g_malloc0(s->nb_snapshots * sizeof(QCowSnapshot));
-
-    for(i = 0; i < s->nb_snapshots; i++) {
-        /* Read statically sized part of the snapshot header */
-        offset = align_offset(offset, 8);
-        ret = bdrv_pread(bs->file, offset, &h, sizeof(h));
-        if (ret < 0) {
-            goto fail;
-        }
-
-        offset += sizeof(h);
-        sn = s->snapshots + i;
-        sn->l1_table_offset = be64_to_cpu(h.l1_table_offset);
-        sn->l1_size = be32_to_cpu(h.l1_size);
-        sn->vm_state_size = be32_to_cpu(h.vm_state_size);
-        sn->date_sec = be32_to_cpu(h.date_sec);
-        sn->date_nsec = be32_to_cpu(h.date_nsec);
-        sn->vm_clock_nsec = be64_to_cpu(h.vm_clock_nsec);
-        extra_data_size = be32_to_cpu(h.extra_data_size);
-
-        id_str_size = be16_to_cpu(h.id_str_size);
-        name_size = be16_to_cpu(h.name_size);
-
-        /* Read extra data */
-        ret = bdrv_pread(bs->file, offset, &extra,
-                         MIN(sizeof(extra), extra_data_size));
-        if (ret < 0) {
-            goto fail;
-        }
-        offset += extra_data_size;
-
-        if (extra_data_size >= 8) {
-            sn->vm_state_size = be64_to_cpu(extra.vm_state_size_large);
-        }
-
-        if (extra_data_size >= 16) {
-            sn->disk_size = be64_to_cpu(extra.disk_size);
-        } else {
-            sn->disk_size = bs->total_sectors * BDRV_SECTOR_SIZE;
-        }
-
-        /* Read snapshot ID */
-        sn->id_str = g_malloc(id_str_size + 1);
-        ret = bdrv_pread(bs->file, offset, sn->id_str, id_str_size);
-        if (ret < 0) {
-            goto fail;
-        }
-        offset += id_str_size;
-        sn->id_str[id_str_size] = '\0';
-
-        /* Read snapshot name */
-        sn->name = g_malloc(name_size + 1);
-        ret = bdrv_pread(bs->file, offset, sn->name, name_size);
-        if (ret < 0) {
-            goto fail;
-        }
-        offset += name_size;
-        sn->name[name_size] = '\0';
-    }
-
-    s->snapshots_size = offset - s->snapshots_offset;
-    return 0;
-
-fail:
-    qcow2_free_snapshots(bs);
-    return ret;
-}
-
-/* add at the end of the file a new list of snapshots */
-static int qcow2_write_snapshots(BlockDriverState *bs)
-{
-    BDRVQcowState *s = bs->opaque;
-    QCowSnapshot *sn;
-    QCowSnapshotHeader h;
-    QCowSnapshotExtraData extra;
-    int i, name_size, id_str_size, snapshots_size;
-    struct {
-        uint32_t nb_snapshots;
-        uint64_t snapshots_offset;
-    } QEMU_PACKED header_data;
-    int64_t offset, snapshots_offset;
-    int ret;
-
-    /* compute the size of the snapshots */
-    offset = 0;
-    for(i = 0; i < s->nb_snapshots; i++) {
-        sn = s->snapshots + i;
-        offset = align_offset(offset, 8);
-        offset += sizeof(h);
-        offset += sizeof(extra);
-        offset += strlen(sn->id_str);
-        offset += strlen(sn->name);
-    }
-    snapshots_size = offset;
-
-    /* Allocate space for the new snapshot list */
-    snapshots_offset = qcow2_alloc_clusters(bs, snapshots_size);
-    offset = snapshots_offset;
-    if (offset < 0) {
-        return offset;
-    }
-    ret = bdrv_flush(bs);
-    if (ret < 0) {
-        return ret;
-    }
-
-    /* Write all snapshots to the new list */
-    for(i = 0; i < s->nb_snapshots; i++) {
-        sn = s->snapshots + i;
-        memset(&h, 0, sizeof(h));
-        h.l1_table_offset = cpu_to_be64(sn->l1_table_offset);
-        h.l1_size = cpu_to_be32(sn->l1_size);
-        /* If it doesn't fit in 32 bit, older implementations should treat it
-         * as a disk-only snapshot rather than truncate the VM state */
-        if (sn->vm_state_size <= 0xffffffff) {
-            h.vm_state_size = cpu_to_be32(sn->vm_state_size);
-        }
-        h.date_sec = cpu_to_be32(sn->date_sec);
-        h.date_nsec = cpu_to_be32(sn->date_nsec);
-        h.vm_clock_nsec = cpu_to_be64(sn->vm_clock_nsec);
-        h.extra_data_size = cpu_to_be32(sizeof(extra));
-
-        memset(&extra, 0, sizeof(extra));
-        extra.vm_state_size_large = cpu_to_be64(sn->vm_state_size);
-        extra.disk_size = cpu_to_be64(sn->disk_size);
-
-        id_str_size = strlen(sn->id_str);
-        name_size = strlen(sn->name);
-        h.id_str_size = cpu_to_be16(id_str_size);
-        h.name_size = cpu_to_be16(name_size);
-        offset = align_offset(offset, 8);
-
-        ret = bdrv_pwrite(bs->file, offset, &h, sizeof(h));
-        if (ret < 0) {
-            goto fail;
-        }
-        offset += sizeof(h);
-
-        ret = bdrv_pwrite(bs->file, offset, &extra, sizeof(extra));
-        if (ret < 0) {
-            goto fail;
-        }
-        offset += sizeof(extra);
-
-        ret = bdrv_pwrite(bs->file, offset, sn->id_str, id_str_size);
-        if (ret < 0) {
-            goto fail;
-        }
-        offset += id_str_size;
-
-        ret = bdrv_pwrite(bs->file, offset, sn->name, name_size);
-        if (ret < 0) {
-            goto fail;
-        }
-        offset += name_size;
-    }
-
-    /*
-     * Update the header to point to the new snapshot table. This requires the
-     * new table and its refcounts to be stable on disk.
-     */
-    ret = bdrv_flush(bs);
-    if (ret < 0) {
-        goto fail;
-    }
-
-    QEMU_BUILD_BUG_ON(offsetof(QCowHeader, snapshots_offset) !=
-        offsetof(QCowHeader, nb_snapshots) + sizeof(header_data.nb_snapshots));
-
-    header_data.nb_snapshots        = cpu_to_be32(s->nb_snapshots);
-    header_data.snapshots_offset    = cpu_to_be64(snapshots_offset);
-
-    ret = bdrv_pwrite_sync(bs->file, offsetof(QCowHeader, nb_snapshots),
-                           &header_data, sizeof(header_data));
-    if (ret < 0) {
-        goto fail;
-    }
-
-    /* free the old snapshot table */
-    qcow2_free_clusters(bs, s->snapshots_offset, s->snapshots_size,
-                        QCOW2_DISCARD_SNAPSHOT);
-    s->snapshots_offset = snapshots_offset;
-    s->snapshots_size = snapshots_size;
-    return 0;
-
-fail:
-    return ret;
-}
-
-static void find_new_snapshot_id(BlockDriverState *bs,
-                                 char *id_str, int id_str_size)
-{
-    BDRVQcowState *s = bs->opaque;
-    QCowSnapshot *sn;
-    int i, id, id_max = 0;
-
-    for(i = 0; i < s->nb_snapshots; i++) {
-        sn = s->snapshots + i;
-        id = strtoul(sn->id_str, NULL, 10);
-        if (id > id_max)
-            id_max = id;
-    }
-    snprintf(id_str, id_str_size, "%d", id_max + 1);
-}
-
-static int find_snapshot_by_id(BlockDriverState *bs, const char *id_str)
-{
-    BDRVQcowState *s = bs->opaque;
-    int i;
-
-    for(i = 0; i < s->nb_snapshots; i++) {
-        if (!strcmp(s->snapshots[i].id_str, id_str))
-            return i;
-    }
-    return -1;
-}
-
-static int find_snapshot_by_id_or_name(BlockDriverState *bs, const char *name)
-{
-    BDRVQcowState *s = bs->opaque;
-    int i, ret;
-
-    ret = find_snapshot_by_id(bs, name);
-    if (ret >= 0)
-        return ret;
-    for(i = 0; i < s->nb_snapshots; i++) {
-        if (!strcmp(s->snapshots[i].name, name))
-            return i;
-    }
-    return -1;
-}
-
-/* if no id is provided, a new one is constructed */
-int qcow2_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info)
-{
-    BDRVQcowState *s = bs->opaque;
-    QCowSnapshot *new_snapshot_list = NULL;
-    QCowSnapshot *old_snapshot_list = NULL;
-    QCowSnapshot sn1, *sn = &sn1;
-    int i, ret;
-    uint64_t *l1_table = NULL;
-    int64_t l1_table_offset;
-
-    memset(sn, 0, sizeof(*sn));
-
-    /* Generate an ID if it wasn't passed */
-    if (sn_info->id_str[0] == '\0') {
-        find_new_snapshot_id(bs, sn_info->id_str, sizeof(sn_info->id_str));
-    }
-
-    /* Check that the ID is unique */
-    if (find_snapshot_by_id(bs, sn_info->id_str) >= 0) {
-        return -EEXIST;
-    }
-
-    /* Populate sn with passed data */
-    sn->id_str = g_strdup(sn_info->id_str);
-    sn->name = g_strdup(sn_info->name);
-
-    sn->disk_size = bs->total_sectors * BDRV_SECTOR_SIZE;
-    sn->vm_state_size = sn_info->vm_state_size;
-    sn->date_sec = sn_info->date_sec;
-    sn->date_nsec = sn_info->date_nsec;
-    sn->vm_clock_nsec = sn_info->vm_clock_nsec;
-
-    /* Allocate the L1 table of the snapshot and copy the current one there. */
-    l1_table_offset = qcow2_alloc_clusters(bs, s->l1_size * sizeof(uint64_t));
-    if (l1_table_offset < 0) {
-        ret = l1_table_offset;
-        goto fail;
-    }
-
-    sn->l1_table_offset = l1_table_offset;
-    sn->l1_size = s->l1_size;
-
-    l1_table = g_malloc(s->l1_size * sizeof(uint64_t));
-    for(i = 0; i < s->l1_size; i++) {
-        l1_table[i] = cpu_to_be64(s->l1_table[i]);
-    }
-
-    ret = bdrv_pwrite(bs->file, sn->l1_table_offset, l1_table,
-                      s->l1_size * sizeof(uint64_t));
-    if (ret < 0) {
-        goto fail;
-    }
-
-    g_free(l1_table);
-    l1_table = NULL;
-
-    /*
-     * Increase the refcounts of all clusters and make sure everything is
-     * stable on disk before updating the snapshot table to contain a pointer
-     * to the new L1 table.
-     */
-    ret = qcow2_update_snapshot_refcount(bs, s->l1_table_offset, s->l1_size, 1);
-    if (ret < 0) {
-        goto fail;
-    }
-
-    /* Append the new snapshot to the snapshot list */
-    new_snapshot_list = g_malloc((s->nb_snapshots + 1) * sizeof(QCowSnapshot));
-    if (s->snapshots) {
-        memcpy(new_snapshot_list, s->snapshots,
-               s->nb_snapshots * sizeof(QCowSnapshot));
-        old_snapshot_list = s->snapshots;
-    }
-    s->snapshots = new_snapshot_list;
-    s->snapshots[s->nb_snapshots++] = *sn;
-
-    ret = qcow2_write_snapshots(bs);
-    if (ret < 0) {
-        g_free(s->snapshots);
-        s->snapshots = old_snapshot_list;
-        goto fail;
-    }
-
-    g_free(old_snapshot_list);
-
-#ifdef DEBUG_ALLOC
-    {
-      BdrvCheckResult result = {0};
-      qcow2_check_refcounts(bs, &result, 0);
-    }
-#endif
-    return 0;
-
-fail:
-    g_free(sn->id_str);
-    g_free(sn->name);
-    g_free(l1_table);
-
-    return ret;
-}
-
-/* copy the snapshot 'snapshot_name' into the current disk image */
-int qcow2_snapshot_goto(BlockDriverState *bs, const char *snapshot_id)
-{
-    BDRVQcowState *s = bs->opaque;
-    QCowSnapshot *sn;
-    int i, snapshot_index;
-    int cur_l1_bytes, sn_l1_bytes;
-    int ret;
-    uint64_t *sn_l1_table = NULL;
-
-    /* Search the snapshot */
-    snapshot_index = find_snapshot_by_id_or_name(bs, snapshot_id);
-    if (snapshot_index < 0) {
-        return -ENOENT;
-    }
-    sn = &s->snapshots[snapshot_index];
-
-    if (sn->disk_size != bs->total_sectors * BDRV_SECTOR_SIZE) {
-        error_report("qcow2: Loading snapshots with different disk "
-            "size is not implemented");
-        ret = -ENOTSUP;
-        goto fail;
-    }
-
-    /*
-     * Make sure that the current L1 table is big enough to contain the whole
-     * L1 table of the snapshot. If the snapshot L1 table is smaller, the
-     * current one must be padded with zeros.
-     */
-    ret = qcow2_grow_l1_table(bs, sn->l1_size, true);
-    if (ret < 0) {
-        goto fail;
-    }
-
-    cur_l1_bytes = s->l1_size * sizeof(uint64_t);
-    sn_l1_bytes = sn->l1_size * sizeof(uint64_t);
-
-    /*
-     * Copy the snapshot L1 table to the current L1 table.
-     *
-     * Before overwriting the old current L1 table on disk, make sure to
-     * increase all refcounts for the clusters referenced by the new one.
-     * Decrease the refcount referenced by the old one only when the L1
-     * table is overwritten.
-     */
-    sn_l1_table = g_malloc0(cur_l1_bytes);
-
-    ret = bdrv_pread(bs->file, sn->l1_table_offset, sn_l1_table, sn_l1_bytes);
-    if (ret < 0) {
-        goto fail;
-    }
-
-    ret = qcow2_update_snapshot_refcount(bs, sn->l1_table_offset,
-                                         sn->l1_size, 1);
-    if (ret < 0) {
-        goto fail;
-    }
-
-    ret = bdrv_pwrite_sync(bs->file, s->l1_table_offset, sn_l1_table,
-                           cur_l1_bytes);
-    if (ret < 0) {
-        goto fail;
-    }
-
-    /*
-     * Decrease refcount of clusters of current L1 table.
-     *
-     * At this point, the in-memory s->l1_table points to the old L1 table,
-     * whereas on disk we already have the new one.
-     *
-     * qcow2_update_snapshot_refcount special cases the current L1 table to use
-     * the in-memory data instead of really using the offset to load a new one,
-     * which is why this works.
-     */
-    ret = qcow2_update_snapshot_refcount(bs, s->l1_table_offset,
-                                         s->l1_size, -1);
-
-    /*
-     * Now update the in-memory L1 table to be in sync with the on-disk one. We
-     * need to do this even if updating refcounts failed.
-     */
-    for(i = 0;i < s->l1_size; i++) {
-        s->l1_table[i] = be64_to_cpu(sn_l1_table[i]);
-    }
-
-    if (ret < 0) {
-        goto fail;
-    }
-
-    g_free(sn_l1_table);
-    sn_l1_table = NULL;
-
-    /*
-     * Update QCOW_OFLAG_COPIED in the active L1 table (it may have changed
-     * when we decreased the refcount of the old snapshot.
-     */
-    ret = qcow2_update_snapshot_refcount(bs, s->l1_table_offset, s->l1_size, 0);
-    if (ret < 0) {
-        goto fail;
-    }
-
-#ifdef DEBUG_ALLOC
-    {
-        BdrvCheckResult result = {0};
-        qcow2_check_refcounts(bs, &result, 0);
-    }
-#endif
-    return 0;
-
-fail:
-    g_free(sn_l1_table);
-    return ret;
-}
-
-int qcow2_snapshot_delete(BlockDriverState *bs, const char *snapshot_id)
-{
-    BDRVQcowState *s = bs->opaque;
-    QCowSnapshot sn;
-    int snapshot_index, ret;
-
-    /* Search the snapshot */
-    snapshot_index = find_snapshot_by_id_or_name(bs, snapshot_id);
-    if (snapshot_index < 0) {
-        return -ENOENT;
-    }
-    sn = s->snapshots[snapshot_index];
-
-    /* Remove it from the snapshot list */
-    memmove(s->snapshots + snapshot_index,
-            s->snapshots + snapshot_index + 1,
-            (s->nb_snapshots - snapshot_index - 1) * sizeof(sn));
-    s->nb_snapshots--;
-    ret = qcow2_write_snapshots(bs);
-    if (ret < 0) {
-        return ret;
-    }
-
-    /*
-     * The snapshot is now unused, clean up. If we fail after this point, we
-     * won't recover but just leak clusters.
-     */
-    g_free(sn.id_str);
-    g_free(sn.name);
-
-    /*
-     * Now decrease the refcounts of clusters referenced by the snapshot and
-     * free the L1 table.
-     */
-    ret = qcow2_update_snapshot_refcount(bs, sn.l1_table_offset,
-                                         sn.l1_size, -1);
-    if (ret < 0) {
-        return ret;
-    }
-    qcow2_free_clusters(bs, sn.l1_table_offset, sn.l1_size * sizeof(uint64_t),
-                        QCOW2_DISCARD_SNAPSHOT);
-
-    /* must update the copied flag on the current cluster offsets */
-    ret = qcow2_update_snapshot_refcount(bs, s->l1_table_offset, s->l1_size, 0);
-    if (ret < 0) {
-        return ret;
-    }
-
-#ifdef DEBUG_ALLOC
-    {
-        BdrvCheckResult result = {0};
-        qcow2_check_refcounts(bs, &result, 0);
-    }
-#endif
-    return 0;
-}
-
-int qcow2_snapshot_list(BlockDriverState *bs, QEMUSnapshotInfo **psn_tab)
-{
-    BDRVQcowState *s = bs->opaque;
-    QEMUSnapshotInfo *sn_tab, *sn_info;
-    QCowSnapshot *sn;
-    int i;
-
-    if (!s->nb_snapshots) {
-        *psn_tab = NULL;
-        return s->nb_snapshots;
-    }
-
-    sn_tab = g_malloc0(s->nb_snapshots * sizeof(QEMUSnapshotInfo));
-    for(i = 0; i < s->nb_snapshots; i++) {
-        sn_info = sn_tab + i;
-        sn = s->snapshots + i;
-        pstrcpy(sn_info->id_str, sizeof(sn_info->id_str),
-                sn->id_str);
-        pstrcpy(sn_info->name, sizeof(sn_info->name),
-                sn->name);
-        sn_info->vm_state_size = sn->vm_state_size;
-        sn_info->date_sec = sn->date_sec;
-        sn_info->date_nsec = sn->date_nsec;
-        sn_info->vm_clock_nsec = sn->vm_clock_nsec;
-    }
-    *psn_tab = sn_tab;
-    return s->nb_snapshots;
-}
-
-int qcow2_snapshot_load_tmp(BlockDriverState *bs, const char *snapshot_name)
-{
-    int i, snapshot_index;
-    BDRVQcowState *s = bs->opaque;
-    QCowSnapshot *sn;
-    uint64_t *new_l1_table;
-    int new_l1_bytes;
-    int ret;
-
-    assert(bs->read_only);
-
-    /* Search the snapshot */
-    snapshot_index = find_snapshot_by_id_or_name(bs, snapshot_name);
-    if (snapshot_index < 0) {
-        return -ENOENT;
-    }
-    sn = &s->snapshots[snapshot_index];
-
-    /* Allocate and read in the snapshot's L1 table */
-    new_l1_bytes = s->l1_size * sizeof(uint64_t);
-    new_l1_table = g_malloc0(align_offset(new_l1_bytes, 512));
-
-    ret = bdrv_pread(bs->file, sn->l1_table_offset, new_l1_table, new_l1_bytes);
-    if (ret < 0) {
-        g_free(new_l1_table);
-        return ret;
-    }
-
-    /* Switch the L1 table */
-    g_free(s->l1_table);
-
-    s->l1_size = sn->l1_size;
-    s->l1_table_offset = sn->l1_table_offset;
-    s->l1_table = new_l1_table;
-
-    for(i = 0;i < s->l1_size; i++) {
-        be64_to_cpus(&s->l1_table[i]);
-    }
-
-    return 0;
-}
diff --git a/contrib/qemu/block/qcow2.c b/contrib/qemu/block/qcow2.c
deleted file mode 100644
index 0eceefe2cd9..00000000000
--- a/contrib/qemu/block/qcow2.c
+++ /dev/null
@@ -1,1825 +0,0 @@
-/*
- * Block driver for the QCOW version 2 format
- *
- * Copyright (c) 2004-2006 Fabrice Bellard
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-#include "qemu-common.h"
-#include "block/block_int.h"
-#include "qemu/module.h"
-#include <zlib.h>
-#include "qemu/aes.h"
-#include "block/qcow2.h"
-#include "qemu/error-report.h"
-#include "qapi/qmp/qerror.h"
-#include "qapi/qmp/qbool.h"
-#include "trace.h"
-
-/*
-  Differences with QCOW:
-
-  - Support for multiple incremental snapshots.
-  - Memory management by reference counts.
-  - Clusters which have a reference count of one have the bit
-    QCOW_OFLAG_COPIED to optimize write performance.
-  - Size of compressed clusters is stored in sectors to reduce bit usage
-    in the cluster offsets.
-  - Support for storing additional data (such as the VM state) in the
-    snapshots.
-  - If a backing store is used, the cluster size is not constrained
-    (could be backported to QCOW).
-  - L2 tables have always a size of one cluster.
-*/
-
-
-typedef struct {
-    uint32_t magic;
-    uint32_t len;
-} QCowExtension;
-
-#define  QCOW2_EXT_MAGIC_END 0
-#define  QCOW2_EXT_MAGIC_BACKING_FORMAT 0xE2792ACA
-#define  QCOW2_EXT_MAGIC_FEATURE_TABLE 0x6803f857
-
-static int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename)
-{
-    const QCowHeader *cow_header = (const void *)buf;
-
-    if (buf_size >= sizeof(QCowHeader) &&
-        be32_to_cpu(cow_header->magic) == QCOW_MAGIC &&
-        be32_to_cpu(cow_header->version) >= 2)
-        return 100;
-    else
-        return 0;
-}
-
-
-/* 
- * read qcow2 extension and fill bs
- * start reading from start_offset
- * finish reading upon magic of value 0 or when end_offset reached
- * unknown magic is skipped (future extension this version knows nothing about)
- * return 0 upon success, non-0 otherwise
- */
-static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset,
-                                 uint64_t end_offset, void **p_feature_table)
-{
-    BDRVQcowState *s = bs->opaque;
-    QCowExtension ext;
-    uint64_t offset;
-    int ret;
-
-#ifdef DEBUG_EXT
-    printf("qcow2_read_extensions: start=%ld end=%ld\n", start_offset, end_offset);
-#endif
-    offset = start_offset;
-    while (offset < end_offset) {
-
-#ifdef DEBUG_EXT
-        /* Sanity check */
-        if (offset > s->cluster_size)
-            printf("qcow2_read_extension: suspicious offset %lu\n", offset);
-
-        printf("attempting to read extended header in offset %lu\n", offset);
-#endif
-
-        if (bdrv_pread(bs->file, offset, &ext, sizeof(ext)) != sizeof(ext)) {
-            fprintf(stderr, "qcow2_read_extension: ERROR: "
-                    "pread fail from offset %" PRIu64 "\n",
-                    offset);
-            return 1;
-        }
-        be32_to_cpus(&ext.magic);
-        be32_to_cpus(&ext.len);
-        offset += sizeof(ext);
-#ifdef DEBUG_EXT
-        printf("ext.magic = 0x%x\n", ext.magic);
-#endif
-        if (ext.len > end_offset - offset) {
-            error_report("Header extension too large");
-            return -EINVAL;
-        }
-
-        switch (ext.magic) {
-        case QCOW2_EXT_MAGIC_END:
-            return 0;
-
-        case QCOW2_EXT_MAGIC_BACKING_FORMAT:
-            if (ext.len >= sizeof(bs->backing_format)) {
-                fprintf(stderr, "ERROR: ext_backing_format: len=%u too large"
-                        " (>=%zu)\n",
-                        ext.len, sizeof(bs->backing_format));
-                return 2;
-            }
-            if (bdrv_pread(bs->file, offset , bs->backing_format,
-                           ext.len) != ext.len)
-                return 3;
-            bs->backing_format[ext.len] = '\0';
-#ifdef DEBUG_EXT
-            printf("Qcow2: Got format extension %s\n", bs->backing_format);
-#endif
-            break;
-
-        case QCOW2_EXT_MAGIC_FEATURE_TABLE:
-            if (p_feature_table != NULL) {
-                void* feature_table = g_malloc0(ext.len + 2 * sizeof(Qcow2Feature));
-                ret = bdrv_pread(bs->file, offset , feature_table, ext.len);
-                if (ret < 0) {
-                    return ret;
-                }
-
-                *p_feature_table = feature_table;
-            }
-            break;
-
-        default:
-            /* unknown magic - save it in case we need to rewrite the header */
-            {
-                Qcow2UnknownHeaderExtension *uext;
-
-                uext = g_malloc0(sizeof(*uext)  + ext.len);
-                uext->magic = ext.magic;
-                uext->len = ext.len;
-                QLIST_INSERT_HEAD(&s->unknown_header_ext, uext, next);
-
-                ret = bdrv_pread(bs->file, offset , uext->data, uext->len);
-                if (ret < 0) {
-                    return ret;
-                }
-            }
-            break;
-        }
-
-        offset += ((ext.len + 7) & ~7);
-    }
-
-    return 0;
-}
-
-static void cleanup_unknown_header_ext(BlockDriverState *bs)
-{
-    BDRVQcowState *s = bs->opaque;
-    Qcow2UnknownHeaderExtension *uext, *next;
-
-    QLIST_FOREACH_SAFE(uext, &s->unknown_header_ext, next, next) {
-        QLIST_REMOVE(uext, next);
-        g_free(uext);
-    }
-}
-
-static void GCC_FMT_ATTR(2, 3) report_unsupported(BlockDriverState *bs,
-    const char *fmt, ...)
-{
-    char msg[64];
-    va_list ap;
-
-    va_start(ap, fmt);
-    vsnprintf(msg, sizeof(msg), fmt, ap);
-    va_end(ap);
-
-    qerror_report(QERR_UNKNOWN_BLOCK_FORMAT_FEATURE,
-        bs->device_name, "qcow2", msg);
-}
-
-static void report_unsupported_feature(BlockDriverState *bs,
-    Qcow2Feature *table, uint64_t mask)
-{
-    while (table && table->name[0] != '\0') {
-        if (table->type == QCOW2_FEAT_TYPE_INCOMPATIBLE) {
-            if (mask & (1 << table->bit)) {
-                report_unsupported(bs, "%.46s",table->name);
-                mask &= ~(1 << table->bit);
-            }
-        }
-        table++;
-    }
-
-    if (mask) {
-        report_unsupported(bs, "Unknown incompatible feature: %" PRIx64, mask);
-    }
-}
-
-/*
- * Sets the dirty bit and flushes afterwards if necessary.
- *
- * The incompatible_features bit is only set if the image file header was
- * updated successfully.  Therefore it is not required to check the return
- * value of this function.
- */
-int qcow2_mark_dirty(BlockDriverState *bs)
-{
-    BDRVQcowState *s = bs->opaque;
-    uint64_t val;
-    int ret;
-
-    assert(s->qcow_version >= 3);
-
-    if (s->incompatible_features & QCOW2_INCOMPAT_DIRTY) {
-        return 0; /* already dirty */
-    }
-
-    val = cpu_to_be64(s->incompatible_features | QCOW2_INCOMPAT_DIRTY);
-    ret = bdrv_pwrite(bs->file, offsetof(QCowHeader, incompatible_features),
-                      &val, sizeof(val));
-    if (ret < 0) {
-        return ret;
-    }
-    ret = bdrv_flush(bs->file);
-    if (ret < 0) {
-        return ret;
-    }
-
-    /* Only treat image as dirty if the header was updated successfully */
-    s->incompatible_features |= QCOW2_INCOMPAT_DIRTY;
-    return 0;
-}
-
-/*
- * Clears the dirty bit and flushes before if necessary.  Only call this
- * function when there are no pending requests, it does not guard against
- * concurrent requests dirtying the image.
- */
-static int qcow2_mark_clean(BlockDriverState *bs)
-{
-    BDRVQcowState *s = bs->opaque;
-
-    if (s->incompatible_features & QCOW2_INCOMPAT_DIRTY) {
-        int ret = bdrv_flush(bs);
-        if (ret < 0) {
-            return ret;
-        }
-
-        s->incompatible_features &= ~QCOW2_INCOMPAT_DIRTY;
-        return qcow2_update_header(bs);
-    }
-    return 0;
-}
-
-static int qcow2_check(BlockDriverState *bs, BdrvCheckResult *result,
-                       BdrvCheckMode fix)
-{
-    int ret = qcow2_check_refcounts(bs, result, fix);
-    if (ret < 0) {
-        return ret;
-    }
-
-    if (fix && result->check_errors == 0 && result->corruptions == 0) {
-        return qcow2_mark_clean(bs);
-    }
-    return ret;
-}
-
-static QemuOptsList qcow2_runtime_opts = {
-    .name = "qcow2",
-    .head = QTAILQ_HEAD_INITIALIZER(qcow2_runtime_opts.head),
-    .desc = {
-        {
-            .name = "lazy_refcounts",
-            .type = QEMU_OPT_BOOL,
-            .help = "Postpone refcount updates",
-        },
-        {
-            .name = QCOW2_OPT_DISCARD_REQUEST,
-            .type = QEMU_OPT_BOOL,
-            .help = "Pass guest discard requests to the layer below",
-        },
-        {
-            .name = QCOW2_OPT_DISCARD_SNAPSHOT,
-            .type = QEMU_OPT_BOOL,
-            .help = "Generate discard requests when snapshot related space "
-                    "is freed",
-        },
-        {
-            .name = QCOW2_OPT_DISCARD_OTHER,
-            .type = QEMU_OPT_BOOL,
-            .help = "Generate discard requests when other clusters are freed",
-        },
-        { /* end of list */ }
-    },
-};
-
-static int qcow2_open(BlockDriverState *bs, QDict *options, int flags)
-{
-    BDRVQcowState *s = bs->opaque;
-    int len, i, ret = 0;
-    QCowHeader header;
-    QemuOpts *opts;
-    Error *local_err = NULL;
-    uint64_t ext_end;
-    uint64_t l1_vm_state_index;
-
-    ret = bdrv_pread(bs->file, 0, &header, sizeof(header));
-    if (ret < 0) {
-        goto fail;
-    }
-    be32_to_cpus(&header.magic);
-    be32_to_cpus(&header.version);
-    be64_to_cpus(&header.backing_file_offset);
-    be32_to_cpus(&header.backing_file_size);
-    be64_to_cpus(&header.size);
-    be32_to_cpus(&header.cluster_bits);
-    be32_to_cpus(&header.crypt_method);
-    be64_to_cpus(&header.l1_table_offset);
-    be32_to_cpus(&header.l1_size);
-    be64_to_cpus(&header.refcount_table_offset);
-    be32_to_cpus(&header.refcount_table_clusters);
-    be64_to_cpus(&header.snapshots_offset);
-    be32_to_cpus(&header.nb_snapshots);
-
-    if (header.magic != QCOW_MAGIC) {
-        ret = -EMEDIUMTYPE;
-        goto fail;
-    }
-    if (header.version < 2 || header.version > 3) {
-        report_unsupported(bs, "QCOW version %d", header.version);
-        ret = -ENOTSUP;
-        goto fail;
-    }
-
-    s->qcow_version = header.version;
-
-    /* Initialise version 3 header fields */
-    if (header.version == 2) {
-        header.incompatible_features    = 0;
-        header.compatible_features      = 0;
-        header.autoclear_features       = 0;
-        header.refcount_order           = 4;
-        header.header_length            = 72;
-    } else {
-        be64_to_cpus(&header.incompatible_features);
-        be64_to_cpus(&header.compatible_features);
-        be64_to_cpus(&header.autoclear_features);
-        be32_to_cpus(&header.refcount_order);
-        be32_to_cpus(&header.header_length);
-    }
-
-    if (header.header_length > sizeof(header)) {
-        s->unknown_header_fields_size = header.header_length - sizeof(header);
-        s->unknown_header_fields = g_malloc(s->unknown_header_fields_size);
-        ret = bdrv_pread(bs->file, sizeof(header), s->unknown_header_fields,
-                         s->unknown_header_fields_size);
-        if (ret < 0) {
-            goto fail;
-        }
-    }
-
-    if (header.backing_file_offset) {
-        ext_end = header.backing_file_offset;
-    } else {
-        ext_end = 1 << header.cluster_bits;
-    }
-
-    /* Handle feature bits */
-    s->incompatible_features    = header.incompatible_features;
-    s->compatible_features      = header.compatible_features;
-    s->autoclear_features       = header.autoclear_features;
-
-    if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK) {
-        void *feature_table = NULL;
-        qcow2_read_extensions(bs, header.header_length, ext_end,
-                              &feature_table);
-        report_unsupported_feature(bs, feature_table,
-                                   s->incompatible_features &
-                                   ~QCOW2_INCOMPAT_MASK);
-        ret = -ENOTSUP;
-        goto fail;
-    }
-
-    /* Check support for various header values */
-    if (header.refcount_order != 4) {
-        report_unsupported(bs, "%d bit reference counts",
-                           1 << header.refcount_order);
-        ret = -ENOTSUP;
-        goto fail;
-    }
-
-    if (header.cluster_bits < MIN_CLUSTER_BITS ||
-        header.cluster_bits > MAX_CLUSTER_BITS) {
-        ret = -EINVAL;
-        goto fail;
-    }
-    if (header.crypt_method > QCOW_CRYPT_AES) {
-        ret = -EINVAL;
-        goto fail;
-    }
-    s->crypt_method_header = header.crypt_method;
-    if (s->crypt_method_header) {
-        bs->encrypted = 1;
-    }
-    s->cluster_bits = header.cluster_bits;
-    s->cluster_size = 1 << s->cluster_bits;
-    s->cluster_sectors = 1 << (s->cluster_bits - 9);
-    s->l2_bits = s->cluster_bits - 3; /* L2 is always one cluster */
-    s->l2_size = 1 << s->l2_bits;
-    bs->total_sectors = header.size / 512;
-    s->csize_shift = (62 - (s->cluster_bits - 8));
-    s->csize_mask = (1 << (s->cluster_bits - 8)) - 1;
-    s->cluster_offset_mask = (1LL << s->csize_shift) - 1;
-    s->refcount_table_offset = header.refcount_table_offset;
-    s->refcount_table_size =
-        header.refcount_table_clusters << (s->cluster_bits - 3);
-
-    s->snapshots_offset = header.snapshots_offset;
-    s->nb_snapshots = header.nb_snapshots;
-
-    /* read the level 1 table */
-    s->l1_size = header.l1_size;
-
-    l1_vm_state_index = size_to_l1(s, header.size);
-    if (l1_vm_state_index > INT_MAX) {
-        ret = -EFBIG;
-        goto fail;
-    }
-    s->l1_vm_state_index = l1_vm_state_index;
-
-    /* the L1 table must contain at least enough entries to put
-       header.size bytes */
-    if (s->l1_size < s->l1_vm_state_index) {
-        ret = -EINVAL;
-        goto fail;
-    }
-    s->l1_table_offset = header.l1_table_offset;
-    if (s->l1_size > 0) {
-        s->l1_table = g_malloc0(
-            align_offset(s->l1_size * sizeof(uint64_t), 512));
-        ret = bdrv_pread(bs->file, s->l1_table_offset, s->l1_table,
-                         s->l1_size * sizeof(uint64_t));
-        if (ret < 0) {
-            goto fail;
-        }
-        for(i = 0;i < s->l1_size; i++) {
-            be64_to_cpus(&s->l1_table[i]);
-        }
-    }
-
-    /* alloc L2 table/refcount block cache */
-    s->l2_table_cache = qcow2_cache_create(bs, L2_CACHE_SIZE);
-    s->refcount_block_cache = qcow2_cache_create(bs, REFCOUNT_CACHE_SIZE);
-
-    s->cluster_cache = g_malloc(s->cluster_size);
-    /* one more sector for decompressed data alignment */
-    s->cluster_data = qemu_blockalign(bs, QCOW_MAX_CRYPT_CLUSTERS * s->cluster_size
-                                  + 512);
-    s->cluster_cache_offset = -1;
-    s->flags = flags;
-
-    ret = qcow2_refcount_init(bs);
-    if (ret != 0) {
-        goto fail;
-    }
-
-    QLIST_INIT(&s->cluster_allocs);
-    QTAILQ_INIT(&s->discards);
-
-    /* read qcow2 extensions */
-    if (qcow2_read_extensions(bs, header.header_length, ext_end, NULL)) {
-        ret = -EINVAL;
-        goto fail;
-    }
-
-    /* read the backing file name */
-    if (header.backing_file_offset != 0) {
-        len = header.backing_file_size;
-        if (len > 1023) {
-            len = 1023;
-        }
-        ret = bdrv_pread(bs->file, header.backing_file_offset,
-                         bs->backing_file, len);
-        if (ret < 0) {
-            goto fail;
-        }
-        bs->backing_file[len] = '\0';
-    }
-
-    ret = qcow2_read_snapshots(bs);
-    if (ret < 0) {
-        goto fail;
-    }
-
-    /* Clear unknown autoclear feature bits */
-    if (!bs->read_only && s->autoclear_features != 0) {
-        s->autoclear_features = 0;
-        ret = qcow2_update_header(bs);
-        if (ret < 0) {
-            goto fail;
-        }
-    }
-
-    /* Initialise locks */
-    qemu_co_mutex_init(&s->lock);
-
-    /* Repair image if dirty */
-    if (!(flags & BDRV_O_CHECK) && !bs->read_only &&
-        (s->incompatible_features & QCOW2_INCOMPAT_DIRTY)) {
-        BdrvCheckResult result = {0};
-
-        ret = qcow2_check(bs, &result, BDRV_FIX_ERRORS);
-        if (ret < 0) {
-            goto fail;
-        }
-    }
-
-    /* Enable lazy_refcounts according to image and command line options */
-    opts = qemu_opts_create_nofail(&qcow2_runtime_opts);
-    qemu_opts_absorb_qdict(opts, options, &local_err);
-    if (error_is_set(&local_err)) {
-        qerror_report_err(local_err);
-        error_free(local_err);
-        ret = -EINVAL;
-        goto fail;
-    }
-
-    s->use_lazy_refcounts = qemu_opt_get_bool(opts, QCOW2_OPT_LAZY_REFCOUNTS,
-        (s->compatible_features & QCOW2_COMPAT_LAZY_REFCOUNTS));
-
-    s->discard_passthrough[QCOW2_DISCARD_NEVER] = false;
-    s->discard_passthrough[QCOW2_DISCARD_ALWAYS] = true;
-    s->discard_passthrough[QCOW2_DISCARD_REQUEST] =
-        qemu_opt_get_bool(opts, QCOW2_OPT_DISCARD_REQUEST,
-                          flags & BDRV_O_UNMAP);
-    s->discard_passthrough[QCOW2_DISCARD_SNAPSHOT] =
-        qemu_opt_get_bool(opts, QCOW2_OPT_DISCARD_SNAPSHOT, true);
-    s->discard_passthrough[QCOW2_DISCARD_OTHER] =
-        qemu_opt_get_bool(opts, QCOW2_OPT_DISCARD_OTHER, false);
-
-    qemu_opts_del(opts);
-
-    if (s->use_lazy_refcounts && s->qcow_version < 3) {
-        qerror_report(ERROR_CLASS_GENERIC_ERROR, "Lazy refcounts require "
-            "a qcow2 image with at least qemu 1.1 compatibility level");
-        ret = -EINVAL;
-        goto fail;
-    }
-
-#ifdef DEBUG_ALLOC
-    {
-        BdrvCheckResult result = {0};
-        qcow2_check_refcounts(bs, &result, 0);
-    }
-#endif
-    return ret;
-
- fail:
-    g_free(s->unknown_header_fields);
-    cleanup_unknown_header_ext(bs);
-    qcow2_free_snapshots(bs);
-    qcow2_refcount_close(bs);
-    g_free(s->l1_table);
-    if (s->l2_table_cache) {
-        qcow2_cache_destroy(bs, s->l2_table_cache);
-    }
-    g_free(s->cluster_cache);
-    qemu_vfree(s->cluster_data);
-    return ret;
-}
-
-static int qcow2_set_key(BlockDriverState *bs, const char *key)
-{
-    BDRVQcowState *s = bs->opaque;
-    uint8_t keybuf[16];
-    int len, i;
-
-    memset(keybuf, 0, 16);
-    len = strlen(key);
-    if (len > 16)
-        len = 16;
-    /* XXX: we could compress the chars to 7 bits to increase
-       entropy */
-    for(i = 0;i < len;i++) {
-        keybuf[i] = key[i];
-    }
-    s->crypt_method = s->crypt_method_header;
-
-    if (AES_set_encrypt_key(keybuf, 128, &s->aes_encrypt_key) != 0)
-        return -1;
-    if (AES_set_decrypt_key(keybuf, 128, &s->aes_decrypt_key) != 0)
-        return -1;
-#if 0
-    /* test */
-    {
-        uint8_t in[16];
-        uint8_t out[16];
-        uint8_t tmp[16];
-        for(i=0;i<16;i++)
-            in[i] = i;
-        AES_encrypt(in, tmp, &s->aes_encrypt_key);
-        AES_decrypt(tmp, out, &s->aes_decrypt_key);
-        for(i = 0; i < 16; i++)
-            printf(" %02x", tmp[i]);
-        printf("\n");
-        for(i = 0; i < 16; i++)
-            printf(" %02x", out[i]);
-        printf("\n");
-    }
-#endif
-    return 0;
-}
-
-/* We have nothing to do for QCOW2 reopen, stubs just return
- * success */
-static int qcow2_reopen_prepare(BDRVReopenState *state,
-                                BlockReopenQueue *queue, Error **errp)
-{
-    return 0;
-}
-
-static int coroutine_fn qcow2_co_is_allocated(BlockDriverState *bs,
-        int64_t sector_num, int nb_sectors, int *pnum)
-{
-    BDRVQcowState *s = bs->opaque;
-    uint64_t cluster_offset;
-    int ret;
-
-    *pnum = nb_sectors;
-    /* FIXME We can get errors here, but the bdrv_co_is_allocated interface
-     * can't pass them on today */
-    qemu_co_mutex_lock(&s->lock);
-    ret = qcow2_get_cluster_offset(bs, sector_num << 9, pnum, &cluster_offset);
-    qemu_co_mutex_unlock(&s->lock);
-    if (ret < 0) {
-        *pnum = 0;
-    }
-
-    return (cluster_offset != 0) || (ret == QCOW2_CLUSTER_ZERO);
-}
-
-/* handle reading after the end of the backing file */
-int qcow2_backing_read1(BlockDriverState *bs, QEMUIOVector *qiov,
-                  int64_t sector_num, int nb_sectors)
-{
-    int n1;
-    if ((sector_num + nb_sectors) <= bs->total_sectors)
-        return nb_sectors;
-    if (sector_num >= bs->total_sectors)
-        n1 = 0;
-    else
-        n1 = bs->total_sectors - sector_num;
-
-    qemu_iovec_memset(qiov, 512 * n1, 0, 512 * (nb_sectors - n1));
-
-    return n1;
-}
-
-static coroutine_fn int qcow2_co_readv(BlockDriverState *bs, int64_t sector_num,
-                          int remaining_sectors, QEMUIOVector *qiov)
-{
-    BDRVQcowState *s = bs->opaque;
-    int index_in_cluster, n1;
-    int ret;
-    int cur_nr_sectors; /* number of sectors in current iteration */
-    uint64_t cluster_offset = 0;
-    uint64_t bytes_done = 0;
-    QEMUIOVector hd_qiov;
-    uint8_t *cluster_data = NULL;
-
-    qemu_iovec_init(&hd_qiov, qiov->niov);
-
-    qemu_co_mutex_lock(&s->lock);
-
-    while (remaining_sectors != 0) {
-
-        /* prepare next request */
-        cur_nr_sectors = remaining_sectors;
-        if (s->crypt_method) {
-            cur_nr_sectors = MIN(cur_nr_sectors,
-                QCOW_MAX_CRYPT_CLUSTERS * s->cluster_sectors);
-        }
-
-        ret = qcow2_get_cluster_offset(bs, sector_num << 9,
-            &cur_nr_sectors, &cluster_offset);
-        if (ret < 0) {
-            goto fail;
-        }
-
-        index_in_cluster = sector_num & (s->cluster_sectors - 1);
-
-        qemu_iovec_reset(&hd_qiov);
-        qemu_iovec_concat(&hd_qiov, qiov, bytes_done,
-            cur_nr_sectors * 512);
-
-        switch (ret) {
-        case QCOW2_CLUSTER_UNALLOCATED:
-
-            if (bs->backing_hd) {
-                /* read from the base image */
-                n1 = qcow2_backing_read1(bs->backing_hd, &hd_qiov,
-                    sector_num, cur_nr_sectors);
-                if (n1 > 0) {
-                    BLKDBG_EVENT(bs->file, BLKDBG_READ_BACKING_AIO);
-                    qemu_co_mutex_unlock(&s->lock);
-                    ret = bdrv_co_readv(bs->backing_hd, sector_num,
-                                        n1, &hd_qiov);
-                    qemu_co_mutex_lock(&s->lock);
-                    if (ret < 0) {
-                        goto fail;
-                    }
-                }
-            } else {
-                /* Note: in this case, no need to wait */
-                qemu_iovec_memset(&hd_qiov, 0, 0, 512 * cur_nr_sectors);
-            }
-            break;
-
-        case QCOW2_CLUSTER_ZERO:
-            qemu_iovec_memset(&hd_qiov, 0, 0, 512 * cur_nr_sectors);
-            break;
-
-        case QCOW2_CLUSTER_COMPRESSED:
-            /* add AIO support for compressed blocks ? */
-            ret = qcow2_decompress_cluster(bs, cluster_offset);
-            if (ret < 0) {
-                goto fail;
-            }
-
-            qemu_iovec_from_buf(&hd_qiov, 0,
-                s->cluster_cache + index_in_cluster * 512,
-                512 * cur_nr_sectors);
-            break;
-
-        case QCOW2_CLUSTER_NORMAL:
-            if ((cluster_offset & 511) != 0) {
-                ret = -EIO;
-                goto fail;
-            }
-
-            if (s->crypt_method) {
-                /*
-                 * For encrypted images, read everything into a temporary
-                 * contiguous buffer on which the AES functions can work.
-                 */
-                if (!cluster_data) {
-                    cluster_data =
-                        qemu_blockalign(bs, QCOW_MAX_CRYPT_CLUSTERS * s->cluster_size);
-                }
-
-                assert(cur_nr_sectors <=
-                    QCOW_MAX_CRYPT_CLUSTERS * s->cluster_sectors);
-                qemu_iovec_reset(&hd_qiov);
-                qemu_iovec_add(&hd_qiov, cluster_data,
-                    512 * cur_nr_sectors);
-            }
-
-            BLKDBG_EVENT(bs->file, BLKDBG_READ_AIO);
-            qemu_co_mutex_unlock(&s->lock);
-            ret = bdrv_co_readv(bs->file,
-                                (cluster_offset >> 9) + index_in_cluster,
-                                cur_nr_sectors, &hd_qiov);
-            qemu_co_mutex_lock(&s->lock);
-            if (ret < 0) {
-                goto fail;
-            }
-            if (s->crypt_method) {
-                qcow2_encrypt_sectors(s, sector_num,  cluster_data,
-                    cluster_data, cur_nr_sectors, 0, &s->aes_decrypt_key);
-                qemu_iovec_from_buf(qiov, bytes_done,
-                    cluster_data, 512 * cur_nr_sectors);
-            }
-            break;
-
-        default:
-            g_assert_not_reached();
-            ret = -EIO;
-            goto fail;
-        }
-
-        remaining_sectors -= cur_nr_sectors;
-        sector_num += cur_nr_sectors;
-        bytes_done += cur_nr_sectors * 512;
-    }
-    ret = 0;
-
-fail:
-    qemu_co_mutex_unlock(&s->lock);
-
-    qemu_iovec_destroy(&hd_qiov);
-    qemu_vfree(cluster_data);
-
-    return ret;
-}
-
-static coroutine_fn int qcow2_co_writev(BlockDriverState *bs,
-                           int64_t sector_num,
-                           int remaining_sectors,
-                           QEMUIOVector *qiov)
-{
-    BDRVQcowState *s = bs->opaque;
-    int index_in_cluster;
-    int n_end;
-    int ret;
-    int cur_nr_sectors; /* number of sectors in current iteration */
-    uint64_t cluster_offset;
-    QEMUIOVector hd_qiov;
-    uint64_t bytes_done = 0;
-    uint8_t *cluster_data = NULL;
-    QCowL2Meta *l2meta = NULL;
-
-    trace_qcow2_writev_start_req(qemu_coroutine_self(), sector_num,
-                                 remaining_sectors);
-
-    qemu_iovec_init(&hd_qiov, qiov->niov);
-
-    s->cluster_cache_offset = -1; /* disable compressed cache */
-
-    qemu_co_mutex_lock(&s->lock);
-
-    while (remaining_sectors != 0) {
-
-        l2meta = NULL;
-
-        trace_qcow2_writev_start_part(qemu_coroutine_self());
-        index_in_cluster = sector_num & (s->cluster_sectors - 1);
-        n_end = index_in_cluster + remaining_sectors;
-        if (s->crypt_method &&
-            n_end > QCOW_MAX_CRYPT_CLUSTERS * s->cluster_sectors) {
-            n_end = QCOW_MAX_CRYPT_CLUSTERS * s->cluster_sectors;
-        }
-
-        ret = qcow2_alloc_cluster_offset(bs, sector_num << 9,
-            index_in_cluster, n_end, &cur_nr_sectors, &cluster_offset, &l2meta);
-        if (ret < 0) {
-            goto fail;
-        }
-
-        assert((cluster_offset & 511) == 0);
-
-        qemu_iovec_reset(&hd_qiov);
-        qemu_iovec_concat(&hd_qiov, qiov, bytes_done,
-            cur_nr_sectors * 512);
-
-        if (s->crypt_method) {
-            if (!cluster_data) {
-                cluster_data = qemu_blockalign(bs, QCOW_MAX_CRYPT_CLUSTERS *
-                                                 s->cluster_size);
-            }
-
-            assert(hd_qiov.size <=
-                   QCOW_MAX_CRYPT_CLUSTERS * s->cluster_size);
-            qemu_iovec_to_buf(&hd_qiov, 0, cluster_data, hd_qiov.size);
-
-            qcow2_encrypt_sectors(s, sector_num, cluster_data,
-                cluster_data, cur_nr_sectors, 1, &s->aes_encrypt_key);
-
-            qemu_iovec_reset(&hd_qiov);
-            qemu_iovec_add(&hd_qiov, cluster_data,
-                cur_nr_sectors * 512);
-        }
-
-        qemu_co_mutex_unlock(&s->lock);
-        BLKDBG_EVENT(bs->file, BLKDBG_WRITE_AIO);
-        trace_qcow2_writev_data(qemu_coroutine_self(),
-                                (cluster_offset >> 9) + index_in_cluster);
-        ret = bdrv_co_writev(bs->file,
-                             (cluster_offset >> 9) + index_in_cluster,
-                             cur_nr_sectors, &hd_qiov);
-        qemu_co_mutex_lock(&s->lock);
-        if (ret < 0) {
-            goto fail;
-        }
-
-        while (l2meta != NULL) {
-            QCowL2Meta *next;
-
-            ret = qcow2_alloc_cluster_link_l2(bs, l2meta);
-            if (ret < 0) {
-                goto fail;
-            }
-
-            /* Take the request off the list of running requests */
-            if (l2meta->nb_clusters != 0) {
-                QLIST_REMOVE(l2meta, next_in_flight);
-            }
-
-            qemu_co_queue_restart_all(&l2meta->dependent_requests);
-
-            next = l2meta->next;
-            g_free(l2meta);
-            l2meta = next;
-        }
-
-        remaining_sectors -= cur_nr_sectors;
-        sector_num += cur_nr_sectors;
-        bytes_done += cur_nr_sectors * 512;
-        trace_qcow2_writev_done_part(qemu_coroutine_self(), cur_nr_sectors);
-    }
-    ret = 0;
-
-fail:
-    qemu_co_mutex_unlock(&s->lock);
-
-    while (l2meta != NULL) {
-        QCowL2Meta *next;
-
-        if (l2meta->nb_clusters != 0) {
-            QLIST_REMOVE(l2meta, next_in_flight);
-        }
-        qemu_co_queue_restart_all(&l2meta->dependent_requests);
-
-        next = l2meta->next;
-        g_free(l2meta);
-        l2meta = next;
-    }
-
-    qemu_iovec_destroy(&hd_qiov);
-    qemu_vfree(cluster_data);
-    trace_qcow2_writev_done_req(qemu_coroutine_self(), ret);
-
-    return ret;
-}
-
-static void qcow2_close(BlockDriverState *bs)
-{
-    BDRVQcowState *s = bs->opaque;
-    g_free(s->l1_table);
-
-    qcow2_cache_flush(bs, s->l2_table_cache);
-    qcow2_cache_flush(bs, s->refcount_block_cache);
-
-    qcow2_mark_clean(bs);
-
-    qcow2_cache_destroy(bs, s->l2_table_cache);
-    qcow2_cache_destroy(bs, s->refcount_block_cache);
-
-    g_free(s->unknown_header_fields);
-    cleanup_unknown_header_ext(bs);
-
-    g_free(s->cluster_cache);
-    qemu_vfree(s->cluster_data);
-    qcow2_refcount_close(bs);
-    qcow2_free_snapshots(bs);
-}
-
-static void qcow2_invalidate_cache(BlockDriverState *bs)
-{
-    BDRVQcowState *s = bs->opaque;
-    int flags = s->flags;
-    AES_KEY aes_encrypt_key;
-    AES_KEY aes_decrypt_key;
-    uint32_t crypt_method = 0;
-    QDict *options;
-
-    /*
-     * Backing files are read-only which makes all of their metadata immutable,
-     * that means we don't have to worry about reopening them here.
-     */
-
-    if (s->crypt_method) {
-        crypt_method = s->crypt_method;
-        memcpy(&aes_encrypt_key, &s->aes_encrypt_key, sizeof(aes_encrypt_key));
-        memcpy(&aes_decrypt_key, &s->aes_decrypt_key, sizeof(aes_decrypt_key));
-    }
-
-    qcow2_close(bs);
-
-    options = qdict_new();
-    qdict_put(options, QCOW2_OPT_LAZY_REFCOUNTS,
-              qbool_from_int(s->use_lazy_refcounts));
-
-    memset(s, 0, sizeof(BDRVQcowState));
-    qcow2_open(bs, options, flags);
-
-    QDECREF(options);
-
-    if (crypt_method) {
-        s->crypt_method = crypt_method;
-        memcpy(&s->aes_encrypt_key, &aes_encrypt_key, sizeof(aes_encrypt_key));
-        memcpy(&s->aes_decrypt_key, &aes_decrypt_key, sizeof(aes_decrypt_key));
-    }
-}
-
-static size_t header_ext_add(char *buf, uint32_t magic, const void *s,
-    size_t len, size_t buflen)
-{
-    QCowExtension *ext_backing_fmt = (QCowExtension*) buf;
-    size_t ext_len = sizeof(QCowExtension) + ((len + 7) & ~7);
-
-    if (buflen < ext_len) {
-        return -ENOSPC;
-    }
-
-    *ext_backing_fmt = (QCowExtension) {
-        .magic  = cpu_to_be32(magic),
-        .len    = cpu_to_be32(len),
-    };
-    memcpy(buf + sizeof(QCowExtension), s, len);
-
-    return ext_len;
-}
-
-/*
- * Updates the qcow2 header, including the variable length parts of it, i.e.
- * the backing file name and all extensions. qcow2 was not designed to allow
- * such changes, so if we run out of space (we can only use the first cluster)
- * this function may fail.
- *
- * Returns 0 on success, -errno in error cases.
- */
-int qcow2_update_header(BlockDriverState *bs)
-{
-    BDRVQcowState *s = bs->opaque;
-    QCowHeader *header;
-    char *buf;
-    size_t buflen = s->cluster_size;
-    int ret;
-    uint64_t total_size;
-    uint32_t refcount_table_clusters;
-    size_t header_length;
-    Qcow2UnknownHeaderExtension *uext;
-
-    buf = qemu_blockalign(bs, buflen);
-
-    /* Header structure */
-    header = (QCowHeader*) buf;
-
-    if (buflen < sizeof(*header)) {
-        ret = -ENOSPC;
-        goto fail;
-    }
-
-    header_length = sizeof(*header) + s->unknown_header_fields_size;
-    total_size = bs->total_sectors * BDRV_SECTOR_SIZE;
-    refcount_table_clusters = s->refcount_table_size >> (s->cluster_bits - 3);
-
-    *header = (QCowHeader) {
-        /* Version 2 fields */
-        .magic                  = cpu_to_be32(QCOW_MAGIC),
-        .version                = cpu_to_be32(s->qcow_version),
-        .backing_file_offset    = 0,
-        .backing_file_size      = 0,
-        .cluster_bits           = cpu_to_be32(s->cluster_bits),
-        .size                   = cpu_to_be64(total_size),
-        .crypt_method           = cpu_to_be32(s->crypt_method_header),
-        .l1_size                = cpu_to_be32(s->l1_size),
-        .l1_table_offset        = cpu_to_be64(s->l1_table_offset),
-        .refcount_table_offset  = cpu_to_be64(s->refcount_table_offset),
-        .refcount_table_clusters = cpu_to_be32(refcount_table_clusters),
-        .nb_snapshots           = cpu_to_be32(s->nb_snapshots),
-        .snapshots_offset       = cpu_to_be64(s->snapshots_offset),
-
-        /* Version 3 fields */
-        .incompatible_features  = cpu_to_be64(s->incompatible_features),
-        .compatible_features    = cpu_to_be64(s->compatible_features),
-        .autoclear_features     = cpu_to_be64(s->autoclear_features),
-        .refcount_order         = cpu_to_be32(3 + REFCOUNT_SHIFT),
-        .header_length          = cpu_to_be32(header_length),
-    };
-
-    /* For older versions, write a shorter header */
-    switch (s->qcow_version) {
-    case 2:
-        ret = offsetof(QCowHeader, incompatible_features);
-        break;
-    case 3:
-        ret = sizeof(*header);
-        break;
-    default:
-        ret = -EINVAL;
-        goto fail;
-    }
-
-    buf += ret;
-    buflen -= ret;
-    memset(buf, 0, buflen);
-
-    /* Preserve any unknown field in the header */
-    if (s->unknown_header_fields_size) {
-        if (buflen < s->unknown_header_fields_size) {
-            ret = -ENOSPC;
-            goto fail;
-        }
-
-        memcpy(buf, s->unknown_header_fields, s->unknown_header_fields_size);
-        buf += s->unknown_header_fields_size;
-        buflen -= s->unknown_header_fields_size;
-    }
-
-    /* Backing file format header extension */
-    if (*bs->backing_format) {
-        ret = header_ext_add(buf, QCOW2_EXT_MAGIC_BACKING_FORMAT,
-                             bs->backing_format, strlen(bs->backing_format),
-                             buflen);
-        if (ret < 0) {
-            goto fail;
-        }
-
-        buf += ret;
-        buflen -= ret;
-    }
-
-    /* Feature table */
-    Qcow2Feature features[] = {
-        {
-            .type = QCOW2_FEAT_TYPE_INCOMPATIBLE,
-            .bit  = QCOW2_INCOMPAT_DIRTY_BITNR,
-            .name = "dirty bit",
-        },
-        {
-            .type = QCOW2_FEAT_TYPE_COMPATIBLE,
-            .bit  = QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR,
-            .name = "lazy refcounts",
-        },
-    };
-
-    ret = header_ext_add(buf, QCOW2_EXT_MAGIC_FEATURE_TABLE,
-                         features, sizeof(features), buflen);
-    if (ret < 0) {
-        goto fail;
-    }
-    buf += ret;
-    buflen -= ret;
-
-    /* Keep unknown header extensions */
-    QLIST_FOREACH(uext, &s->unknown_header_ext, next) {
-        ret = header_ext_add(buf, uext->magic, uext->data, uext->len, buflen);
-        if (ret < 0) {
-            goto fail;
-        }
-
-        buf += ret;
-        buflen -= ret;
-    }
-
-    /* End of header extensions */
-    ret = header_ext_add(buf, QCOW2_EXT_MAGIC_END, NULL, 0, buflen);
-    if (ret < 0) {
-        goto fail;
-    }
-
-    buf += ret;
-    buflen -= ret;
-
-    /* Backing file name */
-    if (*bs->backing_file) {
-        size_t backing_file_len = strlen(bs->backing_file);
-
-        if (buflen < backing_file_len) {
-            ret = -ENOSPC;
-            goto fail;
-        }
-
-        /* Using strncpy is ok here, since buf is not NUL-terminated. */
-        strncpy(buf, bs->backing_file, buflen);
-
-        header->backing_file_offset = cpu_to_be64(buf - ((char*) header));
-        header->backing_file_size   = cpu_to_be32(backing_file_len);
-    }
-
-    /* Write the new header */
-    ret = bdrv_pwrite(bs->file, 0, header, s->cluster_size);
-    if (ret < 0) {
-        goto fail;
-    }
-
-    ret = 0;
-fail:
-    qemu_vfree(header);
-    return ret;
-}
-
-static int qcow2_change_backing_file(BlockDriverState *bs,
-    const char *backing_file, const char *backing_fmt)
-{
-    pstrcpy(bs->backing_file, sizeof(bs->backing_file), backing_file ?: "");
-    pstrcpy(bs->backing_format, sizeof(bs->backing_format), backing_fmt ?: "");
-
-    return qcow2_update_header(bs);
-}
-
-static int preallocate(BlockDriverState *bs)
-{
-    uint64_t nb_sectors;
-    uint64_t offset;
-    uint64_t host_offset = 0;
-    int num;
-    int ret;
-    QCowL2Meta *meta;
-
-    nb_sectors = bdrv_getlength(bs) >> 9;
-    offset = 0;
-
-    while (nb_sectors) {
-        num = MIN(nb_sectors, INT_MAX >> 9);
-        ret = qcow2_alloc_cluster_offset(bs, offset, 0, num, &num,
-                                         &host_offset, &meta);
-        if (ret < 0) {
-            return ret;
-        }
-
-        ret = qcow2_alloc_cluster_link_l2(bs, meta);
-        if (ret < 0) {
-            qcow2_free_any_clusters(bs, meta->alloc_offset, meta->nb_clusters,
-                                    QCOW2_DISCARD_NEVER);
-            return ret;
-        }
-
-        /* There are no dependent requests, but we need to remove our request
-         * from the list of in-flight requests */
-        if (meta != NULL) {
-            QLIST_REMOVE(meta, next_in_flight);
-        }
-
-        /* TODO Preallocate data if requested */
-
-        nb_sectors -= num;
-        offset += num << 9;
-    }
-
-    /*
-     * It is expected that the image file is large enough to actually contain
-     * all of the allocated clusters (otherwise we get failing reads after
-     * EOF). Extend the image to the last allocated sector.
-     */
-    if (host_offset != 0) {
-        uint8_t buf[512];
-        memset(buf, 0, 512);
-        ret = bdrv_write(bs->file, (host_offset >> 9) + num - 1, buf, 1);
-        if (ret < 0) {
-            return ret;
-        }
-    }
-
-    return 0;
-}
-
-static int qcow2_create2(const char *filename, int64_t total_size,
-                         const char *backing_file, const char *backing_format,
-                         int flags, size_t cluster_size, int prealloc,
-                         QEMUOptionParameter *options, int version)
-{
-    /* Calculate cluster_bits */
-    int cluster_bits;
-    cluster_bits = ffs(cluster_size) - 1;
-    if (cluster_bits < MIN_CLUSTER_BITS || cluster_bits > MAX_CLUSTER_BITS ||
-        (1 << cluster_bits) != cluster_size)
-    {
-        error_report(
-            "Cluster size must be a power of two between %d and %dk",
-            1 << MIN_CLUSTER_BITS, 1 << (MAX_CLUSTER_BITS - 10));
-        return -EINVAL;
-    }
-
-    /*
-     * Open the image file and write a minimal qcow2 header.
-     *
-     * We keep things simple and start with a zero-sized image. We also
-     * do without refcount blocks or a L1 table for now. We'll fix the
-     * inconsistency later.
-     *
-     * We do need a refcount table because growing the refcount table means
-     * allocating two new refcount blocks - the seconds of which would be at
-     * 2 GB for 64k clusters, and we don't want to have a 2 GB initial file
-     * size for any qcow2 image.
-     */
-    BlockDriverState* bs;
-    QCowHeader header;
-    uint8_t* refcount_table;
-    int ret;
-
-    ret = bdrv_create_file(filename, options);
-    if (ret < 0) {
-        return ret;
-    }
-
-    ret = bdrv_file_open(&bs, filename, NULL, BDRV_O_RDWR);
-    if (ret < 0) {
-        return ret;
-    }
-
-    /* Write the header */
-    memset(&header, 0, sizeof(header));
-    header.magic = cpu_to_be32(QCOW_MAGIC);
-    header.version = cpu_to_be32(version);
-    header.cluster_bits = cpu_to_be32(cluster_bits);
-    header.size = cpu_to_be64(0);
-    header.l1_table_offset = cpu_to_be64(0);
-    header.l1_size = cpu_to_be32(0);
-    header.refcount_table_offset = cpu_to_be64(cluster_size);
-    header.refcount_table_clusters = cpu_to_be32(1);
-    header.refcount_order = cpu_to_be32(3 + REFCOUNT_SHIFT);
-    header.header_length = cpu_to_be32(sizeof(header));
-
-    if (flags & BLOCK_FLAG_ENCRYPT) {
-        header.crypt_method = cpu_to_be32(QCOW_CRYPT_AES);
-    } else {
-        header.crypt_method = cpu_to_be32(QCOW_CRYPT_NONE);
-    }
-
-    if (flags & BLOCK_FLAG_LAZY_REFCOUNTS) {
-        header.compatible_features |=
-            cpu_to_be64(QCOW2_COMPAT_LAZY_REFCOUNTS);
-    }
-
-    ret = bdrv_pwrite(bs, 0, &header, sizeof(header));
-    if (ret < 0) {
-        goto out;
-    }
-
-    /* Write an empty refcount table */
-    refcount_table = g_malloc0(cluster_size);
-    ret = bdrv_pwrite(bs, cluster_size, refcount_table, cluster_size);
-    g_free(refcount_table);
-
-    if (ret < 0) {
-        goto out;
-    }
-
-    bdrv_close(bs);
-
-    /*
-     * And now open the image and make it consistent first (i.e. increase the
-     * refcount of the cluster that is occupied by the header and the refcount
-     * table)
-     */
-    BlockDriver* drv = bdrv_find_format("qcow2");
-    assert(drv != NULL);
-    ret = bdrv_open(bs, filename, NULL,
-        BDRV_O_RDWR | BDRV_O_CACHE_WB | BDRV_O_NO_FLUSH, drv);
-    if (ret < 0) {
-        goto out;
-    }
-
-    ret = qcow2_alloc_clusters(bs, 2 * cluster_size);
-    if (ret < 0) {
-        goto out;
-
-    } else if (ret != 0) {
-        error_report("Huh, first cluster in empty image is already in use?");
-        abort();
-    }
-
-    /* Okay, now that we have a valid image, let's give it the right size */
-    ret = bdrv_truncate(bs, total_size * BDRV_SECTOR_SIZE);
-    if (ret < 0) {
-        goto out;
-    }
-
-    /* Want a backing file? There you go.*/
-    if (backing_file) {
-        ret = bdrv_change_backing_file(bs, backing_file, backing_format);
-        if (ret < 0) {
-            goto out;
-        }
-    }
-
-    /* And if we're supposed to preallocate metadata, do that now */
-    if (prealloc) {
-        BDRVQcowState *s = bs->opaque;
-        qemu_co_mutex_lock(&s->lock);
-        ret = preallocate(bs);
-        qemu_co_mutex_unlock(&s->lock);
-        if (ret < 0) {
-            goto out;
-        }
-    }
-
-    ret = 0;
-out:
-    bdrv_delete(bs);
-    return ret;
-}
-
-static int qcow2_create(const char *filename, QEMUOptionParameter *options)
-{
-    const char *backing_file = NULL;
-    const char *backing_fmt = NULL;
-    uint64_t sectors = 0;
-    int flags = 0;
-    size_t cluster_size = DEFAULT_CLUSTER_SIZE;
-    int prealloc = 0;
-    int version = 2;
-
-    /* Read out options */
-    while (options && options->name) {
-        if (!strcmp(options->name, BLOCK_OPT_SIZE)) {
-            sectors = options->value.n / 512;
-        } else if (!strcmp(options->name, BLOCK_OPT_BACKING_FILE)) {
-            backing_file = options->value.s;
-        } else if (!strcmp(options->name, BLOCK_OPT_BACKING_FMT)) {
-            backing_fmt = options->value.s;
-        } else if (!strcmp(options->name, BLOCK_OPT_ENCRYPT)) {
-            flags |= options->value.n ? BLOCK_FLAG_ENCRYPT : 0;
-        } else if (!strcmp(options->name, BLOCK_OPT_CLUSTER_SIZE)) {
-            if (options->value.n) {
-                cluster_size = options->value.n;
-            }
-        } else if (!strcmp(options->name, BLOCK_OPT_PREALLOC)) {
-            if (!options->value.s || !strcmp(options->value.s, "off")) {
-                prealloc = 0;
-            } else if (!strcmp(options->value.s, "metadata")) {
-                prealloc = 1;
-            } else {
-                fprintf(stderr, "Invalid preallocation mode: '%s'\n",
-                    options->value.s);
-                return -EINVAL;
-            }
-        } else if (!strcmp(options->name, BLOCK_OPT_COMPAT_LEVEL)) {
-            if (!options->value.s || !strcmp(options->value.s, "0.10")) {
-                version = 2;
-            } else if (!strcmp(options->value.s, "1.1")) {
-                version = 3;
-            } else {
-                fprintf(stderr, "Invalid compatibility level: '%s'\n",
-                    options->value.s);
-                return -EINVAL;
-            }
-        } else if (!strcmp(options->name, BLOCK_OPT_LAZY_REFCOUNTS)) {
-            flags |= options->value.n ? BLOCK_FLAG_LAZY_REFCOUNTS : 0;
-        }
-        options++;
-    }
-
-    if (backing_file && prealloc) {
-        fprintf(stderr, "Backing file and preallocation cannot be used at "
-            "the same time\n");
-        return -EINVAL;
-    }
-
-    if (version < 3 && (flags & BLOCK_FLAG_LAZY_REFCOUNTS)) {
-        fprintf(stderr, "Lazy refcounts only supported with compatibility "
-                "level 1.1 and above (use compat=1.1 or greater)\n");
-        return -EINVAL;
-    }
-
-    return qcow2_create2(filename, sectors, backing_file, backing_fmt, flags,
-                         cluster_size, prealloc, options, version);
-}
-
-static int qcow2_make_empty(BlockDriverState *bs)
-{
-#if 0
-    /* XXX: not correct */
-    BDRVQcowState *s = bs->opaque;
-    uint32_t l1_length = s->l1_size * sizeof(uint64_t);
-    int ret;
-
-    memset(s->l1_table, 0, l1_length);
-    if (bdrv_pwrite(bs->file, s->l1_table_offset, s->l1_table, l1_length) < 0)
-        return -1;
-    ret = bdrv_truncate(bs->file, s->l1_table_offset + l1_length);
-    if (ret < 0)
-        return ret;
-
-    l2_cache_reset(bs);
-#endif
-    return 0;
-}
-
-static coroutine_fn int qcow2_co_write_zeroes(BlockDriverState *bs,
-    int64_t sector_num, int nb_sectors)
-{
-    int ret;
-    BDRVQcowState *s = bs->opaque;
-
-    /* Emulate misaligned zero writes */
-    if (sector_num % s->cluster_sectors || nb_sectors % s->cluster_sectors) {
-        return -ENOTSUP;
-    }
-
-    /* Whatever is left can use real zero clusters */
-    qemu_co_mutex_lock(&s->lock);
-    ret = qcow2_zero_clusters(bs, sector_num << BDRV_SECTOR_BITS,
-        nb_sectors);
-    qemu_co_mutex_unlock(&s->lock);
-
-    return ret;
-}
-
-static coroutine_fn int qcow2_co_discard(BlockDriverState *bs,
-    int64_t sector_num, int nb_sectors)
-{
-    int ret;
-    BDRVQcowState *s = bs->opaque;
-
-    qemu_co_mutex_lock(&s->lock);
-    ret = qcow2_discard_clusters(bs, sector_num << BDRV_SECTOR_BITS,
-        nb_sectors);
-    qemu_co_mutex_unlock(&s->lock);
-    return ret;
-}
-
-static int qcow2_truncate(BlockDriverState *bs, int64_t offset)
-{
-    BDRVQcowState *s = bs->opaque;
-    int64_t new_l1_size;
-    int ret;
-
-    if (offset & 511) {
-        error_report("The new size must be a multiple of 512");
-        return -EINVAL;
-    }
-
-    /* cannot proceed if image has snapshots */
-    if (s->nb_snapshots) {
-        error_report("Can't resize an image which has snapshots");
-        return -ENOTSUP;
-    }
-
-    /* shrinking is currently not supported */
-    if (offset < bs->total_sectors * 512) {
-        error_report("qcow2 doesn't support shrinking images yet");
-        return -ENOTSUP;
-    }
-
-    new_l1_size = size_to_l1(s, offset);
-    ret = qcow2_grow_l1_table(bs, new_l1_size, true);
-    if (ret < 0) {
-        return ret;
-    }
-
-    /* write updated header.size */
-    offset = cpu_to_be64(offset);
-    ret = bdrv_pwrite_sync(bs->file, offsetof(QCowHeader, size),
-                           &offset, sizeof(uint64_t));
-    if (ret < 0) {
-        return ret;
-    }
-
-    s->l1_vm_state_index = new_l1_size;
-    return 0;
-}
-
-/* XXX: put compressed sectors first, then all the cluster aligned
-   tables to avoid losing bytes in alignment */
-static int qcow2_write_compressed(BlockDriverState *bs, int64_t sector_num,
-                                  const uint8_t *buf, int nb_sectors)
-{
-    BDRVQcowState *s = bs->opaque;
-    z_stream strm;
-    int ret, out_len;
-    uint8_t *out_buf;
-    uint64_t cluster_offset;
-
-    if (nb_sectors == 0) {
-        /* align end of file to a sector boundary to ease reading with
-           sector based I/Os */
-        cluster_offset = bdrv_getlength(bs->file);
-        cluster_offset = (cluster_offset + 511) & ~511;
-        bdrv_truncate(bs->file, cluster_offset);
-        return 0;
-    }
-
-    if (nb_sectors != s->cluster_sectors) {
-        ret = -EINVAL;
-
-        /* Zero-pad last write if image size is not cluster aligned */
-        if (sector_num + nb_sectors == bs->total_sectors &&
-            nb_sectors < s->cluster_sectors) {
-            uint8_t *pad_buf = qemu_blockalign(bs, s->cluster_size);
-            memset(pad_buf, 0, s->cluster_size);
-            memcpy(pad_buf, buf, nb_sectors * BDRV_SECTOR_SIZE);
-            ret = qcow2_write_compressed(bs, sector_num,
-                                         pad_buf, s->cluster_sectors);
-            qemu_vfree(pad_buf);
-        }
-        return ret;
-    }
-
-    out_buf = g_malloc(s->cluster_size + (s->cluster_size / 1000) + 128);
-
-    /* best compression, small window, no zlib header */
-    memset(&strm, 0, sizeof(strm));
-    ret = deflateInit2(&strm, Z_DEFAULT_COMPRESSION,
-                       Z_DEFLATED, -12,
-                       9, Z_DEFAULT_STRATEGY);
-    if (ret != 0) {
-        ret = -EINVAL;
-        goto fail;
-    }
-
-    strm.avail_in = s->cluster_size;
-    strm.next_in = (uint8_t *)buf;
-    strm.avail_out = s->cluster_size;
-    strm.next_out = out_buf;
-
-    ret = deflate(&strm, Z_FINISH);
-    if (ret != Z_STREAM_END && ret != Z_OK) {
-        deflateEnd(&strm);
-        ret = -EINVAL;
-        goto fail;
-    }
-    out_len = strm.next_out - out_buf;
-
-    deflateEnd(&strm);
-
-    if (ret != Z_STREAM_END || out_len >= s->cluster_size) {
-        /* could not compress: write normal cluster */
-        ret = bdrv_write(bs, sector_num, buf, s->cluster_sectors);
-        if (ret < 0) {
-            goto fail;
-        }
-    } else {
-        cluster_offset = qcow2_alloc_compressed_cluster_offset(bs,
-            sector_num << 9, out_len);
-        if (!cluster_offset) {
-            ret = -EIO;
-            goto fail;
-        }
-        cluster_offset &= s->cluster_offset_mask;
-        BLKDBG_EVENT(bs->file, BLKDBG_WRITE_COMPRESSED);
-        ret = bdrv_pwrite(bs->file, cluster_offset, out_buf, out_len);
-        if (ret < 0) {
-            goto fail;
-        }
-    }
-
-    ret = 0;
-fail:
-    g_free(out_buf);
-    return ret;
-}
-
-static coroutine_fn int qcow2_co_flush_to_os(BlockDriverState *bs)
-{
-    BDRVQcowState *s = bs->opaque;
-    int ret;
-
-    qemu_co_mutex_lock(&s->lock);
-    ret = qcow2_cache_flush(bs, s->l2_table_cache);
-    if (ret < 0) {
-        qemu_co_mutex_unlock(&s->lock);
-        return ret;
-    }
-
-    if (qcow2_need_accurate_refcounts(s)) {
-        ret = qcow2_cache_flush(bs, s->refcount_block_cache);
-        if (ret < 0) {
-            qemu_co_mutex_unlock(&s->lock);
-            return ret;
-        }
-    }
-    qemu_co_mutex_unlock(&s->lock);
-
-    return 0;
-}
-
-static int64_t qcow2_vm_state_offset(BDRVQcowState *s)
-{
-	return (int64_t)s->l1_vm_state_index << (s->cluster_bits + s->l2_bits);
-}
-
-static int qcow2_get_info(BlockDriverState *bs, BlockDriverInfo *bdi)
-{
-    BDRVQcowState *s = bs->opaque;
-    bdi->cluster_size = s->cluster_size;
-    bdi->vm_state_offset = qcow2_vm_state_offset(s);
-    return 0;
-}
-
-#if 0
-static void dump_refcounts(BlockDriverState *bs)
-{
-    BDRVQcowState *s = bs->opaque;
-    int64_t nb_clusters, k, k1, size;
-    int refcount;
-
-    size = bdrv_getlength(bs->file);
-    nb_clusters = size_to_clusters(s, size);
-    for(k = 0; k < nb_clusters;) {
-        k1 = k;
-        refcount = get_refcount(bs, k);
-        k++;
-        while (k < nb_clusters && get_refcount(bs, k) == refcount)
-            k++;
-        printf("%" PRId64 ": refcount=%d nb=%" PRId64 "\n", k, refcount,
-               k - k1);
-    }
-}
-#endif
-
-static int qcow2_save_vmstate(BlockDriverState *bs, QEMUIOVector *qiov,
-                              int64_t pos)
-{
-    BDRVQcowState *s = bs->opaque;
-    int growable = bs->growable;
-    int ret;
-
-    BLKDBG_EVENT(bs->file, BLKDBG_VMSTATE_SAVE);
-    bs->growable = 1;
-    ret = bdrv_pwritev(bs, qcow2_vm_state_offset(s) + pos, qiov);
-    bs->growable = growable;
-
-    return ret;
-}
-
-static int qcow2_load_vmstate(BlockDriverState *bs, uint8_t *buf,
-                              int64_t pos, int size)
-{
-    BDRVQcowState *s = bs->opaque;
-    int growable = bs->growable;
-    int ret;
-
-    BLKDBG_EVENT(bs->file, BLKDBG_VMSTATE_LOAD);
-    bs->growable = 1;
-    ret = bdrv_pread(bs, qcow2_vm_state_offset(s) + pos, buf, size);
-    bs->growable = growable;
-
-    return ret;
-}
-
-static QEMUOptionParameter qcow2_create_options[] = {
-    {
-        .name = BLOCK_OPT_SIZE,
-        .type = OPT_SIZE,
-        .help = "Virtual disk size"
-    },
-    {
-        .name = BLOCK_OPT_COMPAT_LEVEL,
-        .type = OPT_STRING,
-        .help = "Compatibility level (0.10 or 1.1)"
-    },
-    {
-        .name = BLOCK_OPT_BACKING_FILE,
-        .type = OPT_STRING,
-        .help = "File name of a base image"
-    },
-    {
-        .name = BLOCK_OPT_BACKING_FMT,
-        .type = OPT_STRING,
-        .help = "Image format of the base image"
-    },
-    {
-        .name = BLOCK_OPT_ENCRYPT,
-        .type = OPT_FLAG,
-        .help = "Encrypt the image"
-    },
-    {
-        .name = BLOCK_OPT_CLUSTER_SIZE,
-        .type = OPT_SIZE,
-        .help = "qcow2 cluster size",
-        .value = { .n = DEFAULT_CLUSTER_SIZE },
-    },
-    {
-        .name = BLOCK_OPT_PREALLOC,
-        .type = OPT_STRING,
-        .help = "Preallocation mode (allowed values: off, metadata)"
-    },
-    {
-        .name = BLOCK_OPT_LAZY_REFCOUNTS,
-        .type = OPT_FLAG,
-        .help = "Postpone refcount updates",
-    },
-    { NULL }
-};
-
-static BlockDriver bdrv_qcow2 = {
-    .format_name        = "qcow2",
-    .instance_size      = sizeof(BDRVQcowState),
-    .bdrv_probe         = qcow2_probe,
-    .bdrv_open          = qcow2_open,
-    .bdrv_close         = qcow2_close,
-    .bdrv_reopen_prepare  = qcow2_reopen_prepare,
-    .bdrv_create        = qcow2_create,
-    .bdrv_has_zero_init = bdrv_has_zero_init_1,
-    .bdrv_co_is_allocated = qcow2_co_is_allocated,
-    .bdrv_set_key       = qcow2_set_key,
-    .bdrv_make_empty    = qcow2_make_empty,
-
-    .bdrv_co_readv          = qcow2_co_readv,
-    .bdrv_co_writev         = qcow2_co_writev,
-    .bdrv_co_flush_to_os    = qcow2_co_flush_to_os,
-
-    .bdrv_co_write_zeroes   = qcow2_co_write_zeroes,
-    .bdrv_co_discard        = qcow2_co_discard,
-    .bdrv_truncate          = qcow2_truncate,
-    .bdrv_write_compressed  = qcow2_write_compressed,
-
-    .bdrv_snapshot_create   = qcow2_snapshot_create,
-    .bdrv_snapshot_goto     = qcow2_snapshot_goto,
-    .bdrv_snapshot_delete   = qcow2_snapshot_delete,
-    .bdrv_snapshot_list     = qcow2_snapshot_list,
-    .bdrv_snapshot_load_tmp     = qcow2_snapshot_load_tmp,
-    .bdrv_get_info      = qcow2_get_info,
-
-    .bdrv_save_vmstate    = qcow2_save_vmstate,
-    .bdrv_load_vmstate    = qcow2_load_vmstate,
-
-    .bdrv_change_backing_file   = qcow2_change_backing_file,
-
-    .bdrv_invalidate_cache      = qcow2_invalidate_cache,
-
-    .create_options = qcow2_create_options,
-    .bdrv_check = qcow2_check,
-};
-
-static void bdrv_qcow2_init(void)
-{
-    bdrv_register(&bdrv_qcow2);
-}
-
-block_init(bdrv_qcow2_init);
diff --git a/contrib/qemu/block/qcow2.h b/contrib/qemu/block/qcow2.h
deleted file mode 100644
index 3b2d5cda71f..00000000000
--- a/contrib/qemu/block/qcow2.h
+++ /dev/null
@@ -1,437 +0,0 @@
-/*
- * Block driver for the QCOW version 2 format
- *
- * Copyright (c) 2004-2006 Fabrice Bellard
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-
-#ifndef BLOCK_QCOW2_H
-#define BLOCK_QCOW2_H
-
-#include "qemu/aes.h"
-#include "block/coroutine.h"
-
-//#define DEBUG_ALLOC
-//#define DEBUG_ALLOC2
-//#define DEBUG_EXT
-
-#define QCOW_MAGIC (('Q' << 24) | ('F' << 16) | ('I' << 8) | 0xfb)
-
-#define QCOW_CRYPT_NONE 0
-#define QCOW_CRYPT_AES  1
-
-#define QCOW_MAX_CRYPT_CLUSTERS 32
-
-/* indicate that the refcount of the referenced cluster is exactly one. */
-#define QCOW_OFLAG_COPIED     (1LL << 63)
-/* indicate that the cluster is compressed (they never have the copied flag) */
-#define QCOW_OFLAG_COMPRESSED (1LL << 62)
-/* The cluster reads as all zeros */
-#define QCOW_OFLAG_ZERO (1LL << 0)
-
-#define REFCOUNT_SHIFT 1 /* refcount size is 2 bytes */
-
-#define MIN_CLUSTER_BITS 9
-#define MAX_CLUSTER_BITS 21
-
-#define L2_CACHE_SIZE 16
-
-/* Must be at least 4 to cover all cases of refcount table growth */
-#define REFCOUNT_CACHE_SIZE 4
-
-#define DEFAULT_CLUSTER_SIZE 65536
-
-
-#define QCOW2_OPT_LAZY_REFCOUNTS "lazy_refcounts"
-#define QCOW2_OPT_DISCARD_REQUEST "pass_discard_request"
-#define QCOW2_OPT_DISCARD_SNAPSHOT "pass_discard_snapshot"
-#define QCOW2_OPT_DISCARD_OTHER "pass_discard_other"
-
-typedef struct QCowHeader {
-    uint32_t magic;
-    uint32_t version;
-    uint64_t backing_file_offset;
-    uint32_t backing_file_size;
-    uint32_t cluster_bits;
-    uint64_t size; /* in bytes */
-    uint32_t crypt_method;
-    uint32_t l1_size; /* XXX: save number of clusters instead ? */
-    uint64_t l1_table_offset;
-    uint64_t refcount_table_offset;
-    uint32_t refcount_table_clusters;
-    uint32_t nb_snapshots;
-    uint64_t snapshots_offset;
-
-    /* The following fields are only valid for version >= 3 */
-    uint64_t incompatible_features;
-    uint64_t compatible_features;
-    uint64_t autoclear_features;
-
-    uint32_t refcount_order;
-    uint32_t header_length;
-} QCowHeader;
-
-typedef struct QCowSnapshot {
-    uint64_t l1_table_offset;
-    uint32_t l1_size;
-    char *id_str;
-    char *name;
-    uint64_t disk_size;
-    uint64_t vm_state_size;
-    uint32_t date_sec;
-    uint32_t date_nsec;
-    uint64_t vm_clock_nsec;
-} QCowSnapshot;
-
-struct Qcow2Cache;
-typedef struct Qcow2Cache Qcow2Cache;
-
-typedef struct Qcow2UnknownHeaderExtension {
-    uint32_t magic;
-    uint32_t len;
-    QLIST_ENTRY(Qcow2UnknownHeaderExtension) next;
-    uint8_t data[];
-} Qcow2UnknownHeaderExtension;
-
-enum {
-    QCOW2_FEAT_TYPE_INCOMPATIBLE    = 0,
-    QCOW2_FEAT_TYPE_COMPATIBLE      = 1,
-    QCOW2_FEAT_TYPE_AUTOCLEAR       = 2,
-};
-
-/* Incompatible feature bits */
-enum {
-    QCOW2_INCOMPAT_DIRTY_BITNR   = 0,
-    QCOW2_INCOMPAT_DIRTY         = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
-
-    QCOW2_INCOMPAT_MASK          = QCOW2_INCOMPAT_DIRTY,
-};
-
-/* Compatible feature bits */
-enum {
-    QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR = 0,
-    QCOW2_COMPAT_LAZY_REFCOUNTS       = 1 << QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR,
-
-    QCOW2_COMPAT_FEAT_MASK            = QCOW2_COMPAT_LAZY_REFCOUNTS,
-};
-
-enum qcow2_discard_type {
-    QCOW2_DISCARD_NEVER = 0,
-    QCOW2_DISCARD_ALWAYS,
-    QCOW2_DISCARD_REQUEST,
-    QCOW2_DISCARD_SNAPSHOT,
-    QCOW2_DISCARD_OTHER,
-    QCOW2_DISCARD_MAX
-};
-
-typedef struct Qcow2Feature {
-    uint8_t type;
-    uint8_t bit;
-    char    name[46];
-} QEMU_PACKED Qcow2Feature;
-
-typedef struct Qcow2DiscardRegion {
-    BlockDriverState *bs;
-    uint64_t offset;
-    uint64_t bytes;
-    QTAILQ_ENTRY(Qcow2DiscardRegion) next;
-} Qcow2DiscardRegion;
-
-typedef struct BDRVQcowState {
-    int cluster_bits;
-    int cluster_size;
-    int cluster_sectors;
-    int l2_bits;
-    int l2_size;
-    int l1_size;
-    int l1_vm_state_index;
-    int csize_shift;
-    int csize_mask;
-    uint64_t cluster_offset_mask;
-    uint64_t l1_table_offset;
-    uint64_t *l1_table;
-
-    Qcow2Cache* l2_table_cache;
-    Qcow2Cache* refcount_block_cache;
-
-    uint8_t *cluster_cache;
-    uint8_t *cluster_data;
-    uint64_t cluster_cache_offset;
-    QLIST_HEAD(QCowClusterAlloc, QCowL2Meta) cluster_allocs;
-
-    uint64_t *refcount_table;
-    uint64_t refcount_table_offset;
-    uint32_t refcount_table_size;
-    int64_t free_cluster_index;
-    int64_t free_byte_offset;
-
-    CoMutex lock;
-
-    uint32_t crypt_method; /* current crypt method, 0 if no key yet */
-    uint32_t crypt_method_header;
-    AES_KEY aes_encrypt_key;
-    AES_KEY aes_decrypt_key;
-    uint64_t snapshots_offset;
-    int snapshots_size;
-    int nb_snapshots;
-    QCowSnapshot *snapshots;
-
-    int flags;
-    int qcow_version;
-    bool use_lazy_refcounts;
-
-    bool discard_passthrough[QCOW2_DISCARD_MAX];
-
-    uint64_t incompatible_features;
-    uint64_t compatible_features;
-    uint64_t autoclear_features;
-
-    size_t unknown_header_fields_size;
-    void* unknown_header_fields;
-    QLIST_HEAD(, Qcow2UnknownHeaderExtension) unknown_header_ext;
-    QTAILQ_HEAD (, Qcow2DiscardRegion) discards;
-    bool cache_discards;
-} BDRVQcowState;
-
-/* XXX: use std qcow open function ? */
-typedef struct QCowCreateState {
-    int cluster_size;
-    int cluster_bits;
-    uint16_t *refcount_block;
-    uint64_t *refcount_table;
-    int64_t l1_table_offset;
-    int64_t refcount_table_offset;
-    int64_t refcount_block_offset;
-} QCowCreateState;
-
-struct QCowAIOCB;
-
-typedef struct Qcow2COWRegion {
-    /**
-     * Offset of the COW region in bytes from the start of the first cluster
-     * touched by the request.
-     */
-    uint64_t    offset;
-
-    /** Number of sectors to copy */
-    int         nb_sectors;
-} Qcow2COWRegion;
-
-/**
- * Describes an in-flight (part of a) write request that writes to clusters
- * that are not referenced in their L2 table yet.
- */
-typedef struct QCowL2Meta
-{
-    /** Guest offset of the first newly allocated cluster */
-    uint64_t offset;
-
-    /** Host offset of the first newly allocated cluster */
-    uint64_t alloc_offset;
-
-    /**
-     * Number of sectors from the start of the first allocated cluster to
-     * the end of the (possibly shortened) request
-     */
-    int nb_available;
-
-    /** Number of newly allocated clusters */
-    int nb_clusters;
-
-    /**
-     * Requests that overlap with this allocation and wait to be restarted
-     * when the allocating request has completed.
-     */
-    CoQueue dependent_requests;
-
-    /**
-     * The COW Region between the start of the first allocated cluster and the
-     * area the guest actually writes to.
-     */
-    Qcow2COWRegion cow_start;
-
-    /**
-     * The COW Region between the area the guest actually writes to and the
-     * end of the last allocated cluster.
-     */
-    Qcow2COWRegion cow_end;
-
-    /** Pointer to next L2Meta of the same write request */
-    struct QCowL2Meta *next;
-
-    QLIST_ENTRY(QCowL2Meta) next_in_flight;
-} QCowL2Meta;
-
-enum {
-    QCOW2_CLUSTER_UNALLOCATED,
-    QCOW2_CLUSTER_NORMAL,
-    QCOW2_CLUSTER_COMPRESSED,
-    QCOW2_CLUSTER_ZERO
-};
-
-#define L1E_OFFSET_MASK 0x00ffffffffffff00ULL
-#define L2E_OFFSET_MASK 0x00ffffffffffff00ULL
-#define L2E_COMPRESSED_OFFSET_SIZE_MASK 0x3fffffffffffffffULL
-
-#define REFT_OFFSET_MASK 0xffffffffffffff00ULL
-
-static inline int64_t start_of_cluster(BDRVQcowState *s, int64_t offset)
-{
-    return offset & ~(s->cluster_size - 1);
-}
-
-static inline int64_t offset_into_cluster(BDRVQcowState *s, int64_t offset)
-{
-    return offset & (s->cluster_size - 1);
-}
-
-static inline int size_to_clusters(BDRVQcowState *s, int64_t size)
-{
-    return (size + (s->cluster_size - 1)) >> s->cluster_bits;
-}
-
-static inline int64_t size_to_l1(BDRVQcowState *s, int64_t size)
-{
-    int shift = s->cluster_bits + s->l2_bits;
-    return (size + (1ULL << shift) - 1) >> shift;
-}
-
-static inline int offset_to_l2_index(BDRVQcowState *s, int64_t offset)
-{
-    return (offset >> s->cluster_bits) & (s->l2_size - 1);
-}
-
-static inline int64_t align_offset(int64_t offset, int n)
-{
-    offset = (offset + n - 1) & ~(n - 1);
-    return offset;
-}
-
-static inline int qcow2_get_cluster_type(uint64_t l2_entry)
-{
-    if (l2_entry & QCOW_OFLAG_COMPRESSED) {
-        return QCOW2_CLUSTER_COMPRESSED;
-    } else if (l2_entry & QCOW_OFLAG_ZERO) {
-        return QCOW2_CLUSTER_ZERO;
-    } else if (!(l2_entry & L2E_OFFSET_MASK)) {
-        return QCOW2_CLUSTER_UNALLOCATED;
-    } else {
-        return QCOW2_CLUSTER_NORMAL;
-    }
-}
-
-/* Check whether refcounts are eager or lazy */
-static inline bool qcow2_need_accurate_refcounts(BDRVQcowState *s)
-{
-    return !(s->incompatible_features & QCOW2_INCOMPAT_DIRTY);
-}
-
-static inline uint64_t l2meta_cow_start(QCowL2Meta *m)
-{
-    return m->offset + m->cow_start.offset;
-}
-
-static inline uint64_t l2meta_cow_end(QCowL2Meta *m)
-{
-    return m->offset + m->cow_end.offset
-        + (m->cow_end.nb_sectors << BDRV_SECTOR_BITS);
-}
-
-// FIXME Need qcow2_ prefix to global functions
-
-/* qcow2.c functions */
-int qcow2_backing_read1(BlockDriverState *bs, QEMUIOVector *qiov,
-                  int64_t sector_num, int nb_sectors);
-
-int qcow2_mark_dirty(BlockDriverState *bs);
-int qcow2_update_header(BlockDriverState *bs);
-
-/* qcow2-refcount.c functions */
-int qcow2_refcount_init(BlockDriverState *bs);
-void qcow2_refcount_close(BlockDriverState *bs);
-
-int64_t qcow2_alloc_clusters(BlockDriverState *bs, int64_t size);
-int qcow2_alloc_clusters_at(BlockDriverState *bs, uint64_t offset,
-    int nb_clusters);
-int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size);
-void qcow2_free_clusters(BlockDriverState *bs,
-                          int64_t offset, int64_t size,
-                          enum qcow2_discard_type type);
-void qcow2_free_any_clusters(BlockDriverState *bs, uint64_t l2_entry,
-                             int nb_clusters, enum qcow2_discard_type type);
-
-int qcow2_update_snapshot_refcount(BlockDriverState *bs,
-    int64_t l1_table_offset, int l1_size, int addend);
-
-int qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
-                          BdrvCheckMode fix);
-
-void qcow2_process_discards(BlockDriverState *bs, int ret);
-
-/* qcow2-cluster.c functions */
-int qcow2_grow_l1_table(BlockDriverState *bs, uint64_t min_size,
-                        bool exact_size);
-void qcow2_l2_cache_reset(BlockDriverState *bs);
-int qcow2_decompress_cluster(BlockDriverState *bs, uint64_t cluster_offset);
-void qcow2_encrypt_sectors(BDRVQcowState *s, int64_t sector_num,
-                     uint8_t *out_buf, const uint8_t *in_buf,
-                     int nb_sectors, int enc,
-                     const AES_KEY *key);
-
-int qcow2_get_cluster_offset(BlockDriverState *bs, uint64_t offset,
-    int *num, uint64_t *cluster_offset);
-int qcow2_alloc_cluster_offset(BlockDriverState *bs, uint64_t offset,
-    int n_start, int n_end, int *num, uint64_t *host_offset, QCowL2Meta **m);
-uint64_t qcow2_alloc_compressed_cluster_offset(BlockDriverState *bs,
-                                         uint64_t offset,
-                                         int compressed_size);
-
-int qcow2_alloc_cluster_link_l2(BlockDriverState *bs, QCowL2Meta *m);
-int qcow2_discard_clusters(BlockDriverState *bs, uint64_t offset,
-    int nb_sectors);
-int qcow2_zero_clusters(BlockDriverState *bs, uint64_t offset, int nb_sectors);
-
-/* qcow2-snapshot.c functions */
-int qcow2_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info);
-int qcow2_snapshot_goto(BlockDriverState *bs, const char *snapshot_id);
-int qcow2_snapshot_delete(BlockDriverState *bs, const char *snapshot_id);
-int qcow2_snapshot_list(BlockDriverState *bs, QEMUSnapshotInfo **psn_tab);
-int qcow2_snapshot_load_tmp(BlockDriverState *bs, const char *snapshot_name);
-
-void qcow2_free_snapshots(BlockDriverState *bs);
-int qcow2_read_snapshots(BlockDriverState *bs);
-
-/* qcow2-cache.c functions */
-Qcow2Cache *qcow2_cache_create(BlockDriverState *bs, int num_tables);
-int qcow2_cache_destroy(BlockDriverState* bs, Qcow2Cache *c);
-
-void qcow2_cache_entry_mark_dirty(Qcow2Cache *c, void *table);
-int qcow2_cache_flush(BlockDriverState *bs, Qcow2Cache *c);
-int qcow2_cache_set_dependency(BlockDriverState *bs, Qcow2Cache *c,
-    Qcow2Cache *dependency);
-void qcow2_cache_depends_on_flush(Qcow2Cache *c);
-
-int qcow2_cache_get(BlockDriverState *bs, Qcow2Cache *c, uint64_t offset,
-    void **table);
-int qcow2_cache_get_empty(BlockDriverState *bs, Qcow2Cache *c, uint64_t offset,
-    void **table);
-int qcow2_cache_put(BlockDriverState *bs, Qcow2Cache *c, void **table);
-
-#endif
diff --git a/contrib/qemu/block/qed-check.c b/contrib/qemu/block/qed-check.c
deleted file mode 100644
index b473dcd61f6..00000000000
--- a/contrib/qemu/block/qed-check.c
+++ /dev/null
@@ -1,248 +0,0 @@
-/*
- * QEMU Enhanced Disk Format Consistency Check
- *
- * Copyright IBM, Corp. 2010
- *
- * Authors:
- *  Stefan Hajnoczi   <stefanha@linux.vnet.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#include "qed.h"
-
-typedef struct {
-    BDRVQEDState *s;
-    BdrvCheckResult *result;
-    bool fix;                           /* whether to fix invalid offsets */
-
-    uint64_t nclusters;
-    uint32_t *used_clusters;            /* referenced cluster bitmap */
-
-    QEDRequest request;
-} QEDCheck;
-
-static bool qed_test_bit(uint32_t *bitmap, uint64_t n) {
-    return !!(bitmap[n / 32] & (1 << (n % 32)));
-}
-
-static void qed_set_bit(uint32_t *bitmap, uint64_t n) {
-    bitmap[n / 32] |= 1 << (n % 32);
-}
-
-/**
- * Set bitmap bits for clusters
- *
- * @check:          Check structure
- * @offset:         Starting offset in bytes
- * @n:              Number of clusters
- */
-static bool qed_set_used_clusters(QEDCheck *check, uint64_t offset,
-                                  unsigned int n)
-{
-    uint64_t cluster = qed_bytes_to_clusters(check->s, offset);
-    unsigned int corruptions = 0;
-
-    while (n-- != 0) {
-        /* Clusters should only be referenced once */
-        if (qed_test_bit(check->used_clusters, cluster)) {
-            corruptions++;
-        }
-
-        qed_set_bit(check->used_clusters, cluster);
-        cluster++;
-    }
-
-    check->result->corruptions += corruptions;
-    return corruptions == 0;
-}
-
-/**
- * Check an L2 table
- *
- * @ret:            Number of invalid cluster offsets
- */
-static unsigned int qed_check_l2_table(QEDCheck *check, QEDTable *table)
-{
-    BDRVQEDState *s = check->s;
-    unsigned int i, num_invalid = 0;
-    uint64_t last_offset = 0;
-
-    for (i = 0; i < s->table_nelems; i++) {
-        uint64_t offset = table->offsets[i];
-
-        if (qed_offset_is_unalloc_cluster(offset) ||
-            qed_offset_is_zero_cluster(offset)) {
-            continue;
-        }
-        check->result->bfi.allocated_clusters++;
-        if (last_offset && (last_offset + s->header.cluster_size != offset)) {
-            check->result->bfi.fragmented_clusters++;
-        }
-        last_offset = offset;
-
-        /* Detect invalid cluster offset */
-        if (!qed_check_cluster_offset(s, offset)) {
-            if (check->fix) {
-                table->offsets[i] = 0;
-                check->result->corruptions_fixed++;
-            } else {
-                check->result->corruptions++;
-            }
-
-            num_invalid++;
-            continue;
-        }
-
-        qed_set_used_clusters(check, offset, 1);
-    }
-
-    return num_invalid;
-}
-
-/**
- * Descend tables and check each cluster is referenced once only
- */
-static int qed_check_l1_table(QEDCheck *check, QEDTable *table)
-{
-    BDRVQEDState *s = check->s;
-    unsigned int i, num_invalid_l1 = 0;
-    int ret, last_error = 0;
-
-    /* Mark L1 table clusters used */
-    qed_set_used_clusters(check, s->header.l1_table_offset,
-                          s->header.table_size);
-
-    for (i = 0; i < s->table_nelems; i++) {
-        unsigned int num_invalid_l2;
-        uint64_t offset = table->offsets[i];
-
-        if (qed_offset_is_unalloc_cluster(offset)) {
-            continue;
-        }
-
-        /* Detect invalid L2 offset */
-        if (!qed_check_table_offset(s, offset)) {
-            /* Clear invalid offset */
-            if (check->fix) {
-                table->offsets[i] = 0;
-                check->result->corruptions_fixed++;
-            } else {
-                check->result->corruptions++;
-            }
-
-            num_invalid_l1++;
-            continue;
-        }
-
-        if (!qed_set_used_clusters(check, offset, s->header.table_size)) {
-            continue; /* skip an invalid table */
-        }
-
-        ret = qed_read_l2_table_sync(s, &check->request, offset);
-        if (ret) {
-            check->result->check_errors++;
-            last_error = ret;
-            continue;
-        }
-
-        num_invalid_l2 = qed_check_l2_table(check,
-                                            check->request.l2_table->table);
-
-        /* Write out fixed L2 table */
-        if (num_invalid_l2 > 0 && check->fix) {
-            ret = qed_write_l2_table_sync(s, &check->request, 0,
-                                          s->table_nelems, false);
-            if (ret) {
-                check->result->check_errors++;
-                last_error = ret;
-                continue;
-            }
-        }
-    }
-
-    /* Drop reference to final table */
-    qed_unref_l2_cache_entry(check->request.l2_table);
-    check->request.l2_table = NULL;
-
-    /* Write out fixed L1 table */
-    if (num_invalid_l1 > 0 && check->fix) {
-        ret = qed_write_l1_table_sync(s, 0, s->table_nelems);
-        if (ret) {
-            check->result->check_errors++;
-            last_error = ret;
-        }
-    }
-
-    return last_error;
-}
-
-/**
- * Check for unreferenced (leaked) clusters
- */
-static void qed_check_for_leaks(QEDCheck *check)
-{
-    BDRVQEDState *s = check->s;
-    uint64_t i;
-
-    for (i = s->header.header_size; i < check->nclusters; i++) {
-        if (!qed_test_bit(check->used_clusters, i)) {
-            check->result->leaks++;
-        }
-    }
-}
-
-/**
- * Mark an image clean once it passes check or has been repaired
- */
-static void qed_check_mark_clean(BDRVQEDState *s, BdrvCheckResult *result)
-{
-    /* Skip if there were unfixable corruptions or I/O errors */
-    if (result->corruptions > 0 || result->check_errors > 0) {
-        return;
-    }
-
-    /* Skip if image is already marked clean */
-    if (!(s->header.features & QED_F_NEED_CHECK)) {
-        return;
-    }
-
-    /* Ensure fixes reach storage before clearing check bit */
-    bdrv_flush(s->bs);
-
-    s->header.features &= ~QED_F_NEED_CHECK;
-    qed_write_header_sync(s);
-}
-
-int qed_check(BDRVQEDState *s, BdrvCheckResult *result, bool fix)
-{
-    QEDCheck check = {
-        .s = s,
-        .result = result,
-        .nclusters = qed_bytes_to_clusters(s, s->file_size),
-        .request = { .l2_table = NULL },
-        .fix = fix,
-    };
-    int ret;
-
-    check.used_clusters = g_malloc0(((check.nclusters + 31) / 32) *
-                                       sizeof(check.used_clusters[0]));
-
-    check.result->bfi.total_clusters =
-        (s->header.image_size + s->header.cluster_size - 1) /
-            s->header.cluster_size;
-    ret = qed_check_l1_table(&check, s->l1_table);
-    if (ret == 0) {
-        /* Only check for leaks if entire image was scanned successfully */
-        qed_check_for_leaks(&check);
-
-        if (fix) {
-            qed_check_mark_clean(s, result);
-        }
-    }
-
-    g_free(check.used_clusters);
-    return ret;
-}
diff --git a/contrib/qemu/block/qed-cluster.c b/contrib/qemu/block/qed-cluster.c
deleted file mode 100644
index f64b2af8f7e..00000000000
--- a/contrib/qemu/block/qed-cluster.c
+++ /dev/null
@@ -1,165 +0,0 @@
-/*
- * QEMU Enhanced Disk Format Cluster functions
- *
- * Copyright IBM, Corp. 2010
- *
- * Authors:
- *  Stefan Hajnoczi   <stefanha@linux.vnet.ibm.com>
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#include "qed.h"
-
-/**
- * Count the number of contiguous data clusters
- *
- * @s:              QED state
- * @table:          L2 table
- * @index:          First cluster index
- * @n:              Maximum number of clusters
- * @offset:         Set to first cluster offset
- *
- * This function scans tables for contiguous clusters.  A contiguous run of
- * clusters may be allocated, unallocated, or zero.
- */
-static unsigned int qed_count_contiguous_clusters(BDRVQEDState *s,
-                                                  QEDTable *table,
-                                                  unsigned int index,
-                                                  unsigned int n,
-                                                  uint64_t *offset)
-{
-    unsigned int end = MIN(index + n, s->table_nelems);
-    uint64_t last = table->offsets[index];
-    unsigned int i;
-
-    *offset = last;
-
-    for (i = index + 1; i < end; i++) {
-        if (qed_offset_is_unalloc_cluster(last)) {
-            /* Counting unallocated clusters */
-            if (!qed_offset_is_unalloc_cluster(table->offsets[i])) {
-                break;
-            }
-        } else if (qed_offset_is_zero_cluster(last)) {
-            /* Counting zero clusters */
-            if (!qed_offset_is_zero_cluster(table->offsets[i])) {
-                break;
-            }
-        } else {
-            /* Counting allocated clusters */
-            if (table->offsets[i] != last + s->header.cluster_size) {
-                break;
-            }
-            last = table->offsets[i];
-        }
-    }
-    return i - index;
-}
-
-typedef struct {
-    BDRVQEDState *s;
-    uint64_t pos;
-    size_t len;
-
-    QEDRequest *request;
-
-    /* User callback */
-    QEDFindClusterFunc *cb;
-    void *opaque;
-} QEDFindClusterCB;
-
-static void qed_find_cluster_cb(void *opaque, int ret)
-{
-    QEDFindClusterCB *find_cluster_cb = opaque;
-    BDRVQEDState *s = find_cluster_cb->s;
-    QEDRequest *request = find_cluster_cb->request;
-    uint64_t offset = 0;
-    size_t len = 0;
-    unsigned int index;
-    unsigned int n;
-
-    if (ret) {
-        goto out;
-    }
-
-    index = qed_l2_index(s, find_cluster_cb->pos);
-    n = qed_bytes_to_clusters(s,
-                              qed_offset_into_cluster(s, find_cluster_cb->pos) +
-                              find_cluster_cb->len);
-    n = qed_count_contiguous_clusters(s, request->l2_table->table,
-                                      index, n, &offset);
-
-    if (qed_offset_is_unalloc_cluster(offset)) {
-        ret = QED_CLUSTER_L2;
-    } else if (qed_offset_is_zero_cluster(offset)) {
-        ret = QED_CLUSTER_ZERO;
-    } else if (qed_check_cluster_offset(s, offset)) {
-        ret = QED_CLUSTER_FOUND;
-    } else {
-        ret = -EINVAL;
-    }
-
-    len = MIN(find_cluster_cb->len, n * s->header.cluster_size -
-              qed_offset_into_cluster(s, find_cluster_cb->pos));
-
-out:
-    find_cluster_cb->cb(find_cluster_cb->opaque, ret, offset, len);
-    g_free(find_cluster_cb);
-}
-
-/**
- * Find the offset of a data cluster
- *
- * @s:          QED state
- * @request:    L2 cache entry
- * @pos:        Byte position in device
- * @len:        Number of bytes
- * @cb:         Completion function
- * @opaque:     User data for completion function
- *
- * This function translates a position in the block device to an offset in the
- * image file.  It invokes the cb completion callback to report back the
- * translated offset or unallocated range in the image file.
- *
- * If the L2 table exists, request->l2_table points to the L2 table cache entry
- * and the caller must free the reference when they are finished.  The cache
- * entry is exposed in this way to avoid callers having to read the L2 table
- * again later during request processing.  If request->l2_table is non-NULL it
- * will be unreferenced before taking on the new cache entry.
- */
-void qed_find_cluster(BDRVQEDState *s, QEDRequest *request, uint64_t pos,
-                      size_t len, QEDFindClusterFunc *cb, void *opaque)
-{
-    QEDFindClusterCB *find_cluster_cb;
-    uint64_t l2_offset;
-
-    /* Limit length to L2 boundary.  Requests are broken up at the L2 boundary
-     * so that a request acts on one L2 table at a time.
-     */
-    len = MIN(len, (((pos >> s->l1_shift) + 1) << s->l1_shift) - pos);
-
-    l2_offset = s->l1_table->offsets[qed_l1_index(s, pos)];
-    if (qed_offset_is_unalloc_cluster(l2_offset)) {
-        cb(opaque, QED_CLUSTER_L1, 0, len);
-        return;
-    }
-    if (!qed_check_table_offset(s, l2_offset)) {
-        cb(opaque, -EINVAL, 0, 0);
-        return;
-    }
-
-    find_cluster_cb = g_malloc(sizeof(*find_cluster_cb));
-    find_cluster_cb->s = s;
-    find_cluster_cb->pos = pos;
-    find_cluster_cb->len = len;
-    find_cluster_cb->cb = cb;
-    find_cluster_cb->opaque = opaque;
-    find_cluster_cb->request = request;
-
-    qed_read_l2_table(s, request, l2_offset,
-                      qed_find_cluster_cb, find_cluster_cb);
-}
diff --git a/contrib/qemu/block/qed-gencb.c b/contrib/qemu/block/qed-gencb.c
deleted file mode 100644
index 7d7ac1ffc8e..00000000000
--- a/contrib/qemu/block/qed-gencb.c
+++ /dev/null
@@ -1,32 +0,0 @@
-/*
- * QEMU Enhanced Disk Format
- *
- * Copyright IBM, Corp. 2010
- *
- * Authors:
- *  Stefan Hajnoczi   <stefanha@linux.vnet.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#include "qed.h"
-
-void *gencb_alloc(size_t len, BlockDriverCompletionFunc *cb, void *opaque)
-{
-    GenericCB *gencb = g_malloc(len);
-    gencb->cb = cb;
-    gencb->opaque = opaque;
-    return gencb;
-}
-
-void gencb_complete(void *opaque, int ret)
-{
-    GenericCB *gencb = opaque;
-    BlockDriverCompletionFunc *cb = gencb->cb;
-    void *user_opaque = gencb->opaque;
-
-    g_free(gencb);
-    cb(user_opaque, ret);
-}
diff --git a/contrib/qemu/block/qed-l2-cache.c b/contrib/qemu/block/qed-l2-cache.c
deleted file mode 100644
index e9b2aae44d9..00000000000
--- a/contrib/qemu/block/qed-l2-cache.c
+++ /dev/null
@@ -1,187 +0,0 @@
-/*
- * QEMU Enhanced Disk Format L2 Cache
- *
- * Copyright IBM, Corp. 2010
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-/*
- * L2 table cache usage is as follows:
- *
- * An open image has one L2 table cache that is used to avoid accessing the
- * image file for recently referenced L2 tables.
- *
- * Cluster offset lookup translates the logical offset within the block device
- * to a cluster offset within the image file.  This is done by indexing into
- * the L1 and L2 tables which store cluster offsets.  It is here where the L2
- * table cache serves up recently referenced L2 tables.
- *
- * If there is a cache miss, that L2 table is read from the image file and
- * committed to the cache.  Subsequent accesses to that L2 table will be served
- * from the cache until the table is evicted from the cache.
- *
- * L2 tables are also committed to the cache when new L2 tables are allocated
- * in the image file.  Since the L2 table cache is write-through, the new L2
- * table is first written out to the image file and then committed to the
- * cache.
- *
- * Multiple I/O requests may be using an L2 table cache entry at any given
- * time.  That means an entry may be in use across several requests and
- * reference counting is needed to free the entry at the correct time.  In
- * particular, an entry evicted from the cache will only be freed once all
- * references are dropped.
- *
- * An in-flight I/O request will hold a reference to a L2 table cache entry for
- * the period during which it needs to access the L2 table.  This includes
- * cluster offset lookup, L2 table allocation, and L2 table update when a new
- * data cluster has been allocated.
- *
- * An interesting case occurs when two requests need to access an L2 table that
- * is not in the cache.  Since the operation to read the table from the image
- * file takes some time to complete, both requests may see a cache miss and
- * start reading the L2 table from the image file.  The first to finish will
- * commit its L2 table into the cache.  When the second tries to commit its
- * table will be deleted in favor of the existing cache entry.
- */
-
-#include "trace.h"
-#include "qed.h"
-
-/* Each L2 holds 2GB so this let's us fully cache a 100GB disk */
-#define MAX_L2_CACHE_SIZE 50
-
-/**
- * Initialize the L2 cache
- */
-void qed_init_l2_cache(L2TableCache *l2_cache)
-{
-    QTAILQ_INIT(&l2_cache->entries);
-    l2_cache->n_entries = 0;
-}
-
-/**
- * Free the L2 cache
- */
-void qed_free_l2_cache(L2TableCache *l2_cache)
-{
-    CachedL2Table *entry, *next_entry;
-
-    QTAILQ_FOREACH_SAFE(entry, &l2_cache->entries, node, next_entry) {
-        qemu_vfree(entry->table);
-        g_free(entry);
-    }
-}
-
-/**
- * Allocate an uninitialized entry from the cache
- *
- * The returned entry has a reference count of 1 and is owned by the caller.
- * The caller must allocate the actual table field for this entry and it must
- * be freeable using qemu_vfree().
- */
-CachedL2Table *qed_alloc_l2_cache_entry(L2TableCache *l2_cache)
-{
-    CachedL2Table *entry;
-
-    entry = g_malloc0(sizeof(*entry));
-    entry->ref++;
-
-    trace_qed_alloc_l2_cache_entry(l2_cache, entry);
-
-    return entry;
-}
-
-/**
- * Decrease an entry's reference count and free if necessary when the reference
- * count drops to zero.
- */
-void qed_unref_l2_cache_entry(CachedL2Table *entry)
-{
-    if (!entry) {
-        return;
-    }
-
-    entry->ref--;
-    trace_qed_unref_l2_cache_entry(entry, entry->ref);
-    if (entry->ref == 0) {
-        qemu_vfree(entry->table);
-        g_free(entry);
-    }
-}
-
-/**
- * Find an entry in the L2 cache.  This may return NULL and it's up to the
- * caller to satisfy the cache miss.
- *
- * For a cached entry, this function increases the reference count and returns
- * the entry.
- */
-CachedL2Table *qed_find_l2_cache_entry(L2TableCache *l2_cache, uint64_t offset)
-{
-    CachedL2Table *entry;
-
-    QTAILQ_FOREACH(entry, &l2_cache->entries, node) {
-        if (entry->offset == offset) {
-            trace_qed_find_l2_cache_entry(l2_cache, entry, offset, entry->ref);
-            entry->ref++;
-            return entry;
-        }
-    }
-    return NULL;
-}
-
-/**
- * Commit an L2 cache entry into the cache.  This is meant to be used as part of
- * the process to satisfy a cache miss.  A caller would allocate an entry which
- * is not actually in the L2 cache and then once the entry was valid and
- * present on disk, the entry can be committed into the cache.
- *
- * Since the cache is write-through, it's important that this function is not
- * called until the entry is present on disk and the L1 has been updated to
- * point to the entry.
- *
- * N.B. This function steals a reference to the l2_table from the caller so the
- * caller must obtain a new reference by issuing a call to
- * qed_find_l2_cache_entry().
- */
-void qed_commit_l2_cache_entry(L2TableCache *l2_cache, CachedL2Table *l2_table)
-{
-    CachedL2Table *entry;
-
-    entry = qed_find_l2_cache_entry(l2_cache, l2_table->offset);
-    if (entry) {
-        qed_unref_l2_cache_entry(entry);
-        qed_unref_l2_cache_entry(l2_table);
-        return;
-    }
-
-    /* Evict an unused cache entry so we have space.  If all entries are in use
-     * we can grow the cache temporarily and we try to shrink back down later.
-     */
-    if (l2_cache->n_entries >= MAX_L2_CACHE_SIZE) {
-        CachedL2Table *next;
-        QTAILQ_FOREACH_SAFE(entry, &l2_cache->entries, node, next) {
-            if (entry->ref > 1) {
-                continue;
-            }
-
-            QTAILQ_REMOVE(&l2_cache->entries, entry, node);
-            l2_cache->n_entries--;
-            qed_unref_l2_cache_entry(entry);
-
-            /* Stop evicting when we've shrunk back to max size */
-            if (l2_cache->n_entries < MAX_L2_CACHE_SIZE) {
-                break;
-            }
-        }
-    }
-
-    l2_cache->n_entries++;
-    QTAILQ_INSERT_TAIL(&l2_cache->entries, l2_table, node);
-}
diff --git a/contrib/qemu/block/qed-table.c b/contrib/qemu/block/qed-table.c
deleted file mode 100644
index 76d2dcccf81..00000000000
--- a/contrib/qemu/block/qed-table.c
+++ /dev/null
@@ -1,296 +0,0 @@
-/*
- * QEMU Enhanced Disk Format Table I/O
- *
- * Copyright IBM, Corp. 2010
- *
- * Authors:
- *  Stefan Hajnoczi   <stefanha@linux.vnet.ibm.com>
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#include "trace.h"
-#include "qemu/sockets.h" /* for EINPROGRESS on Windows */
-#include "qed.h"
-
-typedef struct {
-    GenericCB gencb;
-    BDRVQEDState *s;
-    QEDTable *table;
-
-    struct iovec iov;
-    QEMUIOVector qiov;
-} QEDReadTableCB;
-
-static void qed_read_table_cb(void *opaque, int ret)
-{
-    QEDReadTableCB *read_table_cb = opaque;
-    QEDTable *table = read_table_cb->table;
-    int noffsets = read_table_cb->qiov.size / sizeof(uint64_t);
-    int i;
-
-    /* Handle I/O error */
-    if (ret) {
-        goto out;
-    }
-
-    /* Byteswap offsets */
-    for (i = 0; i < noffsets; i++) {
-        table->offsets[i] = le64_to_cpu(table->offsets[i]);
-    }
-
-out:
-    /* Completion */
-    trace_qed_read_table_cb(read_table_cb->s, read_table_cb->table, ret);
-    gencb_complete(&read_table_cb->gencb, ret);
-}
-
-static void qed_read_table(BDRVQEDState *s, uint64_t offset, QEDTable *table,
-                           BlockDriverCompletionFunc *cb, void *opaque)
-{
-    QEDReadTableCB *read_table_cb = gencb_alloc(sizeof(*read_table_cb),
-                                                cb, opaque);
-    QEMUIOVector *qiov = &read_table_cb->qiov;
-
-    trace_qed_read_table(s, offset, table);
-
-    read_table_cb->s = s;
-    read_table_cb->table = table;
-    read_table_cb->iov.iov_base = table->offsets,
-    read_table_cb->iov.iov_len = s->header.cluster_size * s->header.table_size,
-
-    qemu_iovec_init_external(qiov, &read_table_cb->iov, 1);
-    bdrv_aio_readv(s->bs->file, offset / BDRV_SECTOR_SIZE, qiov,
-                   qiov->size / BDRV_SECTOR_SIZE,
-                   qed_read_table_cb, read_table_cb);
-}
-
-typedef struct {
-    GenericCB gencb;
-    BDRVQEDState *s;
-    QEDTable *orig_table;
-    QEDTable *table;
-    bool flush;             /* flush after write? */
-
-    struct iovec iov;
-    QEMUIOVector qiov;
-} QEDWriteTableCB;
-
-static void qed_write_table_cb(void *opaque, int ret)
-{
-    QEDWriteTableCB *write_table_cb = opaque;
-
-    trace_qed_write_table_cb(write_table_cb->s,
-                             write_table_cb->orig_table,
-                             write_table_cb->flush,
-                             ret);
-
-    if (ret) {
-        goto out;
-    }
-
-    if (write_table_cb->flush) {
-        /* We still need to flush first */
-        write_table_cb->flush = false;
-        bdrv_aio_flush(write_table_cb->s->bs, qed_write_table_cb,
-                       write_table_cb);
-        return;
-    }
-
-out:
-    qemu_vfree(write_table_cb->table);
-    gencb_complete(&write_table_cb->gencb, ret);
-}
-
-/**
- * Write out an updated part or all of a table
- *
- * @s:          QED state
- * @offset:     Offset of table in image file, in bytes
- * @table:      Table
- * @index:      Index of first element
- * @n:          Number of elements
- * @flush:      Whether or not to sync to disk
- * @cb:         Completion function
- * @opaque:     Argument for completion function
- */
-static void qed_write_table(BDRVQEDState *s, uint64_t offset, QEDTable *table,
-                            unsigned int index, unsigned int n, bool flush,
-                            BlockDriverCompletionFunc *cb, void *opaque)
-{
-    QEDWriteTableCB *write_table_cb;
-    unsigned int sector_mask = BDRV_SECTOR_SIZE / sizeof(uint64_t) - 1;
-    unsigned int start, end, i;
-    size_t len_bytes;
-
-    trace_qed_write_table(s, offset, table, index, n);
-
-    /* Calculate indices of the first and one after last elements */
-    start = index & ~sector_mask;
-    end = (index + n + sector_mask) & ~sector_mask;
-
-    len_bytes = (end - start) * sizeof(uint64_t);
-
-    write_table_cb = gencb_alloc(sizeof(*write_table_cb), cb, opaque);
-    write_table_cb->s = s;
-    write_table_cb->orig_table = table;
-    write_table_cb->flush = flush;
-    write_table_cb->table = qemu_blockalign(s->bs, len_bytes);
-    write_table_cb->iov.iov_base = write_table_cb->table->offsets;
-    write_table_cb->iov.iov_len = len_bytes;
-    qemu_iovec_init_external(&write_table_cb->qiov, &write_table_cb->iov, 1);
-
-    /* Byteswap table */
-    for (i = start; i < end; i++) {
-        uint64_t le_offset = cpu_to_le64(table->offsets[i]);
-        write_table_cb->table->offsets[i - start] = le_offset;
-    }
-
-    /* Adjust for offset into table */
-    offset += start * sizeof(uint64_t);
-
-    bdrv_aio_writev(s->bs->file, offset / BDRV_SECTOR_SIZE,
-                    &write_table_cb->qiov,
-                    write_table_cb->qiov.size / BDRV_SECTOR_SIZE,
-                    qed_write_table_cb, write_table_cb);
-}
-
-/**
- * Propagate return value from async callback
- */
-static void qed_sync_cb(void *opaque, int ret)
-{
-    *(int *)opaque = ret;
-}
-
-int qed_read_l1_table_sync(BDRVQEDState *s)
-{
-    int ret = -EINPROGRESS;
-
-    qed_read_table(s, s->header.l1_table_offset,
-                   s->l1_table, qed_sync_cb, &ret);
-    while (ret == -EINPROGRESS) {
-        qemu_aio_wait();
-    }
-
-    return ret;
-}
-
-void qed_write_l1_table(BDRVQEDState *s, unsigned int index, unsigned int n,
-                        BlockDriverCompletionFunc *cb, void *opaque)
-{
-    BLKDBG_EVENT(s->bs->file, BLKDBG_L1_UPDATE);
-    qed_write_table(s, s->header.l1_table_offset,
-                    s->l1_table, index, n, false, cb, opaque);
-}
-
-int qed_write_l1_table_sync(BDRVQEDState *s, unsigned int index,
-                            unsigned int n)
-{
-    int ret = -EINPROGRESS;
-
-    qed_write_l1_table(s, index, n, qed_sync_cb, &ret);
-    while (ret == -EINPROGRESS) {
-        qemu_aio_wait();
-    }
-
-    return ret;
-}
-
-typedef struct {
-    GenericCB gencb;
-    BDRVQEDState *s;
-    uint64_t l2_offset;
-    QEDRequest *request;
-} QEDReadL2TableCB;
-
-static void qed_read_l2_table_cb(void *opaque, int ret)
-{
-    QEDReadL2TableCB *read_l2_table_cb = opaque;
-    QEDRequest *request = read_l2_table_cb->request;
-    BDRVQEDState *s = read_l2_table_cb->s;
-    CachedL2Table *l2_table = request->l2_table;
-    uint64_t l2_offset = read_l2_table_cb->l2_offset;
-
-    if (ret) {
-        /* can't trust loaded L2 table anymore */
-        qed_unref_l2_cache_entry(l2_table);
-        request->l2_table = NULL;
-    } else {
-        l2_table->offset = l2_offset;
-
-        qed_commit_l2_cache_entry(&s->l2_cache, l2_table);
-
-        /* This is guaranteed to succeed because we just committed the entry
-         * to the cache.
-         */
-        request->l2_table = qed_find_l2_cache_entry(&s->l2_cache, l2_offset);
-        assert(request->l2_table != NULL);
-    }
-
-    gencb_complete(&read_l2_table_cb->gencb, ret);
-}
-
-void qed_read_l2_table(BDRVQEDState *s, QEDRequest *request, uint64_t offset,
-                       BlockDriverCompletionFunc *cb, void *opaque)
-{
-    QEDReadL2TableCB *read_l2_table_cb;
-
-    qed_unref_l2_cache_entry(request->l2_table);
-
-    /* Check for cached L2 entry */
-    request->l2_table = qed_find_l2_cache_entry(&s->l2_cache, offset);
-    if (request->l2_table) {
-        cb(opaque, 0);
-        return;
-    }
-
-    request->l2_table = qed_alloc_l2_cache_entry(&s->l2_cache);
-    request->l2_table->table = qed_alloc_table(s);
-
-    read_l2_table_cb = gencb_alloc(sizeof(*read_l2_table_cb), cb, opaque);
-    read_l2_table_cb->s = s;
-    read_l2_table_cb->l2_offset = offset;
-    read_l2_table_cb->request = request;
-
-    BLKDBG_EVENT(s->bs->file, BLKDBG_L2_LOAD);
-    qed_read_table(s, offset, request->l2_table->table,
-                   qed_read_l2_table_cb, read_l2_table_cb);
-}
-
-int qed_read_l2_table_sync(BDRVQEDState *s, QEDRequest *request, uint64_t offset)
-{
-    int ret = -EINPROGRESS;
-
-    qed_read_l2_table(s, request, offset, qed_sync_cb, &ret);
-    while (ret == -EINPROGRESS) {
-        qemu_aio_wait();
-    }
-
-    return ret;
-}
-
-void qed_write_l2_table(BDRVQEDState *s, QEDRequest *request,
-                        unsigned int index, unsigned int n, bool flush,
-                        BlockDriverCompletionFunc *cb, void *opaque)
-{
-    BLKDBG_EVENT(s->bs->file, BLKDBG_L2_UPDATE);
-    qed_write_table(s, request->l2_table->offset,
-                    request->l2_table->table, index, n, flush, cb, opaque);
-}
-
-int qed_write_l2_table_sync(BDRVQEDState *s, QEDRequest *request,
-                            unsigned int index, unsigned int n, bool flush)
-{
-    int ret = -EINPROGRESS;
-
-    qed_write_l2_table(s, request, index, n, flush, qed_sync_cb, &ret);
-    while (ret == -EINPROGRESS) {
-        qemu_aio_wait();
-    }
-
-    return ret;
-}
diff --git a/contrib/qemu/block/qed.c b/contrib/qemu/block/qed.c
deleted file mode 100644
index f767b0528ce..00000000000
--- a/contrib/qemu/block/qed.c
+++ /dev/null
@@ -1,1596 +0,0 @@
-/*
- * QEMU Enhanced Disk Format
- *
- * Copyright IBM, Corp. 2010
- *
- * Authors:
- *  Stefan Hajnoczi   <stefanha@linux.vnet.ibm.com>
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#include "qemu/timer.h"
-#include "trace.h"
-#include "qed.h"
-#include "qapi/qmp/qerror.h"
-#include "migration/migration.h"
-
-static void qed_aio_cancel(BlockDriverAIOCB *blockacb)
-{
-    QEDAIOCB *acb = (QEDAIOCB *)blockacb;
-    bool finished = false;
-
-    /* Wait for the request to finish */
-    acb->finished = &finished;
-    while (!finished) {
-        qemu_aio_wait();
-    }
-}
-
-static const AIOCBInfo qed_aiocb_info = {
-    .aiocb_size         = sizeof(QEDAIOCB),
-    .cancel             = qed_aio_cancel,
-};
-
-static int bdrv_qed_probe(const uint8_t *buf, int buf_size,
-                          const char *filename)
-{
-    const QEDHeader *header = (const QEDHeader *)buf;
-
-    if (buf_size < sizeof(*header)) {
-        return 0;
-    }
-    if (le32_to_cpu(header->magic) != QED_MAGIC) {
-        return 0;
-    }
-    return 100;
-}
-
-/**
- * Check whether an image format is raw
- *
- * @fmt:    Backing file format, may be NULL
- */
-static bool qed_fmt_is_raw(const char *fmt)
-{
-    return fmt && strcmp(fmt, "raw") == 0;
-}
-
-static void qed_header_le_to_cpu(const QEDHeader *le, QEDHeader *cpu)
-{
-    cpu->magic = le32_to_cpu(le->magic);
-    cpu->cluster_size = le32_to_cpu(le->cluster_size);
-    cpu->table_size = le32_to_cpu(le->table_size);
-    cpu->header_size = le32_to_cpu(le->header_size);
-    cpu->features = le64_to_cpu(le->features);
-    cpu->compat_features = le64_to_cpu(le->compat_features);
-    cpu->autoclear_features = le64_to_cpu(le->autoclear_features);
-    cpu->l1_table_offset = le64_to_cpu(le->l1_table_offset);
-    cpu->image_size = le64_to_cpu(le->image_size);
-    cpu->backing_filename_offset = le32_to_cpu(le->backing_filename_offset);
-    cpu->backing_filename_size = le32_to_cpu(le->backing_filename_size);
-}
-
-static void qed_header_cpu_to_le(const QEDHeader *cpu, QEDHeader *le)
-{
-    le->magic = cpu_to_le32(cpu->magic);
-    le->cluster_size = cpu_to_le32(cpu->cluster_size);
-    le->table_size = cpu_to_le32(cpu->table_size);
-    le->header_size = cpu_to_le32(cpu->header_size);
-    le->features = cpu_to_le64(cpu->features);
-    le->compat_features = cpu_to_le64(cpu->compat_features);
-    le->autoclear_features = cpu_to_le64(cpu->autoclear_features);
-    le->l1_table_offset = cpu_to_le64(cpu->l1_table_offset);
-    le->image_size = cpu_to_le64(cpu->image_size);
-    le->backing_filename_offset = cpu_to_le32(cpu->backing_filename_offset);
-    le->backing_filename_size = cpu_to_le32(cpu->backing_filename_size);
-}
-
-int qed_write_header_sync(BDRVQEDState *s)
-{
-    QEDHeader le;
-    int ret;
-
-    qed_header_cpu_to_le(&s->header, &le);
-    ret = bdrv_pwrite(s->bs->file, 0, &le, sizeof(le));
-    if (ret != sizeof(le)) {
-        return ret;
-    }
-    return 0;
-}
-
-typedef struct {
-    GenericCB gencb;
-    BDRVQEDState *s;
-    struct iovec iov;
-    QEMUIOVector qiov;
-    int nsectors;
-    uint8_t *buf;
-} QEDWriteHeaderCB;
-
-static void qed_write_header_cb(void *opaque, int ret)
-{
-    QEDWriteHeaderCB *write_header_cb = opaque;
-
-    qemu_vfree(write_header_cb->buf);
-    gencb_complete(write_header_cb, ret);
-}
-
-static void qed_write_header_read_cb(void *opaque, int ret)
-{
-    QEDWriteHeaderCB *write_header_cb = opaque;
-    BDRVQEDState *s = write_header_cb->s;
-
-    if (ret) {
-        qed_write_header_cb(write_header_cb, ret);
-        return;
-    }
-
-    /* Update header */
-    qed_header_cpu_to_le(&s->header, (QEDHeader *)write_header_cb->buf);
-
-    bdrv_aio_writev(s->bs->file, 0, &write_header_cb->qiov,
-                    write_header_cb->nsectors, qed_write_header_cb,
-                    write_header_cb);
-}
-
-/**
- * Update header in-place (does not rewrite backing filename or other strings)
- *
- * This function only updates known header fields in-place and does not affect
- * extra data after the QED header.
- */
-static void qed_write_header(BDRVQEDState *s, BlockDriverCompletionFunc cb,
-                             void *opaque)
-{
-    /* We must write full sectors for O_DIRECT but cannot necessarily generate
-     * the data following the header if an unrecognized compat feature is
-     * active.  Therefore, first read the sectors containing the header, update
-     * them, and write back.
-     */
-
-    int nsectors = (sizeof(QEDHeader) + BDRV_SECTOR_SIZE - 1) /
-                   BDRV_SECTOR_SIZE;
-    size_t len = nsectors * BDRV_SECTOR_SIZE;
-    QEDWriteHeaderCB *write_header_cb = gencb_alloc(sizeof(*write_header_cb),
-                                                    cb, opaque);
-
-    write_header_cb->s = s;
-    write_header_cb->nsectors = nsectors;
-    write_header_cb->buf = qemu_blockalign(s->bs, len);
-    write_header_cb->iov.iov_base = write_header_cb->buf;
-    write_header_cb->iov.iov_len = len;
-    qemu_iovec_init_external(&write_header_cb->qiov, &write_header_cb->iov, 1);
-
-    bdrv_aio_readv(s->bs->file, 0, &write_header_cb->qiov, nsectors,
-                   qed_write_header_read_cb, write_header_cb);
-}
-
-static uint64_t qed_max_image_size(uint32_t cluster_size, uint32_t table_size)
-{
-    uint64_t table_entries;
-    uint64_t l2_size;
-
-    table_entries = (table_size * cluster_size) / sizeof(uint64_t);
-    l2_size = table_entries * cluster_size;
-
-    return l2_size * table_entries;
-}
-
-static bool qed_is_cluster_size_valid(uint32_t cluster_size)
-{
-    if (cluster_size < QED_MIN_CLUSTER_SIZE ||
-        cluster_size > QED_MAX_CLUSTER_SIZE) {
-        return false;
-    }
-    if (cluster_size & (cluster_size - 1)) {
-        return false; /* not power of 2 */
-    }
-    return true;
-}
-
-static bool qed_is_table_size_valid(uint32_t table_size)
-{
-    if (table_size < QED_MIN_TABLE_SIZE ||
-        table_size > QED_MAX_TABLE_SIZE) {
-        return false;
-    }
-    if (table_size & (table_size - 1)) {
-        return false; /* not power of 2 */
-    }
-    return true;
-}
-
-static bool qed_is_image_size_valid(uint64_t image_size, uint32_t cluster_size,
-                                    uint32_t table_size)
-{
-    if (image_size % BDRV_SECTOR_SIZE != 0) {
-        return false; /* not multiple of sector size */
-    }
-    if (image_size > qed_max_image_size(cluster_size, table_size)) {
-        return false; /* image is too large */
-    }
-    return true;
-}
-
-/**
- * Read a string of known length from the image file
- *
- * @file:       Image file
- * @offset:     File offset to start of string, in bytes
- * @n:          String length in bytes
- * @buf:        Destination buffer
- * @buflen:     Destination buffer length in bytes
- * @ret:        0 on success, -errno on failure
- *
- * The string is NUL-terminated.
- */
-static int qed_read_string(BlockDriverState *file, uint64_t offset, size_t n,
-                           char *buf, size_t buflen)
-{
-    int ret;
-    if (n >= buflen) {
-        return -EINVAL;
-    }
-    ret = bdrv_pread(file, offset, buf, n);
-    if (ret < 0) {
-        return ret;
-    }
-    buf[n] = '\0';
-    return 0;
-}
-
-/**
- * Allocate new clusters
- *
- * @s:          QED state
- * @n:          Number of contiguous clusters to allocate
- * @ret:        Offset of first allocated cluster
- *
- * This function only produces the offset where the new clusters should be
- * written.  It updates BDRVQEDState but does not make any changes to the image
- * file.
- */
-static uint64_t qed_alloc_clusters(BDRVQEDState *s, unsigned int n)
-{
-    uint64_t offset = s->file_size;
-    s->file_size += n * s->header.cluster_size;
-    return offset;
-}
-
-QEDTable *qed_alloc_table(BDRVQEDState *s)
-{
-    /* Honor O_DIRECT memory alignment requirements */
-    return qemu_blockalign(s->bs,
-                           s->header.cluster_size * s->header.table_size);
-}
-
-/**
- * Allocate a new zeroed L2 table
- */
-static CachedL2Table *qed_new_l2_table(BDRVQEDState *s)
-{
-    CachedL2Table *l2_table = qed_alloc_l2_cache_entry(&s->l2_cache);
-
-    l2_table->table = qed_alloc_table(s);
-    l2_table->offset = qed_alloc_clusters(s, s->header.table_size);
-
-    memset(l2_table->table->offsets, 0,
-           s->header.cluster_size * s->header.table_size);
-    return l2_table;
-}
-
-static void qed_aio_next_io(void *opaque, int ret);
-
-static void qed_plug_allocating_write_reqs(BDRVQEDState *s)
-{
-    assert(!s->allocating_write_reqs_plugged);
-
-    s->allocating_write_reqs_plugged = true;
-}
-
-static void qed_unplug_allocating_write_reqs(BDRVQEDState *s)
-{
-    QEDAIOCB *acb;
-
-    assert(s->allocating_write_reqs_plugged);
-
-    s->allocating_write_reqs_plugged = false;
-
-    acb = QSIMPLEQ_FIRST(&s->allocating_write_reqs);
-    if (acb) {
-        qed_aio_next_io(acb, 0);
-    }
-}
-
-static void qed_finish_clear_need_check(void *opaque, int ret)
-{
-    /* Do nothing */
-}
-
-static void qed_flush_after_clear_need_check(void *opaque, int ret)
-{
-    BDRVQEDState *s = opaque;
-
-    bdrv_aio_flush(s->bs, qed_finish_clear_need_check, s);
-
-    /* No need to wait until flush completes */
-    qed_unplug_allocating_write_reqs(s);
-}
-
-static void qed_clear_need_check(void *opaque, int ret)
-{
-    BDRVQEDState *s = opaque;
-
-    if (ret) {
-        qed_unplug_allocating_write_reqs(s);
-        return;
-    }
-
-    s->header.features &= ~QED_F_NEED_CHECK;
-    qed_write_header(s, qed_flush_after_clear_need_check, s);
-}
-
-static void qed_need_check_timer_cb(void *opaque)
-{
-    BDRVQEDState *s = opaque;
-
-    /* The timer should only fire when allocating writes have drained */
-    assert(!QSIMPLEQ_FIRST(&s->allocating_write_reqs));
-
-    trace_qed_need_check_timer_cb(s);
-
-    qed_plug_allocating_write_reqs(s);
-
-    /* Ensure writes are on disk before clearing flag */
-    bdrv_aio_flush(s->bs, qed_clear_need_check, s);
-}
-
-static void qed_start_need_check_timer(BDRVQEDState *s)
-{
-    trace_qed_start_need_check_timer(s);
-
-    /* Use vm_clock so we don't alter the image file while suspended for
-     * migration.
-     */
-    qemu_mod_timer(s->need_check_timer, qemu_get_clock_ns(vm_clock) +
-                   get_ticks_per_sec() * QED_NEED_CHECK_TIMEOUT);
-}
-
-/* It's okay to call this multiple times or when no timer is started */
-static void qed_cancel_need_check_timer(BDRVQEDState *s)
-{
-    trace_qed_cancel_need_check_timer(s);
-    qemu_del_timer(s->need_check_timer);
-}
-
-static void bdrv_qed_rebind(BlockDriverState *bs)
-{
-    BDRVQEDState *s = bs->opaque;
-    s->bs = bs;
-}
-
-static int bdrv_qed_open(BlockDriverState *bs, QDict *options, int flags)
-{
-    BDRVQEDState *s = bs->opaque;
-    QEDHeader le_header;
-    int64_t file_size;
-    int ret;
-
-    s->bs = bs;
-    QSIMPLEQ_INIT(&s->allocating_write_reqs);
-
-    ret = bdrv_pread(bs->file, 0, &le_header, sizeof(le_header));
-    if (ret < 0) {
-        return ret;
-    }
-    qed_header_le_to_cpu(&le_header, &s->header);
-
-    if (s->header.magic != QED_MAGIC) {
-        return -EMEDIUMTYPE;
-    }
-    if (s->header.features & ~QED_FEATURE_MASK) {
-        /* image uses unsupported feature bits */
-        char buf[64];
-        snprintf(buf, sizeof(buf), "%" PRIx64,
-            s->header.features & ~QED_FEATURE_MASK);
-        qerror_report(QERR_UNKNOWN_BLOCK_FORMAT_FEATURE,
-            bs->device_name, "QED", buf);
-        return -ENOTSUP;
-    }
-    if (!qed_is_cluster_size_valid(s->header.cluster_size)) {
-        return -EINVAL;
-    }
-
-    /* Round down file size to the last cluster */
-    file_size = bdrv_getlength(bs->file);
-    if (file_size < 0) {
-        return file_size;
-    }
-    s->file_size = qed_start_of_cluster(s, file_size);
-
-    if (!qed_is_table_size_valid(s->header.table_size)) {
-        return -EINVAL;
-    }
-    if (!qed_is_image_size_valid(s->header.image_size,
-                                 s->header.cluster_size,
-                                 s->header.table_size)) {
-        return -EINVAL;
-    }
-    if (!qed_check_table_offset(s, s->header.l1_table_offset)) {
-        return -EINVAL;
-    }
-
-    s->table_nelems = (s->header.cluster_size * s->header.table_size) /
-                      sizeof(uint64_t);
-    s->l2_shift = ffs(s->header.cluster_size) - 1;
-    s->l2_mask = s->table_nelems - 1;
-    s->l1_shift = s->l2_shift + ffs(s->table_nelems) - 1;
-
-    if ((s->header.features & QED_F_BACKING_FILE)) {
-        if ((uint64_t)s->header.backing_filename_offset +
-            s->header.backing_filename_size >
-            s->header.cluster_size * s->header.header_size) {
-            return -EINVAL;
-        }
-
-        ret = qed_read_string(bs->file, s->header.backing_filename_offset,
-                              s->header.backing_filename_size, bs->backing_file,
-                              sizeof(bs->backing_file));
-        if (ret < 0) {
-            return ret;
-        }
-
-        if (s->header.features & QED_F_BACKING_FORMAT_NO_PROBE) {
-            pstrcpy(bs->backing_format, sizeof(bs->backing_format), "raw");
-        }
-    }
-
-    /* Reset unknown autoclear feature bits.  This is a backwards
-     * compatibility mechanism that allows images to be opened by older
-     * programs, which "knock out" unknown feature bits.  When an image is
-     * opened by a newer program again it can detect that the autoclear
-     * feature is no longer valid.
-     */
-    if ((s->header.autoclear_features & ~QED_AUTOCLEAR_FEATURE_MASK) != 0 &&
-        !bdrv_is_read_only(bs->file) && !(flags & BDRV_O_INCOMING)) {
-        s->header.autoclear_features &= QED_AUTOCLEAR_FEATURE_MASK;
-
-        ret = qed_write_header_sync(s);
-        if (ret) {
-            return ret;
-        }
-
-        /* From here on only known autoclear feature bits are valid */
-        bdrv_flush(bs->file);
-    }
-
-    s->l1_table = qed_alloc_table(s);
-    qed_init_l2_cache(&s->l2_cache);
-
-    ret = qed_read_l1_table_sync(s);
-    if (ret) {
-        goto out;
-    }
-
-    /* If image was not closed cleanly, check consistency */
-    if (!(flags & BDRV_O_CHECK) && (s->header.features & QED_F_NEED_CHECK)) {
-        /* Read-only images cannot be fixed.  There is no risk of corruption
-         * since write operations are not possible.  Therefore, allow
-         * potentially inconsistent images to be opened read-only.  This can
-         * aid data recovery from an otherwise inconsistent image.
-         */
-        if (!bdrv_is_read_only(bs->file) &&
-            !(flags & BDRV_O_INCOMING)) {
-            BdrvCheckResult result = {0};
-
-            ret = qed_check(s, &result, true);
-            if (ret) {
-                goto out;
-            }
-        }
-    }
-
-    s->need_check_timer = qemu_new_timer_ns(vm_clock,
-                                            qed_need_check_timer_cb, s);
-
-out:
-    if (ret) {
-        qed_free_l2_cache(&s->l2_cache);
-        qemu_vfree(s->l1_table);
-    }
-    return ret;
-}
-
-/* We have nothing to do for QED reopen, stubs just return
- * success */
-static int bdrv_qed_reopen_prepare(BDRVReopenState *state,
-                                   BlockReopenQueue *queue, Error **errp)
-{
-    return 0;
-}
-
-static void bdrv_qed_close(BlockDriverState *bs)
-{
-    BDRVQEDState *s = bs->opaque;
-
-    qed_cancel_need_check_timer(s);
-    qemu_free_timer(s->need_check_timer);
-
-    /* Ensure writes reach stable storage */
-    bdrv_flush(bs->file);
-
-    /* Clean shutdown, no check required on next open */
-    if (s->header.features & QED_F_NEED_CHECK) {
-        s->header.features &= ~QED_F_NEED_CHECK;
-        qed_write_header_sync(s);
-    }
-
-    qed_free_l2_cache(&s->l2_cache);
-    qemu_vfree(s->l1_table);
-}
-
-static int qed_create(const char *filename, uint32_t cluster_size,
-                      uint64_t image_size, uint32_t table_size,
-                      const char *backing_file, const char *backing_fmt)
-{
-    QEDHeader header = {
-        .magic = QED_MAGIC,
-        .cluster_size = cluster_size,
-        .table_size = table_size,
-        .header_size = 1,
-        .features = 0,
-        .compat_features = 0,
-        .l1_table_offset = cluster_size,
-        .image_size = image_size,
-    };
-    QEDHeader le_header;
-    uint8_t *l1_table = NULL;
-    size_t l1_size = header.cluster_size * header.table_size;
-    int ret = 0;
-    BlockDriverState *bs = NULL;
-
-    ret = bdrv_create_file(filename, NULL);
-    if (ret < 0) {
-        return ret;
-    }
-
-    ret = bdrv_file_open(&bs, filename, NULL, BDRV_O_RDWR | BDRV_O_CACHE_WB);
-    if (ret < 0) {
-        return ret;
-    }
-
-    /* File must start empty and grow, check truncate is supported */
-    ret = bdrv_truncate(bs, 0);
-    if (ret < 0) {
-        goto out;
-    }
-
-    if (backing_file) {
-        header.features |= QED_F_BACKING_FILE;
-        header.backing_filename_offset = sizeof(le_header);
-        header.backing_filename_size = strlen(backing_file);
-
-        if (qed_fmt_is_raw(backing_fmt)) {
-            header.features |= QED_F_BACKING_FORMAT_NO_PROBE;
-        }
-    }
-
-    qed_header_cpu_to_le(&header, &le_header);
-    ret = bdrv_pwrite(bs, 0, &le_header, sizeof(le_header));
-    if (ret < 0) {
-        goto out;
-    }
-    ret = bdrv_pwrite(bs, sizeof(le_header), backing_file,
-                      header.backing_filename_size);
-    if (ret < 0) {
-        goto out;
-    }
-
-    l1_table = g_malloc0(l1_size);
-    ret = bdrv_pwrite(bs, header.l1_table_offset, l1_table, l1_size);
-    if (ret < 0) {
-        goto out;
-    }
-
-    ret = 0; /* success */
-out:
-    g_free(l1_table);
-    bdrv_delete(bs);
-    return ret;
-}
-
-static int bdrv_qed_create(const char *filename, QEMUOptionParameter *options)
-{
-    uint64_t image_size = 0;
-    uint32_t cluster_size = QED_DEFAULT_CLUSTER_SIZE;
-    uint32_t table_size = QED_DEFAULT_TABLE_SIZE;
-    const char *backing_file = NULL;
-    const char *backing_fmt = NULL;
-
-    while (options && options->name) {
-        if (!strcmp(options->name, BLOCK_OPT_SIZE)) {
-            image_size = options->value.n;
-        } else if (!strcmp(options->name, BLOCK_OPT_BACKING_FILE)) {
-            backing_file = options->value.s;
-        } else if (!strcmp(options->name, BLOCK_OPT_BACKING_FMT)) {
-            backing_fmt = options->value.s;
-        } else if (!strcmp(options->name, BLOCK_OPT_CLUSTER_SIZE)) {
-            if (options->value.n) {
-                cluster_size = options->value.n;
-            }
-        } else if (!strcmp(options->name, BLOCK_OPT_TABLE_SIZE)) {
-            if (options->value.n) {
-                table_size = options->value.n;
-            }
-        }
-        options++;
-    }
-
-    if (!qed_is_cluster_size_valid(cluster_size)) {
-        fprintf(stderr, "QED cluster size must be within range [%u, %u] and power of 2\n",
-                QED_MIN_CLUSTER_SIZE, QED_MAX_CLUSTER_SIZE);
-        return -EINVAL;
-    }
-    if (!qed_is_table_size_valid(table_size)) {
-        fprintf(stderr, "QED table size must be within range [%u, %u] and power of 2\n",
-                QED_MIN_TABLE_SIZE, QED_MAX_TABLE_SIZE);
-        return -EINVAL;
-    }
-    if (!qed_is_image_size_valid(image_size, cluster_size, table_size)) {
-        fprintf(stderr, "QED image size must be a non-zero multiple of "
-                        "cluster size and less than %" PRIu64 " bytes\n",
-                qed_max_image_size(cluster_size, table_size));
-        return -EINVAL;
-    }
-
-    return qed_create(filename, cluster_size, image_size, table_size,
-                      backing_file, backing_fmt);
-}
-
-typedef struct {
-    Coroutine *co;
-    int is_allocated;
-    int *pnum;
-} QEDIsAllocatedCB;
-
-static void qed_is_allocated_cb(void *opaque, int ret, uint64_t offset, size_t len)
-{
-    QEDIsAllocatedCB *cb = opaque;
-    *cb->pnum = len / BDRV_SECTOR_SIZE;
-    cb->is_allocated = (ret == QED_CLUSTER_FOUND || ret == QED_CLUSTER_ZERO);
-    if (cb->co) {
-        qemu_coroutine_enter(cb->co, NULL);
-    }
-}
-
-static int coroutine_fn bdrv_qed_co_is_allocated(BlockDriverState *bs,
-                                                 int64_t sector_num,
-                                                 int nb_sectors, int *pnum)
-{
-    BDRVQEDState *s = bs->opaque;
-    uint64_t pos = (uint64_t)sector_num * BDRV_SECTOR_SIZE;
-    size_t len = (size_t)nb_sectors * BDRV_SECTOR_SIZE;
-    QEDIsAllocatedCB cb = {
-        .is_allocated = -1,
-        .pnum = pnum,
-    };
-    QEDRequest request = { .l2_table = NULL };
-
-    qed_find_cluster(s, &request, pos, len, qed_is_allocated_cb, &cb);
-
-    /* Now sleep if the callback wasn't invoked immediately */
-    while (cb.is_allocated == -1) {
-        cb.co = qemu_coroutine_self();
-        qemu_coroutine_yield();
-    }
-
-    qed_unref_l2_cache_entry(request.l2_table);
-
-    return cb.is_allocated;
-}
-
-static int bdrv_qed_make_empty(BlockDriverState *bs)
-{
-    return -ENOTSUP;
-}
-
-static BDRVQEDState *acb_to_s(QEDAIOCB *acb)
-{
-    return acb->common.bs->opaque;
-}
-
-/**
- * Read from the backing file or zero-fill if no backing file
- *
- * @s:          QED state
- * @pos:        Byte position in device
- * @qiov:       Destination I/O vector
- * @cb:         Completion function
- * @opaque:     User data for completion function
- *
- * This function reads qiov->size bytes starting at pos from the backing file.
- * If there is no backing file then zeroes are read.
- */
-static void qed_read_backing_file(BDRVQEDState *s, uint64_t pos,
-                                  QEMUIOVector *qiov,
-                                  BlockDriverCompletionFunc *cb, void *opaque)
-{
-    uint64_t backing_length = 0;
-    size_t size;
-
-    /* If there is a backing file, get its length.  Treat the absence of a
-     * backing file like a zero length backing file.
-     */
-    if (s->bs->backing_hd) {
-        int64_t l = bdrv_getlength(s->bs->backing_hd);
-        if (l < 0) {
-            cb(opaque, l);
-            return;
-        }
-        backing_length = l;
-    }
-
-    /* Zero all sectors if reading beyond the end of the backing file */
-    if (pos >= backing_length ||
-        pos + qiov->size > backing_length) {
-        qemu_iovec_memset(qiov, 0, 0, qiov->size);
-    }
-
-    /* Complete now if there are no backing file sectors to read */
-    if (pos >= backing_length) {
-        cb(opaque, 0);
-        return;
-    }
-
-    /* If the read straddles the end of the backing file, shorten it */
-    size = MIN((uint64_t)backing_length - pos, qiov->size);
-
-    BLKDBG_EVENT(s->bs->file, BLKDBG_READ_BACKING_AIO);
-    bdrv_aio_readv(s->bs->backing_hd, pos / BDRV_SECTOR_SIZE,
-                   qiov, size / BDRV_SECTOR_SIZE, cb, opaque);
-}
-
-typedef struct {
-    GenericCB gencb;
-    BDRVQEDState *s;
-    QEMUIOVector qiov;
-    struct iovec iov;
-    uint64_t offset;
-} CopyFromBackingFileCB;
-
-static void qed_copy_from_backing_file_cb(void *opaque, int ret)
-{
-    CopyFromBackingFileCB *copy_cb = opaque;
-    qemu_vfree(copy_cb->iov.iov_base);
-    gencb_complete(&copy_cb->gencb, ret);
-}
-
-static void qed_copy_from_backing_file_write(void *opaque, int ret)
-{
-    CopyFromBackingFileCB *copy_cb = opaque;
-    BDRVQEDState *s = copy_cb->s;
-
-    if (ret) {
-        qed_copy_from_backing_file_cb(copy_cb, ret);
-        return;
-    }
-
-    BLKDBG_EVENT(s->bs->file, BLKDBG_COW_WRITE);
-    bdrv_aio_writev(s->bs->file, copy_cb->offset / BDRV_SECTOR_SIZE,
-                    &copy_cb->qiov, copy_cb->qiov.size / BDRV_SECTOR_SIZE,
-                    qed_copy_from_backing_file_cb, copy_cb);
-}
-
-/**
- * Copy data from backing file into the image
- *
- * @s:          QED state
- * @pos:        Byte position in device
- * @len:        Number of bytes
- * @offset:     Byte offset in image file
- * @cb:         Completion function
- * @opaque:     User data for completion function
- */
-static void qed_copy_from_backing_file(BDRVQEDState *s, uint64_t pos,
-                                       uint64_t len, uint64_t offset,
-                                       BlockDriverCompletionFunc *cb,
-                                       void *opaque)
-{
-    CopyFromBackingFileCB *copy_cb;
-
-    /* Skip copy entirely if there is no work to do */
-    if (len == 0) {
-        cb(opaque, 0);
-        return;
-    }
-
-    copy_cb = gencb_alloc(sizeof(*copy_cb), cb, opaque);
-    copy_cb->s = s;
-    copy_cb->offset = offset;
-    copy_cb->iov.iov_base = qemu_blockalign(s->bs, len);
-    copy_cb->iov.iov_len = len;
-    qemu_iovec_init_external(&copy_cb->qiov, &copy_cb->iov, 1);
-
-    qed_read_backing_file(s, pos, &copy_cb->qiov,
-                          qed_copy_from_backing_file_write, copy_cb);
-}
-
-/**
- * Link one or more contiguous clusters into a table
- *
- * @s:              QED state
- * @table:          L2 table
- * @index:          First cluster index
- * @n:              Number of contiguous clusters
- * @cluster:        First cluster offset
- *
- * The cluster offset may be an allocated byte offset in the image file, the
- * zero cluster marker, or the unallocated cluster marker.
- */
-static void qed_update_l2_table(BDRVQEDState *s, QEDTable *table, int index,
-                                unsigned int n, uint64_t cluster)
-{
-    int i;
-    for (i = index; i < index + n; i++) {
-        table->offsets[i] = cluster;
-        if (!qed_offset_is_unalloc_cluster(cluster) &&
-            !qed_offset_is_zero_cluster(cluster)) {
-            cluster += s->header.cluster_size;
-        }
-    }
-}
-
-static void qed_aio_complete_bh(void *opaque)
-{
-    QEDAIOCB *acb = opaque;
-    BlockDriverCompletionFunc *cb = acb->common.cb;
-    void *user_opaque = acb->common.opaque;
-    int ret = acb->bh_ret;
-    bool *finished = acb->finished;
-
-    qemu_bh_delete(acb->bh);
-    qemu_aio_release(acb);
-
-    /* Invoke callback */
-    cb(user_opaque, ret);
-
-    /* Signal cancel completion */
-    if (finished) {
-        *finished = true;
-    }
-}
-
-static void qed_aio_complete(QEDAIOCB *acb, int ret)
-{
-    BDRVQEDState *s = acb_to_s(acb);
-
-    trace_qed_aio_complete(s, acb, ret);
-
-    /* Free resources */
-    qemu_iovec_destroy(&acb->cur_qiov);
-    qed_unref_l2_cache_entry(acb->request.l2_table);
-
-    /* Free the buffer we may have allocated for zero writes */
-    if (acb->flags & QED_AIOCB_ZERO) {
-        qemu_vfree(acb->qiov->iov[0].iov_base);
-        acb->qiov->iov[0].iov_base = NULL;
-    }
-
-    /* Arrange for a bh to invoke the completion function */
-    acb->bh_ret = ret;
-    acb->bh = qemu_bh_new(qed_aio_complete_bh, acb);
-    qemu_bh_schedule(acb->bh);
-
-    /* Start next allocating write request waiting behind this one.  Note that
-     * requests enqueue themselves when they first hit an unallocated cluster
-     * but they wait until the entire request is finished before waking up the
-     * next request in the queue.  This ensures that we don't cycle through
-     * requests multiple times but rather finish one at a time completely.
-     */
-    if (acb == QSIMPLEQ_FIRST(&s->allocating_write_reqs)) {
-        QSIMPLEQ_REMOVE_HEAD(&s->allocating_write_reqs, next);
-        acb = QSIMPLEQ_FIRST(&s->allocating_write_reqs);
-        if (acb) {
-            qed_aio_next_io(acb, 0);
-        } else if (s->header.features & QED_F_NEED_CHECK) {
-            qed_start_need_check_timer(s);
-        }
-    }
-}
-
-/**
- * Commit the current L2 table to the cache
- */
-static void qed_commit_l2_update(void *opaque, int ret)
-{
-    QEDAIOCB *acb = opaque;
-    BDRVQEDState *s = acb_to_s(acb);
-    CachedL2Table *l2_table = acb->request.l2_table;
-    uint64_t l2_offset = l2_table->offset;
-
-    qed_commit_l2_cache_entry(&s->l2_cache, l2_table);
-
-    /* This is guaranteed to succeed because we just committed the entry to the
-     * cache.
-     */
-    acb->request.l2_table = qed_find_l2_cache_entry(&s->l2_cache, l2_offset);
-    assert(acb->request.l2_table != NULL);
-
-    qed_aio_next_io(opaque, ret);
-}
-
-/**
- * Update L1 table with new L2 table offset and write it out
- */
-static void qed_aio_write_l1_update(void *opaque, int ret)
-{
-    QEDAIOCB *acb = opaque;
-    BDRVQEDState *s = acb_to_s(acb);
-    int index;
-
-    if (ret) {
-        qed_aio_complete(acb, ret);
-        return;
-    }
-
-    index = qed_l1_index(s, acb->cur_pos);
-    s->l1_table->offsets[index] = acb->request.l2_table->offset;
-
-    qed_write_l1_table(s, index, 1, qed_commit_l2_update, acb);
-}
-
-/**
- * Update L2 table with new cluster offsets and write them out
- */
-static void qed_aio_write_l2_update(QEDAIOCB *acb, int ret, uint64_t offset)
-{
-    BDRVQEDState *s = acb_to_s(acb);
-    bool need_alloc = acb->find_cluster_ret == QED_CLUSTER_L1;
-    int index;
-
-    if (ret) {
-        goto err;
-    }
-
-    if (need_alloc) {
-        qed_unref_l2_cache_entry(acb->request.l2_table);
-        acb->request.l2_table = qed_new_l2_table(s);
-    }
-
-    index = qed_l2_index(s, acb->cur_pos);
-    qed_update_l2_table(s, acb->request.l2_table->table, index, acb->cur_nclusters,
-                         offset);
-
-    if (need_alloc) {
-        /* Write out the whole new L2 table */
-        qed_write_l2_table(s, &acb->request, 0, s->table_nelems, true,
-                            qed_aio_write_l1_update, acb);
-    } else {
-        /* Write out only the updated part of the L2 table */
-        qed_write_l2_table(s, &acb->request, index, acb->cur_nclusters, false,
-                            qed_aio_next_io, acb);
-    }
-    return;
-
-err:
-    qed_aio_complete(acb, ret);
-}
-
-static void qed_aio_write_l2_update_cb(void *opaque, int ret)
-{
-    QEDAIOCB *acb = opaque;
-    qed_aio_write_l2_update(acb, ret, acb->cur_cluster);
-}
-
-/**
- * Flush new data clusters before updating the L2 table
- *
- * This flush is necessary when a backing file is in use.  A crash during an
- * allocating write could result in empty clusters in the image.  If the write
- * only touched a subregion of the cluster, then backing image sectors have
- * been lost in the untouched region.  The solution is to flush after writing a
- * new data cluster and before updating the L2 table.
- */
-static void qed_aio_write_flush_before_l2_update(void *opaque, int ret)
-{
-    QEDAIOCB *acb = opaque;
-    BDRVQEDState *s = acb_to_s(acb);
-
-    if (!bdrv_aio_flush(s->bs->file, qed_aio_write_l2_update_cb, opaque)) {
-        qed_aio_complete(acb, -EIO);
-    }
-}
-
-/**
- * Write data to the image file
- */
-static void qed_aio_write_main(void *opaque, int ret)
-{
-    QEDAIOCB *acb = opaque;
-    BDRVQEDState *s = acb_to_s(acb);
-    uint64_t offset = acb->cur_cluster +
-                      qed_offset_into_cluster(s, acb->cur_pos);
-    BlockDriverCompletionFunc *next_fn;
-
-    trace_qed_aio_write_main(s, acb, ret, offset, acb->cur_qiov.size);
-
-    if (ret) {
-        qed_aio_complete(acb, ret);
-        return;
-    }
-
-    if (acb->find_cluster_ret == QED_CLUSTER_FOUND) {
-        next_fn = qed_aio_next_io;
-    } else {
-        if (s->bs->backing_hd) {
-            next_fn = qed_aio_write_flush_before_l2_update;
-        } else {
-            next_fn = qed_aio_write_l2_update_cb;
-        }
-    }
-
-    BLKDBG_EVENT(s->bs->file, BLKDBG_WRITE_AIO);
-    bdrv_aio_writev(s->bs->file, offset / BDRV_SECTOR_SIZE,
-                    &acb->cur_qiov, acb->cur_qiov.size / BDRV_SECTOR_SIZE,
-                    next_fn, acb);
-}
-
-/**
- * Populate back untouched region of new data cluster
- */
-static void qed_aio_write_postfill(void *opaque, int ret)
-{
-    QEDAIOCB *acb = opaque;
-    BDRVQEDState *s = acb_to_s(acb);
-    uint64_t start = acb->cur_pos + acb->cur_qiov.size;
-    uint64_t len =
-        qed_start_of_cluster(s, start + s->header.cluster_size - 1) - start;
-    uint64_t offset = acb->cur_cluster +
-                      qed_offset_into_cluster(s, acb->cur_pos) +
-                      acb->cur_qiov.size;
-
-    if (ret) {
-        qed_aio_complete(acb, ret);
-        return;
-    }
-
-    trace_qed_aio_write_postfill(s, acb, start, len, offset);
-    qed_copy_from_backing_file(s, start, len, offset,
-                                qed_aio_write_main, acb);
-}
-
-/**
- * Populate front untouched region of new data cluster
- */
-static void qed_aio_write_prefill(void *opaque, int ret)
-{
-    QEDAIOCB *acb = opaque;
-    BDRVQEDState *s = acb_to_s(acb);
-    uint64_t start = qed_start_of_cluster(s, acb->cur_pos);
-    uint64_t len = qed_offset_into_cluster(s, acb->cur_pos);
-
-    trace_qed_aio_write_prefill(s, acb, start, len, acb->cur_cluster);
-    qed_copy_from_backing_file(s, start, len, acb->cur_cluster,
-                                qed_aio_write_postfill, acb);
-}
-
-/**
- * Check if the QED_F_NEED_CHECK bit should be set during allocating write
- */
-static bool qed_should_set_need_check(BDRVQEDState *s)
-{
-    /* The flush before L2 update path ensures consistency */
-    if (s->bs->backing_hd) {
-        return false;
-    }
-
-    return !(s->header.features & QED_F_NEED_CHECK);
-}
-
-static void qed_aio_write_zero_cluster(void *opaque, int ret)
-{
-    QEDAIOCB *acb = opaque;
-
-    if (ret) {
-        qed_aio_complete(acb, ret);
-        return;
-    }
-
-    qed_aio_write_l2_update(acb, 0, 1);
-}
-
-/**
- * Write new data cluster
- *
- * @acb:        Write request
- * @len:        Length in bytes
- *
- * This path is taken when writing to previously unallocated clusters.
- */
-static void qed_aio_write_alloc(QEDAIOCB *acb, size_t len)
-{
-    BDRVQEDState *s = acb_to_s(acb);
-    BlockDriverCompletionFunc *cb;
-
-    /* Cancel timer when the first allocating request comes in */
-    if (QSIMPLEQ_EMPTY(&s->allocating_write_reqs)) {
-        qed_cancel_need_check_timer(s);
-    }
-
-    /* Freeze this request if another allocating write is in progress */
-    if (acb != QSIMPLEQ_FIRST(&s->allocating_write_reqs)) {
-        QSIMPLEQ_INSERT_TAIL(&s->allocating_write_reqs, acb, next);
-    }
-    if (acb != QSIMPLEQ_FIRST(&s->allocating_write_reqs) ||
-        s->allocating_write_reqs_plugged) {
-        return; /* wait for existing request to finish */
-    }
-
-    acb->cur_nclusters = qed_bytes_to_clusters(s,
-            qed_offset_into_cluster(s, acb->cur_pos) + len);
-    qemu_iovec_concat(&acb->cur_qiov, acb->qiov, acb->qiov_offset, len);
-
-    if (acb->flags & QED_AIOCB_ZERO) {
-        /* Skip ahead if the clusters are already zero */
-        if (acb->find_cluster_ret == QED_CLUSTER_ZERO) {
-            qed_aio_next_io(acb, 0);
-            return;
-        }
-
-        cb = qed_aio_write_zero_cluster;
-    } else {
-        cb = qed_aio_write_prefill;
-        acb->cur_cluster = qed_alloc_clusters(s, acb->cur_nclusters);
-    }
-
-    if (qed_should_set_need_check(s)) {
-        s->header.features |= QED_F_NEED_CHECK;
-        qed_write_header(s, cb, acb);
-    } else {
-        cb(acb, 0);
-    }
-}
-
-/**
- * Write data cluster in place
- *
- * @acb:        Write request
- * @offset:     Cluster offset in bytes
- * @len:        Length in bytes
- *
- * This path is taken when writing to already allocated clusters.
- */
-static void qed_aio_write_inplace(QEDAIOCB *acb, uint64_t offset, size_t len)
-{
-    /* Allocate buffer for zero writes */
-    if (acb->flags & QED_AIOCB_ZERO) {
-        struct iovec *iov = acb->qiov->iov;
-
-        if (!iov->iov_base) {
-            iov->iov_base = qemu_blockalign(acb->common.bs, iov->iov_len);
-            memset(iov->iov_base, 0, iov->iov_len);
-        }
-    }
-
-    /* Calculate the I/O vector */
-    acb->cur_cluster = offset;
-    qemu_iovec_concat(&acb->cur_qiov, acb->qiov, acb->qiov_offset, len);
-
-    /* Do the actual write */
-    qed_aio_write_main(acb, 0);
-}
-
-/**
- * Write data cluster
- *
- * @opaque:     Write request
- * @ret:        QED_CLUSTER_FOUND, QED_CLUSTER_L2, QED_CLUSTER_L1,
- *              or -errno
- * @offset:     Cluster offset in bytes
- * @len:        Length in bytes
- *
- * Callback from qed_find_cluster().
- */
-static void qed_aio_write_data(void *opaque, int ret,
-                               uint64_t offset, size_t len)
-{
-    QEDAIOCB *acb = opaque;
-
-    trace_qed_aio_write_data(acb_to_s(acb), acb, ret, offset, len);
-
-    acb->find_cluster_ret = ret;
-
-    switch (ret) {
-    case QED_CLUSTER_FOUND:
-        qed_aio_write_inplace(acb, offset, len);
-        break;
-
-    case QED_CLUSTER_L2:
-    case QED_CLUSTER_L1:
-    case QED_CLUSTER_ZERO:
-        qed_aio_write_alloc(acb, len);
-        break;
-
-    default:
-        qed_aio_complete(acb, ret);
-        break;
-    }
-}
-
-/**
- * Read data cluster
- *
- * @opaque:     Read request
- * @ret:        QED_CLUSTER_FOUND, QED_CLUSTER_L2, QED_CLUSTER_L1,
- *              or -errno
- * @offset:     Cluster offset in bytes
- * @len:        Length in bytes
- *
- * Callback from qed_find_cluster().
- */
-static void qed_aio_read_data(void *opaque, int ret,
-                              uint64_t offset, size_t len)
-{
-    QEDAIOCB *acb = opaque;
-    BDRVQEDState *s = acb_to_s(acb);
-    BlockDriverState *bs = acb->common.bs;
-
-    /* Adjust offset into cluster */
-    offset += qed_offset_into_cluster(s, acb->cur_pos);
-
-    trace_qed_aio_read_data(s, acb, ret, offset, len);
-
-    if (ret < 0) {
-        goto err;
-    }
-
-    qemu_iovec_concat(&acb->cur_qiov, acb->qiov, acb->qiov_offset, len);
-
-    /* Handle zero cluster and backing file reads */
-    if (ret == QED_CLUSTER_ZERO) {
-        qemu_iovec_memset(&acb->cur_qiov, 0, 0, acb->cur_qiov.size);
-        qed_aio_next_io(acb, 0);
-        return;
-    } else if (ret != QED_CLUSTER_FOUND) {
-        qed_read_backing_file(s, acb->cur_pos, &acb->cur_qiov,
-                              qed_aio_next_io, acb);
-        return;
-    }
-
-    BLKDBG_EVENT(bs->file, BLKDBG_READ_AIO);
-    bdrv_aio_readv(bs->file, offset / BDRV_SECTOR_SIZE,
-                   &acb->cur_qiov, acb->cur_qiov.size / BDRV_SECTOR_SIZE,
-                   qed_aio_next_io, acb);
-    return;
-
-err:
-    qed_aio_complete(acb, ret);
-}
-
-/**
- * Begin next I/O or complete the request
- */
-static void qed_aio_next_io(void *opaque, int ret)
-{
-    QEDAIOCB *acb = opaque;
-    BDRVQEDState *s = acb_to_s(acb);
-    QEDFindClusterFunc *io_fn = (acb->flags & QED_AIOCB_WRITE) ?
-                                qed_aio_write_data : qed_aio_read_data;
-
-    trace_qed_aio_next_io(s, acb, ret, acb->cur_pos + acb->cur_qiov.size);
-
-    /* Handle I/O error */
-    if (ret) {
-        qed_aio_complete(acb, ret);
-        return;
-    }
-
-    acb->qiov_offset += acb->cur_qiov.size;
-    acb->cur_pos += acb->cur_qiov.size;
-    qemu_iovec_reset(&acb->cur_qiov);
-
-    /* Complete request */
-    if (acb->cur_pos >= acb->end_pos) {
-        qed_aio_complete(acb, 0);
-        return;
-    }
-
-    /* Find next cluster and start I/O */
-    qed_find_cluster(s, &acb->request,
-                      acb->cur_pos, acb->end_pos - acb->cur_pos,
-                      io_fn, acb);
-}
-
-static BlockDriverAIOCB *qed_aio_setup(BlockDriverState *bs,
-                                       int64_t sector_num,
-                                       QEMUIOVector *qiov, int nb_sectors,
-                                       BlockDriverCompletionFunc *cb,
-                                       void *opaque, int flags)
-{
-    QEDAIOCB *acb = qemu_aio_get(&qed_aiocb_info, bs, cb, opaque);
-
-    trace_qed_aio_setup(bs->opaque, acb, sector_num, nb_sectors,
-                        opaque, flags);
-
-    acb->flags = flags;
-    acb->finished = NULL;
-    acb->qiov = qiov;
-    acb->qiov_offset = 0;
-    acb->cur_pos = (uint64_t)sector_num * BDRV_SECTOR_SIZE;
-    acb->end_pos = acb->cur_pos + nb_sectors * BDRV_SECTOR_SIZE;
-    acb->request.l2_table = NULL;
-    qemu_iovec_init(&acb->cur_qiov, qiov->niov);
-
-    /* Start request */
-    qed_aio_next_io(acb, 0);
-    return &acb->common;
-}
-
-static BlockDriverAIOCB *bdrv_qed_aio_readv(BlockDriverState *bs,
-                                            int64_t sector_num,
-                                            QEMUIOVector *qiov, int nb_sectors,
-                                            BlockDriverCompletionFunc *cb,
-                                            void *opaque)
-{
-    return qed_aio_setup(bs, sector_num, qiov, nb_sectors, cb, opaque, 0);
-}
-
-static BlockDriverAIOCB *bdrv_qed_aio_writev(BlockDriverState *bs,
-                                             int64_t sector_num,
-                                             QEMUIOVector *qiov, int nb_sectors,
-                                             BlockDriverCompletionFunc *cb,
-                                             void *opaque)
-{
-    return qed_aio_setup(bs, sector_num, qiov, nb_sectors, cb,
-                         opaque, QED_AIOCB_WRITE);
-}
-
-typedef struct {
-    Coroutine *co;
-    int ret;
-    bool done;
-} QEDWriteZeroesCB;
-
-static void coroutine_fn qed_co_write_zeroes_cb(void *opaque, int ret)
-{
-    QEDWriteZeroesCB *cb = opaque;
-
-    cb->done = true;
-    cb->ret = ret;
-    if (cb->co) {
-        qemu_coroutine_enter(cb->co, NULL);
-    }
-}
-
-static int coroutine_fn bdrv_qed_co_write_zeroes(BlockDriverState *bs,
-                                                 int64_t sector_num,
-                                                 int nb_sectors)
-{
-    BlockDriverAIOCB *blockacb;
-    BDRVQEDState *s = bs->opaque;
-    QEDWriteZeroesCB cb = { .done = false };
-    QEMUIOVector qiov;
-    struct iovec iov;
-
-    /* Refuse if there are untouched backing file sectors */
-    if (bs->backing_hd) {
-        if (qed_offset_into_cluster(s, sector_num * BDRV_SECTOR_SIZE) != 0) {
-            return -ENOTSUP;
-        }
-        if (qed_offset_into_cluster(s, nb_sectors * BDRV_SECTOR_SIZE) != 0) {
-            return -ENOTSUP;
-        }
-    }
-
-    /* Zero writes start without an I/O buffer.  If a buffer becomes necessary
-     * then it will be allocated during request processing.
-     */
-    iov.iov_base = NULL,
-    iov.iov_len  = nb_sectors * BDRV_SECTOR_SIZE,
-
-    qemu_iovec_init_external(&qiov, &iov, 1);
-    blockacb = qed_aio_setup(bs, sector_num, &qiov, nb_sectors,
-                             qed_co_write_zeroes_cb, &cb,
-                             QED_AIOCB_WRITE | QED_AIOCB_ZERO);
-    if (!blockacb) {
-        return -EIO;
-    }
-    if (!cb.done) {
-        cb.co = qemu_coroutine_self();
-        qemu_coroutine_yield();
-    }
-    assert(cb.done);
-    return cb.ret;
-}
-
-static int bdrv_qed_truncate(BlockDriverState *bs, int64_t offset)
-{
-    BDRVQEDState *s = bs->opaque;
-    uint64_t old_image_size;
-    int ret;
-
-    if (!qed_is_image_size_valid(offset, s->header.cluster_size,
-                                 s->header.table_size)) {
-        return -EINVAL;
-    }
-
-    /* Shrinking is currently not supported */
-    if ((uint64_t)offset < s->header.image_size) {
-        return -ENOTSUP;
-    }
-
-    old_image_size = s->header.image_size;
-    s->header.image_size = offset;
-    ret = qed_write_header_sync(s);
-    if (ret < 0) {
-        s->header.image_size = old_image_size;
-    }
-    return ret;
-}
-
-static int64_t bdrv_qed_getlength(BlockDriverState *bs)
-{
-    BDRVQEDState *s = bs->opaque;
-    return s->header.image_size;
-}
-
-static int bdrv_qed_get_info(BlockDriverState *bs, BlockDriverInfo *bdi)
-{
-    BDRVQEDState *s = bs->opaque;
-
-    memset(bdi, 0, sizeof(*bdi));
-    bdi->cluster_size = s->header.cluster_size;
-    bdi->is_dirty = s->header.features & QED_F_NEED_CHECK;
-    return 0;
-}
-
-static int bdrv_qed_change_backing_file(BlockDriverState *bs,
-                                        const char *backing_file,
-                                        const char *backing_fmt)
-{
-    BDRVQEDState *s = bs->opaque;
-    QEDHeader new_header, le_header;
-    void *buffer;
-    size_t buffer_len, backing_file_len;
-    int ret;
-
-    /* Refuse to set backing filename if unknown compat feature bits are
-     * active.  If the image uses an unknown compat feature then we may not
-     * know the layout of data following the header structure and cannot safely
-     * add a new string.
-     */
-    if (backing_file && (s->header.compat_features &
-                         ~QED_COMPAT_FEATURE_MASK)) {
-        return -ENOTSUP;
-    }
-
-    memcpy(&new_header, &s->header, sizeof(new_header));
-
-    new_header.features &= ~(QED_F_BACKING_FILE |
-                             QED_F_BACKING_FORMAT_NO_PROBE);
-
-    /* Adjust feature flags */
-    if (backing_file) {
-        new_header.features |= QED_F_BACKING_FILE;
-
-        if (qed_fmt_is_raw(backing_fmt)) {
-            new_header.features |= QED_F_BACKING_FORMAT_NO_PROBE;
-        }
-    }
-
-    /* Calculate new header size */
-    backing_file_len = 0;
-
-    if (backing_file) {
-        backing_file_len = strlen(backing_file);
-    }
-
-    buffer_len = sizeof(new_header);
-    new_header.backing_filename_offset = buffer_len;
-    new_header.backing_filename_size = backing_file_len;
-    buffer_len += backing_file_len;
-
-    /* Make sure we can rewrite header without failing */
-    if (buffer_len > new_header.header_size * new_header.cluster_size) {
-        return -ENOSPC;
-    }
-
-    /* Prepare new header */
-    buffer = g_malloc(buffer_len);
-
-    qed_header_cpu_to_le(&new_header, &le_header);
-    memcpy(buffer, &le_header, sizeof(le_header));
-    buffer_len = sizeof(le_header);
-
-    if (backing_file) {
-        memcpy(buffer + buffer_len, backing_file, backing_file_len);
-        buffer_len += backing_file_len;
-    }
-
-    /* Write new header */
-    ret = bdrv_pwrite_sync(bs->file, 0, buffer, buffer_len);
-    g_free(buffer);
-    if (ret == 0) {
-        memcpy(&s->header, &new_header, sizeof(new_header));
-    }
-    return ret;
-}
-
-static void bdrv_qed_invalidate_cache(BlockDriverState *bs)
-{
-    BDRVQEDState *s = bs->opaque;
-
-    bdrv_qed_close(bs);
-    memset(s, 0, sizeof(BDRVQEDState));
-    bdrv_qed_open(bs, NULL, bs->open_flags);
-}
-
-static int bdrv_qed_check(BlockDriverState *bs, BdrvCheckResult *result,
-                          BdrvCheckMode fix)
-{
-    BDRVQEDState *s = bs->opaque;
-
-    return qed_check(s, result, !!fix);
-}
-
-static QEMUOptionParameter qed_create_options[] = {
-    {
-        .name = BLOCK_OPT_SIZE,
-        .type = OPT_SIZE,
-        .help = "Virtual disk size (in bytes)"
-    }, {
-        .name = BLOCK_OPT_BACKING_FILE,
-        .type = OPT_STRING,
-        .help = "File name of a base image"
-    }, {
-        .name = BLOCK_OPT_BACKING_FMT,
-        .type = OPT_STRING,
-        .help = "Image format of the base image"
-    }, {
-        .name = BLOCK_OPT_CLUSTER_SIZE,
-        .type = OPT_SIZE,
-        .help = "Cluster size (in bytes)",
-        .value = { .n = QED_DEFAULT_CLUSTER_SIZE },
-    }, {
-        .name = BLOCK_OPT_TABLE_SIZE,
-        .type = OPT_SIZE,
-        .help = "L1/L2 table size (in clusters)"
-    },
-    { /* end of list */ }
-};
-
-static BlockDriver bdrv_qed = {
-    .format_name              = "qed",
-    .instance_size            = sizeof(BDRVQEDState),
-    .create_options           = qed_create_options,
-
-    .bdrv_probe               = bdrv_qed_probe,
-    .bdrv_rebind              = bdrv_qed_rebind,
-    .bdrv_open                = bdrv_qed_open,
-    .bdrv_close               = bdrv_qed_close,
-    .bdrv_reopen_prepare      = bdrv_qed_reopen_prepare,
-    .bdrv_create              = bdrv_qed_create,
-    .bdrv_has_zero_init       = bdrv_has_zero_init_1,
-    .bdrv_co_is_allocated     = bdrv_qed_co_is_allocated,
-    .bdrv_make_empty          = bdrv_qed_make_empty,
-    .bdrv_aio_readv           = bdrv_qed_aio_readv,
-    .bdrv_aio_writev          = bdrv_qed_aio_writev,
-    .bdrv_co_write_zeroes     = bdrv_qed_co_write_zeroes,
-    .bdrv_truncate            = bdrv_qed_truncate,
-    .bdrv_getlength           = bdrv_qed_getlength,
-    .bdrv_get_info            = bdrv_qed_get_info,
-    .bdrv_change_backing_file = bdrv_qed_change_backing_file,
-    .bdrv_invalidate_cache    = bdrv_qed_invalidate_cache,
-    .bdrv_check               = bdrv_qed_check,
-};
-
-static void bdrv_qed_init(void)
-{
-    bdrv_register(&bdrv_qed);
-}
-
-block_init(bdrv_qed_init);
diff --git a/contrib/qemu/block/qed.h b/contrib/qemu/block/qed.h
deleted file mode 100644
index 2b4ddedf313..00000000000
--- a/contrib/qemu/block/qed.h
+++ /dev/null
@@ -1,344 +0,0 @@
-/*
- * QEMU Enhanced Disk Format
- *
- * Copyright IBM, Corp. 2010
- *
- * Authors:
- *  Stefan Hajnoczi   <stefanha@linux.vnet.ibm.com>
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#ifndef BLOCK_QED_H
-#define BLOCK_QED_H
-
-#include "block/block_int.h"
-
-/* The layout of a QED file is as follows:
- *
- * +--------+----------+----------+----------+-----+
- * | header | L1 table | cluster0 | cluster1 | ... |
- * +--------+----------+----------+----------+-----+
- *
- * There is a 2-level pagetable for cluster allocation:
- *
- *                     +----------+
- *                     | L1 table |
- *                     +----------+
- *                ,------'  |  '------.
- *           +----------+   |    +----------+
- *           | L2 table |  ...   | L2 table |
- *           +----------+        +----------+
- *       ,------'  |  '------.
- *  +----------+   |    +----------+
- *  |   Data   |  ...   |   Data   |
- *  +----------+        +----------+
- *
- * The L1 table is fixed size and always present.  L2 tables are allocated on
- * demand.  The L1 table size determines the maximum possible image size; it
- * can be influenced using the cluster_size and table_size values.
- *
- * All fields are little-endian on disk.
- */
-
-enum {
-    QED_MAGIC = 'Q' | 'E' << 8 | 'D' << 16 | '\0' << 24,
-
-    /* The image supports a backing file */
-    QED_F_BACKING_FILE = 0x01,
-
-    /* The image needs a consistency check before use */
-    QED_F_NEED_CHECK = 0x02,
-
-    /* The backing file format must not be probed, treat as raw image */
-    QED_F_BACKING_FORMAT_NO_PROBE = 0x04,
-
-    /* Feature bits must be used when the on-disk format changes */
-    QED_FEATURE_MASK = QED_F_BACKING_FILE | /* supported feature bits */
-                       QED_F_NEED_CHECK |
-                       QED_F_BACKING_FORMAT_NO_PROBE,
-    QED_COMPAT_FEATURE_MASK = 0,            /* supported compat feature bits */
-    QED_AUTOCLEAR_FEATURE_MASK = 0,         /* supported autoclear feature bits */
-
-    /* Data is stored in groups of sectors called clusters.  Cluster size must
-     * be large to avoid keeping too much metadata.  I/O requests that have
-     * sub-cluster size will require read-modify-write.
-     */
-    QED_MIN_CLUSTER_SIZE = 4 * 1024, /* in bytes */
-    QED_MAX_CLUSTER_SIZE = 64 * 1024 * 1024,
-    QED_DEFAULT_CLUSTER_SIZE = 64 * 1024,
-
-    /* Allocated clusters are tracked using a 2-level pagetable.  Table size is
-     * a multiple of clusters so large maximum image sizes can be supported
-     * without jacking up the cluster size too much.
-     */
-    QED_MIN_TABLE_SIZE = 1,        /* in clusters */
-    QED_MAX_TABLE_SIZE = 16,
-    QED_DEFAULT_TABLE_SIZE = 4,
-
-    /* Delay to flush and clean image after last allocating write completes */
-    QED_NEED_CHECK_TIMEOUT = 5,    /* in seconds */
-};
-
-typedef struct {
-    uint32_t magic;                 /* QED\0 */
-
-    uint32_t cluster_size;          /* in bytes */
-    uint32_t table_size;            /* for L1 and L2 tables, in clusters */
-    uint32_t header_size;           /* in clusters */
-
-    uint64_t features;              /* format feature bits */
-    uint64_t compat_features;       /* compatible feature bits */
-    uint64_t autoclear_features;    /* self-resetting feature bits */
-
-    uint64_t l1_table_offset;       /* in bytes */
-    uint64_t image_size;            /* total logical image size, in bytes */
-
-    /* if (features & QED_F_BACKING_FILE) */
-    uint32_t backing_filename_offset; /* in bytes from start of header */
-    uint32_t backing_filename_size;   /* in bytes */
-} QEDHeader;
-
-typedef struct {
-    uint64_t offsets[0];            /* in bytes */
-} QEDTable;
-
-/* The L2 cache is a simple write-through cache for L2 structures */
-typedef struct CachedL2Table {
-    QEDTable *table;
-    uint64_t offset;    /* offset=0 indicates an invalidate entry */
-    QTAILQ_ENTRY(CachedL2Table) node;
-    int ref;
-} CachedL2Table;
-
-typedef struct {
-    QTAILQ_HEAD(, CachedL2Table) entries;
-    unsigned int n_entries;
-} L2TableCache;
-
-typedef struct QEDRequest {
-    CachedL2Table *l2_table;
-} QEDRequest;
-
-enum {
-    QED_AIOCB_WRITE = 0x0001,       /* read or write? */
-    QED_AIOCB_ZERO  = 0x0002,       /* zero write, used with QED_AIOCB_WRITE */
-};
-
-typedef struct QEDAIOCB {
-    BlockDriverAIOCB common;
-    QEMUBH *bh;
-    int bh_ret;                     /* final return status for completion bh */
-    QSIMPLEQ_ENTRY(QEDAIOCB) next;  /* next request */
-    int flags;                      /* QED_AIOCB_* bits ORed together */
-    bool *finished;                 /* signal for cancel completion */
-    uint64_t end_pos;               /* request end on block device, in bytes */
-
-    /* User scatter-gather list */
-    QEMUIOVector *qiov;
-    size_t qiov_offset;             /* byte count already processed */
-
-    /* Current cluster scatter-gather list */
-    QEMUIOVector cur_qiov;
-    uint64_t cur_pos;               /* position on block device, in bytes */
-    uint64_t cur_cluster;           /* cluster offset in image file */
-    unsigned int cur_nclusters;     /* number of clusters being accessed */
-    int find_cluster_ret;           /* used for L1/L2 update */
-
-    QEDRequest request;
-} QEDAIOCB;
-
-typedef struct {
-    BlockDriverState *bs;           /* device */
-    uint64_t file_size;             /* length of image file, in bytes */
-
-    QEDHeader header;               /* always cpu-endian */
-    QEDTable *l1_table;
-    L2TableCache l2_cache;          /* l2 table cache */
-    uint32_t table_nelems;
-    uint32_t l1_shift;
-    uint32_t l2_shift;
-    uint32_t l2_mask;
-
-    /* Allocating write request queue */
-    QSIMPLEQ_HEAD(, QEDAIOCB) allocating_write_reqs;
-    bool allocating_write_reqs_plugged;
-
-    /* Periodic flush and clear need check flag */
-    QEMUTimer *need_check_timer;
-} BDRVQEDState;
-
-enum {
-    QED_CLUSTER_FOUND,         /* cluster found */
-    QED_CLUSTER_ZERO,          /* zero cluster found */
-    QED_CLUSTER_L2,            /* cluster missing in L2 */
-    QED_CLUSTER_L1,            /* cluster missing in L1 */
-};
-
-/**
- * qed_find_cluster() completion callback
- *
- * @opaque:     User data for completion callback
- * @ret:        QED_CLUSTER_FOUND   Success
- *              QED_CLUSTER_L2      Data cluster unallocated in L2
- *              QED_CLUSTER_L1      L2 unallocated in L1
- *              -errno              POSIX error occurred
- * @offset:     Data cluster offset
- * @len:        Contiguous bytes starting from cluster offset
- *
- * This function is invoked when qed_find_cluster() completes.
- *
- * On success ret is QED_CLUSTER_FOUND and offset/len are a contiguous range
- * in the image file.
- *
- * On failure ret is QED_CLUSTER_L2 or QED_CLUSTER_L1 for missing L2 or L1
- * table offset, respectively.  len is number of contiguous unallocated bytes.
- */
-typedef void QEDFindClusterFunc(void *opaque, int ret, uint64_t offset, size_t len);
-
-/**
- * Generic callback for chaining async callbacks
- */
-typedef struct {
-    BlockDriverCompletionFunc *cb;
-    void *opaque;
-} GenericCB;
-
-void *gencb_alloc(size_t len, BlockDriverCompletionFunc *cb, void *opaque);
-void gencb_complete(void *opaque, int ret);
-
-/**
- * Header functions
- */
-int qed_write_header_sync(BDRVQEDState *s);
-
-/**
- * L2 cache functions
- */
-void qed_init_l2_cache(L2TableCache *l2_cache);
-void qed_free_l2_cache(L2TableCache *l2_cache);
-CachedL2Table *qed_alloc_l2_cache_entry(L2TableCache *l2_cache);
-void qed_unref_l2_cache_entry(CachedL2Table *entry);
-CachedL2Table *qed_find_l2_cache_entry(L2TableCache *l2_cache, uint64_t offset);
-void qed_commit_l2_cache_entry(L2TableCache *l2_cache, CachedL2Table *l2_table);
-
-/**
- * Table I/O functions
- */
-int qed_read_l1_table_sync(BDRVQEDState *s);
-void qed_write_l1_table(BDRVQEDState *s, unsigned int index, unsigned int n,
-                        BlockDriverCompletionFunc *cb, void *opaque);
-int qed_write_l1_table_sync(BDRVQEDState *s, unsigned int index,
-                            unsigned int n);
-int qed_read_l2_table_sync(BDRVQEDState *s, QEDRequest *request,
-                           uint64_t offset);
-void qed_read_l2_table(BDRVQEDState *s, QEDRequest *request, uint64_t offset,
-                       BlockDriverCompletionFunc *cb, void *opaque);
-void qed_write_l2_table(BDRVQEDState *s, QEDRequest *request,
-                        unsigned int index, unsigned int n, bool flush,
-                        BlockDriverCompletionFunc *cb, void *opaque);
-int qed_write_l2_table_sync(BDRVQEDState *s, QEDRequest *request,
-                            unsigned int index, unsigned int n, bool flush);
-
-/**
- * Cluster functions
- */
-void qed_find_cluster(BDRVQEDState *s, QEDRequest *request, uint64_t pos,
-                      size_t len, QEDFindClusterFunc *cb, void *opaque);
-
-/**
- * Consistency check
- */
-int qed_check(BDRVQEDState *s, BdrvCheckResult *result, bool fix);
-
-QEDTable *qed_alloc_table(BDRVQEDState *s);
-
-/**
- * Round down to the start of a cluster
- */
-static inline uint64_t qed_start_of_cluster(BDRVQEDState *s, uint64_t offset)
-{
-    return offset & ~(uint64_t)(s->header.cluster_size - 1);
-}
-
-static inline uint64_t qed_offset_into_cluster(BDRVQEDState *s, uint64_t offset)
-{
-    return offset & (s->header.cluster_size - 1);
-}
-
-static inline uint64_t qed_bytes_to_clusters(BDRVQEDState *s, uint64_t bytes)
-{
-    return qed_start_of_cluster(s, bytes + (s->header.cluster_size - 1)) /
-           (s->header.cluster_size - 1);
-}
-
-static inline unsigned int qed_l1_index(BDRVQEDState *s, uint64_t pos)
-{
-    return pos >> s->l1_shift;
-}
-
-static inline unsigned int qed_l2_index(BDRVQEDState *s, uint64_t pos)
-{
-    return (pos >> s->l2_shift) & s->l2_mask;
-}
-
-/**
- * Test if a cluster offset is valid
- */
-static inline bool qed_check_cluster_offset(BDRVQEDState *s, uint64_t offset)
-{
-    uint64_t header_size = (uint64_t)s->header.header_size *
-                           s->header.cluster_size;
-
-    if (offset & (s->header.cluster_size - 1)) {
-        return false;
-    }
-    return offset >= header_size && offset < s->file_size;
-}
-
-/**
- * Test if a table offset is valid
- */
-static inline bool qed_check_table_offset(BDRVQEDState *s, uint64_t offset)
-{
-    uint64_t end_offset = offset + (s->header.table_size - 1) *
-                          s->header.cluster_size;
-
-    /* Overflow check */
-    if (end_offset <= offset) {
-        return false;
-    }
-
-    return qed_check_cluster_offset(s, offset) &&
-           qed_check_cluster_offset(s, end_offset);
-}
-
-static inline bool qed_offset_is_cluster_aligned(BDRVQEDState *s,
-                                                 uint64_t offset)
-{
-    if (qed_offset_into_cluster(s, offset)) {
-        return false;
-    }
-    return true;
-}
-
-static inline bool qed_offset_is_unalloc_cluster(uint64_t offset)
-{
-    if (offset == 0) {
-        return true;
-    }
-    return false;
-}
-
-static inline bool qed_offset_is_zero_cluster(uint64_t offset)
-{
-    if (offset == 1) {
-        return true;
-    }
-    return false;
-}
-
-#endif /* BLOCK_QED_H */
diff --git a/contrib/qemu/block/snapshot.c b/contrib/qemu/block/snapshot.c
deleted file mode 100644
index 6c6d9deea1f..00000000000
--- a/contrib/qemu/block/snapshot.c
+++ /dev/null
@@ -1,157 +0,0 @@
-/*
- * Block layer snapshot related functions
- *
- * Copyright (c) 2003-2008 Fabrice Bellard
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-
-#include "block/snapshot.h"
-#include "block/block_int.h"
-
-int bdrv_snapshot_find(BlockDriverState *bs, QEMUSnapshotInfo *sn_info,
-                       const char *name)
-{
-    QEMUSnapshotInfo *sn_tab, *sn;
-    int nb_sns, i, ret;
-
-    ret = -ENOENT;
-    nb_sns = bdrv_snapshot_list(bs, &sn_tab);
-    if (nb_sns < 0) {
-        return ret;
-    }
-    for (i = 0; i < nb_sns; i++) {
-        sn = &sn_tab[i];
-        if (!strcmp(sn->id_str, name) || !strcmp(sn->name, name)) {
-            *sn_info = *sn;
-            ret = 0;
-            break;
-        }
-    }
-    g_free(sn_tab);
-    return ret;
-}
-
-int bdrv_can_snapshot(BlockDriverState *bs)
-{
-    BlockDriver *drv = bs->drv;
-    if (!drv || !bdrv_is_inserted(bs) || bdrv_is_read_only(bs)) {
-        return 0;
-    }
-
-    if (!drv->bdrv_snapshot_create) {
-        if (bs->file != NULL) {
-            return bdrv_can_snapshot(bs->file);
-        }
-        return 0;
-    }
-
-    return 1;
-}
-
-int bdrv_snapshot_create(BlockDriverState *bs,
-                         QEMUSnapshotInfo *sn_info)
-{
-    BlockDriver *drv = bs->drv;
-    if (!drv) {
-        return -ENOMEDIUM;
-    }
-    if (drv->bdrv_snapshot_create) {
-        return drv->bdrv_snapshot_create(bs, sn_info);
-    }
-    if (bs->file) {
-        return bdrv_snapshot_create(bs->file, sn_info);
-    }
-    return -ENOTSUP;
-}
-
-int bdrv_snapshot_goto(BlockDriverState *bs,
-                       const char *snapshot_id)
-{
-    BlockDriver *drv = bs->drv;
-    int ret, open_ret;
-
-    if (!drv) {
-        return -ENOMEDIUM;
-    }
-    if (drv->bdrv_snapshot_goto) {
-        return drv->bdrv_snapshot_goto(bs, snapshot_id);
-    }
-
-    if (bs->file) {
-        drv->bdrv_close(bs);
-        ret = bdrv_snapshot_goto(bs->file, snapshot_id);
-        open_ret = drv->bdrv_open(bs, NULL, bs->open_flags);
-        if (open_ret < 0) {
-            bdrv_delete(bs->file);
-            bs->drv = NULL;
-            return open_ret;
-        }
-        return ret;
-    }
-
-    return -ENOTSUP;
-}
-
-int bdrv_snapshot_delete(BlockDriverState *bs, const char *snapshot_id)
-{
-    BlockDriver *drv = bs->drv;
-    if (!drv) {
-        return -ENOMEDIUM;
-    }
-    if (drv->bdrv_snapshot_delete) {
-        return drv->bdrv_snapshot_delete(bs, snapshot_id);
-    }
-    if (bs->file) {
-        return bdrv_snapshot_delete(bs->file, snapshot_id);
-    }
-    return -ENOTSUP;
-}
-
-int bdrv_snapshot_list(BlockDriverState *bs,
-                       QEMUSnapshotInfo **psn_info)
-{
-    BlockDriver *drv = bs->drv;
-    if (!drv) {
-        return -ENOMEDIUM;
-    }
-    if (drv->bdrv_snapshot_list) {
-        return drv->bdrv_snapshot_list(bs, psn_info);
-    }
-    if (bs->file) {
-        return bdrv_snapshot_list(bs->file, psn_info);
-    }
-    return -ENOTSUP;
-}
-
-int bdrv_snapshot_load_tmp(BlockDriverState *bs,
-        const char *snapshot_name)
-{
-    BlockDriver *drv = bs->drv;
-    if (!drv) {
-        return -ENOMEDIUM;
-    }
-    if (!bs->read_only) {
-        return -EINVAL;
-    }
-    if (drv->bdrv_snapshot_load_tmp) {
-        return drv->bdrv_snapshot_load_tmp(bs, snapshot_name);
-    }
-    return -ENOTSUP;
-}
diff --git a/contrib/qemu/config-host.h b/contrib/qemu/config-host.h
deleted file mode 100644
index 6b5c8da1243..00000000000
--- a/contrib/qemu/config-host.h
+++ /dev/null
@@ -1,72 +0,0 @@
-/* Automatically generated by create_config - do not modify */
-#define CONFIG_QEMU_CONFDIR "/usr/local/etc/qemu"
-#define CONFIG_QEMU_DATADIR "/usr/local/share/qemu"
-#define CONFIG_QEMU_DOCDIR "/usr/local/share/doc/qemu"
-#define CONFIG_QEMU_LOCALSTATEDIR "/usr/local/var"
-#define CONFIG_QEMU_HELPERDIR "/usr/local/libexec"
-#define CONFIG_QEMU_LOCALEDIR "/usr/local/share/locale"
-#define HOST_X86_64 1
-#define CONFIG_QEMU_LDST_OPTIMIZATION 1
-#define CONFIG_POSIX 1
-#define CONFIG_LINUX 1
-#define CONFIG_SLIRP 1
-#define CONFIG_SMBD_COMMAND "/usr/sbin/smbd"
-#define CONFIG_AUDIO_DRIVERS \
-    &oss_audio_driver,\
-
-#define CONFIG_OSS 1
-#define CONFIG_BDRV_RW_WHITELIST\
-    NULL
-#define CONFIG_BDRV_RO_WHITELIST\
-    NULL
-#define CONFIG_VNC 1
-#define CONFIG_VNC_TLS 1
-#define CONFIG_VNC_SASL 1
-#define CONFIG_VNC_WS 1
-#define CONFIG_FNMATCH 1
-#define CONFIG_UUID 1
-#define CONFIG_XFS 1
-#define QEMU_VERSION "1.5.50"
-#define QEMU_PKGVERSION ""
-#define CONFIG_CURSES 1
-#define CONFIG_UTIMENSAT 1
-#define CONFIG_PIPE2 1
-#define CONFIG_ACCEPT4 1
-#define CONFIG_SPLICE 1
-#define CONFIG_EVENTFD 1
-#define CONFIG_FALLOCATE 1
-#define CONFIG_FALLOCATE_PUNCH_HOLE 1
-#define CONFIG_SYNC_FILE_RANGE 1
-#define CONFIG_FIEMAP 1
-#define CONFIG_DUP3 1
-#define CONFIG_EPOLL 1
-#define CONFIG_EPOLL_CREATE1 1
-#define CONFIG_EPOLL_PWAIT 1
-#define CONFIG_SENDFILE 1
-#define CONFIG_INOTIFY 1
-#define CONFIG_INOTIFY1 1
-#define CONFIG_BYTESWAP_H 1
-#define CONFIG_CURL 1
-#define CONFIG_LINUX_AIO 1
-#define CONFIG_ATTR 1
-#define CONFIG_VHOST_SCSI 1
-#define CONFIG_IOVEC 1
-#define CONFIG_PREADV 1
-#define CONFIG_FDT 1
-#define CONFIG_SIGNALFD 1
-#define CONFIG_FDATASYNC 1
-#define CONFIG_MADVISE 1
-#define CONFIG_POSIX_MADVISE 1
-#define CONFIG_SIGEV_THREAD_ID 1
-#define CONFIG_UNAME_RELEASE ""
-#define CONFIG_QOM_CAST_DEBUG 1
-#define CONFIG_COROUTINE_BACKEND ucontext
-#define CONFIG_OPEN_BY_HANDLE 1
-#define CONFIG_LINUX_MAGIC_H 1
-#define CONFIG_PRAGMA_DIAGNOSTIC_AVAILABLE 1
-#define CONFIG_HAS_ENVIRON 1
-#define CONFIG_CPUID_H 1
-#define CONFIG_VIRTIO_BLK_DATA_PLANE $(CONFIG_VIRTIO)
-#define CONFIG_TRACE_NOP 1
-#define CONFIG_TRACE_FILE trace
-#define CONFIG_TRACE_DEFAULT 1
diff --git a/contrib/qemu/coroutine-ucontext.c b/contrib/qemu/coroutine-ucontext.c
deleted file mode 100644
index 4bf2cde279b..00000000000
--- a/contrib/qemu/coroutine-ucontext.c
+++ /dev/null
@@ -1,225 +0,0 @@
-/*
- * ucontext coroutine initialization code
- *
- * Copyright (C) 2006  Anthony Liguori <anthony@codemonkey.ws>
- * Copyright (C) 2011  Kevin Wolf <kwolf@redhat.com>
- *
- * This library is free software; you can redistribute it and/or
- * modify it under the terms of the GNU Lesser General Public
- * License as published by the Free Software Foundation; either
- * version 2.0 of the License, or (at your option) any later version.
- *
- * This library is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
- * Lesser General Public License for more details.
- *
- * You should have received a copy of the GNU Lesser General Public
- * License along with this library; if not, see <http://www.gnu.org/licenses/>.
- */
-
-/* XXX Is there a nicer way to disable glibc's stack check for longjmp? */
-#ifdef _FORTIFY_SOURCE
-#undef _FORTIFY_SOURCE
-#endif
-#include <stdlib.h>
-#include <setjmp.h>
-#include <stdint.h>
-#include <pthread.h>
-#include <ucontext.h>
-#include "qemu-common.h"
-#include "block/coroutine_int.h"
-
-#ifdef CONFIG_VALGRIND_H
-#include <valgrind/valgrind.h>
-#endif
-
-typedef struct {
-    Coroutine base;
-    void *stack;
-    sigjmp_buf env;
-
-#ifdef CONFIG_VALGRIND_H
-    unsigned int valgrind_stack_id;
-#endif
-
-} CoroutineUContext;
-
-/**
- * Per-thread coroutine bookkeeping
- */
-typedef struct {
-    /** Currently executing coroutine */
-    Coroutine *current;
-
-    /** The default coroutine */
-    CoroutineUContext leader;
-} CoroutineThreadState;
-
-static pthread_key_t thread_state_key;
-
-/*
- * va_args to makecontext() must be type 'int', so passing
- * the pointer we need may require several int args. This
- * union is a quick hack to let us do that
- */
-union cc_arg {
-    void *p;
-    int i[2];
-};
-
-static CoroutineThreadState *coroutine_get_thread_state(void)
-{
-    CoroutineThreadState *s = pthread_getspecific(thread_state_key);
-
-    if (!s) {
-        s = g_malloc0(sizeof(*s));
-        s->current = &s->leader.base;
-        pthread_setspecific(thread_state_key, s);
-    }
-    return s;
-}
-
-static void qemu_coroutine_thread_cleanup(void *opaque)
-{
-    CoroutineThreadState *s = opaque;
-
-    g_free(s);
-}
-
-static void __attribute__((constructor)) coroutine_init(void)
-{
-    int ret;
-
-    ret = pthread_key_create(&thread_state_key, qemu_coroutine_thread_cleanup);
-    if (ret != 0) {
-        fprintf(stderr, "unable to create leader key: %s\n", strerror(errno));
-        abort();
-    }
-}
-
-static void coroutine_trampoline(int i0, int i1)
-{
-    union cc_arg arg;
-    CoroutineUContext *self;
-    Coroutine *co;
-
-    arg.i[0] = i0;
-    arg.i[1] = i1;
-    self = arg.p;
-    co = &self->base;
-
-    /* Initialize longjmp environment and switch back the caller */
-    if (!sigsetjmp(self->env, 0)) {
-        siglongjmp(*(sigjmp_buf *)co->entry_arg, 1);
-    }
-
-    while (true) {
-        co->entry(co->entry_arg);
-        qemu_coroutine_switch(co, co->caller, COROUTINE_TERMINATE);
-    }
-}
-
-Coroutine *qemu_coroutine_new(void)
-{
-    const size_t stack_size = 1 << 20;
-    CoroutineUContext *co;
-    ucontext_t old_uc, uc;
-    sigjmp_buf old_env;
-    union cc_arg arg = {0};
-
-    /* The ucontext functions preserve signal masks which incurs a
-     * system call overhead.  sigsetjmp(buf, 0)/siglongjmp() does not
-     * preserve signal masks but only works on the current stack.
-     * Since we need a way to create and switch to a new stack, use
-     * the ucontext functions for that but sigsetjmp()/siglongjmp() for
-     * everything else.
-     */
-
-    if (getcontext(&uc) == -1) {
-        abort();
-    }
-
-    co = g_malloc0(sizeof(*co));
-    co->stack = g_malloc(stack_size);
-    co->base.entry_arg = &old_env; /* stash away our jmp_buf */
-
-    uc.uc_link = &old_uc;
-    uc.uc_stack.ss_sp = co->stack;
-    uc.uc_stack.ss_size = stack_size;
-    uc.uc_stack.ss_flags = 0;
-
-#ifdef CONFIG_VALGRIND_H
-    co->valgrind_stack_id =
-        VALGRIND_STACK_REGISTER(co->stack, co->stack + stack_size);
-#endif
-
-    arg.p = co;
-
-    makecontext(&uc, (void (*)(void))coroutine_trampoline,
-                2, arg.i[0], arg.i[1]);
-
-    /* swapcontext() in, siglongjmp() back out */
-    if (!sigsetjmp(old_env, 0)) {
-        swapcontext(&old_uc, &uc);
-    }
-    return &co->base;
-}
-
-#ifdef CONFIG_VALGRIND_H
-#ifdef CONFIG_PRAGMA_DIAGNOSTIC_AVAILABLE
-/* Work around an unused variable in the valgrind.h macro... */
-#pragma GCC diagnostic push
-#pragma GCC diagnostic ignored "-Wunused-but-set-variable"
-#endif
-static inline void valgrind_stack_deregister(CoroutineUContext *co)
-{
-    VALGRIND_STACK_DEREGISTER(co->valgrind_stack_id);
-}
-#ifdef CONFIG_PRAGMA_DIAGNOSTIC_AVAILABLE
-#pragma GCC diagnostic pop
-#endif
-#endif
-
-void qemu_coroutine_delete(Coroutine *co_)
-{
-    CoroutineUContext *co = DO_UPCAST(CoroutineUContext, base, co_);
-
-#ifdef CONFIG_VALGRIND_H
-    valgrind_stack_deregister(co);
-#endif
-
-    g_free(co->stack);
-    g_free(co);
-}
-
-CoroutineAction qemu_coroutine_switch(Coroutine *from_, Coroutine *to_,
-                                      CoroutineAction action)
-{
-    CoroutineUContext *from = DO_UPCAST(CoroutineUContext, base, from_);
-    CoroutineUContext *to = DO_UPCAST(CoroutineUContext, base, to_);
-    CoroutineThreadState *s = coroutine_get_thread_state();
-    int ret;
-
-    s->current = to_;
-
-    ret = sigsetjmp(from->env, 0);
-    if (ret == 0) {
-        siglongjmp(to->env, action);
-    }
-    return ret;
-}
-
-Coroutine *qemu_coroutine_self(void)
-{
-    CoroutineThreadState *s = coroutine_get_thread_state();
-
-    return s->current;
-}
-
-bool qemu_in_coroutine(void)
-{
-    CoroutineThreadState *s = pthread_getspecific(thread_state_key);
-
-    return s && s->current->caller;
-}
diff --git a/contrib/qemu/include/block/aio.h b/contrib/qemu/include/block/aio.h
deleted file mode 100644
index 183679374fa..00000000000
--- a/contrib/qemu/include/block/aio.h
+++ /dev/null
@@ -1,247 +0,0 @@
-/*
- * QEMU aio implementation
- *
- * Copyright IBM, Corp. 2008
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU GPL, version 2.  See
- * the COPYING file in the top-level directory.
- *
- */
-
-#ifndef QEMU_AIO_H
-#define QEMU_AIO_H
-
-#include "qemu-common.h"
-#include "qemu/queue.h"
-#include "qemu/event_notifier.h"
-
-typedef struct BlockDriverAIOCB BlockDriverAIOCB;
-typedef void BlockDriverCompletionFunc(void *opaque, int ret);
-
-typedef struct AIOCBInfo {
-    void (*cancel)(BlockDriverAIOCB *acb);
-    size_t aiocb_size;
-} AIOCBInfo;
-
-struct BlockDriverAIOCB {
-    const AIOCBInfo *aiocb_info;
-    BlockDriverState *bs;
-    BlockDriverCompletionFunc *cb;
-    void *opaque;
-};
-
-void *qemu_aio_get(const AIOCBInfo *aiocb_info, BlockDriverState *bs,
-                   BlockDriverCompletionFunc *cb, void *opaque);
-void qemu_aio_release(void *p);
-
-typedef struct AioHandler AioHandler;
-typedef void QEMUBHFunc(void *opaque);
-typedef void IOHandler(void *opaque);
-
-typedef struct AioContext {
-    GSource source;
-
-    /* The list of registered AIO handlers */
-    QLIST_HEAD(, AioHandler) aio_handlers;
-
-    /* This is a simple lock used to protect the aio_handlers list.
-     * Specifically, it's used to ensure that no callbacks are removed while
-     * we're walking and dispatching callbacks.
-     */
-    int walking_handlers;
-
-    /* Anchor of the list of Bottom Halves belonging to the context */
-    struct QEMUBH *first_bh;
-
-    /* A simple lock used to protect the first_bh list, and ensure that
-     * no callbacks are removed while we're walking and dispatching callbacks.
-     */
-    int walking_bh;
-
-    /* Used for aio_notify.  */
-    EventNotifier notifier;
-
-    /* GPollFDs for aio_poll() */
-    GArray *pollfds;
-
-    /* Thread pool for performing work and receiving completion callbacks */
-    struct ThreadPool *thread_pool;
-} AioContext;
-
-/* Returns 1 if there are still outstanding AIO requests; 0 otherwise */
-typedef int (AioFlushEventNotifierHandler)(EventNotifier *e);
-
-/**
- * aio_context_new: Allocate a new AioContext.
- *
- * AioContext provide a mini event-loop that can be waited on synchronously.
- * They also provide bottom halves, a service to execute a piece of code
- * as soon as possible.
- */
-AioContext *aio_context_new(void);
-
-/**
- * aio_context_ref:
- * @ctx: The AioContext to operate on.
- *
- * Add a reference to an AioContext.
- */
-void aio_context_ref(AioContext *ctx);
-
-/**
- * aio_context_unref:
- * @ctx: The AioContext to operate on.
- *
- * Drop a reference to an AioContext.
- */
-void aio_context_unref(AioContext *ctx);
-
-/**
- * aio_bh_new: Allocate a new bottom half structure.
- *
- * Bottom halves are lightweight callbacks whose invocation is guaranteed
- * to be wait-free, thread-safe and signal-safe.  The #QEMUBH structure
- * is opaque and must be allocated prior to its use.
- */
-QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque);
-
-/**
- * aio_notify: Force processing of pending events.
- *
- * Similar to signaling a condition variable, aio_notify forces
- * aio_wait to exit, so that the next call will re-examine pending events.
- * The caller of aio_notify will usually call aio_wait again very soon,
- * or go through another iteration of the GLib main loop.  Hence, aio_notify
- * also has the side effect of recalculating the sets of file descriptors
- * that the main loop waits for.
- *
- * Calling aio_notify is rarely necessary, because for example scheduling
- * a bottom half calls it already.
- */
-void aio_notify(AioContext *ctx);
-
-/**
- * aio_bh_poll: Poll bottom halves for an AioContext.
- *
- * These are internal functions used by the QEMU main loop.
- */
-int aio_bh_poll(AioContext *ctx);
-
-/**
- * qemu_bh_schedule: Schedule a bottom half.
- *
- * Scheduling a bottom half interrupts the main loop and causes the
- * execution of the callback that was passed to qemu_bh_new.
- *
- * Bottom halves that are scheduled from a bottom half handler are instantly
- * invoked.  This can create an infinite loop if a bottom half handler
- * schedules itself.
- *
- * @bh: The bottom half to be scheduled.
- */
-void qemu_bh_schedule(QEMUBH *bh);
-
-/**
- * qemu_bh_cancel: Cancel execution of a bottom half.
- *
- * Canceling execution of a bottom half undoes the effect of calls to
- * qemu_bh_schedule without freeing its resources yet.  While cancellation
- * itself is also wait-free and thread-safe, it can of course race with the
- * loop that executes bottom halves unless you are holding the iothread
- * mutex.  This makes it mostly useless if you are not holding the mutex.
- *
- * @bh: The bottom half to be canceled.
- */
-void qemu_bh_cancel(QEMUBH *bh);
-
-/**
- *qemu_bh_delete: Cancel execution of a bottom half and free its resources.
- *
- * Deleting a bottom half frees the memory that was allocated for it by
- * qemu_bh_new.  It also implies canceling the bottom half if it was
- * scheduled.
- *
- * @bh: The bottom half to be deleted.
- */
-void qemu_bh_delete(QEMUBH *bh);
-
-/* Return whether there are any pending callbacks from the GSource
- * attached to the AioContext.
- *
- * This is used internally in the implementation of the GSource.
- */
-bool aio_pending(AioContext *ctx);
-
-/* Progress in completing AIO work to occur.  This can issue new pending
- * aio as a result of executing I/O completion or bh callbacks.
- *
- * If there is no pending AIO operation or completion (bottom half),
- * return false.  If there are pending AIO operations of bottom halves,
- * return true.
- *
- * If there are no pending bottom halves, but there are pending AIO
- * operations, it may not be possible to make any progress without
- * blocking.  If @blocking is true, this function will wait until one
- * or more AIO events have completed, to ensure something has moved
- * before returning.
- */
-bool aio_poll(AioContext *ctx, bool blocking);
-
-#ifdef CONFIG_POSIX
-/* Returns 1 if there are still outstanding AIO requests; 0 otherwise */
-typedef int (AioFlushHandler)(void *opaque);
-
-/* Register a file descriptor and associated callbacks.  Behaves very similarly
- * to qemu_set_fd_handler2.  Unlike qemu_set_fd_handler2, these callbacks will
- * be invoked when using qemu_aio_wait().
- *
- * Code that invokes AIO completion functions should rely on this function
- * instead of qemu_set_fd_handler[2].
- */
-void aio_set_fd_handler(AioContext *ctx,
-                        int fd,
-                        IOHandler *io_read,
-                        IOHandler *io_write,
-                        AioFlushHandler *io_flush,
-                        void *opaque);
-#endif
-
-/* Register an event notifier and associated callbacks.  Behaves very similarly
- * to event_notifier_set_handler.  Unlike event_notifier_set_handler, these callbacks
- * will be invoked when using qemu_aio_wait().
- *
- * Code that invokes AIO completion functions should rely on this function
- * instead of event_notifier_set_handler.
- */
-void aio_set_event_notifier(AioContext *ctx,
-                            EventNotifier *notifier,
-                            EventNotifierHandler *io_read,
-                            AioFlushEventNotifierHandler *io_flush);
-
-/* Return a GSource that lets the main loop poll the file descriptors attached
- * to this AioContext.
- */
-GSource *aio_get_g_source(AioContext *ctx);
-
-/* Return the ThreadPool bound to this AioContext */
-struct ThreadPool *aio_get_thread_pool(AioContext *ctx);
-
-/* Functions to operate on the main QEMU AioContext.  */
-
-bool qemu_aio_wait(void);
-void qemu_aio_set_event_notifier(EventNotifier *notifier,
-                                 EventNotifierHandler *io_read,
-                                 AioFlushEventNotifierHandler *io_flush);
-
-#ifdef CONFIG_POSIX
-void qemu_aio_set_fd_handler(int fd,
-                             IOHandler *io_read,
-                             IOHandler *io_write,
-                             AioFlushHandler *io_flush,
-                             void *opaque);
-#endif
-
-#endif
diff --git a/contrib/qemu/include/block/block.h b/contrib/qemu/include/block/block.h
deleted file mode 100644
index b6b9014a9ce..00000000000
--- a/contrib/qemu/include/block/block.h
+++ /dev/null
@@ -1,443 +0,0 @@
-#ifndef BLOCK_H
-#define BLOCK_H
-
-#include "block/aio.h"
-#include "qemu-common.h"
-#include "qemu/option.h"
-#include "block/coroutine.h"
-#include "qapi/qmp/qobject.h"
-#include "qapi-types.h"
-
-/* block.c */
-typedef struct BlockDriver BlockDriver;
-typedef struct BlockJob BlockJob;
-
-typedef struct BlockDriverInfo {
-    /* in bytes, 0 if irrelevant */
-    int cluster_size;
-    /* offset at which the VM state can be saved (0 if not possible) */
-    int64_t vm_state_offset;
-    bool is_dirty;
-} BlockDriverInfo;
-
-typedef struct BlockFragInfo {
-    uint64_t allocated_clusters;
-    uint64_t total_clusters;
-    uint64_t fragmented_clusters;
-    uint64_t compressed_clusters;
-} BlockFragInfo;
-
-/* Callbacks for block device models */
-typedef struct BlockDevOps {
-    /*
-     * Runs when virtual media changed (monitor commands eject, change)
-     * Argument load is true on load and false on eject.
-     * Beware: doesn't run when a host device's physical media
-     * changes.  Sure would be useful if it did.
-     * Device models with removable media must implement this callback.
-     */
-    void (*change_media_cb)(void *opaque, bool load);
-    /*
-     * Runs when an eject request is issued from the monitor, the tray
-     * is closed, and the medium is locked.
-     * Device models that do not implement is_medium_locked will not need
-     * this callback.  Device models that can lock the medium or tray might
-     * want to implement the callback and unlock the tray when "force" is
-     * true, even if they do not support eject requests.
-     */
-    void (*eject_request_cb)(void *opaque, bool force);
-    /*
-     * Is the virtual tray open?
-     * Device models implement this only when the device has a tray.
-     */
-    bool (*is_tray_open)(void *opaque);
-    /*
-     * Is the virtual medium locked into the device?
-     * Device models implement this only when device has such a lock.
-     */
-    bool (*is_medium_locked)(void *opaque);
-    /*
-     * Runs when the size changed (e.g. monitor command block_resize)
-     */
-    void (*resize_cb)(void *opaque);
-} BlockDevOps;
-
-#define BDRV_O_RDWR        0x0002
-#define BDRV_O_SNAPSHOT    0x0008 /* open the file read only and save writes in a snapshot */
-#define BDRV_O_NOCACHE     0x0020 /* do not use the host page cache */
-#define BDRV_O_CACHE_WB    0x0040 /* use write-back caching */
-#define BDRV_O_NATIVE_AIO  0x0080 /* use native AIO instead of the thread pool */
-#define BDRV_O_NO_BACKING  0x0100 /* don't open the backing file */
-#define BDRV_O_NO_FLUSH    0x0200 /* disable flushing on this disk */
-#define BDRV_O_COPY_ON_READ 0x0400 /* copy read backing sectors into image */
-#define BDRV_O_INCOMING    0x0800  /* consistency hint for incoming migration */
-#define BDRV_O_CHECK       0x1000  /* open solely for consistency check */
-#define BDRV_O_ALLOW_RDWR  0x2000  /* allow reopen to change from r/o to r/w */
-#define BDRV_O_UNMAP       0x4000  /* execute guest UNMAP/TRIM operations */
-
-#define BDRV_O_CACHE_MASK  (BDRV_O_NOCACHE | BDRV_O_CACHE_WB | BDRV_O_NO_FLUSH)
-
-#define BDRV_SECTOR_BITS   9
-#define BDRV_SECTOR_SIZE   (1ULL << BDRV_SECTOR_BITS)
-#define BDRV_SECTOR_MASK   ~(BDRV_SECTOR_SIZE - 1)
-
-typedef enum {
-    BDRV_ACTION_REPORT, BDRV_ACTION_IGNORE, BDRV_ACTION_STOP
-} BlockErrorAction;
-
-typedef QSIMPLEQ_HEAD(BlockReopenQueue, BlockReopenQueueEntry) BlockReopenQueue;
-
-typedef struct BDRVReopenState {
-    BlockDriverState *bs;
-    int flags;
-    void *opaque;
-} BDRVReopenState;
-
-
-void bdrv_iostatus_enable(BlockDriverState *bs);
-void bdrv_iostatus_reset(BlockDriverState *bs);
-void bdrv_iostatus_disable(BlockDriverState *bs);
-bool bdrv_iostatus_is_enabled(const BlockDriverState *bs);
-void bdrv_iostatus_set_err(BlockDriverState *bs, int error);
-void bdrv_info_print(Monitor *mon, const QObject *data);
-void bdrv_info(Monitor *mon, QObject **ret_data);
-void bdrv_stats_print(Monitor *mon, const QObject *data);
-void bdrv_info_stats(Monitor *mon, QObject **ret_data);
-
-/* disk I/O throttling */
-void bdrv_io_limits_enable(BlockDriverState *bs);
-void bdrv_io_limits_disable(BlockDriverState *bs);
-bool bdrv_io_limits_enabled(BlockDriverState *bs);
-
-void bdrv_init(void);
-void bdrv_init_with_whitelist(void);
-BlockDriver *bdrv_find_protocol(const char *filename,
-                                bool allow_protocol_prefix);
-BlockDriver *bdrv_find_format(const char *format_name);
-BlockDriver *bdrv_find_whitelisted_format(const char *format_name,
-                                          bool readonly);
-int bdrv_create(BlockDriver *drv, const char* filename,
-    QEMUOptionParameter *options);
-int bdrv_create_file(const char* filename, QEMUOptionParameter *options);
-BlockDriverState *bdrv_new(const char *device_name);
-void bdrv_make_anon(BlockDriverState *bs);
-void bdrv_swap(BlockDriverState *bs_new, BlockDriverState *bs_old);
-void bdrv_append(BlockDriverState *bs_new, BlockDriverState *bs_top);
-void bdrv_delete(BlockDriverState *bs);
-int bdrv_parse_cache_flags(const char *mode, int *flags);
-int bdrv_parse_discard_flags(const char *mode, int *flags);
-int bdrv_file_open(BlockDriverState **pbs, const char *filename,
-                   QDict *options, int flags);
-int bdrv_open_backing_file(BlockDriverState *bs, QDict *options);
-int bdrv_open(BlockDriverState *bs, const char *filename, QDict *options,
-              int flags, BlockDriver *drv);
-BlockReopenQueue *bdrv_reopen_queue(BlockReopenQueue *bs_queue,
-                                    BlockDriverState *bs, int flags);
-int bdrv_reopen_multiple(BlockReopenQueue *bs_queue, Error **errp);
-int bdrv_reopen(BlockDriverState *bs, int bdrv_flags, Error **errp);
-int bdrv_reopen_prepare(BDRVReopenState *reopen_state,
-                        BlockReopenQueue *queue, Error **errp);
-void bdrv_reopen_commit(BDRVReopenState *reopen_state);
-void bdrv_reopen_abort(BDRVReopenState *reopen_state);
-void bdrv_close(BlockDriverState *bs);
-void bdrv_add_close_notifier(BlockDriverState *bs, Notifier *notify);
-int bdrv_attach_dev(BlockDriverState *bs, void *dev);
-void bdrv_attach_dev_nofail(BlockDriverState *bs, void *dev);
-void bdrv_detach_dev(BlockDriverState *bs, void *dev);
-void *bdrv_get_attached_dev(BlockDriverState *bs);
-void bdrv_set_dev_ops(BlockDriverState *bs, const BlockDevOps *ops,
-                      void *opaque);
-void bdrv_dev_eject_request(BlockDriverState *bs, bool force);
-bool bdrv_dev_has_removable_media(BlockDriverState *bs);
-bool bdrv_dev_is_tray_open(BlockDriverState *bs);
-bool bdrv_dev_is_medium_locked(BlockDriverState *bs);
-int bdrv_read(BlockDriverState *bs, int64_t sector_num,
-              uint8_t *buf, int nb_sectors);
-int bdrv_read_unthrottled(BlockDriverState *bs, int64_t sector_num,
-                          uint8_t *buf, int nb_sectors);
-int bdrv_write(BlockDriverState *bs, int64_t sector_num,
-               const uint8_t *buf, int nb_sectors);
-int bdrv_writev(BlockDriverState *bs, int64_t sector_num, QEMUIOVector *qiov);
-int bdrv_pread(BlockDriverState *bs, int64_t offset,
-               void *buf, int count);
-int bdrv_pwrite(BlockDriverState *bs, int64_t offset,
-                const void *buf, int count);
-int bdrv_pwritev(BlockDriverState *bs, int64_t offset, QEMUIOVector *qiov);
-int bdrv_pwrite_sync(BlockDriverState *bs, int64_t offset,
-    const void *buf, int count);
-int coroutine_fn bdrv_co_readv(BlockDriverState *bs, int64_t sector_num,
-    int nb_sectors, QEMUIOVector *qiov);
-int coroutine_fn bdrv_co_copy_on_readv(BlockDriverState *bs,
-    int64_t sector_num, int nb_sectors, QEMUIOVector *qiov);
-int coroutine_fn bdrv_co_writev(BlockDriverState *bs, int64_t sector_num,
-    int nb_sectors, QEMUIOVector *qiov);
-/*
- * Efficiently zero a region of the disk image.  Note that this is a regular
- * I/O request like read or write and should have a reasonable size.  This
- * function is not suitable for zeroing the entire image in a single request
- * because it may allocate memory for the entire region.
- */
-int coroutine_fn bdrv_co_write_zeroes(BlockDriverState *bs, int64_t sector_num,
-    int nb_sectors);
-int coroutine_fn bdrv_co_is_allocated(BlockDriverState *bs, int64_t sector_num,
-    int nb_sectors, int *pnum);
-int coroutine_fn bdrv_co_is_allocated_above(BlockDriverState *top,
-                                            BlockDriverState *base,
-                                            int64_t sector_num,
-                                            int nb_sectors, int *pnum);
-BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
-    const char *backing_file);
-int bdrv_get_backing_file_depth(BlockDriverState *bs);
-int bdrv_truncate(BlockDriverState *bs, int64_t offset);
-int64_t bdrv_getlength(BlockDriverState *bs);
-int64_t bdrv_get_allocated_file_size(BlockDriverState *bs);
-void bdrv_get_geometry(BlockDriverState *bs, uint64_t *nb_sectors_ptr);
-int bdrv_commit(BlockDriverState *bs);
-int bdrv_commit_all(void);
-int bdrv_change_backing_file(BlockDriverState *bs,
-    const char *backing_file, const char *backing_fmt);
-void bdrv_register(BlockDriver *bdrv);
-int bdrv_drop_intermediate(BlockDriverState *active, BlockDriverState *top,
-                           BlockDriverState *base);
-BlockDriverState *bdrv_find_overlay(BlockDriverState *active,
-                                    BlockDriverState *bs);
-BlockDriverState *bdrv_find_base(BlockDriverState *bs);
-
-
-typedef struct BdrvCheckResult {
-    int corruptions;
-    int leaks;
-    int check_errors;
-    int corruptions_fixed;
-    int leaks_fixed;
-    int64_t image_end_offset;
-    BlockFragInfo bfi;
-} BdrvCheckResult;
-
-typedef enum {
-    BDRV_FIX_LEAKS    = 1,
-    BDRV_FIX_ERRORS   = 2,
-} BdrvCheckMode;
-
-int bdrv_check(BlockDriverState *bs, BdrvCheckResult *res, BdrvCheckMode fix);
-
-/* async block I/O */
-typedef void BlockDriverDirtyHandler(BlockDriverState *bs, int64_t sector,
-                                     int sector_num);
-BlockDriverAIOCB *bdrv_aio_readv(BlockDriverState *bs, int64_t sector_num,
-                                 QEMUIOVector *iov, int nb_sectors,
-                                 BlockDriverCompletionFunc *cb, void *opaque);
-BlockDriverAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
-                                  QEMUIOVector *iov, int nb_sectors,
-                                  BlockDriverCompletionFunc *cb, void *opaque);
-BlockDriverAIOCB *bdrv_aio_flush(BlockDriverState *bs,
-                                 BlockDriverCompletionFunc *cb, void *opaque);
-BlockDriverAIOCB *bdrv_aio_discard(BlockDriverState *bs,
-                                   int64_t sector_num, int nb_sectors,
-                                   BlockDriverCompletionFunc *cb, void *opaque);
-void bdrv_aio_cancel(BlockDriverAIOCB *acb);
-
-typedef struct BlockRequest {
-    /* Fields to be filled by multiwrite caller */
-    int64_t sector;
-    int nb_sectors;
-    QEMUIOVector *qiov;
-    BlockDriverCompletionFunc *cb;
-    void *opaque;
-
-    /* Filled by multiwrite implementation */
-    int error;
-} BlockRequest;
-
-int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs,
-    int num_reqs);
-
-/* sg packet commands */
-int bdrv_ioctl(BlockDriverState *bs, unsigned long int req, void *buf);
-BlockDriverAIOCB *bdrv_aio_ioctl(BlockDriverState *bs,
-        unsigned long int req, void *buf,
-        BlockDriverCompletionFunc *cb, void *opaque);
-
-/* Invalidate any cached metadata used by image formats */
-void bdrv_invalidate_cache(BlockDriverState *bs);
-void bdrv_invalidate_cache_all(void);
-
-void bdrv_clear_incoming_migration_all(void);
-
-/* Ensure contents are flushed to disk.  */
-int bdrv_flush(BlockDriverState *bs);
-int coroutine_fn bdrv_co_flush(BlockDriverState *bs);
-int bdrv_flush_all(void);
-void bdrv_close_all(void);
-void bdrv_drain_all(void);
-
-int bdrv_discard(BlockDriverState *bs, int64_t sector_num, int nb_sectors);
-int bdrv_co_discard(BlockDriverState *bs, int64_t sector_num, int nb_sectors);
-int bdrv_has_zero_init_1(BlockDriverState *bs);
-int bdrv_has_zero_init(BlockDriverState *bs);
-int bdrv_is_allocated(BlockDriverState *bs, int64_t sector_num, int nb_sectors,
-                      int *pnum);
-int bdrv_is_allocated_above(BlockDriverState *top, BlockDriverState *base,
-                            int64_t sector_num, int nb_sectors, int *pnum);
-
-void bdrv_set_on_error(BlockDriverState *bs, BlockdevOnError on_read_error,
-                       BlockdevOnError on_write_error);
-BlockdevOnError bdrv_get_on_error(BlockDriverState *bs, bool is_read);
-BlockErrorAction bdrv_get_error_action(BlockDriverState *bs, bool is_read, int error);
-void bdrv_error_action(BlockDriverState *bs, BlockErrorAction action,
-                       bool is_read, int error);
-int bdrv_is_read_only(BlockDriverState *bs);
-int bdrv_is_sg(BlockDriverState *bs);
-int bdrv_enable_write_cache(BlockDriverState *bs);
-void bdrv_set_enable_write_cache(BlockDriverState *bs, bool wce);
-int bdrv_is_inserted(BlockDriverState *bs);
-int bdrv_media_changed(BlockDriverState *bs);
-void bdrv_lock_medium(BlockDriverState *bs, bool locked);
-void bdrv_eject(BlockDriverState *bs, bool eject_flag);
-const char *bdrv_get_format_name(BlockDriverState *bs);
-BlockDriverState *bdrv_find(const char *name);
-BlockDriverState *bdrv_next(BlockDriverState *bs);
-void bdrv_iterate(void (*it)(void *opaque, BlockDriverState *bs),
-                  void *opaque);
-int bdrv_is_encrypted(BlockDriverState *bs);
-int bdrv_key_required(BlockDriverState *bs);
-int bdrv_set_key(BlockDriverState *bs, const char *key);
-int bdrv_query_missing_keys(void);
-void bdrv_iterate_format(void (*it)(void *opaque, const char *name),
-                         void *opaque);
-const char *bdrv_get_device_name(BlockDriverState *bs);
-int bdrv_get_flags(BlockDriverState *bs);
-int bdrv_write_compressed(BlockDriverState *bs, int64_t sector_num,
-                          const uint8_t *buf, int nb_sectors);
-int bdrv_get_info(BlockDriverState *bs, BlockDriverInfo *bdi);
-void bdrv_round_to_clusters(BlockDriverState *bs,
-                            int64_t sector_num, int nb_sectors,
-                            int64_t *cluster_sector_num,
-                            int *cluster_nb_sectors);
-
-const char *bdrv_get_encrypted_filename(BlockDriverState *bs);
-void bdrv_get_backing_filename(BlockDriverState *bs,
-                               char *filename, int filename_size);
-void bdrv_get_full_backing_filename(BlockDriverState *bs,
-                                    char *dest, size_t sz);
-int bdrv_is_snapshot(BlockDriverState *bs);
-
-int path_is_absolute(const char *path);
-void path_combine(char *dest, int dest_size,
-                  const char *base_path,
-                  const char *filename);
-
-int bdrv_writev_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos);
-int bdrv_save_vmstate(BlockDriverState *bs, const uint8_t *buf,
-                      int64_t pos, int size);
-
-int bdrv_load_vmstate(BlockDriverState *bs, uint8_t *buf,
-                      int64_t pos, int size);
-
-void bdrv_img_create(const char *filename, const char *fmt,
-                     const char *base_filename, const char *base_fmt,
-                     char *options, uint64_t img_size, int flags,
-                     Error **errp, bool quiet);
-
-void bdrv_set_buffer_alignment(BlockDriverState *bs, int align);
-void *qemu_blockalign(BlockDriverState *bs, size_t size);
-bool bdrv_qiov_is_aligned(BlockDriverState *bs, QEMUIOVector *qiov);
-
-struct HBitmapIter;
-void bdrv_set_dirty_tracking(BlockDriverState *bs, int granularity);
-int bdrv_get_dirty(BlockDriverState *bs, int64_t sector);
-void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector, int nr_sectors);
-void bdrv_reset_dirty(BlockDriverState *bs, int64_t cur_sector, int nr_sectors);
-void bdrv_dirty_iter_init(BlockDriverState *bs, struct HBitmapIter *hbi);
-int64_t bdrv_get_dirty_count(BlockDriverState *bs);
-
-void bdrv_enable_copy_on_read(BlockDriverState *bs);
-void bdrv_disable_copy_on_read(BlockDriverState *bs);
-
-void bdrv_set_in_use(BlockDriverState *bs, int in_use);
-int bdrv_in_use(BlockDriverState *bs);
-
-#ifdef CONFIG_LINUX_AIO
-int raw_get_aio_fd(BlockDriverState *bs);
-#else
-static inline int raw_get_aio_fd(BlockDriverState *bs)
-{
-    return -ENOTSUP;
-}
-#endif
-
-enum BlockAcctType {
-    BDRV_ACCT_READ,
-    BDRV_ACCT_WRITE,
-    BDRV_ACCT_FLUSH,
-    BDRV_MAX_IOTYPE,
-};
-
-typedef struct BlockAcctCookie {
-    int64_t bytes;
-    int64_t start_time_ns;
-    enum BlockAcctType type;
-} BlockAcctCookie;
-
-void bdrv_acct_start(BlockDriverState *bs, BlockAcctCookie *cookie,
-        int64_t bytes, enum BlockAcctType type);
-void bdrv_acct_done(BlockDriverState *bs, BlockAcctCookie *cookie);
-
-typedef enum {
-    BLKDBG_L1_UPDATE,
-
-    BLKDBG_L1_GROW_ALLOC_TABLE,
-    BLKDBG_L1_GROW_WRITE_TABLE,
-    BLKDBG_L1_GROW_ACTIVATE_TABLE,
-
-    BLKDBG_L2_LOAD,
-    BLKDBG_L2_UPDATE,
-    BLKDBG_L2_UPDATE_COMPRESSED,
-    BLKDBG_L2_ALLOC_COW_READ,
-    BLKDBG_L2_ALLOC_WRITE,
-
-    BLKDBG_READ_AIO,
-    BLKDBG_READ_BACKING_AIO,
-    BLKDBG_READ_COMPRESSED,
-
-    BLKDBG_WRITE_AIO,
-    BLKDBG_WRITE_COMPRESSED,
-
-    BLKDBG_VMSTATE_LOAD,
-    BLKDBG_VMSTATE_SAVE,
-
-    BLKDBG_COW_READ,
-    BLKDBG_COW_WRITE,
-
-    BLKDBG_REFTABLE_LOAD,
-    BLKDBG_REFTABLE_GROW,
-
-    BLKDBG_REFBLOCK_LOAD,
-    BLKDBG_REFBLOCK_UPDATE,
-    BLKDBG_REFBLOCK_UPDATE_PART,
-    BLKDBG_REFBLOCK_ALLOC,
-    BLKDBG_REFBLOCK_ALLOC_HOOKUP,
-    BLKDBG_REFBLOCK_ALLOC_WRITE,
-    BLKDBG_REFBLOCK_ALLOC_WRITE_BLOCKS,
-    BLKDBG_REFBLOCK_ALLOC_WRITE_TABLE,
-    BLKDBG_REFBLOCK_ALLOC_SWITCH_TABLE,
-
-    BLKDBG_CLUSTER_ALLOC,
-    BLKDBG_CLUSTER_ALLOC_BYTES,
-    BLKDBG_CLUSTER_FREE,
-
-    BLKDBG_FLUSH_TO_OS,
-    BLKDBG_FLUSH_TO_DISK,
-
-    BLKDBG_EVENT_MAX,
-} BlkDebugEvent;
-
-#define BLKDBG_EVENT(bs, evt) bdrv_debug_event(bs, evt)
-void bdrv_debug_event(BlockDriverState *bs, BlkDebugEvent event);
-
-int bdrv_debug_breakpoint(BlockDriverState *bs, const char *event,
-                           const char *tag);
-int bdrv_debug_resume(BlockDriverState *bs, const char *tag);
-bool bdrv_debug_is_suspended(BlockDriverState *bs, const char *tag);
-
-#endif
diff --git a/contrib/qemu/include/block/block_int.h b/contrib/qemu/include/block/block_int.h
deleted file mode 100644
index c6ac871e210..00000000000
--- a/contrib/qemu/include/block/block_int.h
+++ /dev/null
@@ -1,421 +0,0 @@
-/*
- * QEMU System Emulator block driver
- *
- * Copyright (c) 2003 Fabrice Bellard
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-#ifndef BLOCK_INT_H
-#define BLOCK_INT_H
-
-#include "block/block.h"
-#include "qemu/option.h"
-#include "qemu/queue.h"
-#include "block/coroutine.h"
-#include "qemu/timer.h"
-#include "qapi-types.h"
-#include "qapi/qmp/qerror.h"
-#include "monitor/monitor.h"
-#include "qemu/hbitmap.h"
-#include "block/snapshot.h"
-
-#define BLOCK_FLAG_ENCRYPT          1
-#define BLOCK_FLAG_COMPAT6          4
-#define BLOCK_FLAG_LAZY_REFCOUNTS   8
-
-#define BLOCK_IO_LIMIT_READ     0
-#define BLOCK_IO_LIMIT_WRITE    1
-#define BLOCK_IO_LIMIT_TOTAL    2
-
-#define BLOCK_IO_SLICE_TIME     100000000
-#define NANOSECONDS_PER_SECOND  1000000000.0
-
-#define BLOCK_OPT_SIZE              "size"
-#define BLOCK_OPT_ENCRYPT           "encryption"
-#define BLOCK_OPT_COMPAT6           "compat6"
-#define BLOCK_OPT_BACKING_FILE      "backing_file"
-#define BLOCK_OPT_BACKING_FMT       "backing_fmt"
-#define BLOCK_OPT_CLUSTER_SIZE      "cluster_size"
-#define BLOCK_OPT_TABLE_SIZE        "table_size"
-#define BLOCK_OPT_PREALLOC          "preallocation"
-#define BLOCK_OPT_SUBFMT            "subformat"
-#define BLOCK_OPT_COMPAT_LEVEL      "compat"
-#define BLOCK_OPT_LAZY_REFCOUNTS    "lazy_refcounts"
-#define BLOCK_OPT_ADAPTER_TYPE      "adapter_type"
-
-typedef struct BdrvTrackedRequest {
-    BlockDriverState *bs;
-    int64_t sector_num;
-    int nb_sectors;
-    bool is_write;
-    QLIST_ENTRY(BdrvTrackedRequest) list;
-    Coroutine *co; /* owner, used for deadlock detection */
-    CoQueue wait_queue; /* coroutines blocked on this request */
-} BdrvTrackedRequest;
-
-
-typedef struct BlockIOLimit {
-    int64_t bps[3];
-    int64_t iops[3];
-} BlockIOLimit;
-
-typedef struct BlockIOBaseValue {
-    uint64_t bytes[2];
-    uint64_t ios[2];
-} BlockIOBaseValue;
-
-struct BlockDriver {
-    const char *format_name;
-    int instance_size;
-    int (*bdrv_probe)(const uint8_t *buf, int buf_size, const char *filename);
-    int (*bdrv_probe_device)(const char *filename);
-
-    /* Any driver implementing this callback is expected to be able to handle
-     * NULL file names in its .bdrv_open() implementation */
-    void (*bdrv_parse_filename)(const char *filename, QDict *options, Error **errp);
-
-    /* For handling image reopen for split or non-split files */
-    int (*bdrv_reopen_prepare)(BDRVReopenState *reopen_state,
-                               BlockReopenQueue *queue, Error **errp);
-    void (*bdrv_reopen_commit)(BDRVReopenState *reopen_state);
-    void (*bdrv_reopen_abort)(BDRVReopenState *reopen_state);
-
-    int (*bdrv_open)(BlockDriverState *bs, QDict *options, int flags);
-    int (*bdrv_file_open)(BlockDriverState *bs, QDict *options, int flags);
-    int (*bdrv_read)(BlockDriverState *bs, int64_t sector_num,
-                     uint8_t *buf, int nb_sectors);
-    int (*bdrv_write)(BlockDriverState *bs, int64_t sector_num,
-                      const uint8_t *buf, int nb_sectors);
-    void (*bdrv_close)(BlockDriverState *bs);
-    void (*bdrv_rebind)(BlockDriverState *bs);
-    int (*bdrv_create)(const char *filename, QEMUOptionParameter *options);
-    int (*bdrv_set_key)(BlockDriverState *bs, const char *key);
-    int (*bdrv_make_empty)(BlockDriverState *bs);
-    /* aio */
-    BlockDriverAIOCB *(*bdrv_aio_readv)(BlockDriverState *bs,
-        int64_t sector_num, QEMUIOVector *qiov, int nb_sectors,
-        BlockDriverCompletionFunc *cb, void *opaque);
-    BlockDriverAIOCB *(*bdrv_aio_writev)(BlockDriverState *bs,
-        int64_t sector_num, QEMUIOVector *qiov, int nb_sectors,
-        BlockDriverCompletionFunc *cb, void *opaque);
-    BlockDriverAIOCB *(*bdrv_aio_flush)(BlockDriverState *bs,
-        BlockDriverCompletionFunc *cb, void *opaque);
-    BlockDriverAIOCB *(*bdrv_aio_discard)(BlockDriverState *bs,
-        int64_t sector_num, int nb_sectors,
-        BlockDriverCompletionFunc *cb, void *opaque);
-
-    int coroutine_fn (*bdrv_co_readv)(BlockDriverState *bs,
-        int64_t sector_num, int nb_sectors, QEMUIOVector *qiov);
-    int coroutine_fn (*bdrv_co_writev)(BlockDriverState *bs,
-        int64_t sector_num, int nb_sectors, QEMUIOVector *qiov);
-    /*
-     * Efficiently zero a region of the disk image.  Typically an image format
-     * would use a compact metadata representation to implement this.  This
-     * function pointer may be NULL and .bdrv_co_writev() will be called
-     * instead.
-     */
-    int coroutine_fn (*bdrv_co_write_zeroes)(BlockDriverState *bs,
-        int64_t sector_num, int nb_sectors);
-    int coroutine_fn (*bdrv_co_discard)(BlockDriverState *bs,
-        int64_t sector_num, int nb_sectors);
-    int coroutine_fn (*bdrv_co_is_allocated)(BlockDriverState *bs,
-        int64_t sector_num, int nb_sectors, int *pnum);
-
-    /*
-     * Invalidate any cached meta-data.
-     */
-    void (*bdrv_invalidate_cache)(BlockDriverState *bs);
-
-    /*
-     * Flushes all data that was already written to the OS all the way down to
-     * the disk (for example raw-posix calls fsync()).
-     */
-    int coroutine_fn (*bdrv_co_flush_to_disk)(BlockDriverState *bs);
-
-    /*
-     * Flushes all internal caches to the OS. The data may still sit in a
-     * writeback cache of the host OS, but it will survive a crash of the qemu
-     * process.
-     */
-    int coroutine_fn (*bdrv_co_flush_to_os)(BlockDriverState *bs);
-
-    const char *protocol_name;
-    int (*bdrv_truncate)(BlockDriverState *bs, int64_t offset);
-    int64_t (*bdrv_getlength)(BlockDriverState *bs);
-    int64_t (*bdrv_get_allocated_file_size)(BlockDriverState *bs);
-    int (*bdrv_write_compressed)(BlockDriverState *bs, int64_t sector_num,
-                                 const uint8_t *buf, int nb_sectors);
-
-    int (*bdrv_snapshot_create)(BlockDriverState *bs,
-                                QEMUSnapshotInfo *sn_info);
-    int (*bdrv_snapshot_goto)(BlockDriverState *bs,
-                              const char *snapshot_id);
-    int (*bdrv_snapshot_delete)(BlockDriverState *bs, const char *snapshot_id);
-    int (*bdrv_snapshot_list)(BlockDriverState *bs,
-                              QEMUSnapshotInfo **psn_info);
-    int (*bdrv_snapshot_load_tmp)(BlockDriverState *bs,
-                                  const char *snapshot_name);
-    int (*bdrv_get_info)(BlockDriverState *bs, BlockDriverInfo *bdi);
-
-    int (*bdrv_save_vmstate)(BlockDriverState *bs, QEMUIOVector *qiov,
-                             int64_t pos);
-    int (*bdrv_load_vmstate)(BlockDriverState *bs, uint8_t *buf,
-                             int64_t pos, int size);
-
-    int (*bdrv_change_backing_file)(BlockDriverState *bs,
-        const char *backing_file, const char *backing_fmt);
-
-    /* removable device specific */
-    int (*bdrv_is_inserted)(BlockDriverState *bs);
-    int (*bdrv_media_changed)(BlockDriverState *bs);
-    void (*bdrv_eject)(BlockDriverState *bs, bool eject_flag);
-    void (*bdrv_lock_medium)(BlockDriverState *bs, bool locked);
-
-    /* to control generic scsi devices */
-    int (*bdrv_ioctl)(BlockDriverState *bs, unsigned long int req, void *buf);
-    BlockDriverAIOCB *(*bdrv_aio_ioctl)(BlockDriverState *bs,
-        unsigned long int req, void *buf,
-        BlockDriverCompletionFunc *cb, void *opaque);
-
-    /* List of options for creating images, terminated by name == NULL */
-    QEMUOptionParameter *create_options;
-
-
-    /*
-     * Returns 0 for completed check, -errno for internal errors.
-     * The check results are stored in result.
-     */
-    int (*bdrv_check)(BlockDriverState* bs, BdrvCheckResult *result,
-        BdrvCheckMode fix);
-
-    void (*bdrv_debug_event)(BlockDriverState *bs, BlkDebugEvent event);
-
-    /* TODO Better pass a option string/QDict/QemuOpts to add any rule? */
-    int (*bdrv_debug_breakpoint)(BlockDriverState *bs, const char *event,
-        const char *tag);
-    int (*bdrv_debug_resume)(BlockDriverState *bs, const char *tag);
-    bool (*bdrv_debug_is_suspended)(BlockDriverState *bs, const char *tag);
-
-    /*
-     * Returns 1 if newly created images are guaranteed to contain only
-     * zeros, 0 otherwise.
-     */
-    int (*bdrv_has_zero_init)(BlockDriverState *bs);
-
-    QLIST_ENTRY(BlockDriver) list;
-};
-
-/*
- * Note: the function bdrv_append() copies and swaps contents of
- * BlockDriverStates, so if you add new fields to this struct, please
- * inspect bdrv_append() to determine if the new fields need to be
- * copied as well.
- */
-struct BlockDriverState {
-    int64_t total_sectors; /* if we are reading a disk image, give its
-                              size in sectors */
-    int read_only; /* if true, the media is read only */
-    int open_flags; /* flags used to open the file, re-used for re-open */
-    int encrypted; /* if true, the media is encrypted */
-    int valid_key; /* if true, a valid encryption key has been set */
-    int sg;        /* if true, the device is a /dev/sg* */
-    int copy_on_read; /* if true, copy read backing sectors into image
-                         note this is a reference count */
-
-    BlockDriver *drv; /* NULL means no media */
-    void *opaque;
-
-    void *dev;                  /* attached device model, if any */
-    /* TODO change to DeviceState when all users are qdevified */
-    const BlockDevOps *dev_ops;
-    void *dev_opaque;
-
-    char filename[1024];
-    char backing_file[1024]; /* if non zero, the image is a diff of
-                                this file image */
-    char backing_format[16]; /* if non-zero and backing_file exists */
-    int is_temporary;
-
-    BlockDriverState *backing_hd;
-    BlockDriverState *file;
-
-    NotifierList close_notifiers;
-
-    /* Callback before write request is processed */
-    NotifierWithReturnList before_write_notifiers;
-
-    /* number of in-flight copy-on-read requests */
-    unsigned int copy_on_read_in_flight;
-
-    /* the time for latest disk I/O */
-    int64_t slice_start;
-    int64_t slice_end;
-    BlockIOLimit io_limits;
-    BlockIOBaseValue slice_submitted;
-    CoQueue      throttled_reqs;
-    QEMUTimer    *block_timer;
-    bool         io_limits_enabled;
-
-    /* I/O stats (display with "info blockstats"). */
-    uint64_t nr_bytes[BDRV_MAX_IOTYPE];
-    uint64_t nr_ops[BDRV_MAX_IOTYPE];
-    uint64_t total_time_ns[BDRV_MAX_IOTYPE];
-    uint64_t wr_highest_sector;
-
-    /* Whether the disk can expand beyond total_sectors */
-    int growable;
-
-    /* the memory alignment required for the buffers handled by this driver */
-    int buffer_alignment;
-
-    /* do we need to tell the quest if we have a volatile write cache? */
-    int enable_write_cache;
-
-    /* NOTE: the following infos are only hints for real hardware
-       drivers. They are not used by the block driver */
-    BlockdevOnError on_read_error, on_write_error;
-    bool iostatus_enabled;
-    BlockDeviceIoStatus iostatus;
-    char device_name[32];
-    HBitmap *dirty_bitmap;
-    int in_use; /* users other than guest access, eg. block migration */
-    QTAILQ_ENTRY(BlockDriverState) list;
-
-    QLIST_HEAD(, BdrvTrackedRequest) tracked_requests;
-
-    /* long-running background operation */
-    BlockJob *job;
-
-    QDict *options;
-};
-
-int get_tmp_filename(char *filename, int size);
-
-void bdrv_set_io_limits(BlockDriverState *bs,
-                        BlockIOLimit *io_limits);
-
-/**
- * bdrv_add_before_write_notifier:
- *
- * Register a callback that is invoked before write requests are processed but
- * after any throttling or waiting for overlapping requests.
- */
-void bdrv_add_before_write_notifier(BlockDriverState *bs,
-                                    NotifierWithReturn *notifier);
-
-/**
- * bdrv_get_aio_context:
- *
- * Returns: the currently bound #AioContext
- */
-AioContext *bdrv_get_aio_context(BlockDriverState *bs);
-
-#ifdef _WIN32
-int is_windows_drive(const char *filename);
-#endif
-void bdrv_emit_qmp_error_event(const BlockDriverState *bdrv,
-                               enum MonitorEvent ev,
-                               BlockErrorAction action, bool is_read);
-
-/**
- * stream_start:
- * @bs: Block device to operate on.
- * @base: Block device that will become the new base, or %NULL to
- * flatten the whole backing file chain onto @bs.
- * @base_id: The file name that will be written to @bs as the new
- * backing file if the job completes.  Ignored if @base is %NULL.
- * @speed: The maximum speed, in bytes per second, or 0 for unlimited.
- * @on_error: The action to take upon error.
- * @cb: Completion function for the job.
- * @opaque: Opaque pointer value passed to @cb.
- * @errp: Error object.
- *
- * Start a streaming operation on @bs.  Clusters that are unallocated
- * in @bs, but allocated in any image between @base and @bs (both
- * exclusive) will be written to @bs.  At the end of a successful
- * streaming job, the backing file of @bs will be changed to
- * @base_id in the written image and to @base in the live BlockDriverState.
- */
-void stream_start(BlockDriverState *bs, BlockDriverState *base,
-                  const char *base_id, int64_t speed, BlockdevOnError on_error,
-                  BlockDriverCompletionFunc *cb,
-                  void *opaque, Error **errp);
-
-/**
- * commit_start:
- * @bs: Top Block device
- * @base: Block device that will be written into, and become the new top
- * @speed: The maximum speed, in bytes per second, or 0 for unlimited.
- * @on_error: The action to take upon error.
- * @cb: Completion function for the job.
- * @opaque: Opaque pointer value passed to @cb.
- * @errp: Error object.
- *
- */
-void commit_start(BlockDriverState *bs, BlockDriverState *base,
-                 BlockDriverState *top, int64_t speed,
-                 BlockdevOnError on_error, BlockDriverCompletionFunc *cb,
-                 void *opaque, Error **errp);
-
-/*
- * mirror_start:
- * @bs: Block device to operate on.
- * @target: Block device to write to.
- * @speed: The maximum speed, in bytes per second, or 0 for unlimited.
- * @granularity: The chosen granularity for the dirty bitmap.
- * @buf_size: The amount of data that can be in flight at one time.
- * @mode: Whether to collapse all images in the chain to the target.
- * @on_source_error: The action to take upon error reading from the source.
- * @on_target_error: The action to take upon error writing to the target.
- * @cb: Completion function for the job.
- * @opaque: Opaque pointer value passed to @cb.
- * @errp: Error object.
- *
- * Start a mirroring operation on @bs.  Clusters that are allocated
- * in @bs will be written to @bs until the job is cancelled or
- * manually completed.  At the end of a successful mirroring job,
- * @bs will be switched to read from @target.
- */
-void mirror_start(BlockDriverState *bs, BlockDriverState *target,
-                  int64_t speed, int64_t granularity, int64_t buf_size,
-                  MirrorSyncMode mode, BlockdevOnError on_source_error,
-                  BlockdevOnError on_target_error,
-                  BlockDriverCompletionFunc *cb,
-                  void *opaque, Error **errp);
-
-/*
- * backup_start:
- * @bs: Block device to operate on.
- * @target: Block device to write to.
- * @speed: The maximum speed, in bytes per second, or 0 for unlimited.
- * @on_source_error: The action to take upon error reading from the source.
- * @on_target_error: The action to take upon error writing to the target.
- * @cb: Completion function for the job.
- * @opaque: Opaque pointer value passed to @cb.
- *
- * Start a backup operation on @bs.  Clusters in @bs are written to @target
- * until the job is cancelled or manually completed.
- */
-void backup_start(BlockDriverState *bs, BlockDriverState *target,
-                  int64_t speed, BlockdevOnError on_source_error,
-                  BlockdevOnError on_target_error,
-                  BlockDriverCompletionFunc *cb, void *opaque,
-                  Error **errp);
-
-#endif /* BLOCK_INT_H */
diff --git a/contrib/qemu/include/block/blockjob.h b/contrib/qemu/include/block/blockjob.h
deleted file mode 100644
index c290d07bba0..00000000000
--- a/contrib/qemu/include/block/blockjob.h
+++ /dev/null
@@ -1,278 +0,0 @@
-/*
- * Declarations for long-running block device operations
- *
- * Copyright (c) 2011 IBM Corp.
- * Copyright (c) 2012 Red Hat, Inc.
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-#ifndef BLOCKJOB_H
-#define BLOCKJOB_H 1
-
-#include "block/block.h"
-
-/**
- * BlockJobType:
- *
- * A class type for block job objects.
- */
-typedef struct BlockJobType {
-    /** Derived BlockJob struct size */
-    size_t instance_size;
-
-    /** String describing the operation, part of query-block-jobs QMP API */
-    const char *job_type;
-
-    /** Optional callback for job types that support setting a speed limit */
-    void (*set_speed)(BlockJob *job, int64_t speed, Error **errp);
-
-    /** Optional callback for job types that need to forward I/O status reset */
-    void (*iostatus_reset)(BlockJob *job);
-
-    /**
-     * Optional callback for job types whose completion must be triggered
-     * manually.
-     */
-    void (*complete)(BlockJob *job, Error **errp);
-} BlockJobType;
-
-/**
- * BlockJob:
- *
- * Long-running operation on a BlockDriverState.
- */
-struct BlockJob {
-    /** The job type, including the job vtable.  */
-    const BlockJobType *job_type;
-
-    /** The block device on which the job is operating.  */
-    BlockDriverState *bs;
-
-    /**
-     * The coroutine that executes the job.  If not NULL, it is
-     * reentered when busy is false and the job is cancelled.
-     */
-    Coroutine *co;
-
-    /**
-     * Set to true if the job should cancel itself.  The flag must
-     * always be tested just before toggling the busy flag from false
-     * to true.  After a job has been cancelled, it should only yield
-     * if #qemu_aio_wait will ("sooner or later") reenter the coroutine.
-     */
-    bool cancelled;
-
-    /**
-     * Set to true if the job is either paused, or will pause itself
-     * as soon as possible (if busy == true).
-     */
-    bool paused;
-
-    /**
-     * Set to false by the job while it is in a quiescent state, where
-     * no I/O is pending and the job has yielded on any condition
-     * that is not detected by #qemu_aio_wait, such as a timer.
-     */
-    bool busy;
-
-    /** Status that is published by the query-block-jobs QMP API */
-    BlockDeviceIoStatus iostatus;
-
-    /** Offset that is published by the query-block-jobs QMP API */
-    int64_t offset;
-
-    /** Length that is published by the query-block-jobs QMP API */
-    int64_t len;
-
-    /** Speed that was set with @block_job_set_speed.  */
-    int64_t speed;
-
-    /** The completion function that will be called when the job completes.  */
-    BlockDriverCompletionFunc *cb;
-
-    /** The opaque value that is passed to the completion function.  */
-    void *opaque;
-};
-
-/**
- * block_job_create:
- * @job_type: The class object for the newly-created job.
- * @bs: The block
- * @speed: The maximum speed, in bytes per second, or 0 for unlimited.
- * @cb: Completion function for the job.
- * @opaque: Opaque pointer value passed to @cb.
- * @errp: Error object.
- *
- * Create a new long-running block device job and return it.  The job
- * will call @cb asynchronously when the job completes.  Note that
- * @bs may have been closed at the time the @cb it is called.  If
- * this is the case, the job may be reported as either cancelled or
- * completed.
- *
- * This function is not part of the public job interface; it should be
- * called from a wrapper that is specific to the job type.
- */
-void *block_job_create(const BlockJobType *job_type, BlockDriverState *bs,
-                       int64_t speed, BlockDriverCompletionFunc *cb,
-                       void *opaque, Error **errp);
-
-/**
- * block_job_sleep_ns:
- * @job: The job that calls the function.
- * @clock: The clock to sleep on.
- * @ns: How many nanoseconds to stop for.
- *
- * Put the job to sleep (assuming that it wasn't canceled) for @ns
- * nanoseconds.  Canceling the job will interrupt the wait immediately.
- */
-void block_job_sleep_ns(BlockJob *job, QEMUClock *clock, int64_t ns);
-
-/**
- * block_job_completed:
- * @job: The job being completed.
- * @ret: The status code.
- *
- * Call the completion function that was registered at creation time, and
- * free @job.
- */
-void block_job_completed(BlockJob *job, int ret);
-
-/**
- * block_job_set_speed:
- * @job: The job to set the speed for.
- * @speed: The new value
- * @errp: Error object.
- *
- * Set a rate-limiting parameter for the job; the actual meaning may
- * vary depending on the job type.
- */
-void block_job_set_speed(BlockJob *job, int64_t speed, Error **errp);
-
-/**
- * block_job_cancel:
- * @job: The job to be canceled.
- *
- * Asynchronously cancel the specified job.
- */
-void block_job_cancel(BlockJob *job);
-
-/**
- * block_job_complete:
- * @job: The job to be completed.
- * @errp: Error object.
- *
- * Asynchronously complete the specified job.
- */
-void block_job_complete(BlockJob *job, Error **errp);
-
-/**
- * block_job_is_cancelled:
- * @job: The job being queried.
- *
- * Returns whether the job is scheduled for cancellation.
- */
-bool block_job_is_cancelled(BlockJob *job);
-
-/**
- * block_job_query:
- * @job: The job to get information about.
- *
- * Return information about a job.
- */
-BlockJobInfo *block_job_query(BlockJob *job);
-
-/**
- * block_job_pause:
- * @job: The job to be paused.
- *
- * Asynchronously pause the specified job.
- */
-void block_job_pause(BlockJob *job);
-
-/**
- * block_job_resume:
- * @job: The job to be resumed.
- *
- * Resume the specified job.
- */
-void block_job_resume(BlockJob *job);
-
-/**
- * qobject_from_block_job:
- * @job: The job whose information is requested.
- *
- * Return a QDict corresponding to @job's query-block-jobs entry.
- */
-QObject *qobject_from_block_job(BlockJob *job);
-
-/**
- * block_job_ready:
- * @job: The job which is now ready to complete.
- *
- * Send a BLOCK_JOB_READY event for the specified job.
- */
-void block_job_ready(BlockJob *job);
-
-/**
- * block_job_is_paused:
- * @job: The job being queried.
- *
- * Returns whether the job is currently paused, or will pause
- * as soon as it reaches a sleeping point.
- */
-bool block_job_is_paused(BlockJob *job);
-
-/**
- * block_job_cancel_sync:
- * @job: The job to be canceled.
- *
- * Synchronously cancel the job.  The completion callback is called
- * before the function returns.  The job may actually complete
- * instead of canceling itself; the circumstances under which this
- * happens depend on the kind of job that is active.
- *
- * Returns the return value from the job if the job actually completed
- * during the call, or -ECANCELED if it was canceled.
- */
-int block_job_cancel_sync(BlockJob *job);
-
-/**
- * block_job_iostatus_reset:
- * @job: The job whose I/O status should be reset.
- *
- * Reset I/O status on @job and on BlockDriverState objects it uses,
- * other than job->bs.
- */
-void block_job_iostatus_reset(BlockJob *job);
-
-/**
- * block_job_error_action:
- * @job: The job to signal an error for.
- * @bs: The block device on which to set an I/O error.
- * @on_err: The error action setting.
- * @is_read: Whether the operation was a read.
- * @error: The error that was reported.
- *
- * Report an I/O error for a block job and possibly stop the VM.  Return the
- * action that was selected based on @on_err and @error.
- */
-BlockErrorAction block_job_error_action(BlockJob *job, BlockDriverState *bs,
-                                        BlockdevOnError on_err,
-                                        int is_read, int error);
-#endif
diff --git a/contrib/qemu/include/block/coroutine.h b/contrib/qemu/include/block/coroutine.h
deleted file mode 100644
index 377805a3b08..00000000000
--- a/contrib/qemu/include/block/coroutine.h
+++ /dev/null
@@ -1,218 +0,0 @@
-/*
- * QEMU coroutine implementation
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Stefan Hajnoczi    <stefanha@linux.vnet.ibm.com>
- *  Kevin Wolf         <kwolf@redhat.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#ifndef QEMU_COROUTINE_H
-#define QEMU_COROUTINE_H
-
-#include <stdbool.h>
-#include "qemu/queue.h"
-#include "qemu/timer.h"
-
-/**
- * Coroutines are a mechanism for stack switching and can be used for
- * cooperative userspace threading.  These functions provide a simple but
- * useful flavor of coroutines that is suitable for writing sequential code,
- * rather than callbacks, for operations that need to give up control while
- * waiting for events to complete.
- *
- * These functions are re-entrant and may be used outside the global mutex.
- */
-
-/**
- * Mark a function that executes in coroutine context
- *
- * Functions that execute in coroutine context cannot be called directly from
- * normal functions.  In the future it would be nice to enable compiler or
- * static checker support for catching such errors.  This annotation might make
- * it possible and in the meantime it serves as documentation.
- *
- * For example:
- *
- *   static void coroutine_fn foo(void) {
- *       ....
- *   }
- */
-#define coroutine_fn
-
-typedef struct Coroutine Coroutine;
-
-/**
- * Coroutine entry point
- *
- * When the coroutine is entered for the first time, opaque is passed in as an
- * argument.
- *
- * When this function returns, the coroutine is destroyed automatically and
- * execution continues in the caller who last entered the coroutine.
- */
-typedef void coroutine_fn CoroutineEntry(void *opaque);
-
-/**
- * Create a new coroutine
- *
- * Use qemu_coroutine_enter() to actually transfer control to the coroutine.
- */
-Coroutine *qemu_coroutine_create(CoroutineEntry *entry);
-
-/**
- * Transfer control to a coroutine
- *
- * The opaque argument is passed as the argument to the entry point when
- * entering the coroutine for the first time.  It is subsequently ignored.
- */
-void qemu_coroutine_enter(Coroutine *coroutine, void *opaque);
-
-/**
- * Transfer control back to a coroutine's caller
- *
- * This function does not return until the coroutine is re-entered using
- * qemu_coroutine_enter().
- */
-void coroutine_fn qemu_coroutine_yield(void);
-
-/**
- * Get the currently executing coroutine
- */
-Coroutine *coroutine_fn qemu_coroutine_self(void);
-
-/**
- * Return whether or not currently inside a coroutine
- *
- * This can be used to write functions that work both when in coroutine context
- * and when not in coroutine context.  Note that such functions cannot use the
- * coroutine_fn annotation since they work outside coroutine context.
- */
-bool qemu_in_coroutine(void);
-
-
-
-/**
- * CoQueues are a mechanism to queue coroutines in order to continue executing
- * them later. They provide the fundamental primitives on which coroutine locks
- * are built.
- */
-typedef struct CoQueue {
-    QTAILQ_HEAD(, Coroutine) entries;
-    AioContext *ctx;
-} CoQueue;
-
-/**
- * Initialise a CoQueue. This must be called before any other operation is used
- * on the CoQueue.
- */
-void qemu_co_queue_init(CoQueue *queue);
-
-/**
- * Adds the current coroutine to the CoQueue and transfers control to the
- * caller of the coroutine.
- */
-void coroutine_fn qemu_co_queue_wait(CoQueue *queue);
-
-/**
- * Adds the current coroutine to the head of the CoQueue and transfers control to the
- * caller of the coroutine.
- */
-void coroutine_fn qemu_co_queue_wait_insert_head(CoQueue *queue);
-
-/**
- * Restarts the next coroutine in the CoQueue and removes it from the queue.
- *
- * Returns true if a coroutine was restarted, false if the queue is empty.
- */
-bool qemu_co_queue_next(CoQueue *queue);
-
-/**
- * Restarts all coroutines in the CoQueue and leaves the queue empty.
- */
-void qemu_co_queue_restart_all(CoQueue *queue);
-
-/**
- * Checks if the CoQueue is empty.
- */
-bool qemu_co_queue_empty(CoQueue *queue);
-
-
-/**
- * Provides a mutex that can be used to synchronise coroutines
- */
-typedef struct CoMutex {
-    bool locked;
-    CoQueue queue;
-} CoMutex;
-
-/**
- * Initialises a CoMutex. This must be called before any other operation is used
- * on the CoMutex.
- */
-void qemu_co_mutex_init(CoMutex *mutex);
-
-/**
- * Locks the mutex. If the lock cannot be taken immediately, control is
- * transferred to the caller of the current coroutine.
- */
-void coroutine_fn qemu_co_mutex_lock(CoMutex *mutex);
-
-/**
- * Unlocks the mutex and schedules the next coroutine that was waiting for this
- * lock to be run.
- */
-void coroutine_fn qemu_co_mutex_unlock(CoMutex *mutex);
-
-typedef struct CoRwlock {
-    bool writer;
-    int reader;
-    CoQueue queue;
-} CoRwlock;
-
-/**
- * Initialises a CoRwlock. This must be called before any other operation
- * is used on the CoRwlock
- */
-void qemu_co_rwlock_init(CoRwlock *lock);
-
-/**
- * Read locks the CoRwlock. If the lock cannot be taken immediately because
- * of a parallel writer, control is transferred to the caller of the current
- * coroutine.
- */
-void qemu_co_rwlock_rdlock(CoRwlock *lock);
-
-/**
- * Write Locks the mutex. If the lock cannot be taken immediately because
- * of a parallel reader, control is transferred to the caller of the current
- * coroutine.
- */
-void qemu_co_rwlock_wrlock(CoRwlock *lock);
-
-/**
- * Unlocks the read/write lock and schedules the next coroutine that was
- * waiting for this lock to be run.
- */
-void qemu_co_rwlock_unlock(CoRwlock *lock);
-
-/**
- * Yield the coroutine for a given duration
- *
- * Note this function uses timers and hence only works when a main loop is in
- * use.  See main-loop.h and do not use from qemu-tool programs.
- */
-void coroutine_fn co_sleep_ns(QEMUClock *clock, int64_t ns);
-
-/**
- * Yield until a file descriptor becomes readable
- *
- * Note that this function clobbers the handlers for the file descriptor.
- */
-void coroutine_fn yield_until_fd_readable(int fd);
-#endif /* QEMU_COROUTINE_H */
diff --git a/contrib/qemu/include/block/coroutine_int.h b/contrib/qemu/include/block/coroutine_int.h
deleted file mode 100644
index f133d65af86..00000000000
--- a/contrib/qemu/include/block/coroutine_int.h
+++ /dev/null
@@ -1,53 +0,0 @@
-/*
- * Coroutine internals
- *
- * Copyright (c) 2011 Kevin Wolf <kwolf@redhat.com>
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-
-#ifndef QEMU_COROUTINE_INT_H
-#define QEMU_COROUTINE_INT_H
-
-#include "qemu/queue.h"
-#include "block/coroutine.h"
-
-typedef enum {
-    COROUTINE_YIELD = 1,
-    COROUTINE_TERMINATE = 2,
-} CoroutineAction;
-
-struct Coroutine {
-    CoroutineEntry *entry;
-    void *entry_arg;
-    Coroutine *caller;
-    QSLIST_ENTRY(Coroutine) pool_next;
-
-    /* Coroutines that should be woken up when we yield or terminate */
-    QTAILQ_HEAD(, Coroutine) co_queue_wakeup;
-    QTAILQ_ENTRY(Coroutine) co_queue_next;
-};
-
-Coroutine *qemu_coroutine_new(void);
-void qemu_coroutine_delete(Coroutine *co);
-CoroutineAction qemu_coroutine_switch(Coroutine *from, Coroutine *to,
-                                      CoroutineAction action);
-void coroutine_fn qemu_co_queue_run_restart(Coroutine *co);
-
-#endif
diff --git a/contrib/qemu/include/block/snapshot.h b/contrib/qemu/include/block/snapshot.h
deleted file mode 100644
index eaf61f0326e..00000000000
--- a/contrib/qemu/include/block/snapshot.h
+++ /dev/null
@@ -1,53 +0,0 @@
-/*
- * Block layer snapshot related functions
- *
- * Copyright (c) 2003-2008 Fabrice Bellard
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-
-#ifndef SNAPSHOT_H
-#define SNAPSHOT_H
-
-#include "qemu-common.h"
-
-typedef struct QEMUSnapshotInfo {
-    char id_str[128]; /* unique snapshot id */
-    /* the following fields are informative. They are not needed for
-       the consistency of the snapshot */
-    char name[256]; /* user chosen name */
-    uint64_t vm_state_size; /* VM state info size */
-    uint32_t date_sec; /* UTC date of the snapshot */
-    uint32_t date_nsec;
-    uint64_t vm_clock_nsec; /* VM clock relative to boot */
-} QEMUSnapshotInfo;
-
-int bdrv_snapshot_find(BlockDriverState *bs, QEMUSnapshotInfo *sn_info,
-                       const char *name);
-int bdrv_can_snapshot(BlockDriverState *bs);
-int bdrv_snapshot_create(BlockDriverState *bs,
-                         QEMUSnapshotInfo *sn_info);
-int bdrv_snapshot_goto(BlockDriverState *bs,
-                       const char *snapshot_id);
-int bdrv_snapshot_delete(BlockDriverState *bs, const char *snapshot_id);
-int bdrv_snapshot_list(BlockDriverState *bs,
-                       QEMUSnapshotInfo **psn_info);
-int bdrv_snapshot_load_tmp(BlockDriverState *bs,
-                           const char *snapshot_name);
-#endif
diff --git a/contrib/qemu/include/config.h b/contrib/qemu/include/config.h
deleted file mode 100644
index e20f78696a1..00000000000
--- a/contrib/qemu/include/config.h
+++ /dev/null
@@ -1,2 +0,0 @@
-#include "config-host.h"
-#include "config-target.h"
diff --git a/contrib/qemu/include/exec/cpu-common.h b/contrib/qemu/include/exec/cpu-common.h
deleted file mode 100644
index e4996e19c32..00000000000
--- a/contrib/qemu/include/exec/cpu-common.h
+++ /dev/null
@@ -1,124 +0,0 @@
-#ifndef CPU_COMMON_H
-#define CPU_COMMON_H 1
-
-/* CPU interfaces that are target independent.  */
-
-#ifndef CONFIG_USER_ONLY
-#include "exec/hwaddr.h"
-#endif
-
-#ifndef NEED_CPU_H
-#include "exec/poison.h"
-#endif
-
-#include "qemu/bswap.h"
-#include "qemu/queue.h"
-
-/**
- * CPUListState:
- * @cpu_fprintf: Print function.
- * @file: File to print to using @cpu_fprint.
- *
- * State commonly used for iterating over CPU models.
- */
-typedef struct CPUListState {
-    fprintf_function cpu_fprintf;
-    FILE *file;
-} CPUListState;
-
-#if !defined(CONFIG_USER_ONLY)
-
-enum device_endian {
-    DEVICE_NATIVE_ENDIAN,
-    DEVICE_BIG_ENDIAN,
-    DEVICE_LITTLE_ENDIAN,
-};
-
-/* address in the RAM (different from a physical address) */
-#if defined(CONFIG_XEN_BACKEND)
-typedef uint64_t ram_addr_t;
-#  define RAM_ADDR_MAX UINT64_MAX
-#  define RAM_ADDR_FMT "%" PRIx64
-#else
-typedef uintptr_t ram_addr_t;
-#  define RAM_ADDR_MAX UINTPTR_MAX
-#  define RAM_ADDR_FMT "%" PRIxPTR
-#endif
-
-/* memory API */
-
-typedef void CPUWriteMemoryFunc(void *opaque, hwaddr addr, uint32_t value);
-typedef uint32_t CPUReadMemoryFunc(void *opaque, hwaddr addr);
-
-void qemu_ram_remap(ram_addr_t addr, ram_addr_t length);
-/* This should not be used by devices.  */
-MemoryRegion *qemu_ram_addr_from_host(void *ptr, ram_addr_t *ram_addr);
-void qemu_ram_set_idstr(ram_addr_t addr, const char *name, DeviceState *dev);
-
-void cpu_physical_memory_rw(hwaddr addr, uint8_t *buf,
-                            int len, int is_write);
-static inline void cpu_physical_memory_read(hwaddr addr,
-                                            void *buf, int len)
-{
-    cpu_physical_memory_rw(addr, buf, len, 0);
-}
-static inline void cpu_physical_memory_write(hwaddr addr,
-                                             const void *buf, int len)
-{
-    cpu_physical_memory_rw(addr, (void *)buf, len, 1);
-}
-void *cpu_physical_memory_map(hwaddr addr,
-                              hwaddr *plen,
-                              int is_write);
-void cpu_physical_memory_unmap(void *buffer, hwaddr len,
-                               int is_write, hwaddr access_len);
-void *cpu_register_map_client(void *opaque, void (*callback)(void *opaque));
-
-bool cpu_physical_memory_is_io(hwaddr phys_addr);
-
-/* Coalesced MMIO regions are areas where write operations can be reordered.
- * This usually implies that write operations are side-effect free.  This allows
- * batching which can make a major impact on performance when using
- * virtualization.
- */
-void qemu_flush_coalesced_mmio_buffer(void);
-
-uint32_t ldub_phys(hwaddr addr);
-uint32_t lduw_le_phys(hwaddr addr);
-uint32_t lduw_be_phys(hwaddr addr);
-uint32_t ldl_le_phys(hwaddr addr);
-uint32_t ldl_be_phys(hwaddr addr);
-uint64_t ldq_le_phys(hwaddr addr);
-uint64_t ldq_be_phys(hwaddr addr);
-void stb_phys(hwaddr addr, uint32_t val);
-void stw_le_phys(hwaddr addr, uint32_t val);
-void stw_be_phys(hwaddr addr, uint32_t val);
-void stl_le_phys(hwaddr addr, uint32_t val);
-void stl_be_phys(hwaddr addr, uint32_t val);
-void stq_le_phys(hwaddr addr, uint64_t val);
-void stq_be_phys(hwaddr addr, uint64_t val);
-
-#ifdef NEED_CPU_H
-uint32_t lduw_phys(hwaddr addr);
-uint32_t ldl_phys(hwaddr addr);
-uint64_t ldq_phys(hwaddr addr);
-void stl_phys_notdirty(hwaddr addr, uint32_t val);
-void stw_phys(hwaddr addr, uint32_t val);
-void stl_phys(hwaddr addr, uint32_t val);
-void stq_phys(hwaddr addr, uint64_t val);
-#endif
-
-void cpu_physical_memory_write_rom(hwaddr addr,
-                                   const uint8_t *buf, int len);
-
-extern struct MemoryRegion io_mem_rom;
-extern struct MemoryRegion io_mem_notdirty;
-
-typedef void (RAMBlockIterFunc)(void *host_addr,
-    ram_addr_t offset, ram_addr_t length, void *opaque);
-
-void qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque);
-
-#endif
-
-#endif /* !CPU_COMMON_H */
diff --git a/contrib/qemu/include/exec/hwaddr.h b/contrib/qemu/include/exec/hwaddr.h
deleted file mode 100644
index c9eb78fba18..00000000000
--- a/contrib/qemu/include/exec/hwaddr.h
+++ /dev/null
@@ -1,20 +0,0 @@
-/* Define hwaddr if it exists.  */
-
-#ifndef HWADDR_H
-#define HWADDR_H
-
-#define HWADDR_BITS 64
-/* hwaddr is the type of a physical address (its size can
-   be different from 'target_ulong').  */
-
-typedef uint64_t hwaddr;
-#define HWADDR_MAX UINT64_MAX
-#define TARGET_FMT_plx "%016" PRIx64
-#define HWADDR_PRId PRId64
-#define HWADDR_PRIi PRIi64
-#define HWADDR_PRIo PRIo64
-#define HWADDR_PRIu PRIu64
-#define HWADDR_PRIx PRIx64
-#define HWADDR_PRIX PRIX64
-
-#endif
diff --git a/contrib/qemu/include/exec/poison.h b/contrib/qemu/include/exec/poison.h
deleted file mode 100644
index 2341a750413..00000000000
--- a/contrib/qemu/include/exec/poison.h
+++ /dev/null
@@ -1,63 +0,0 @@
-/* Poison identifiers that should not be used when building
-   target independent device code.  */
-
-#ifndef HW_POISON_H
-#define HW_POISON_H
-#ifdef __GNUC__
-
-#pragma GCC poison TARGET_I386
-#pragma GCC poison TARGET_X86_64
-#pragma GCC poison TARGET_ALPHA
-#pragma GCC poison TARGET_ARM
-#pragma GCC poison TARGET_CRIS
-#pragma GCC poison TARGET_LM32
-#pragma GCC poison TARGET_M68K
-#pragma GCC poison TARGET_MIPS
-#pragma GCC poison TARGET_MIPS64
-#pragma GCC poison TARGET_OPENRISC
-#pragma GCC poison TARGET_PPC
-#pragma GCC poison TARGET_PPCEMB
-#pragma GCC poison TARGET_PPC64
-#pragma GCC poison TARGET_ABI32
-#pragma GCC poison TARGET_SH4
-#pragma GCC poison TARGET_SPARC
-#pragma GCC poison TARGET_SPARC64
-
-#pragma GCC poison TARGET_WORDS_BIGENDIAN
-#pragma GCC poison BSWAP_NEEDED
-
-#pragma GCC poison TARGET_LONG_BITS
-#pragma GCC poison TARGET_FMT_lx
-#pragma GCC poison TARGET_FMT_ld
-
-#pragma GCC poison TARGET_PAGE_SIZE
-#pragma GCC poison TARGET_PAGE_MASK
-#pragma GCC poison TARGET_PAGE_BITS
-#pragma GCC poison TARGET_PAGE_ALIGN
-
-#pragma GCC poison CPUArchState
-#pragma GCC poison env
-
-#pragma GCC poison lduw_phys
-#pragma GCC poison ldl_phys
-#pragma GCC poison ldq_phys
-#pragma GCC poison stl_phys_notdirty
-#pragma GCC poison stw_phys
-#pragma GCC poison stl_phys
-#pragma GCC poison stq_phys
-
-#pragma GCC poison CPU_INTERRUPT_HARD
-#pragma GCC poison CPU_INTERRUPT_EXITTB
-#pragma GCC poison CPU_INTERRUPT_HALT
-#pragma GCC poison CPU_INTERRUPT_DEBUG
-#pragma GCC poison CPU_INTERRUPT_TGT_EXT_0
-#pragma GCC poison CPU_INTERRUPT_TGT_EXT_1
-#pragma GCC poison CPU_INTERRUPT_TGT_EXT_2
-#pragma GCC poison CPU_INTERRUPT_TGT_EXT_3
-#pragma GCC poison CPU_INTERRUPT_TGT_EXT_4
-#pragma GCC poison CPU_INTERRUPT_TGT_INT_0
-#pragma GCC poison CPU_INTERRUPT_TGT_INT_1
-#pragma GCC poison CPU_INTERRUPT_TGT_INT_2
-
-#endif
-#endif
diff --git a/contrib/qemu/include/fpu/softfloat.h b/contrib/qemu/include/fpu/softfloat.h
deleted file mode 100644
index f3927e2419f..00000000000
--- a/contrib/qemu/include/fpu/softfloat.h
+++ /dev/null
@@ -1,641 +0,0 @@
-/*
- * QEMU float support
- *
- * Derived from SoftFloat.
- */
-
-/*============================================================================
-
-This C header file is part of the SoftFloat IEC/IEEE Floating-point Arithmetic
-Package, Release 2b.
-
-Written by John R. Hauser.  This work was made possible in part by the
-International Computer Science Institute, located at Suite 600, 1947 Center
-Street, Berkeley, California 94704.  Funding was partially provided by the
-National Science Foundation under grant MIP-9311980.  The original version
-of this code was written as part of a project to build a fixed-point vector
-processor in collaboration with the University of California at Berkeley,
-overseen by Profs. Nelson Morgan and John Wawrzynek.  More information
-is available through the Web page `http://www.cs.berkeley.edu/~jhauser/
-arithmetic/SoftFloat.html'.
-
-THIS SOFTWARE IS DISTRIBUTED AS IS, FOR FREE.  Although reasonable effort has
-been made to avoid it, THIS SOFTWARE MAY CONTAIN FAULTS THAT WILL AT TIMES
-RESULT IN INCORRECT BEHAVIOR.  USE OF THIS SOFTWARE IS RESTRICTED TO PERSONS
-AND ORGANIZATIONS WHO CAN AND WILL TAKE FULL RESPONSIBILITY FOR ALL LOSSES,
-COSTS, OR OTHER PROBLEMS THEY INCUR DUE TO THE SOFTWARE, AND WHO FURTHERMORE
-EFFECTIVELY INDEMNIFY JOHN HAUSER AND THE INTERNATIONAL COMPUTER SCIENCE
-INSTITUTE (possibly via similar legal warning) AGAINST ALL LOSSES, COSTS, OR
-OTHER PROBLEMS INCURRED BY THEIR CUSTOMERS AND CLIENTS DUE TO THE SOFTWARE.
-
-Derivative works are acceptable, even for commercial purposes, so long as
-(1) the source code for the derivative work includes prominent notice that
-the work is derivative, and (2) the source code includes prominent notice with
-these four paragraphs for those parts of this code that are retained.
-
-=============================================================================*/
-
-#ifndef SOFTFLOAT_H
-#define SOFTFLOAT_H
-
-#if defined(CONFIG_SOLARIS) && defined(CONFIG_NEEDS_LIBSUNMATH)
-#include <sunmath.h>
-#endif
-
-#include <inttypes.h>
-#include "config-host.h"
-#include "qemu/osdep.h"
-
-/*----------------------------------------------------------------------------
-| Each of the following `typedef's defines the most convenient type that holds
-| integers of at least as many bits as specified.  For example, `uint8' should
-| be the most convenient type that can hold unsigned integers of as many as
-| 8 bits.  The `flag' type must be able to hold either a 0 or 1.  For most
-| implementations of C, `flag', `uint8', and `int8' should all be `typedef'ed
-| to the same as `int'.
-*----------------------------------------------------------------------------*/
-typedef uint8_t flag;
-typedef uint8_t uint8;
-typedef int8_t int8;
-typedef unsigned int uint32;
-typedef signed int int32;
-typedef uint64_t uint64;
-typedef int64_t int64;
-
-#define LIT64( a ) a##LL
-#define INLINE static inline
-
-#define STATUS_PARAM , float_status *status
-#define STATUS(field) status->field
-#define STATUS_VAR , status
-
-/*----------------------------------------------------------------------------
-| Software IEC/IEEE floating-point ordering relations
-*----------------------------------------------------------------------------*/
-enum {
-    float_relation_less      = -1,
-    float_relation_equal     =  0,
-    float_relation_greater   =  1,
-    float_relation_unordered =  2
-};
-
-/*----------------------------------------------------------------------------
-| Software IEC/IEEE floating-point types.
-*----------------------------------------------------------------------------*/
-/* Use structures for soft-float types.  This prevents accidentally mixing
-   them with native int/float types.  A sufficiently clever compiler and
-   sane ABI should be able to see though these structs.  However
-   x86/gcc 3.x seems to struggle a bit, so leave them disabled by default.  */
-//#define USE_SOFTFLOAT_STRUCT_TYPES
-#ifdef USE_SOFTFLOAT_STRUCT_TYPES
-typedef struct {
-    uint16_t v;
-} float16;
-#define float16_val(x) (((float16)(x)).v)
-#define make_float16(x) __extension__ ({ float16 f16_val = {x}; f16_val; })
-#define const_float16(x) { x }
-typedef struct {
-    uint32_t v;
-} float32;
-/* The cast ensures an error if the wrong type is passed.  */
-#define float32_val(x) (((float32)(x)).v)
-#define make_float32(x) __extension__ ({ float32 f32_val = {x}; f32_val; })
-#define const_float32(x) { x }
-typedef struct {
-    uint64_t v;
-} float64;
-#define float64_val(x) (((float64)(x)).v)
-#define make_float64(x) __extension__ ({ float64 f64_val = {x}; f64_val; })
-#define const_float64(x) { x }
-#else
-typedef uint16_t float16;
-typedef uint32_t float32;
-typedef uint64_t float64;
-#define float16_val(x) (x)
-#define float32_val(x) (x)
-#define float64_val(x) (x)
-#define make_float16(x) (x)
-#define make_float32(x) (x)
-#define make_float64(x) (x)
-#define const_float16(x) (x)
-#define const_float32(x) (x)
-#define const_float64(x) (x)
-#endif
-typedef struct {
-    uint64_t low;
-    uint16_t high;
-} floatx80;
-#define make_floatx80(exp, mant) ((floatx80) { mant, exp })
-#define make_floatx80_init(exp, mant) { .low = mant, .high = exp }
-typedef struct {
-#ifdef HOST_WORDS_BIGENDIAN
-    uint64_t high, low;
-#else
-    uint64_t low, high;
-#endif
-} float128;
-#define make_float128(high_, low_) ((float128) { .high = high_, .low = low_ })
-#define make_float128_init(high_, low_) { .high = high_, .low = low_ }
-
-/*----------------------------------------------------------------------------
-| Software IEC/IEEE floating-point underflow tininess-detection mode.
-*----------------------------------------------------------------------------*/
-enum {
-    float_tininess_after_rounding  = 0,
-    float_tininess_before_rounding = 1
-};
-
-/*----------------------------------------------------------------------------
-| Software IEC/IEEE floating-point rounding mode.
-*----------------------------------------------------------------------------*/
-enum {
-    float_round_nearest_even = 0,
-    float_round_down         = 1,
-    float_round_up           = 2,
-    float_round_to_zero      = 3
-};
-
-/*----------------------------------------------------------------------------
-| Software IEC/IEEE floating-point exception flags.
-*----------------------------------------------------------------------------*/
-enum {
-    float_flag_invalid   =  1,
-    float_flag_divbyzero =  4,
-    float_flag_overflow  =  8,
-    float_flag_underflow = 16,
-    float_flag_inexact   = 32,
-    float_flag_input_denormal = 64,
-    float_flag_output_denormal = 128
-};
-
-typedef struct float_status {
-    signed char float_detect_tininess;
-    signed char float_rounding_mode;
-    signed char float_exception_flags;
-    signed char floatx80_rounding_precision;
-    /* should denormalised results go to zero and set the inexact flag? */
-    flag flush_to_zero;
-    /* should denormalised inputs go to zero and set the input_denormal flag? */
-    flag flush_inputs_to_zero;
-    flag default_nan_mode;
-} float_status;
-
-void set_float_rounding_mode(int val STATUS_PARAM);
-void set_float_exception_flags(int val STATUS_PARAM);
-INLINE void set_float_detect_tininess(int val STATUS_PARAM)
-{
-    STATUS(float_detect_tininess) = val;
-}
-INLINE void set_flush_to_zero(flag val STATUS_PARAM)
-{
-    STATUS(flush_to_zero) = val;
-}
-INLINE void set_flush_inputs_to_zero(flag val STATUS_PARAM)
-{
-    STATUS(flush_inputs_to_zero) = val;
-}
-INLINE void set_default_nan_mode(flag val STATUS_PARAM)
-{
-    STATUS(default_nan_mode) = val;
-}
-INLINE int get_float_exception_flags(float_status *status)
-{
-    return STATUS(float_exception_flags);
-}
-void set_floatx80_rounding_precision(int val STATUS_PARAM);
-
-/*----------------------------------------------------------------------------
-| Routine to raise any or all of the software IEC/IEEE floating-point
-| exception flags.
-*----------------------------------------------------------------------------*/
-void float_raise( int8 flags STATUS_PARAM);
-
-/*----------------------------------------------------------------------------
-| Options to indicate which negations to perform in float*_muladd()
-| Using these differs from negating an input or output before calling
-| the muladd function in that this means that a NaN doesn't have its
-| sign bit inverted before it is propagated.
-*----------------------------------------------------------------------------*/
-enum {
-    float_muladd_negate_c = 1,
-    float_muladd_negate_product = 2,
-    float_muladd_negate_result = 4,
-};
-
-/*----------------------------------------------------------------------------
-| Software IEC/IEEE integer-to-floating-point conversion routines.
-*----------------------------------------------------------------------------*/
-float32 int32_to_float32( int32 STATUS_PARAM );
-float64 int32_to_float64( int32 STATUS_PARAM );
-float32 uint32_to_float32( uint32 STATUS_PARAM );
-float64 uint32_to_float64( uint32 STATUS_PARAM );
-floatx80 int32_to_floatx80( int32 STATUS_PARAM );
-float128 int32_to_float128( int32 STATUS_PARAM );
-float32 int64_to_float32( int64 STATUS_PARAM );
-float32 uint64_to_float32( uint64 STATUS_PARAM );
-float64 int64_to_float64( int64 STATUS_PARAM );
-float64 uint64_to_float64( uint64 STATUS_PARAM );
-floatx80 int64_to_floatx80( int64 STATUS_PARAM );
-float128 int64_to_float128( int64 STATUS_PARAM );
-float128 uint64_to_float128( uint64 STATUS_PARAM );
-
-/*----------------------------------------------------------------------------
-| Software half-precision conversion routines.
-*----------------------------------------------------------------------------*/
-float16 float32_to_float16( float32, flag STATUS_PARAM );
-float32 float16_to_float32( float16, flag STATUS_PARAM );
-
-/*----------------------------------------------------------------------------
-| Software half-precision operations.
-*----------------------------------------------------------------------------*/
-int float16_is_quiet_nan( float16 );
-int float16_is_signaling_nan( float16 );
-float16 float16_maybe_silence_nan( float16 );
-
-INLINE int float16_is_any_nan(float16 a)
-{
-    return ((float16_val(a) & ~0x8000) > 0x7c00);
-}
-
-/*----------------------------------------------------------------------------
-| The pattern for a default generated half-precision NaN.
-*----------------------------------------------------------------------------*/
-extern const float16 float16_default_nan;
-
-/*----------------------------------------------------------------------------
-| Software IEC/IEEE single-precision conversion routines.
-*----------------------------------------------------------------------------*/
-int_fast16_t float32_to_int16_round_to_zero(float32 STATUS_PARAM);
-uint_fast16_t float32_to_uint16_round_to_zero(float32 STATUS_PARAM);
-int32 float32_to_int32( float32 STATUS_PARAM );
-int32 float32_to_int32_round_to_zero( float32 STATUS_PARAM );
-uint32 float32_to_uint32( float32 STATUS_PARAM );
-uint32 float32_to_uint32_round_to_zero( float32 STATUS_PARAM );
-int64 float32_to_int64( float32 STATUS_PARAM );
-int64 float32_to_int64_round_to_zero( float32 STATUS_PARAM );
-float64 float32_to_float64( float32 STATUS_PARAM );
-floatx80 float32_to_floatx80( float32 STATUS_PARAM );
-float128 float32_to_float128( float32 STATUS_PARAM );
-
-/*----------------------------------------------------------------------------
-| Software IEC/IEEE single-precision operations.
-*----------------------------------------------------------------------------*/
-float32 float32_round_to_int( float32 STATUS_PARAM );
-float32 float32_add( float32, float32 STATUS_PARAM );
-float32 float32_sub( float32, float32 STATUS_PARAM );
-float32 float32_mul( float32, float32 STATUS_PARAM );
-float32 float32_div( float32, float32 STATUS_PARAM );
-float32 float32_rem( float32, float32 STATUS_PARAM );
-float32 float32_muladd(float32, float32, float32, int STATUS_PARAM);
-float32 float32_sqrt( float32 STATUS_PARAM );
-float32 float32_exp2( float32 STATUS_PARAM );
-float32 float32_log2( float32 STATUS_PARAM );
-int float32_eq( float32, float32 STATUS_PARAM );
-int float32_le( float32, float32 STATUS_PARAM );
-int float32_lt( float32, float32 STATUS_PARAM );
-int float32_unordered( float32, float32 STATUS_PARAM );
-int float32_eq_quiet( float32, float32 STATUS_PARAM );
-int float32_le_quiet( float32, float32 STATUS_PARAM );
-int float32_lt_quiet( float32, float32 STATUS_PARAM );
-int float32_unordered_quiet( float32, float32 STATUS_PARAM );
-int float32_compare( float32, float32 STATUS_PARAM );
-int float32_compare_quiet( float32, float32 STATUS_PARAM );
-float32 float32_min(float32, float32 STATUS_PARAM);
-float32 float32_max(float32, float32 STATUS_PARAM);
-int float32_is_quiet_nan( float32 );
-int float32_is_signaling_nan( float32 );
-float32 float32_maybe_silence_nan( float32 );
-float32 float32_scalbn( float32, int STATUS_PARAM );
-
-INLINE float32 float32_abs(float32 a)
-{
-    /* Note that abs does *not* handle NaN specially, nor does
-     * it flush denormal inputs to zero.
-     */
-    return make_float32(float32_val(a) & 0x7fffffff);
-}
-
-INLINE float32 float32_chs(float32 a)
-{
-    /* Note that chs does *not* handle NaN specially, nor does
-     * it flush denormal inputs to zero.
-     */
-    return make_float32(float32_val(a) ^ 0x80000000);
-}
-
-INLINE int float32_is_infinity(float32 a)
-{
-    return (float32_val(a) & 0x7fffffff) == 0x7f800000;
-}
-
-INLINE int float32_is_neg(float32 a)
-{
-    return float32_val(a) >> 31;
-}
-
-INLINE int float32_is_zero(float32 a)
-{
-    return (float32_val(a) & 0x7fffffff) == 0;
-}
-
-INLINE int float32_is_any_nan(float32 a)
-{
-    return ((float32_val(a) & ~(1 << 31)) > 0x7f800000UL);
-}
-
-INLINE int float32_is_zero_or_denormal(float32 a)
-{
-    return (float32_val(a) & 0x7f800000) == 0;
-}
-
-INLINE float32 float32_set_sign(float32 a, int sign)
-{
-    return make_float32((float32_val(a) & 0x7fffffff) | (sign << 31));
-}
-
-#define float32_zero make_float32(0)
-#define float32_one make_float32(0x3f800000)
-#define float32_ln2 make_float32(0x3f317218)
-#define float32_pi make_float32(0x40490fdb)
-#define float32_half make_float32(0x3f000000)
-#define float32_infinity make_float32(0x7f800000)
-
-
-/*----------------------------------------------------------------------------
-| The pattern for a default generated single-precision NaN.
-*----------------------------------------------------------------------------*/
-extern const float32 float32_default_nan;
-
-/*----------------------------------------------------------------------------
-| Software IEC/IEEE double-precision conversion routines.
-*----------------------------------------------------------------------------*/
-int_fast16_t float64_to_int16_round_to_zero(float64 STATUS_PARAM);
-uint_fast16_t float64_to_uint16_round_to_zero(float64 STATUS_PARAM);
-int32 float64_to_int32( float64 STATUS_PARAM );
-int32 float64_to_int32_round_to_zero( float64 STATUS_PARAM );
-uint32 float64_to_uint32( float64 STATUS_PARAM );
-uint32 float64_to_uint32_round_to_zero( float64 STATUS_PARAM );
-int64 float64_to_int64( float64 STATUS_PARAM );
-int64 float64_to_int64_round_to_zero( float64 STATUS_PARAM );
-uint64 float64_to_uint64 (float64 a STATUS_PARAM);
-uint64 float64_to_uint64_round_to_zero (float64 a STATUS_PARAM);
-float32 float64_to_float32( float64 STATUS_PARAM );
-floatx80 float64_to_floatx80( float64 STATUS_PARAM );
-float128 float64_to_float128( float64 STATUS_PARAM );
-
-/*----------------------------------------------------------------------------
-| Software IEC/IEEE double-precision operations.
-*----------------------------------------------------------------------------*/
-float64 float64_round_to_int( float64 STATUS_PARAM );
-float64 float64_trunc_to_int( float64 STATUS_PARAM );
-float64 float64_add( float64, float64 STATUS_PARAM );
-float64 float64_sub( float64, float64 STATUS_PARAM );
-float64 float64_mul( float64, float64 STATUS_PARAM );
-float64 float64_div( float64, float64 STATUS_PARAM );
-float64 float64_rem( float64, float64 STATUS_PARAM );
-float64 float64_muladd(float64, float64, float64, int STATUS_PARAM);
-float64 float64_sqrt( float64 STATUS_PARAM );
-float64 float64_log2( float64 STATUS_PARAM );
-int float64_eq( float64, float64 STATUS_PARAM );
-int float64_le( float64, float64 STATUS_PARAM );
-int float64_lt( float64, float64 STATUS_PARAM );
-int float64_unordered( float64, float64 STATUS_PARAM );
-int float64_eq_quiet( float64, float64 STATUS_PARAM );
-int float64_le_quiet( float64, float64 STATUS_PARAM );
-int float64_lt_quiet( float64, float64 STATUS_PARAM );
-int float64_unordered_quiet( float64, float64 STATUS_PARAM );
-int float64_compare( float64, float64 STATUS_PARAM );
-int float64_compare_quiet( float64, float64 STATUS_PARAM );
-float64 float64_min(float64, float64 STATUS_PARAM);
-float64 float64_max(float64, float64 STATUS_PARAM);
-int float64_is_quiet_nan( float64 a );
-int float64_is_signaling_nan( float64 );
-float64 float64_maybe_silence_nan( float64 );
-float64 float64_scalbn( float64, int STATUS_PARAM );
-
-INLINE float64 float64_abs(float64 a)
-{
-    /* Note that abs does *not* handle NaN specially, nor does
-     * it flush denormal inputs to zero.
-     */
-    return make_float64(float64_val(a) & 0x7fffffffffffffffLL);
-}
-
-INLINE float64 float64_chs(float64 a)
-{
-    /* Note that chs does *not* handle NaN specially, nor does
-     * it flush denormal inputs to zero.
-     */
-    return make_float64(float64_val(a) ^ 0x8000000000000000LL);
-}
-
-INLINE int float64_is_infinity(float64 a)
-{
-    return (float64_val(a) & 0x7fffffffffffffffLL ) == 0x7ff0000000000000LL;
-}
-
-INLINE int float64_is_neg(float64 a)
-{
-    return float64_val(a) >> 63;
-}
-
-INLINE int float64_is_zero(float64 a)
-{
-    return (float64_val(a) & 0x7fffffffffffffffLL) == 0;
-}
-
-INLINE int float64_is_any_nan(float64 a)
-{
-    return ((float64_val(a) & ~(1ULL << 63)) > 0x7ff0000000000000ULL);
-}
-
-INLINE int float64_is_zero_or_denormal(float64 a)
-{
-    return (float64_val(a) & 0x7ff0000000000000LL) == 0;
-}
-
-INLINE float64 float64_set_sign(float64 a, int sign)
-{
-    return make_float64((float64_val(a) & 0x7fffffffffffffffULL)
-                        | ((int64_t)sign << 63));
-}
-
-#define float64_zero make_float64(0)
-#define float64_one make_float64(0x3ff0000000000000LL)
-#define float64_ln2 make_float64(0x3fe62e42fefa39efLL)
-#define float64_pi make_float64(0x400921fb54442d18LL)
-#define float64_half make_float64(0x3fe0000000000000LL)
-#define float64_infinity make_float64(0x7ff0000000000000LL)
-
-/*----------------------------------------------------------------------------
-| The pattern for a default generated double-precision NaN.
-*----------------------------------------------------------------------------*/
-extern const float64 float64_default_nan;
-
-/*----------------------------------------------------------------------------
-| Software IEC/IEEE extended double-precision conversion routines.
-*----------------------------------------------------------------------------*/
-int32 floatx80_to_int32( floatx80 STATUS_PARAM );
-int32 floatx80_to_int32_round_to_zero( floatx80 STATUS_PARAM );
-int64 floatx80_to_int64( floatx80 STATUS_PARAM );
-int64 floatx80_to_int64_round_to_zero( floatx80 STATUS_PARAM );
-float32 floatx80_to_float32( floatx80 STATUS_PARAM );
-float64 floatx80_to_float64( floatx80 STATUS_PARAM );
-float128 floatx80_to_float128( floatx80 STATUS_PARAM );
-
-/*----------------------------------------------------------------------------
-| Software IEC/IEEE extended double-precision operations.
-*----------------------------------------------------------------------------*/
-floatx80 floatx80_round_to_int( floatx80 STATUS_PARAM );
-floatx80 floatx80_add( floatx80, floatx80 STATUS_PARAM );
-floatx80 floatx80_sub( floatx80, floatx80 STATUS_PARAM );
-floatx80 floatx80_mul( floatx80, floatx80 STATUS_PARAM );
-floatx80 floatx80_div( floatx80, floatx80 STATUS_PARAM );
-floatx80 floatx80_rem( floatx80, floatx80 STATUS_PARAM );
-floatx80 floatx80_sqrt( floatx80 STATUS_PARAM );
-int floatx80_eq( floatx80, floatx80 STATUS_PARAM );
-int floatx80_le( floatx80, floatx80 STATUS_PARAM );
-int floatx80_lt( floatx80, floatx80 STATUS_PARAM );
-int floatx80_unordered( floatx80, floatx80 STATUS_PARAM );
-int floatx80_eq_quiet( floatx80, floatx80 STATUS_PARAM );
-int floatx80_le_quiet( floatx80, floatx80 STATUS_PARAM );
-int floatx80_lt_quiet( floatx80, floatx80 STATUS_PARAM );
-int floatx80_unordered_quiet( floatx80, floatx80 STATUS_PARAM );
-int floatx80_compare( floatx80, floatx80 STATUS_PARAM );
-int floatx80_compare_quiet( floatx80, floatx80 STATUS_PARAM );
-int floatx80_is_quiet_nan( floatx80 );
-int floatx80_is_signaling_nan( floatx80 );
-floatx80 floatx80_maybe_silence_nan( floatx80 );
-floatx80 floatx80_scalbn( floatx80, int STATUS_PARAM );
-
-INLINE floatx80 floatx80_abs(floatx80 a)
-{
-    a.high &= 0x7fff;
-    return a;
-}
-
-INLINE floatx80 floatx80_chs(floatx80 a)
-{
-    a.high ^= 0x8000;
-    return a;
-}
-
-INLINE int floatx80_is_infinity(floatx80 a)
-{
-    return (a.high & 0x7fff) == 0x7fff && a.low == 0x8000000000000000LL;
-}
-
-INLINE int floatx80_is_neg(floatx80 a)
-{
-    return a.high >> 15;
-}
-
-INLINE int floatx80_is_zero(floatx80 a)
-{
-    return (a.high & 0x7fff) == 0 && a.low == 0;
-}
-
-INLINE int floatx80_is_zero_or_denormal(floatx80 a)
-{
-    return (a.high & 0x7fff) == 0;
-}
-
-INLINE int floatx80_is_any_nan(floatx80 a)
-{
-    return ((a.high & 0x7fff) == 0x7fff) && (a.low<<1);
-}
-
-#define floatx80_zero make_floatx80(0x0000, 0x0000000000000000LL)
-#define floatx80_one make_floatx80(0x3fff, 0x8000000000000000LL)
-#define floatx80_ln2 make_floatx80(0x3ffe, 0xb17217f7d1cf79acLL)
-#define floatx80_pi make_floatx80(0x4000, 0xc90fdaa22168c235LL)
-#define floatx80_half make_floatx80(0x3ffe, 0x8000000000000000LL)
-#define floatx80_infinity make_floatx80(0x7fff, 0x8000000000000000LL)
-
-/*----------------------------------------------------------------------------
-| The pattern for a default generated extended double-precision NaN.
-*----------------------------------------------------------------------------*/
-extern const floatx80 floatx80_default_nan;
-
-/*----------------------------------------------------------------------------
-| Software IEC/IEEE quadruple-precision conversion routines.
-*----------------------------------------------------------------------------*/
-int32 float128_to_int32( float128 STATUS_PARAM );
-int32 float128_to_int32_round_to_zero( float128 STATUS_PARAM );
-int64 float128_to_int64( float128 STATUS_PARAM );
-int64 float128_to_int64_round_to_zero( float128 STATUS_PARAM );
-float32 float128_to_float32( float128 STATUS_PARAM );
-float64 float128_to_float64( float128 STATUS_PARAM );
-floatx80 float128_to_floatx80( float128 STATUS_PARAM );
-
-/*----------------------------------------------------------------------------
-| Software IEC/IEEE quadruple-precision operations.
-*----------------------------------------------------------------------------*/
-float128 float128_round_to_int( float128 STATUS_PARAM );
-float128 float128_add( float128, float128 STATUS_PARAM );
-float128 float128_sub( float128, float128 STATUS_PARAM );
-float128 float128_mul( float128, float128 STATUS_PARAM );
-float128 float128_div( float128, float128 STATUS_PARAM );
-float128 float128_rem( float128, float128 STATUS_PARAM );
-float128 float128_sqrt( float128 STATUS_PARAM );
-int float128_eq( float128, float128 STATUS_PARAM );
-int float128_le( float128, float128 STATUS_PARAM );
-int float128_lt( float128, float128 STATUS_PARAM );
-int float128_unordered( float128, float128 STATUS_PARAM );
-int float128_eq_quiet( float128, float128 STATUS_PARAM );
-int float128_le_quiet( float128, float128 STATUS_PARAM );
-int float128_lt_quiet( float128, float128 STATUS_PARAM );
-int float128_unordered_quiet( float128, float128 STATUS_PARAM );
-int float128_compare( float128, float128 STATUS_PARAM );
-int float128_compare_quiet( float128, float128 STATUS_PARAM );
-int float128_is_quiet_nan( float128 );
-int float128_is_signaling_nan( float128 );
-float128 float128_maybe_silence_nan( float128 );
-float128 float128_scalbn( float128, int STATUS_PARAM );
-
-INLINE float128 float128_abs(float128 a)
-{
-    a.high &= 0x7fffffffffffffffLL;
-    return a;
-}
-
-INLINE float128 float128_chs(float128 a)
-{
-    a.high ^= 0x8000000000000000LL;
-    return a;
-}
-
-INLINE int float128_is_infinity(float128 a)
-{
-    return (a.high & 0x7fffffffffffffffLL) == 0x7fff000000000000LL && a.low == 0;
-}
-
-INLINE int float128_is_neg(float128 a)
-{
-    return a.high >> 63;
-}
-
-INLINE int float128_is_zero(float128 a)
-{
-    return (a.high & 0x7fffffffffffffffLL) == 0 && a.low == 0;
-}
-
-INLINE int float128_is_zero_or_denormal(float128 a)
-{
-    return (a.high & 0x7fff000000000000LL) == 0;
-}
-
-INLINE int float128_is_any_nan(float128 a)
-{
-    return ((a.high >> 48) & 0x7fff) == 0x7fff &&
-        ((a.low != 0) || ((a.high & 0xffffffffffffLL) != 0));
-}
-
-#define float128_zero make_float128(0, 0)
-
-/*----------------------------------------------------------------------------
-| The pattern for a default generated quadruple-precision NaN.
-*----------------------------------------------------------------------------*/
-extern const float128 float128_default_nan;
-
-#endif /* !SOFTFLOAT_H */
diff --git a/contrib/qemu/include/glib-compat.h b/contrib/qemu/include/glib-compat.h
deleted file mode 100644
index 8aa77afd626..00000000000
--- a/contrib/qemu/include/glib-compat.h
+++ /dev/null
@@ -1,27 +0,0 @@
-/*
- * GLIB Compatibility Functions
- *
- * Copyright IBM, Corp. 2013
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU GPL, version 2 or later.
- * See the COPYING file in the top-level directory.
- *
- */
-
-#ifndef QEMU_GLIB_COMPAT_H
-#define QEMU_GLIB_COMPAT_H
-
-#include <glib.h>
-
-#if !GLIB_CHECK_VERSION(2, 14, 0)
-static inline guint g_timeout_add_seconds(guint interval, GSourceFunc function,
-                                          gpointer data)
-{
-    return g_timeout_add(interval * 1000, function, data);
-}
-#endif
-
-#endif
diff --git a/contrib/qemu/include/migration/migration.h b/contrib/qemu/include/migration/migration.h
deleted file mode 100644
index bc9fde0b2ab..00000000000
--- a/contrib/qemu/include/migration/migration.h
+++ /dev/null
@@ -1,157 +0,0 @@
-/*
- * QEMU live migration
- *
- * Copyright IBM, Corp. 2008
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU GPL, version 2.  See
- * the COPYING file in the top-level directory.
- *
- */
-
-#ifndef QEMU_MIGRATION_H
-#define QEMU_MIGRATION_H
-
-#include "qapi/qmp/qdict.h"
-#include "qemu-common.h"
-#include "qemu/thread.h"
-#include "qemu/notify.h"
-#include "qapi/error.h"
-#include "migration/vmstate.h"
-#include "qapi-types.h"
-#include "exec/cpu-common.h"
-
-struct MigrationParams {
-    bool blk;
-    bool shared;
-};
-
-typedef struct MigrationState MigrationState;
-
-struct MigrationState
-{
-    int64_t bandwidth_limit;
-    size_t bytes_xfer;
-    size_t xfer_limit;
-    QemuThread thread;
-    QEMUBH *cleanup_bh;
-    QEMUFile *file;
-
-    int state;
-    MigrationParams params;
-    double mbps;
-    int64_t total_time;
-    int64_t downtime;
-    int64_t expected_downtime;
-    int64_t dirty_pages_rate;
-    int64_t dirty_bytes_rate;
-    bool enabled_capabilities[MIGRATION_CAPABILITY_MAX];
-    int64_t xbzrle_cache_size;
-};
-
-void process_incoming_migration(QEMUFile *f);
-
-void qemu_start_incoming_migration(const char *uri, Error **errp);
-
-uint64_t migrate_max_downtime(void);
-
-void do_info_migrate_print(Monitor *mon, const QObject *data);
-
-void do_info_migrate(Monitor *mon, QObject **ret_data);
-
-void exec_start_incoming_migration(const char *host_port, Error **errp);
-
-void exec_start_outgoing_migration(MigrationState *s, const char *host_port, Error **errp);
-
-void tcp_start_incoming_migration(const char *host_port, Error **errp);
-
-void tcp_start_outgoing_migration(MigrationState *s, const char *host_port, Error **errp);
-
-void unix_start_incoming_migration(const char *path, Error **errp);
-
-void unix_start_outgoing_migration(MigrationState *s, const char *path, Error **errp);
-
-void fd_start_incoming_migration(const char *path, Error **errp);
-
-void fd_start_outgoing_migration(MigrationState *s, const char *fdname, Error **errp);
-
-void migrate_fd_error(MigrationState *s);
-
-void migrate_fd_connect(MigrationState *s);
-
-int migrate_fd_close(MigrationState *s);
-
-void add_migration_state_change_notifier(Notifier *notify);
-void remove_migration_state_change_notifier(Notifier *notify);
-bool migration_is_active(MigrationState *);
-bool migration_has_finished(MigrationState *);
-bool migration_has_failed(MigrationState *);
-MigrationState *migrate_get_current(void);
-
-uint64_t ram_bytes_remaining(void);
-uint64_t ram_bytes_transferred(void);
-uint64_t ram_bytes_total(void);
-
-void acct_update_position(QEMUFile *f, size_t size, bool zero);
-
-extern SaveVMHandlers savevm_ram_handlers;
-
-uint64_t dup_mig_bytes_transferred(void);
-uint64_t dup_mig_pages_transferred(void);
-uint64_t skipped_mig_bytes_transferred(void);
-uint64_t skipped_mig_pages_transferred(void);
-uint64_t norm_mig_bytes_transferred(void);
-uint64_t norm_mig_pages_transferred(void);
-uint64_t xbzrle_mig_bytes_transferred(void);
-uint64_t xbzrle_mig_pages_transferred(void);
-uint64_t xbzrle_mig_pages_overflow(void);
-uint64_t xbzrle_mig_pages_cache_miss(void);
-
-/**
- * @migrate_add_blocker - prevent migration from proceeding
- *
- * @reason - an error to be returned whenever migration is attempted
- */
-void migrate_add_blocker(Error *reason);
-
-/**
- * @migrate_del_blocker - remove a blocking error from migration
- *
- * @reason - the error blocking migration
- */
-void migrate_del_blocker(Error *reason);
-
-bool migrate_rdma_pin_all(void);
-
-bool migrate_auto_converge(void);
-
-int xbzrle_encode_buffer(uint8_t *old_buf, uint8_t *new_buf, int slen,
-                         uint8_t *dst, int dlen);
-int xbzrle_decode_buffer(uint8_t *src, int slen, uint8_t *dst, int dlen);
-
-int migrate_use_xbzrle(void);
-int64_t migrate_xbzrle_cache_size(void);
-
-int64_t xbzrle_cache_resize(int64_t new_size);
-
-void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
-void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
-void ram_control_load_hook(QEMUFile *f, uint64_t flags);
-
-/* Whenever this is found in the data stream, the flags
- * will be passed to ram_control_load_hook in the incoming-migration
- * side. This lets before_ram_iterate/after_ram_iterate add
- * transport-specific sections to the RAM migration data.
- */
-#define RAM_SAVE_FLAG_HOOK     0x80
-
-#define RAM_SAVE_CONTROL_NOT_SUPP -1000
-#define RAM_SAVE_CONTROL_DELAYED  -2000
-
-size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
-                             ram_addr_t offset, size_t size,
-                             int *bytes_sent);
-
-#endif
diff --git a/contrib/qemu/include/migration/qemu-file.h b/contrib/qemu/include/migration/qemu-file.h
deleted file mode 100644
index 0f757fbeb63..00000000000
--- a/contrib/qemu/include/migration/qemu-file.h
+++ /dev/null
@@ -1,266 +0,0 @@
-/*
- * QEMU System Emulator
- *
- * Copyright (c) 2003-2008 Fabrice Bellard
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-#ifndef QEMU_FILE_H
-#define QEMU_FILE_H 1
-#include "exec/cpu-common.h"
-
-/* This function writes a chunk of data to a file at the given position.
- * The pos argument can be ignored if the file is only being used for
- * streaming.  The handler should try to write all of the data it can.
- */
-typedef int (QEMUFilePutBufferFunc)(void *opaque, const uint8_t *buf,
-                                    int64_t pos, int size);
-
-/* Read a chunk of data from a file at the given position.  The pos argument
- * can be ignored if the file is only be used for streaming.  The number of
- * bytes actually read should be returned.
- */
-typedef int (QEMUFileGetBufferFunc)(void *opaque, uint8_t *buf,
-                                    int64_t pos, int size);
-
-/* Close a file
- *
- * Return negative error number on error, 0 or positive value on success.
- *
- * The meaning of return value on success depends on the specific back-end being
- * used.
- */
-typedef int (QEMUFileCloseFunc)(void *opaque);
-
-/* Called to return the OS file descriptor associated to the QEMUFile.
- */
-typedef int (QEMUFileGetFD)(void *opaque);
-
-/*
- * This function writes an iovec to file.
- */
-typedef ssize_t (QEMUFileWritevBufferFunc)(void *opaque, struct iovec *iov,
-                                           int iovcnt, int64_t pos);
-
-/*
- * This function provides hooks around different
- * stages of RAM migration.
- */
-typedef int (QEMURamHookFunc)(QEMUFile *f, void *opaque, uint64_t flags);
-
-/*
- * Constants used by ram_control_* hooks
- */
-#define RAM_CONTROL_SETUP    0
-#define RAM_CONTROL_ROUND    1
-#define RAM_CONTROL_HOOK     2
-#define RAM_CONTROL_FINISH   3
-
-/*
- * This function allows override of where the RAM page
- * is saved (such as RDMA, for example.)
- */
-typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque,
-                               ram_addr_t block_offset,
-                               ram_addr_t offset,
-                               size_t size,
-                               int *bytes_sent);
-
-typedef struct QEMUFileOps {
-    QEMUFilePutBufferFunc *put_buffer;
-    QEMUFileGetBufferFunc *get_buffer;
-    QEMUFileCloseFunc *close;
-    QEMUFileGetFD *get_fd;
-    QEMUFileWritevBufferFunc *writev_buffer;
-    QEMURamHookFunc *before_ram_iterate;
-    QEMURamHookFunc *after_ram_iterate;
-    QEMURamHookFunc *hook_ram_load;
-    QEMURamSaveFunc *save_page;
-} QEMUFileOps;
-
-QEMUFile *qemu_fopen_ops(void *opaque, const QEMUFileOps *ops);
-QEMUFile *qemu_fopen(const char *filename, const char *mode);
-QEMUFile *qemu_fdopen(int fd, const char *mode);
-QEMUFile *qemu_fopen_socket(int fd, const char *mode);
-QEMUFile *qemu_popen_cmd(const char *command, const char *mode);
-int qemu_get_fd(QEMUFile *f);
-int qemu_fclose(QEMUFile *f);
-int64_t qemu_ftell(QEMUFile *f);
-void qemu_put_buffer(QEMUFile *f, const uint8_t *buf, int size);
-void qemu_put_byte(QEMUFile *f, int v);
-/*
- * put_buffer without copying the buffer.
- * The buffer should be available till it is sent asynchronously.
- */
-void qemu_put_buffer_async(QEMUFile *f, const uint8_t *buf, int size);
-bool qemu_file_mode_is_not_valid(const char *mode);
-
-static inline void qemu_put_ubyte(QEMUFile *f, unsigned int v)
-{
-    qemu_put_byte(f, (int)v);
-}
-
-#define qemu_put_sbyte qemu_put_byte
-
-void qemu_put_be16(QEMUFile *f, unsigned int v);
-void qemu_put_be32(QEMUFile *f, unsigned int v);
-void qemu_put_be64(QEMUFile *f, uint64_t v);
-int qemu_get_buffer(QEMUFile *f, uint8_t *buf, int size);
-int qemu_get_byte(QEMUFile *f);
-void qemu_update_position(QEMUFile *f, size_t size);
-
-static inline unsigned int qemu_get_ubyte(QEMUFile *f)
-{
-    return (unsigned int)qemu_get_byte(f);
-}
-
-#define qemu_get_sbyte qemu_get_byte
-
-unsigned int qemu_get_be16(QEMUFile *f);
-unsigned int qemu_get_be32(QEMUFile *f);
-uint64_t qemu_get_be64(QEMUFile *f);
-
-int qemu_file_rate_limit(QEMUFile *f);
-void qemu_file_reset_rate_limit(QEMUFile *f);
-void qemu_file_set_rate_limit(QEMUFile *f, int64_t new_rate);
-int64_t qemu_file_get_rate_limit(QEMUFile *f);
-int qemu_file_get_error(QEMUFile *f);
-void qemu_fflush(QEMUFile *f);
-
-static inline void qemu_put_be64s(QEMUFile *f, const uint64_t *pv)
-{
-    qemu_put_be64(f, *pv);
-}
-
-static inline void qemu_put_be32s(QEMUFile *f, const uint32_t *pv)
-{
-    qemu_put_be32(f, *pv);
-}
-
-static inline void qemu_put_be16s(QEMUFile *f, const uint16_t *pv)
-{
-    qemu_put_be16(f, *pv);
-}
-
-static inline void qemu_put_8s(QEMUFile *f, const uint8_t *pv)
-{
-    qemu_put_byte(f, *pv);
-}
-
-static inline void qemu_get_be64s(QEMUFile *f, uint64_t *pv)
-{
-    *pv = qemu_get_be64(f);
-}
-
-static inline void qemu_get_be32s(QEMUFile *f, uint32_t *pv)
-{
-    *pv = qemu_get_be32(f);
-}
-
-static inline void qemu_get_be16s(QEMUFile *f, uint16_t *pv)
-{
-    *pv = qemu_get_be16(f);
-}
-
-static inline void qemu_get_8s(QEMUFile *f, uint8_t *pv)
-{
-    *pv = qemu_get_byte(f);
-}
-
-// Signed versions for type safety
-static inline void qemu_put_sbuffer(QEMUFile *f, const int8_t *buf, int size)
-{
-    qemu_put_buffer(f, (const uint8_t *)buf, size);
-}
-
-static inline void qemu_put_sbe16(QEMUFile *f, int v)
-{
-    qemu_put_be16(f, (unsigned int)v);
-}
-
-static inline void qemu_put_sbe32(QEMUFile *f, int v)
-{
-    qemu_put_be32(f, (unsigned int)v);
-}
-
-static inline void qemu_put_sbe64(QEMUFile *f, int64_t v)
-{
-    qemu_put_be64(f, (uint64_t)v);
-}
-
-static inline size_t qemu_get_sbuffer(QEMUFile *f, int8_t *buf, int size)
-{
-    return qemu_get_buffer(f, (uint8_t *)buf, size);
-}
-
-static inline int qemu_get_sbe16(QEMUFile *f)
-{
-    return (int)qemu_get_be16(f);
-}
-
-static inline int qemu_get_sbe32(QEMUFile *f)
-{
-    return (int)qemu_get_be32(f);
-}
-
-static inline int64_t qemu_get_sbe64(QEMUFile *f)
-{
-    return (int64_t)qemu_get_be64(f);
-}
-
-static inline void qemu_put_s8s(QEMUFile *f, const int8_t *pv)
-{
-    qemu_put_8s(f, (const uint8_t *)pv);
-}
-
-static inline void qemu_put_sbe16s(QEMUFile *f, const int16_t *pv)
-{
-    qemu_put_be16s(f, (const uint16_t *)pv);
-}
-
-static inline void qemu_put_sbe32s(QEMUFile *f, const int32_t *pv)
-{
-    qemu_put_be32s(f, (const uint32_t *)pv);
-}
-
-static inline void qemu_put_sbe64s(QEMUFile *f, const int64_t *pv)
-{
-    qemu_put_be64s(f, (const uint64_t *)pv);
-}
-
-static inline void qemu_get_s8s(QEMUFile *f, int8_t *pv)
-{
-    qemu_get_8s(f, (uint8_t *)pv);
-}
-
-static inline void qemu_get_sbe16s(QEMUFile *f, int16_t *pv)
-{
-    qemu_get_be16s(f, (uint16_t *)pv);
-}
-
-static inline void qemu_get_sbe32s(QEMUFile *f, int32_t *pv)
-{
-    qemu_get_be32s(f, (uint32_t *)pv);
-}
-
-static inline void qemu_get_sbe64s(QEMUFile *f, int64_t *pv)
-{
-    qemu_get_be64s(f, (uint64_t *)pv);
-}
-#endif
diff --git a/contrib/qemu/include/migration/vmstate.h b/contrib/qemu/include/migration/vmstate.h
deleted file mode 100644
index 1c31b5d6fb5..00000000000
--- a/contrib/qemu/include/migration/vmstate.h
+++ /dev/null
@@ -1,740 +0,0 @@
-/*
- * QEMU migration/snapshot declarations
- *
- * Copyright (c) 2009-2011 Red Hat, Inc.
- *
- * Original author: Juan Quintela <quintela@redhat.com>
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-#ifndef QEMU_VMSTATE_H
-#define QEMU_VMSTATE_H 1
-
-#ifndef CONFIG_USER_ONLY
-#include <migration/qemu-file.h>
-#endif
-
-typedef void SaveStateHandler(QEMUFile *f, void *opaque);
-typedef int LoadStateHandler(QEMUFile *f, void *opaque, int version_id);
-
-typedef struct SaveVMHandlers {
-    /* This runs inside the iothread lock.  */
-    void (*set_params)(const MigrationParams *params, void * opaque);
-    SaveStateHandler *save_state;
-
-    void (*cancel)(void *opaque);
-    int (*save_live_complete)(QEMUFile *f, void *opaque);
-
-    /* This runs both outside and inside the iothread lock.  */
-    bool (*is_active)(void *opaque);
-
-    /* This runs outside the iothread lock in the migration case, and
-     * within the lock in the savevm case.  The callback had better only
-     * use data that is local to the migration thread or protected
-     * by other locks.
-     */
-    int (*save_live_iterate)(QEMUFile *f, void *opaque);
-
-    /* This runs outside the iothread lock!  */
-    int (*save_live_setup)(QEMUFile *f, void *opaque);
-    uint64_t (*save_live_pending)(QEMUFile *f, void *opaque, uint64_t max_size);
-
-    LoadStateHandler *load_state;
-} SaveVMHandlers;
-
-int register_savevm(DeviceState *dev,
-                    const char *idstr,
-                    int instance_id,
-                    int version_id,
-                    SaveStateHandler *save_state,
-                    LoadStateHandler *load_state,
-                    void *opaque);
-
-int register_savevm_live(DeviceState *dev,
-                         const char *idstr,
-                         int instance_id,
-                         int version_id,
-                         SaveVMHandlers *ops,
-                         void *opaque);
-
-void unregister_savevm(DeviceState *dev, const char *idstr, void *opaque);
-void register_device_unmigratable(DeviceState *dev, const char *idstr,
-                                                                void *opaque);
-
-
-typedef struct VMStateInfo VMStateInfo;
-typedef struct VMStateDescription VMStateDescription;
-
-struct VMStateInfo {
-    const char *name;
-    int (*get)(QEMUFile *f, void *pv, size_t size);
-    void (*put)(QEMUFile *f, void *pv, size_t size);
-};
-
-enum VMStateFlags {
-    VMS_SINGLE           = 0x001,
-    VMS_POINTER          = 0x002,
-    VMS_ARRAY            = 0x004,
-    VMS_STRUCT           = 0x008,
-    VMS_VARRAY_INT32     = 0x010,  /* Array with size in int32_t field*/
-    VMS_BUFFER           = 0x020,  /* static sized buffer */
-    VMS_ARRAY_OF_POINTER = 0x040,
-    VMS_VARRAY_UINT16    = 0x080,  /* Array with size in uint16_t field */
-    VMS_VBUFFER          = 0x100,  /* Buffer with size in int32_t field */
-    VMS_MULTIPLY         = 0x200,  /* multiply "size" field by field_size */
-    VMS_VARRAY_UINT8     = 0x400,  /* Array with size in uint8_t field*/
-    VMS_VARRAY_UINT32    = 0x800,  /* Array with size in uint32_t field*/
-};
-
-typedef struct {
-    const char *name;
-    size_t offset;
-    size_t size;
-    size_t start;
-    int num;
-    size_t num_offset;
-    size_t size_offset;
-    const VMStateInfo *info;
-    enum VMStateFlags flags;
-    const VMStateDescription *vmsd;
-    int version_id;
-    bool (*field_exists)(void *opaque, int version_id);
-} VMStateField;
-
-typedef struct VMStateSubsection {
-    const VMStateDescription *vmsd;
-    bool (*needed)(void *opaque);
-} VMStateSubsection;
-
-struct VMStateDescription {
-    const char *name;
-    int unmigratable;
-    int version_id;
-    int minimum_version_id;
-    int minimum_version_id_old;
-    LoadStateHandler *load_state_old;
-    int (*pre_load)(void *opaque);
-    int (*post_load)(void *opaque, int version_id);
-    void (*pre_save)(void *opaque);
-    VMStateField *fields;
-    const VMStateSubsection *subsections;
-};
-
-#ifdef CONFIG_USER_ONLY
-extern const VMStateDescription vmstate_dummy;
-#endif
-
-extern const VMStateInfo vmstate_info_bool;
-
-extern const VMStateInfo vmstate_info_int8;
-extern const VMStateInfo vmstate_info_int16;
-extern const VMStateInfo vmstate_info_int32;
-extern const VMStateInfo vmstate_info_int64;
-
-extern const VMStateInfo vmstate_info_uint8_equal;
-extern const VMStateInfo vmstate_info_uint16_equal;
-extern const VMStateInfo vmstate_info_int32_equal;
-extern const VMStateInfo vmstate_info_uint32_equal;
-extern const VMStateInfo vmstate_info_uint64_equal;
-extern const VMStateInfo vmstate_info_int32_le;
-
-extern const VMStateInfo vmstate_info_uint8;
-extern const VMStateInfo vmstate_info_uint16;
-extern const VMStateInfo vmstate_info_uint32;
-extern const VMStateInfo vmstate_info_uint64;
-
-extern const VMStateInfo vmstate_info_float64;
-
-extern const VMStateInfo vmstate_info_timer;
-extern const VMStateInfo vmstate_info_buffer;
-extern const VMStateInfo vmstate_info_unused_buffer;
-extern const VMStateInfo vmstate_info_bitmap;
-
-#define type_check_2darray(t1,t2,n,m) ((t1(*)[n][m])0 - (t2*)0)
-#define type_check_array(t1,t2,n) ((t1(*)[n])0 - (t2*)0)
-#define type_check_pointer(t1,t2) ((t1**)0 - (t2*)0)
-
-#define vmstate_offset_value(_state, _field, _type)                  \
-    (offsetof(_state, _field) +                                      \
-     type_check(_type, typeof_field(_state, _field)))
-
-#define vmstate_offset_pointer(_state, _field, _type)                \
-    (offsetof(_state, _field) +                                      \
-     type_check_pointer(_type, typeof_field(_state, _field)))
-
-#define vmstate_offset_array(_state, _field, _type, _num)            \
-    (offsetof(_state, _field) +                                      \
-     type_check_array(_type, typeof_field(_state, _field), _num))
-
-#define vmstate_offset_2darray(_state, _field, _type, _n1, _n2)      \
-    (offsetof(_state, _field) +                                      \
-     type_check_2darray(_type, typeof_field(_state, _field), _n1, _n2))
-
-#define vmstate_offset_sub_array(_state, _field, _type, _start)      \
-    (offsetof(_state, _field[_start]))
-
-#define vmstate_offset_buffer(_state, _field)                        \
-    vmstate_offset_array(_state, _field, uint8_t,                    \
-                         sizeof(typeof_field(_state, _field)))
-
-#define VMSTATE_SINGLE_TEST(_field, _state, _test, _version, _info, _type) { \
-    .name         = (stringify(_field)),                             \
-    .version_id   = (_version),                                      \
-    .field_exists = (_test),                                         \
-    .size         = sizeof(_type),                                   \
-    .info         = &(_info),                                        \
-    .flags        = VMS_SINGLE,                                      \
-    .offset       = vmstate_offset_value(_state, _field, _type),     \
-}
-
-#define VMSTATE_POINTER(_field, _state, _version, _info, _type) {    \
-    .name       = (stringify(_field)),                               \
-    .version_id = (_version),                                        \
-    .info       = &(_info),                                          \
-    .size       = sizeof(_type),                                     \
-    .flags      = VMS_SINGLE|VMS_POINTER,                            \
-    .offset     = vmstate_offset_value(_state, _field, _type),       \
-}
-
-#define VMSTATE_POINTER_TEST(_field, _state, _test, _info, _type) {  \
-    .name       = (stringify(_field)),                               \
-    .info       = &(_info),                                          \
-    .field_exists = (_test),                                         \
-    .size       = sizeof(_type),                                     \
-    .flags      = VMS_SINGLE|VMS_POINTER,                            \
-    .offset     = vmstate_offset_value(_state, _field, _type),       \
-}
-
-#define VMSTATE_ARRAY(_field, _state, _num, _version, _info, _type) {\
-    .name       = (stringify(_field)),                               \
-    .version_id = (_version),                                        \
-    .num        = (_num),                                            \
-    .info       = &(_info),                                          \
-    .size       = sizeof(_type),                                     \
-    .flags      = VMS_ARRAY,                                         \
-    .offset     = vmstate_offset_array(_state, _field, _type, _num), \
-}
-
-#define VMSTATE_2DARRAY(_field, _state, _n1, _n2, _version, _info, _type) { \
-    .name       = (stringify(_field)),                                      \
-    .version_id = (_version),                                               \
-    .num        = (_n1) * (_n2),                                            \
-    .info       = &(_info),                                                 \
-    .size       = sizeof(_type),                                            \
-    .flags      = VMS_ARRAY,                                                \
-    .offset     = vmstate_offset_2darray(_state, _field, _type, _n1, _n2),  \
-}
-
-#define VMSTATE_ARRAY_TEST(_field, _state, _num, _test, _info, _type) {\
-    .name         = (stringify(_field)),                              \
-    .field_exists = (_test),                                          \
-    .num          = (_num),                                           \
-    .info         = &(_info),                                         \
-    .size         = sizeof(_type),                                    \
-    .flags        = VMS_ARRAY,                                        \
-    .offset       = vmstate_offset_array(_state, _field, _type, _num),\
-}
-
-#define VMSTATE_SUB_ARRAY(_field, _state, _start, _num, _version, _info, _type) { \
-    .name       = (stringify(_field)),                               \
-    .version_id = (_version),                                        \
-    .num        = (_num),                                            \
-    .info       = &(_info),                                          \
-    .size       = sizeof(_type),                                     \
-    .flags      = VMS_ARRAY,                                         \
-    .offset     = vmstate_offset_sub_array(_state, _field, _type, _start), \
-}
-
-#define VMSTATE_ARRAY_INT32_UNSAFE(_field, _state, _field_num, _info, _type) {\
-    .name       = (stringify(_field)),                               \
-    .num_offset = vmstate_offset_value(_state, _field_num, int32_t), \
-    .info       = &(_info),                                          \
-    .size       = sizeof(_type),                                     \
-    .flags      = VMS_VARRAY_INT32,                                  \
-    .offset     = offsetof(_state, _field),                          \
-}
-
-#define VMSTATE_VARRAY_INT32(_field, _state, _field_num, _version, _info, _type) {\
-    .name       = (stringify(_field)),                               \
-    .version_id = (_version),                                        \
-    .num_offset = vmstate_offset_value(_state, _field_num, int32_t), \
-    .info       = &(_info),                                          \
-    .size       = sizeof(_type),                                     \
-    .flags      = VMS_VARRAY_INT32|VMS_POINTER,                      \
-    .offset     = vmstate_offset_pointer(_state, _field, _type),     \
-}
-
-#define VMSTATE_VARRAY_UINT32(_field, _state, _field_num, _version, _info, _type) {\
-    .name       = (stringify(_field)),                               \
-    .version_id = (_version),                                        \
-    .num_offset = vmstate_offset_value(_state, _field_num, uint32_t),\
-    .info       = &(_info),                                          \
-    .size       = sizeof(_type),                                     \
-    .flags      = VMS_VARRAY_UINT32|VMS_POINTER,                     \
-    .offset     = vmstate_offset_pointer(_state, _field, _type),     \
-}
-
-#define VMSTATE_VARRAY_UINT16_UNSAFE(_field, _state, _field_num, _version, _info, _type) {\
-    .name       = (stringify(_field)),                               \
-    .version_id = (_version),                                        \
-    .num_offset = vmstate_offset_value(_state, _field_num, uint16_t),\
-    .info       = &(_info),                                          \
-    .size       = sizeof(_type),                                     \
-    .flags      = VMS_VARRAY_UINT16,                                 \
-    .offset     = offsetof(_state, _field),                          \
-}
-
-#define VMSTATE_STRUCT_TEST(_field, _state, _test, _version, _vmsd, _type) { \
-    .name         = (stringify(_field)),                             \
-    .version_id   = (_version),                                      \
-    .field_exists = (_test),                                         \
-    .vmsd         = &(_vmsd),                                        \
-    .size         = sizeof(_type),                                   \
-    .flags        = VMS_STRUCT,                                      \
-    .offset       = vmstate_offset_value(_state, _field, _type),     \
-}
-
-#define VMSTATE_STRUCT_POINTER_TEST(_field, _state, _test, _vmsd, _type) { \
-    .name         = (stringify(_field)),                             \
-    .field_exists = (_test),                                         \
-    .vmsd         = &(_vmsd),                                        \
-    .size         = sizeof(_type),                                   \
-    .flags        = VMS_STRUCT|VMS_POINTER,                          \
-    .offset       = vmstate_offset_value(_state, _field, _type),     \
-}
-
-#define VMSTATE_ARRAY_OF_POINTER(_field, _state, _num, _version, _info, _type) {\
-    .name       = (stringify(_field)),                               \
-    .version_id = (_version),                                        \
-    .num        = (_num),                                            \
-    .info       = &(_info),                                          \
-    .size       = sizeof(_type),                                     \
-    .flags      = VMS_ARRAY|VMS_ARRAY_OF_POINTER,                    \
-    .offset     = vmstate_offset_array(_state, _field, _type, _num), \
-}
-
-#define VMSTATE_STRUCT_ARRAY_TEST(_field, _state, _num, _test, _version, _vmsd, _type) { \
-    .name         = (stringify(_field)),                             \
-    .num          = (_num),                                          \
-    .field_exists = (_test),                                         \
-    .version_id   = (_version),                                      \
-    .vmsd         = &(_vmsd),                                        \
-    .size         = sizeof(_type),                                   \
-    .flags        = VMS_STRUCT|VMS_ARRAY,                            \
-    .offset       = vmstate_offset_array(_state, _field, _type, _num),\
-}
-
-#define VMSTATE_STRUCT_VARRAY_UINT8(_field, _state, _field_num, _version, _vmsd, _type) { \
-    .name       = (stringify(_field)),                               \
-    .num_offset = vmstate_offset_value(_state, _field_num, uint8_t), \
-    .version_id = (_version),                                        \
-    .vmsd       = &(_vmsd),                                          \
-    .size       = sizeof(_type),                                     \
-    .flags      = VMS_STRUCT|VMS_VARRAY_UINT8,                       \
-    .offset     = offsetof(_state, _field),                          \
-}
-
-#define VMSTATE_STRUCT_VARRAY_POINTER_INT32(_field, _state, _field_num, _vmsd, _type) { \
-    .name       = (stringify(_field)),                               \
-    .version_id = 0,                                                 \
-    .num_offset = vmstate_offset_value(_state, _field_num, int32_t), \
-    .size       = sizeof(_type),                                     \
-    .vmsd       = &(_vmsd),                                          \
-    .flags      = VMS_POINTER | VMS_VARRAY_INT32 | VMS_STRUCT,       \
-    .offset     = vmstate_offset_pointer(_state, _field, _type),     \
-}
-
-#define VMSTATE_STRUCT_VARRAY_POINTER_UINT32(_field, _state, _field_num, _vmsd, _type) { \
-    .name       = (stringify(_field)),                               \
-    .version_id = 0,                                                 \
-    .num_offset = vmstate_offset_value(_state, _field_num, uint32_t),\
-    .size       = sizeof(_type),                                     \
-    .vmsd       = &(_vmsd),                                          \
-    .flags      = VMS_POINTER | VMS_VARRAY_INT32 | VMS_STRUCT,       \
-    .offset     = vmstate_offset_pointer(_state, _field, _type),     \
-}
-
-#define VMSTATE_STRUCT_VARRAY_POINTER_UINT16(_field, _state, _field_num, _vmsd, _type) { \
-    .name       = (stringify(_field)),                               \
-    .version_id = 0,                                                 \
-    .num_offset = vmstate_offset_value(_state, _field_num, uint16_t),\
-    .size       = sizeof(_type),                                     \
-    .vmsd       = &(_vmsd),                                          \
-    .flags      = VMS_POINTER | VMS_VARRAY_UINT16 | VMS_STRUCT,      \
-    .offset     = vmstate_offset_pointer(_state, _field, _type),     \
-}
-
-#define VMSTATE_STRUCT_VARRAY_INT32(_field, _state, _field_num, _version, _vmsd, _type) { \
-    .name       = (stringify(_field)),                               \
-    .num_offset = vmstate_offset_value(_state, _field_num, int32_t), \
-    .version_id = (_version),                                        \
-    .vmsd       = &(_vmsd),                                          \
-    .size       = sizeof(_type),                                     \
-    .flags      = VMS_STRUCT|VMS_VARRAY_INT32,                       \
-    .offset     = offsetof(_state, _field),                          \
-}
-
-#define VMSTATE_STRUCT_VARRAY_UINT32(_field, _state, _field_num, _version, _vmsd, _type) { \
-    .name       = (stringify(_field)),                               \
-    .num_offset = vmstate_offset_value(_state, _field_num, uint32_t), \
-    .version_id = (_version),                                        \
-    .vmsd       = &(_vmsd),                                          \
-    .size       = sizeof(_type),                                     \
-    .flags      = VMS_STRUCT|VMS_VARRAY_UINT32,                      \
-    .offset     = offsetof(_state, _field),                          \
-}
-
-#define VMSTATE_STATIC_BUFFER(_field, _state, _version, _test, _start, _size) { \
-    .name         = (stringify(_field)),                             \
-    .version_id   = (_version),                                      \
-    .field_exists = (_test),                                         \
-    .size         = (_size - _start),                                \
-    .info         = &vmstate_info_buffer,                            \
-    .flags        = VMS_BUFFER,                                      \
-    .offset       = vmstate_offset_buffer(_state, _field) + _start,  \
-}
-
-#define VMSTATE_VBUFFER_MULTIPLY(_field, _state, _version, _test, _start, _field_size, _multiply) { \
-    .name         = (stringify(_field)),                             \
-    .version_id   = (_version),                                      \
-    .field_exists = (_test),                                         \
-    .size_offset  = vmstate_offset_value(_state, _field_size, uint32_t),\
-    .size         = (_multiply),                                      \
-    .info         = &vmstate_info_buffer,                            \
-    .flags        = VMS_VBUFFER|VMS_POINTER|VMS_MULTIPLY,            \
-    .offset       = offsetof(_state, _field),                        \
-    .start        = (_start),                                        \
-}
-
-#define VMSTATE_VBUFFER(_field, _state, _version, _test, _start, _field_size) { \
-    .name         = (stringify(_field)),                             \
-    .version_id   = (_version),                                      \
-    .field_exists = (_test),                                         \
-    .size_offset  = vmstate_offset_value(_state, _field_size, int32_t),\
-    .info         = &vmstate_info_buffer,                            \
-    .flags        = VMS_VBUFFER|VMS_POINTER,                         \
-    .offset       = offsetof(_state, _field),                        \
-    .start        = (_start),                                        \
-}
-
-#define VMSTATE_VBUFFER_UINT32(_field, _state, _version, _test, _start, _field_size) { \
-    .name         = (stringify(_field)),                             \
-    .version_id   = (_version),                                      \
-    .field_exists = (_test),                                         \
-    .size_offset  = vmstate_offset_value(_state, _field_size, uint32_t),\
-    .info         = &vmstate_info_buffer,                            \
-    .flags        = VMS_VBUFFER|VMS_POINTER,                         \
-    .offset       = offsetof(_state, _field),                        \
-    .start        = (_start),                                        \
-}
-
-#define VMSTATE_BUFFER_UNSAFE_INFO(_field, _state, _version, _info, _size) { \
-    .name       = (stringify(_field)),                               \
-    .version_id = (_version),                                        \
-    .size       = (_size),                                           \
-    .info       = &(_info),                                          \
-    .flags      = VMS_BUFFER,                                        \
-    .offset     = offsetof(_state, _field),                          \
-}
-
-#define VMSTATE_BUFFER_POINTER_UNSAFE(_field, _state, _version, _size) { \
-    .name       = (stringify(_field)),                               \
-    .version_id = (_version),                                        \
-    .size       = (_size),                                           \
-    .info       = &vmstate_info_buffer,                              \
-    .flags      = VMS_BUFFER|VMS_POINTER,                            \
-    .offset     = offsetof(_state, _field),                          \
-}
-
-#define VMSTATE_UNUSED_BUFFER(_test, _version, _size) {              \
-    .name         = "unused",                                        \
-    .field_exists = (_test),                                         \
-    .version_id   = (_version),                                      \
-    .size         = (_size),                                         \
-    .info         = &vmstate_info_unused_buffer,                     \
-    .flags        = VMS_BUFFER,                                      \
-}
-
-/* _field_size should be a int32_t field in the _state struct giving the
- * size of the bitmap _field in bits.
- */
-#define VMSTATE_BITMAP(_field, _state, _version, _field_size) {      \
-    .name         = (stringify(_field)),                             \
-    .version_id   = (_version),                                      \
-    .size_offset  = vmstate_offset_value(_state, _field_size, int32_t),\
-    .info         = &vmstate_info_bitmap,                            \
-    .flags        = VMS_VBUFFER|VMS_POINTER,                         \
-    .offset       = offsetof(_state, _field),                        \
-}
-
-/* _f : field name
-   _f_n : num of elements field_name
-   _n : num of elements
-   _s : struct state name
-   _v : version
-*/
-
-#define VMSTATE_SINGLE(_field, _state, _version, _info, _type)        \
-    VMSTATE_SINGLE_TEST(_field, _state, NULL, _version, _info, _type)
-
-#define VMSTATE_STRUCT(_field, _state, _version, _vmsd, _type)        \
-    VMSTATE_STRUCT_TEST(_field, _state, NULL, _version, _vmsd, _type)
-
-#define VMSTATE_STRUCT_POINTER(_field, _state, _vmsd, _type)          \
-    VMSTATE_STRUCT_POINTER_TEST(_field, _state, NULL, _vmsd, _type)
-
-#define VMSTATE_STRUCT_ARRAY(_field, _state, _num, _version, _vmsd, _type) \
-    VMSTATE_STRUCT_ARRAY_TEST(_field, _state, _num, NULL, _version,   \
-            _vmsd, _type)
-
-#define VMSTATE_BOOL_V(_f, _s, _v)                                    \
-    VMSTATE_SINGLE(_f, _s, _v, vmstate_info_bool, bool)
-
-#define VMSTATE_INT8_V(_f, _s, _v)                                    \
-    VMSTATE_SINGLE(_f, _s, _v, vmstate_info_int8, int8_t)
-#define VMSTATE_INT16_V(_f, _s, _v)                                   \
-    VMSTATE_SINGLE(_f, _s, _v, vmstate_info_int16, int16_t)
-#define VMSTATE_INT32_V(_f, _s, _v)                                   \
-    VMSTATE_SINGLE(_f, _s, _v, vmstate_info_int32, int32_t)
-#define VMSTATE_INT64_V(_f, _s, _v)                                   \
-    VMSTATE_SINGLE(_f, _s, _v, vmstate_info_int64, int64_t)
-
-#define VMSTATE_UINT8_V(_f, _s, _v)                                   \
-    VMSTATE_SINGLE(_f, _s, _v, vmstate_info_uint8, uint8_t)
-#define VMSTATE_UINT16_V(_f, _s, _v)                                  \
-    VMSTATE_SINGLE(_f, _s, _v, vmstate_info_uint16, uint16_t)
-#define VMSTATE_UINT32_V(_f, _s, _v)                                  \
-    VMSTATE_SINGLE(_f, _s, _v, vmstate_info_uint32, uint32_t)
-#define VMSTATE_UINT64_V(_f, _s, _v)                                  \
-    VMSTATE_SINGLE(_f, _s, _v, vmstate_info_uint64, uint64_t)
-
-#define VMSTATE_BOOL(_f, _s)                                          \
-    VMSTATE_BOOL_V(_f, _s, 0)
-
-#define VMSTATE_INT8(_f, _s)                                          \
-    VMSTATE_INT8_V(_f, _s, 0)
-#define VMSTATE_INT16(_f, _s)                                         \
-    VMSTATE_INT16_V(_f, _s, 0)
-#define VMSTATE_INT32(_f, _s)                                         \
-    VMSTATE_INT32_V(_f, _s, 0)
-#define VMSTATE_INT64(_f, _s)                                         \
-    VMSTATE_INT64_V(_f, _s, 0)
-
-#define VMSTATE_UINT8(_f, _s)                                         \
-    VMSTATE_UINT8_V(_f, _s, 0)
-#define VMSTATE_UINT16(_f, _s)                                        \
-    VMSTATE_UINT16_V(_f, _s, 0)
-#define VMSTATE_UINT32(_f, _s)                                        \
-    VMSTATE_UINT32_V(_f, _s, 0)
-#define VMSTATE_UINT64(_f, _s)                                        \
-    VMSTATE_UINT64_V(_f, _s, 0)
-
-#define VMSTATE_UINT8_EQUAL(_f, _s)                                   \
-    VMSTATE_SINGLE(_f, _s, 0, vmstate_info_uint8_equal, uint8_t)
-
-#define VMSTATE_UINT16_EQUAL(_f, _s)                                  \
-    VMSTATE_SINGLE(_f, _s, 0, vmstate_info_uint16_equal, uint16_t)
-
-#define VMSTATE_UINT16_EQUAL_V(_f, _s, _v)                            \
-    VMSTATE_SINGLE(_f, _s, _v, vmstate_info_uint16_equal, uint16_t)
-
-#define VMSTATE_INT32_EQUAL(_f, _s)                                   \
-    VMSTATE_SINGLE(_f, _s, 0, vmstate_info_int32_equal, int32_t)
-
-#define VMSTATE_UINT32_EQUAL_V(_f, _s, _v)                            \
-    VMSTATE_SINGLE(_f, _s, _v, vmstate_info_uint32_equal, uint32_t)
-
-#define VMSTATE_UINT32_EQUAL(_f, _s)                                  \
-    VMSTATE_UINT32_EQUAL_V(_f, _s, 0)
-
-#define VMSTATE_UINT64_EQUAL_V(_f, _s, _v)                            \
-    VMSTATE_SINGLE(_f, _s, _v, vmstate_info_uint64_equal, uint64_t)
-
-#define VMSTATE_UINT64_EQUAL(_f, _s)                                  \
-    VMSTATE_UINT64_EQUAL_V(_f, _s, 0)
-
-#define VMSTATE_INT32_LE(_f, _s)                                   \
-    VMSTATE_SINGLE(_f, _s, 0, vmstate_info_int32_le, int32_t)
-
-#define VMSTATE_UINT8_TEST(_f, _s, _t)                               \
-    VMSTATE_SINGLE_TEST(_f, _s, _t, 0, vmstate_info_uint8, uint8_t)
-
-#define VMSTATE_UINT16_TEST(_f, _s, _t)                               \
-    VMSTATE_SINGLE_TEST(_f, _s, _t, 0, vmstate_info_uint16, uint16_t)
-
-#define VMSTATE_UINT32_TEST(_f, _s, _t)                                  \
-    VMSTATE_SINGLE_TEST(_f, _s, _t, 0, vmstate_info_uint32, uint32_t)
-
-
-#define VMSTATE_FLOAT64_V(_f, _s, _v)                                 \
-    VMSTATE_SINGLE(_f, _s, _v, vmstate_info_float64, float64)
-
-#define VMSTATE_FLOAT64(_f, _s)                                       \
-    VMSTATE_FLOAT64_V(_f, _s, 0)
-
-#define VMSTATE_TIMER_TEST(_f, _s, _test)                             \
-    VMSTATE_POINTER_TEST(_f, _s, _test, vmstate_info_timer, QEMUTimer *)
-
-#define VMSTATE_TIMER_V(_f, _s, _v)                                   \
-    VMSTATE_POINTER(_f, _s, _v, vmstate_info_timer, QEMUTimer *)
-
-#define VMSTATE_TIMER(_f, _s)                                         \
-    VMSTATE_TIMER_V(_f, _s, 0)
-
-#define VMSTATE_TIMER_ARRAY(_f, _s, _n)                              \
-    VMSTATE_ARRAY_OF_POINTER(_f, _s, _n, 0, vmstate_info_timer, QEMUTimer *)
-
-#define VMSTATE_BOOL_ARRAY_V(_f, _s, _n, _v)                         \
-    VMSTATE_ARRAY(_f, _s, _n, _v, vmstate_info_bool, bool)
-
-#define VMSTATE_BOOL_ARRAY(_f, _s, _n)                               \
-    VMSTATE_BOOL_ARRAY_V(_f, _s, _n, 0)
-
-#define VMSTATE_UINT16_ARRAY_V(_f, _s, _n, _v)                         \
-    VMSTATE_ARRAY(_f, _s, _n, _v, vmstate_info_uint16, uint16_t)
-
-#define VMSTATE_UINT16_2DARRAY_V(_f, _s, _n1, _n2, _v)                \
-    VMSTATE_2DARRAY(_f, _s, _n1, _n2, _v, vmstate_info_uint16, uint16_t)
-
-#define VMSTATE_UINT16_ARRAY(_f, _s, _n)                               \
-    VMSTATE_UINT16_ARRAY_V(_f, _s, _n, 0)
-
-#define VMSTATE_UINT16_2DARRAY(_f, _s, _n1, _n2)                      \
-    VMSTATE_UINT16_2DARRAY_V(_f, _s, _n1, _n2, 0)
-
-#define VMSTATE_UINT8_2DARRAY_V(_f, _s, _n1, _n2, _v)                 \
-    VMSTATE_2DARRAY(_f, _s, _n1, _n2, _v, vmstate_info_uint8, uint8_t)
-
-#define VMSTATE_UINT8_ARRAY_V(_f, _s, _n, _v)                         \
-    VMSTATE_ARRAY(_f, _s, _n, _v, vmstate_info_uint8, uint8_t)
-
-#define VMSTATE_UINT8_ARRAY(_f, _s, _n)                               \
-    VMSTATE_UINT8_ARRAY_V(_f, _s, _n, 0)
-
-#define VMSTATE_UINT8_2DARRAY(_f, _s, _n1, _n2)                       \
-    VMSTATE_UINT8_2DARRAY_V(_f, _s, _n1, _n2, 0)
-
-#define VMSTATE_UINT32_ARRAY_V(_f, _s, _n, _v)                        \
-    VMSTATE_ARRAY(_f, _s, _n, _v, vmstate_info_uint32, uint32_t)
-
-#define VMSTATE_UINT32_ARRAY(_f, _s, _n)                              \
-    VMSTATE_UINT32_ARRAY_V(_f, _s, _n, 0)
-
-#define VMSTATE_UINT64_ARRAY_V(_f, _s, _n, _v)                        \
-    VMSTATE_ARRAY(_f, _s, _n, _v, vmstate_info_uint64, uint64_t)
-
-#define VMSTATE_UINT64_ARRAY(_f, _s, _n)                              \
-    VMSTATE_UINT64_ARRAY_V(_f, _s, _n, 0)
-
-#define VMSTATE_INT16_ARRAY_V(_f, _s, _n, _v)                         \
-    VMSTATE_ARRAY(_f, _s, _n, _v, vmstate_info_int16, int16_t)
-
-#define VMSTATE_INT16_ARRAY(_f, _s, _n)                               \
-    VMSTATE_INT16_ARRAY_V(_f, _s, _n, 0)
-
-#define VMSTATE_INT32_ARRAY_V(_f, _s, _n, _v)                         \
-    VMSTATE_ARRAY(_f, _s, _n, _v, vmstate_info_int32, int32_t)
-
-#define VMSTATE_INT32_ARRAY(_f, _s, _n)                               \
-    VMSTATE_INT32_ARRAY_V(_f, _s, _n, 0)
-
-#define VMSTATE_UINT32_SUB_ARRAY(_f, _s, _start, _num)                \
-    VMSTATE_SUB_ARRAY(_f, _s, _start, _num, 0, vmstate_info_uint32, uint32_t)
-
-#define VMSTATE_UINT32_ARRAY(_f, _s, _n)                              \
-    VMSTATE_UINT32_ARRAY_V(_f, _s, _n, 0)
-
-#define VMSTATE_INT64_ARRAY_V(_f, _s, _n, _v)                         \
-    VMSTATE_ARRAY(_f, _s, _n, _v, vmstate_info_int64, int64_t)
-
-#define VMSTATE_INT64_ARRAY(_f, _s, _n)                               \
-    VMSTATE_INT64_ARRAY_V(_f, _s, _n, 0)
-
-#define VMSTATE_FLOAT64_ARRAY_V(_f, _s, _n, _v)                       \
-    VMSTATE_ARRAY(_f, _s, _n, _v, vmstate_info_float64, float64)
-
-#define VMSTATE_FLOAT64_ARRAY(_f, _s, _n)                             \
-    VMSTATE_FLOAT64_ARRAY_V(_f, _s, _n, 0)
-
-#define VMSTATE_BUFFER_V(_f, _s, _v)                                  \
-    VMSTATE_STATIC_BUFFER(_f, _s, _v, NULL, 0, sizeof(typeof_field(_s, _f)))
-
-#define VMSTATE_BUFFER(_f, _s)                                        \
-    VMSTATE_BUFFER_V(_f, _s, 0)
-
-#define VMSTATE_PARTIAL_BUFFER(_f, _s, _size)                         \
-    VMSTATE_STATIC_BUFFER(_f, _s, 0, NULL, 0, _size)
-
-#define VMSTATE_BUFFER_START_MIDDLE(_f, _s, _start) \
-    VMSTATE_STATIC_BUFFER(_f, _s, 0, NULL, _start, sizeof(typeof_field(_s, _f)))
-
-#define VMSTATE_PARTIAL_VBUFFER(_f, _s, _size)                        \
-    VMSTATE_VBUFFER(_f, _s, 0, NULL, 0, _size)
-
-#define VMSTATE_PARTIAL_VBUFFER_UINT32(_f, _s, _size)                        \
-    VMSTATE_VBUFFER_UINT32(_f, _s, 0, NULL, 0, _size)
-
-#define VMSTATE_SUB_VBUFFER(_f, _s, _start, _size)                    \
-    VMSTATE_VBUFFER(_f, _s, 0, NULL, _start, _size)
-
-#define VMSTATE_BUFFER_TEST(_f, _s, _test)                            \
-    VMSTATE_STATIC_BUFFER(_f, _s, 0, _test, 0, sizeof(typeof_field(_s, _f)))
-
-#define VMSTATE_BUFFER_UNSAFE(_field, _state, _version, _size)        \
-    VMSTATE_BUFFER_UNSAFE_INFO(_field, _state, _version, vmstate_info_buffer, _size)
-
-#define VMSTATE_UNUSED_V(_v, _size)                                   \
-    VMSTATE_UNUSED_BUFFER(NULL, _v, _size)
-
-#define VMSTATE_UNUSED(_size)                                         \
-    VMSTATE_UNUSED_V(0, _size)
-
-#define VMSTATE_UNUSED_TEST(_test, _size)                             \
-    VMSTATE_UNUSED_BUFFER(_test, 0, _size)
-
-#define VMSTATE_END_OF_LIST()                                         \
-    {}
-
-int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
-                       void *opaque, int version_id);
-void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
-                        void *opaque);
-
-int vmstate_register_with_alias_id(DeviceState *dev, int instance_id,
-                                   const VMStateDescription *vmsd,
-                                   void *base, int alias_id,
-                                   int required_for_version);
-
-static inline int vmstate_register(DeviceState *dev, int instance_id,
-                                   const VMStateDescription *vmsd,
-                                   void *opaque)
-{
-    return vmstate_register_with_alias_id(dev, instance_id, vmsd,
-                                          opaque, -1, 0);
-}
-
-void vmstate_unregister(DeviceState *dev, const VMStateDescription *vmsd,
-                        void *opaque);
-
-struct MemoryRegion;
-void vmstate_register_ram(struct MemoryRegion *memory, DeviceState *dev);
-void vmstate_unregister_ram(struct MemoryRegion *memory, DeviceState *dev);
-void vmstate_register_ram_global(struct MemoryRegion *memory);
-
-#endif
diff --git a/contrib/qemu/include/monitor/monitor.h b/contrib/qemu/include/monitor/monitor.h
deleted file mode 100644
index 1942cc42fe2..00000000000
--- a/contrib/qemu/include/monitor/monitor.h
+++ /dev/null
@@ -1,104 +0,0 @@
-#ifndef MONITOR_H
-#define MONITOR_H
-
-#include "qemu-common.h"
-#include "qapi/qmp/qerror.h"
-#include "qapi/qmp/qdict.h"
-#include "block/block.h"
-#include "monitor/readline.h"
-
-extern Monitor *cur_mon;
-extern Monitor *default_mon;
-
-/* flags for monitor_init */
-#define MONITOR_IS_DEFAULT    0x01
-#define MONITOR_USE_READLINE  0x02
-#define MONITOR_USE_CONTROL   0x04
-#define MONITOR_USE_PRETTY    0x08
-
-/* flags for monitor commands */
-#define MONITOR_CMD_ASYNC       0x0001
-
-/* QMP events */
-typedef enum MonitorEvent {
-    QEVENT_SHUTDOWN,
-    QEVENT_RESET,
-    QEVENT_POWERDOWN,
-    QEVENT_STOP,
-    QEVENT_RESUME,
-    QEVENT_VNC_CONNECTED,
-    QEVENT_VNC_INITIALIZED,
-    QEVENT_VNC_DISCONNECTED,
-    QEVENT_BLOCK_IO_ERROR,
-    QEVENT_RTC_CHANGE,
-    QEVENT_WATCHDOG,
-    QEVENT_SPICE_CONNECTED,
-    QEVENT_SPICE_INITIALIZED,
-    QEVENT_SPICE_DISCONNECTED,
-    QEVENT_BLOCK_JOB_COMPLETED,
-    QEVENT_BLOCK_JOB_CANCELLED,
-    QEVENT_BLOCK_JOB_ERROR,
-    QEVENT_BLOCK_JOB_READY,
-    QEVENT_DEVICE_DELETED,
-    QEVENT_DEVICE_TRAY_MOVED,
-    QEVENT_NIC_RX_FILTER_CHANGED,
-    QEVENT_SUSPEND,
-    QEVENT_SUSPEND_DISK,
-    QEVENT_WAKEUP,
-    QEVENT_BALLOON_CHANGE,
-    QEVENT_SPICE_MIGRATE_COMPLETED,
-    QEVENT_GUEST_PANICKED,
-
-    /* Add to 'monitor_event_names' array in monitor.c when
-     * defining new events here */
-
-    QEVENT_MAX,
-} MonitorEvent;
-
-int monitor_cur_is_qmp(void);
-
-void monitor_protocol_event(MonitorEvent event, QObject *data);
-void monitor_init(CharDriverState *chr, int flags);
-
-int monitor_suspend(Monitor *mon);
-void monitor_resume(Monitor *mon);
-
-int monitor_read_bdrv_key_start(Monitor *mon, BlockDriverState *bs,
-                                BlockDriverCompletionFunc *completion_cb,
-                                void *opaque);
-int monitor_read_block_device_key(Monitor *mon, const char *device,
-                                  BlockDriverCompletionFunc *completion_cb,
-                                  void *opaque);
-
-int monitor_get_fd(Monitor *mon, const char *fdname, Error **errp);
-int monitor_handle_fd_param(Monitor *mon, const char *fdname);
-
-void monitor_vprintf(Monitor *mon, const char *fmt, va_list ap)
-    GCC_FMT_ATTR(2, 0);
-void monitor_printf(Monitor *mon, const char *fmt, ...) GCC_FMT_ATTR(2, 3);
-void monitor_print_filename(Monitor *mon, const char *filename);
-void monitor_flush(Monitor *mon);
-int monitor_set_cpu(int cpu_index);
-int monitor_get_cpu_index(void);
-
-typedef void (MonitorCompletion)(void *opaque, QObject *ret_data);
-
-void monitor_set_error(Monitor *mon, QError *qerror);
-void monitor_read_command(Monitor *mon, int show_prompt);
-ReadLineState *monitor_get_rs(Monitor *mon);
-int monitor_read_password(Monitor *mon, ReadLineFunc *readline_func,
-                          void *opaque);
-
-int qmp_qom_set(Monitor *mon, const QDict *qdict, QObject **ret);
-
-int qmp_qom_get(Monitor *mon, const QDict *qdict, QObject **ret);
-
-AddfdInfo *monitor_fdset_add_fd(int fd, bool has_fdset_id, int64_t fdset_id,
-                                bool has_opaque, const char *opaque,
-                                Error **errp);
-int monitor_fdset_get_fd(int64_t fdset_id, int flags);
-int monitor_fdset_dup_fd_add(int64_t fdset_id, int dup_fd);
-int monitor_fdset_dup_fd_remove(int dup_fd);
-int monitor_fdset_dup_fd_find(int dup_fd);
-
-#endif /* !MONITOR_H */
diff --git a/contrib/qemu/include/monitor/readline.h b/contrib/qemu/include/monitor/readline.h
deleted file mode 100644
index fc9806ecf1a..00000000000
--- a/contrib/qemu/include/monitor/readline.h
+++ /dev/null
@@ -1,55 +0,0 @@
-#ifndef READLINE_H
-#define READLINE_H
-
-#include "qemu-common.h"
-
-#define READLINE_CMD_BUF_SIZE 4095
-#define READLINE_MAX_CMDS 64
-#define READLINE_MAX_COMPLETIONS 256
-
-typedef void ReadLineFunc(Monitor *mon, const char *str, void *opaque);
-typedef void ReadLineCompletionFunc(const char *cmdline);
-
-typedef struct ReadLineState {
-    char cmd_buf[READLINE_CMD_BUF_SIZE + 1];
-    int cmd_buf_index;
-    int cmd_buf_size;
-
-    char last_cmd_buf[READLINE_CMD_BUF_SIZE + 1];
-    int last_cmd_buf_index;
-    int last_cmd_buf_size;
-
-    int esc_state;
-    int esc_param;
-
-    char *history[READLINE_MAX_CMDS];
-    int hist_entry;
-
-    ReadLineCompletionFunc *completion_finder;
-    char *completions[READLINE_MAX_COMPLETIONS];
-    int nb_completions;
-    int completion_index;
-
-    ReadLineFunc *readline_func;
-    void *readline_opaque;
-    int read_password;
-    char prompt[256];
-    Monitor *mon;
-} ReadLineState;
-
-void readline_add_completion(ReadLineState *rs, const char *str);
-void readline_set_completion_index(ReadLineState *rs, int completion_index);
-
-const char *readline_get_history(ReadLineState *rs, unsigned int index);
-
-void readline_handle_byte(ReadLineState *rs, int ch);
-
-void readline_start(ReadLineState *rs, const char *prompt, int read_password,
-                    ReadLineFunc *readline_func, void *opaque);
-void readline_restart(ReadLineState *rs);
-void readline_show_prompt(ReadLineState *rs);
-
-ReadLineState *readline_init(Monitor *mon,
-                             ReadLineCompletionFunc *completion_finder);
-
-#endif /* !READLINE_H */
diff --git a/contrib/qemu/include/qapi/error.h b/contrib/qemu/include/qapi/error.h
deleted file mode 100644
index ffd1cea4772..00000000000
--- a/contrib/qemu/include/qapi/error.h
+++ /dev/null
@@ -1,85 +0,0 @@
-/*
- * QEMU Error Objects
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.  See
- * the COPYING.LIB file in the top-level directory.
- */
-#ifndef ERROR_H
-#define ERROR_H
-
-#include "qemu/compiler.h"
-#include "qapi-types.h"
-#include <stdbool.h>
-
-/**
- * A class representing internal errors within QEMU.  An error has a ErrorClass
- * code and a human message.
- */
-typedef struct Error Error;
-
-/**
- * Set an indirect pointer to an error given a ErrorClass value and a
- * printf-style human message.  This function is not meant to be used outside
- * of QEMU.
- */
-void error_set(Error **err, ErrorClass err_class, const char *fmt, ...) GCC_FMT_ATTR(3, 4);
-
-/**
- * Set an indirect pointer to an error given a ErrorClass value and a
- * printf-style human message, followed by a strerror() string if
- * @os_error is not zero.
- */
-void error_set_errno(Error **err, int os_error, ErrorClass err_class, const char *fmt, ...) GCC_FMT_ATTR(4, 5);
-
-/**
- * Same as error_set(), but sets a generic error
- */
-#define error_setg(err, fmt, ...) \
-    error_set(err, ERROR_CLASS_GENERIC_ERROR, fmt, ## __VA_ARGS__)
-#define error_setg_errno(err, os_error, fmt, ...) \
-    error_set_errno(err, os_error, ERROR_CLASS_GENERIC_ERROR, fmt, ## __VA_ARGS__)
-
-/**
- * Helper for open() errors
- */
-void error_setg_file_open(Error **errp, int os_errno, const char *filename);
-
-/**
- * Returns true if an indirect pointer to an error is pointing to a valid
- * error object.
- */
-bool error_is_set(Error **err);
-
-/*
- * Get the error class of an error object.
- */
-ErrorClass error_get_class(const Error *err);
-
-/**
- * Returns an exact copy of the error passed as an argument.
- */
-Error *error_copy(const Error *err);
-
-/**
- * Get a human readable representation of an error object.
- */
-const char *error_get_pretty(Error *err);
-
-/**
- * Propagate an error to an indirect pointer to an error.  This function will
- * always transfer ownership of the error reference and handles the case where
- * dst_err is NULL correctly.  Errors after the first are discarded.
- */
-void error_propagate(Error **dst_err, Error *local_err);
-
-/**
- * Free an error object.
- */
-void error_free(Error *err);
-
-#endif
diff --git a/contrib/qemu/include/qapi/qmp/json-lexer.h b/contrib/qemu/include/qapi/qmp/json-lexer.h
deleted file mode 100644
index cdff0460a83..00000000000
--- a/contrib/qemu/include/qapi/qmp/json-lexer.h
+++ /dev/null
@@ -1,51 +0,0 @@
-/*
- * JSON lexer
- *
- * Copyright IBM, Corp. 2009
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#ifndef QEMU_JSON_LEXER_H
-#define QEMU_JSON_LEXER_H
-
-#include "qapi/qmp/qstring.h"
-#include "qapi/qmp/qlist.h"
-
-typedef enum json_token_type {
-    JSON_OPERATOR = 100,
-    JSON_INTEGER,
-    JSON_FLOAT,
-    JSON_KEYWORD,
-    JSON_STRING,
-    JSON_ESCAPE,
-    JSON_SKIP,
-    JSON_ERROR,
-} JSONTokenType;
-
-typedef struct JSONLexer JSONLexer;
-
-typedef void (JSONLexerEmitter)(JSONLexer *, QString *, JSONTokenType, int x, int y);
-
-struct JSONLexer
-{
-    JSONLexerEmitter *emit;
-    int state;
-    QString *token;
-    int x, y;
-};
-
-void json_lexer_init(JSONLexer *lexer, JSONLexerEmitter func);
-
-int json_lexer_feed(JSONLexer *lexer, const char *buffer, size_t size);
-
-int json_lexer_flush(JSONLexer *lexer);
-
-void json_lexer_destroy(JSONLexer *lexer);
-
-#endif
diff --git a/contrib/qemu/include/qapi/qmp/json-parser.h b/contrib/qemu/include/qapi/qmp/json-parser.h
deleted file mode 100644
index 44d88f34685..00000000000
--- a/contrib/qemu/include/qapi/qmp/json-parser.h
+++ /dev/null
@@ -1,24 +0,0 @@
-/*
- * JSON Parser 
- *
- * Copyright IBM, Corp. 2009
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#ifndef QEMU_JSON_PARSER_H
-#define QEMU_JSON_PARSER_H
-
-#include "qemu-common.h"
-#include "qapi/qmp/qlist.h"
-#include "qapi/error.h"
-
-QObject *json_parser_parse(QList *tokens, va_list *ap);
-QObject *json_parser_parse_err(QList *tokens, va_list *ap, Error **errp);
-
-#endif
diff --git a/contrib/qemu/include/qapi/qmp/json-streamer.h b/contrib/qemu/include/qapi/qmp/json-streamer.h
deleted file mode 100644
index 823f7d7fa42..00000000000
--- a/contrib/qemu/include/qapi/qmp/json-streamer.h
+++ /dev/null
@@ -1,40 +0,0 @@
-/*
- * JSON streaming support
- *
- * Copyright IBM, Corp. 2009
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#ifndef QEMU_JSON_STREAMER_H
-#define QEMU_JSON_STREAMER_H
-
-#include "qapi/qmp/qlist.h"
-#include "qapi/qmp/json-lexer.h"
-
-typedef struct JSONMessageParser
-{
-    void (*emit)(struct JSONMessageParser *parser, QList *tokens);
-    JSONLexer lexer;
-    int brace_count;
-    int bracket_count;
-    QList *tokens;
-    uint64_t token_size;
-} JSONMessageParser;
-
-void json_message_parser_init(JSONMessageParser *parser,
-                              void (*func)(JSONMessageParser *, QList *));
-
-int json_message_parser_feed(JSONMessageParser *parser,
-                             const char *buffer, size_t size);
-
-int json_message_parser_flush(JSONMessageParser *parser);
-
-void json_message_parser_destroy(JSONMessageParser *parser);
-
-#endif
diff --git a/contrib/qemu/include/qapi/qmp/qbool.h b/contrib/qemu/include/qapi/qmp/qbool.h
deleted file mode 100644
index c4eaab9bb9f..00000000000
--- a/contrib/qemu/include/qapi/qmp/qbool.h
+++ /dev/null
@@ -1,29 +0,0 @@
-/*
- * QBool Module
- *
- * Copyright IBM, Corp. 2009
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#ifndef QBOOL_H
-#define QBOOL_H
-
-#include <stdint.h>
-#include "qapi/qmp/qobject.h"
-
-typedef struct QBool {
-    QObject_HEAD;
-    int value;
-} QBool;
-
-QBool *qbool_from_int(int value);
-int qbool_get_int(const QBool *qb);
-QBool *qobject_to_qbool(const QObject *obj);
-
-#endif /* QBOOL_H */
diff --git a/contrib/qemu/include/qapi/qmp/qdict.h b/contrib/qemu/include/qapi/qmp/qdict.h
deleted file mode 100644
index 685b2e3fcbb..00000000000
--- a/contrib/qemu/include/qapi/qmp/qdict.h
+++ /dev/null
@@ -1,69 +0,0 @@
-/*
- * QDict Module
- *
- * Copyright (C) 2009 Red Hat Inc.
- *
- * Authors:
- *  Luiz Capitulino <lcapitulino@redhat.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- */
-
-#ifndef QDICT_H
-#define QDICT_H
-
-#include "qapi/qmp/qobject.h"
-#include "qapi/qmp/qlist.h"
-#include "qemu/queue.h"
-#include <stdint.h>
-
-#define QDICT_BUCKET_MAX 512
-
-typedef struct QDictEntry {
-    char *key;
-    QObject *value;
-    QLIST_ENTRY(QDictEntry) next;
-} QDictEntry;
-
-typedef struct QDict {
-    QObject_HEAD;
-    size_t size;
-    QLIST_HEAD(,QDictEntry) table[QDICT_BUCKET_MAX];
-} QDict;
-
-/* Object API */
-QDict *qdict_new(void);
-const char *qdict_entry_key(const QDictEntry *entry);
-QObject *qdict_entry_value(const QDictEntry *entry);
-size_t qdict_size(const QDict *qdict);
-void qdict_put_obj(QDict *qdict, const char *key, QObject *value);
-void qdict_del(QDict *qdict, const char *key);
-int qdict_haskey(const QDict *qdict, const char *key);
-QObject *qdict_get(const QDict *qdict, const char *key);
-QDict *qobject_to_qdict(const QObject *obj);
-void qdict_iter(const QDict *qdict,
-                void (*iter)(const char *key, QObject *obj, void *opaque),
-                void *opaque);
-const QDictEntry *qdict_first(const QDict *qdict);
-const QDictEntry *qdict_next(const QDict *qdict, const QDictEntry *entry);
-
-/* Helper to qdict_put_obj(), accepts any object */
-#define qdict_put(qdict, key, obj) \
-        qdict_put_obj(qdict, key, QOBJECT(obj))
-
-/* High level helpers */
-double qdict_get_double(const QDict *qdict, const char *key);
-int64_t qdict_get_int(const QDict *qdict, const char *key);
-int qdict_get_bool(const QDict *qdict, const char *key);
-QList *qdict_get_qlist(const QDict *qdict, const char *key);
-QDict *qdict_get_qdict(const QDict *qdict, const char *key);
-const char *qdict_get_str(const QDict *qdict, const char *key);
-int64_t qdict_get_try_int(const QDict *qdict, const char *key,
-                          int64_t def_value);
-int qdict_get_try_bool(const QDict *qdict, const char *key, int def_value);
-const char *qdict_get_try_str(const QDict *qdict, const char *key);
-
-QDict *qdict_clone_shallow(const QDict *src);
-
-#endif /* QDICT_H */
diff --git a/contrib/qemu/include/qapi/qmp/qerror.h b/contrib/qemu/include/qapi/qmp/qerror.h
deleted file mode 100644
index c30c2f6d7a3..00000000000
--- a/contrib/qemu/include/qapi/qmp/qerror.h
+++ /dev/null
@@ -1,249 +0,0 @@
-/*
- * QError Module
- *
- * Copyright (C) 2009 Red Hat Inc.
- *
- * Authors:
- *  Luiz Capitulino <lcapitulino@redhat.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- */
-#ifndef QERROR_H
-#define QERROR_H
-
-#include "qapi/qmp/qdict.h"
-#include "qapi/qmp/qstring.h"
-#include "qemu/error-report.h"
-#include "qapi/error.h"
-#include "qapi-types.h"
-#include <stdarg.h>
-
-typedef struct QError {
-    QObject_HEAD;
-    Location loc;
-    char *err_msg;
-    ErrorClass err_class;
-} QError;
-
-QString *qerror_human(const QError *qerror);
-void qerror_report(ErrorClass err_class, const char *fmt, ...) GCC_FMT_ATTR(2, 3);
-void qerror_report_err(Error *err);
-void assert_no_error(Error *err);
-
-/*
- * QError class list
- * Please keep the definitions in alphabetical order.
- * Use scripts/check-qerror.sh to check.
- */
-#define QERR_ADD_CLIENT_FAILED \
-    ERROR_CLASS_GENERIC_ERROR, "Could not add client"
-
-#define QERR_AMBIGUOUS_PATH \
-    ERROR_CLASS_GENERIC_ERROR, "Path '%s' does not uniquely identify an object"
-
-#define QERR_BAD_BUS_FOR_DEVICE \
-    ERROR_CLASS_GENERIC_ERROR, "Device '%s' can't go on a %s bus"
-
-#define QERR_BASE_NOT_FOUND \
-    ERROR_CLASS_GENERIC_ERROR, "Base '%s' not found"
-
-#define QERR_BLOCK_JOB_NOT_ACTIVE \
-    ERROR_CLASS_DEVICE_NOT_ACTIVE, "No active block job on device '%s'"
-
-#define QERR_BLOCK_JOB_PAUSED \
-    ERROR_CLASS_GENERIC_ERROR, "The block job for device '%s' is currently paused"
-
-#define QERR_BLOCK_JOB_NOT_READY \
-    ERROR_CLASS_GENERIC_ERROR, "The active block job for device '%s' cannot be completed"
-
-#define QERR_BLOCK_FORMAT_FEATURE_NOT_SUPPORTED \
-    ERROR_CLASS_GENERIC_ERROR, "Block format '%s' used by device '%s' does not support feature '%s'"
-
-#define QERR_BUFFER_OVERRUN \
-    ERROR_CLASS_GENERIC_ERROR, "An internal buffer overran"
-
-#define QERR_BUS_NO_HOTPLUG \
-    ERROR_CLASS_GENERIC_ERROR, "Bus '%s' does not support hotplugging"
-
-#define QERR_BUS_NOT_FOUND \
-    ERROR_CLASS_GENERIC_ERROR, "Bus '%s' not found"
-
-#define QERR_COMMAND_DISABLED \
-    ERROR_CLASS_GENERIC_ERROR, "The command %s has been disabled for this instance"
-
-#define QERR_COMMAND_NOT_FOUND \
-    ERROR_CLASS_COMMAND_NOT_FOUND, "The command %s has not been found"
-
-#define QERR_DEVICE_ENCRYPTED \
-    ERROR_CLASS_DEVICE_ENCRYPTED, "'%s' (%s) is encrypted"
-
-#define QERR_DEVICE_FEATURE_BLOCKS_MIGRATION \
-    ERROR_CLASS_GENERIC_ERROR, "Migration is disabled when using feature '%s' in device '%s'"
-
-#define QERR_DEVICE_HAS_NO_MEDIUM \
-    ERROR_CLASS_GENERIC_ERROR, "Device '%s' has no medium"
-
-#define QERR_DEVICE_INIT_FAILED \
-    ERROR_CLASS_GENERIC_ERROR, "Device '%s' could not be initialized"
-
-#define QERR_DEVICE_IN_USE \
-    ERROR_CLASS_GENERIC_ERROR, "Device '%s' is in use"
-
-#define QERR_DEVICE_IS_READ_ONLY \
-    ERROR_CLASS_GENERIC_ERROR, "Device '%s' is read only"
-
-#define QERR_DEVICE_LOCKED \
-    ERROR_CLASS_GENERIC_ERROR, "Device '%s' is locked"
-
-#define QERR_DEVICE_MULTIPLE_BUSSES \
-    ERROR_CLASS_GENERIC_ERROR, "Device '%s' has multiple child busses"
-
-#define QERR_DEVICE_NO_BUS \
-    ERROR_CLASS_GENERIC_ERROR, "Device '%s' has no child bus"
-
-#define QERR_DEVICE_NO_HOTPLUG \
-    ERROR_CLASS_GENERIC_ERROR, "Device '%s' does not support hotplugging"
-
-#define QERR_DEVICE_NOT_ACTIVE \
-    ERROR_CLASS_DEVICE_NOT_ACTIVE, "Device '%s' has not been activated"
-
-#define QERR_DEVICE_NOT_ENCRYPTED \
-    ERROR_CLASS_GENERIC_ERROR, "Device '%s' is not encrypted"
-
-#define QERR_DEVICE_NOT_FOUND \
-    ERROR_CLASS_DEVICE_NOT_FOUND, "Device '%s' not found"
-
-#define QERR_DEVICE_NOT_REMOVABLE \
-    ERROR_CLASS_GENERIC_ERROR, "Device '%s' is not removable"
-
-#define QERR_DUPLICATE_ID \
-    ERROR_CLASS_GENERIC_ERROR, "Duplicate ID '%s' for %s"
-
-#define QERR_FD_NOT_FOUND \
-    ERROR_CLASS_GENERIC_ERROR, "File descriptor named '%s' not found"
-
-#define QERR_FD_NOT_SUPPLIED \
-    ERROR_CLASS_GENERIC_ERROR, "No file descriptor supplied via SCM_RIGHTS"
-
-#define QERR_FEATURE_DISABLED \
-    ERROR_CLASS_GENERIC_ERROR, "The feature '%s' is not enabled"
-
-#define QERR_INVALID_BLOCK_FORMAT \
-    ERROR_CLASS_GENERIC_ERROR, "Invalid block format '%s'"
-
-#define QERR_INVALID_OPTION_GROUP \
-    ERROR_CLASS_GENERIC_ERROR, "There is no option group '%s'"
-
-#define QERR_INVALID_PARAMETER \
-    ERROR_CLASS_GENERIC_ERROR, "Invalid parameter '%s'"
-
-#define QERR_INVALID_PARAMETER_COMBINATION \
-    ERROR_CLASS_GENERIC_ERROR, "Invalid parameter combination"
-
-#define QERR_INVALID_PARAMETER_TYPE \
-    ERROR_CLASS_GENERIC_ERROR, "Invalid parameter type for '%s', expected: %s"
-
-#define QERR_INVALID_PARAMETER_VALUE \
-    ERROR_CLASS_GENERIC_ERROR, "Parameter '%s' expects %s"
-
-#define QERR_INVALID_PASSWORD \
-    ERROR_CLASS_GENERIC_ERROR, "Password incorrect"
-
-#define QERR_IO_ERROR \
-    ERROR_CLASS_GENERIC_ERROR, "An IO error has occurred"
-
-#define QERR_JSON_PARSE_ERROR \
-    ERROR_CLASS_GENERIC_ERROR, "JSON parse error, %s"
-
-#define QERR_JSON_PARSING \
-    ERROR_CLASS_GENERIC_ERROR, "Invalid JSON syntax"
-
-#define QERR_KVM_MISSING_CAP \
-    ERROR_CLASS_K_V_M_MISSING_CAP, "Using KVM without %s, %s unavailable"
-
-#define QERR_MIGRATION_ACTIVE \
-    ERROR_CLASS_GENERIC_ERROR, "There's a migration process in progress"
-
-#define QERR_MIGRATION_NOT_SUPPORTED \
-    ERROR_CLASS_GENERIC_ERROR, "State blocked by non-migratable device '%s'"
-
-#define QERR_MISSING_PARAMETER \
-    ERROR_CLASS_GENERIC_ERROR, "Parameter '%s' is missing"
-
-#define QERR_NO_BUS_FOR_DEVICE \
-    ERROR_CLASS_GENERIC_ERROR, "No '%s' bus found for device '%s'"
-
-#define QERR_NOT_SUPPORTED \
-    ERROR_CLASS_GENERIC_ERROR, "Not supported"
-
-#define QERR_PERMISSION_DENIED \
-    ERROR_CLASS_GENERIC_ERROR, "Insufficient permission to perform this operation"
-
-#define QERR_PROPERTY_NOT_FOUND \
-    ERROR_CLASS_GENERIC_ERROR, "Property '%s.%s' not found"
-
-#define QERR_PROPERTY_VALUE_BAD \
-    ERROR_CLASS_GENERIC_ERROR, "Property '%s.%s' doesn't take value '%s'"
-
-#define QERR_PROPERTY_VALUE_IN_USE \
-    ERROR_CLASS_GENERIC_ERROR, "Property '%s.%s' can't take value '%s', it's in use"
-
-#define QERR_PROPERTY_VALUE_NOT_FOUND \
-    ERROR_CLASS_GENERIC_ERROR, "Property '%s.%s' can't find value '%s'"
-
-#define QERR_PROPERTY_VALUE_NOT_POWER_OF_2 \
-    ERROR_CLASS_GENERIC_ERROR, "Property %s.%s doesn't take value '%" PRId64 "', it's not a power of 2"
-
-#define QERR_PROPERTY_VALUE_OUT_OF_RANGE \
-    ERROR_CLASS_GENERIC_ERROR, "Property %s.%s doesn't take value %" PRId64 " (minimum: %" PRId64 ", maximum: %" PRId64 ")"
-
-#define QERR_QGA_COMMAND_FAILED \
-    ERROR_CLASS_GENERIC_ERROR, "Guest agent command failed, error was '%s'"
-
-#define QERR_QGA_LOGGING_FAILED \
-    ERROR_CLASS_GENERIC_ERROR, "Guest agent failed to log non-optional log statement"
-
-#define QERR_QMP_BAD_INPUT_OBJECT \
-    ERROR_CLASS_GENERIC_ERROR, "Expected '%s' in QMP input"
-
-#define QERR_QMP_BAD_INPUT_OBJECT_MEMBER \
-    ERROR_CLASS_GENERIC_ERROR, "QMP input object member '%s' expects '%s'"
-
-#define QERR_QMP_EXTRA_MEMBER \
-    ERROR_CLASS_GENERIC_ERROR, "QMP input object member '%s' is unexpected"
-
-#define QERR_RESET_REQUIRED \
-    ERROR_CLASS_GENERIC_ERROR, "Resetting the Virtual Machine is required"
-
-#define QERR_SET_PASSWD_FAILED \
-    ERROR_CLASS_GENERIC_ERROR, "Could not set password"
-
-#define QERR_TOO_MANY_FILES \
-    ERROR_CLASS_GENERIC_ERROR, "Too many open files"
-
-#define QERR_UNDEFINED_ERROR \
-    ERROR_CLASS_GENERIC_ERROR, "An undefined error has occurred"
-
-#define QERR_UNKNOWN_BLOCK_FORMAT_FEATURE \
-    ERROR_CLASS_GENERIC_ERROR, "'%s' uses a %s feature which is not supported by this qemu version: %s"
-
-#define QERR_UNSUPPORTED \
-    ERROR_CLASS_GENERIC_ERROR, "this feature or command is not currently supported"
-
-#define QERR_VIRTFS_FEATURE_BLOCKS_MIGRATION \
-    ERROR_CLASS_GENERIC_ERROR, "Migration is disabled when VirtFS export path '%s' is mounted in the guest using mount_tag '%s'"
-
-#define QERR_SOCKET_CONNECT_FAILED \
-    ERROR_CLASS_GENERIC_ERROR, "Failed to connect to socket"
-
-#define QERR_SOCKET_LISTEN_FAILED \
-    ERROR_CLASS_GENERIC_ERROR, "Failed to set socket to listening mode"
-
-#define QERR_SOCKET_BIND_FAILED \
-    ERROR_CLASS_GENERIC_ERROR, "Failed to bind socket"
-
-#define QERR_SOCKET_CREATE_FAILED \
-    ERROR_CLASS_GENERIC_ERROR, "Failed to create socket"
-
-#endif /* QERROR_H */
diff --git a/contrib/qemu/include/qapi/qmp/qfloat.h b/contrib/qemu/include/qapi/qmp/qfloat.h
deleted file mode 100644
index a8658443dc2..00000000000
--- a/contrib/qemu/include/qapi/qmp/qfloat.h
+++ /dev/null
@@ -1,29 +0,0 @@
-/*
- * QFloat Module
- *
- * Copyright IBM, Corp. 2009
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#ifndef QFLOAT_H
-#define QFLOAT_H
-
-#include <stdint.h>
-#include "qapi/qmp/qobject.h"
-
-typedef struct QFloat {
-    QObject_HEAD;
-    double value;
-} QFloat;
-
-QFloat *qfloat_from_double(double value);
-double qfloat_get_double(const QFloat *qi);
-QFloat *qobject_to_qfloat(const QObject *obj);
-
-#endif /* QFLOAT_H */
diff --git a/contrib/qemu/include/qapi/qmp/qint.h b/contrib/qemu/include/qapi/qmp/qint.h
deleted file mode 100644
index 48a41b0f2ae..00000000000
--- a/contrib/qemu/include/qapi/qmp/qint.h
+++ /dev/null
@@ -1,28 +0,0 @@
-/*
- * QInt Module
- *
- * Copyright (C) 2009 Red Hat Inc.
- *
- * Authors:
- *  Luiz Capitulino <lcapitulino@redhat.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- */
-
-#ifndef QINT_H
-#define QINT_H
-
-#include <stdint.h>
-#include "qapi/qmp/qobject.h"
-
-typedef struct QInt {
-    QObject_HEAD;
-    int64_t value;
-} QInt;
-
-QInt *qint_from_int(int64_t value);
-int64_t qint_get_int(const QInt *qi);
-QInt *qobject_to_qint(const QObject *obj);
-
-#endif /* QINT_H */
diff --git a/contrib/qemu/include/qapi/qmp/qjson.h b/contrib/qemu/include/qapi/qmp/qjson.h
deleted file mode 100644
index 73351ed6d68..00000000000
--- a/contrib/qemu/include/qapi/qmp/qjson.h
+++ /dev/null
@@ -1,29 +0,0 @@
-/*
- * QObject JSON integration
- *
- * Copyright IBM, Corp. 2009
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#ifndef QJSON_H
-#define QJSON_H
-
-#include <stdarg.h>
-#include "qemu/compiler.h"
-#include "qapi/qmp/qobject.h"
-#include "qapi/qmp/qstring.h"
-
-QObject *qobject_from_json(const char *string) GCC_FMT_ATTR(1, 0);
-QObject *qobject_from_jsonf(const char *string, ...) GCC_FMT_ATTR(1, 2);
-QObject *qobject_from_jsonv(const char *string, va_list *ap) GCC_FMT_ATTR(1, 0);
-
-QString *qobject_to_json(const QObject *obj);
-QString *qobject_to_json_pretty(const QObject *obj);
-
-#endif /* QJSON_H */
diff --git a/contrib/qemu/include/qapi/qmp/qlist.h b/contrib/qemu/include/qapi/qmp/qlist.h
deleted file mode 100644
index 6cc4831df3e..00000000000
--- a/contrib/qemu/include/qapi/qmp/qlist.h
+++ /dev/null
@@ -1,63 +0,0 @@
-/*
- * QList Module
- *
- * Copyright (C) 2009 Red Hat Inc.
- *
- * Authors:
- *  Luiz Capitulino <lcapitulino@redhat.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- */
-
-#ifndef QLIST_H
-#define QLIST_H
-
-#include "qapi/qmp/qobject.h"
-#include "qemu/queue.h"
-
-typedef struct QListEntry {
-    QObject *value;
-    QTAILQ_ENTRY(QListEntry) next;
-} QListEntry;
-
-typedef struct QList {
-    QObject_HEAD;
-    QTAILQ_HEAD(,QListEntry) head;
-} QList;
-
-#define qlist_append(qlist, obj) \
-        qlist_append_obj(qlist, QOBJECT(obj))
-
-#define QLIST_FOREACH_ENTRY(qlist, var)             \
-        for ((var) = ((qlist)->head.tqh_first);     \
-            (var);                                  \
-            (var) = ((var)->next.tqe_next))
-
-static inline QObject *qlist_entry_obj(const QListEntry *entry)
-{
-    return entry->value;
-}
-
-QList *qlist_new(void);
-QList *qlist_copy(QList *src);
-void qlist_append_obj(QList *qlist, QObject *obj);
-void qlist_iter(const QList *qlist,
-                void (*iter)(QObject *obj, void *opaque), void *opaque);
-QObject *qlist_pop(QList *qlist);
-QObject *qlist_peek(QList *qlist);
-int qlist_empty(const QList *qlist);
-size_t qlist_size(const QList *qlist);
-QList *qobject_to_qlist(const QObject *obj);
-
-static inline const QListEntry *qlist_first(const QList *qlist)
-{
-    return QTAILQ_FIRST(&qlist->head);
-}
-
-static inline const QListEntry *qlist_next(const QListEntry *entry)
-{
-    return QTAILQ_NEXT(entry, next);
-}
-
-#endif /* QLIST_H */
diff --git a/contrib/qemu/include/qapi/qmp/qobject.h b/contrib/qemu/include/qapi/qmp/qobject.h
deleted file mode 100644
index 9124649ed20..00000000000
--- a/contrib/qemu/include/qapi/qmp/qobject.h
+++ /dev/null
@@ -1,112 +0,0 @@
-/*
- * QEMU Object Model.
- *
- * Based on ideas by Avi Kivity <avi@redhat.com>
- *
- * Copyright (C) 2009 Red Hat Inc.
- *
- * Authors:
- *  Luiz Capitulino <lcapitulino@redhat.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- * QObject Reference Counts Terminology
- * ------------------------------------
- *
- *  - Returning references: A function that returns an object may
- *  return it as either a weak or a strong reference.  If the reference
- *  is strong, you are responsible for calling QDECREF() on the reference
- *  when you are done.
- *
- *  If the reference is weak, the owner of the reference may free it at
- *  any time in the future.  Before storing the reference anywhere, you
- *  should call QINCREF() to make the reference strong.
- *
- *  - Transferring ownership: when you transfer ownership of a reference
- *  by calling a function, you are no longer responsible for calling
- *  QDECREF() when the reference is no longer needed.  In other words,
- *  when the function returns you must behave as if the reference to the
- *  passed object was weak.
- */
-#ifndef QOBJECT_H
-#define QOBJECT_H
-
-#include <stddef.h>
-#include <assert.h>
-
-typedef enum {
-    QTYPE_NONE,
-    QTYPE_QINT,
-    QTYPE_QSTRING,
-    QTYPE_QDICT,
-    QTYPE_QLIST,
-    QTYPE_QFLOAT,
-    QTYPE_QBOOL,
-    QTYPE_QERROR,
-} qtype_code;
-
-struct QObject;
-
-typedef struct QType {
-    qtype_code code;
-    void (*destroy)(struct QObject *);
-} QType;
-
-typedef struct QObject {
-    const QType *type;
-    size_t refcnt;
-} QObject;
-
-/* Objects definitions must include this */
-#define QObject_HEAD  \
-    QObject base
-
-/* Get the 'base' part of an object */
-#define QOBJECT(obj) (&(obj)->base)
-
-/* High-level interface for qobject_incref() */
-#define QINCREF(obj)      \
-    qobject_incref(QOBJECT(obj))
-
-/* High-level interface for qobject_decref() */
-#define QDECREF(obj)              \
-    qobject_decref(obj ? QOBJECT(obj) : NULL)
-
-/* Initialize an object to default values */
-#define QOBJECT_INIT(obj, qtype_type)   \
-    obj->base.refcnt = 1;               \
-    obj->base.type   = qtype_type
-
-/**
- * qobject_incref(): Increment QObject's reference count
- */
-static inline void qobject_incref(QObject *obj)
-{
-    if (obj)
-        obj->refcnt++;
-}
-
-/**
- * qobject_decref(): Decrement QObject's reference count, deallocate
- * when it reaches zero
- */
-static inline void qobject_decref(QObject *obj)
-{
-    if (obj && --obj->refcnt == 0) {
-        assert(obj->type != NULL);
-        assert(obj->type->destroy != NULL);
-        obj->type->destroy(obj);
-    }
-}
-
-/**
- * qobject_type(): Return the QObject's type
- */
-static inline qtype_code qobject_type(const QObject *obj)
-{
-    assert(obj->type != NULL);
-    return obj->type->code;
-}
-
-#endif /* QOBJECT_H */
diff --git a/contrib/qemu/include/qapi/qmp/qstring.h b/contrib/qemu/include/qapi/qmp/qstring.h
deleted file mode 100644
index 1bc3666107c..00000000000
--- a/contrib/qemu/include/qapi/qmp/qstring.h
+++ /dev/null
@@ -1,36 +0,0 @@
-/*
- * QString Module
- *
- * Copyright (C) 2009 Red Hat Inc.
- *
- * Authors:
- *  Luiz Capitulino <lcapitulino@redhat.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- */
-
-#ifndef QSTRING_H
-#define QSTRING_H
-
-#include <stdint.h>
-#include "qapi/qmp/qobject.h"
-
-typedef struct QString {
-    QObject_HEAD;
-    char *string;
-    size_t length;
-    size_t capacity;
-} QString;
-
-QString *qstring_new(void);
-QString *qstring_from_str(const char *str);
-QString *qstring_from_substr(const char *str, int start, int end);
-size_t qstring_get_length(const QString *qstring);
-const char *qstring_get_str(const QString *qstring);
-void qstring_append_int(QString *qstring, int64_t value);
-void qstring_append(QString *qstring, const char *str);
-void qstring_append_chr(QString *qstring, int c);
-QString *qobject_to_qstring(const QObject *obj);
-
-#endif /* QSTRING_H */
diff --git a/contrib/qemu/include/qapi/qmp/types.h b/contrib/qemu/include/qapi/qmp/types.h
deleted file mode 100644
index 7782ec5a602..00000000000
--- a/contrib/qemu/include/qapi/qmp/types.h
+++ /dev/null
@@ -1,25 +0,0 @@
-/*
- * Include all QEMU objects.
- *
- * Copyright (C) 2009 Red Hat Inc.
- *
- * Authors:
- *  Luiz Capitulino <lcapitulino@redhat.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- */
-
-#ifndef QEMU_OBJECTS_H
-#define QEMU_OBJECTS_H
-
-#include "qapi/qmp/qobject.h"
-#include "qapi/qmp/qint.h"
-#include "qapi/qmp/qfloat.h"
-#include "qapi/qmp/qbool.h"
-#include "qapi/qmp/qstring.h"
-#include "qapi/qmp/qdict.h"
-#include "qapi/qmp/qlist.h"
-#include "qapi/qmp/qjson.h"
-
-#endif /* QEMU_OBJECTS_H */
diff --git a/contrib/qemu/include/qemu-common.h b/contrib/qemu/include/qemu-common.h
deleted file mode 100644
index 6948bb91774..00000000000
--- a/contrib/qemu/include/qemu-common.h
+++ /dev/null
@@ -1,478 +0,0 @@
-
-/* Common header file that is included by all of QEMU.
- *
- * This file is supposed to be included only by .c files. No header file should
- * depend on qemu-common.h, as this would easily lead to circular header
- * dependencies.
- *
- * If a header file uses a definition from qemu-common.h, that definition
- * must be moved to a separate header file, and the header that uses it
- * must include that header.
- */
-#ifndef QEMU_COMMON_H
-#define QEMU_COMMON_H
-
-#include "qemu/compiler.h"
-#include "config-host.h"
-#include "qemu/typedefs.h"
-
-#if defined(__arm__) || defined(__sparc__) || defined(__mips__) || defined(__hppa__) || defined(__ia64__)
-#define WORDS_ALIGNED
-#endif
-
-#define TFR(expr) do { if ((expr) != -1) break; } while (errno == EINTR)
-
-/* we put basic includes here to avoid repeating them in device drivers */
-#include <stdlib.h>
-#include <stdio.h>
-#include <stdarg.h>
-#include <stdbool.h>
-#include <string.h>
-#include <strings.h>
-#include <inttypes.h>
-#include <limits.h>
-#include <time.h>
-#include <ctype.h>
-#include <errno.h>
-#include <unistd.h>
-#include <fcntl.h>
-#include <sys/stat.h>
-#include <sys/time.h>
-#include <assert.h>
-#include <signal.h>
-#include "glib-compat.h"
-
-#ifdef _WIN32
-#include "sysemu/os-win32.h"
-#endif
-
-#ifdef CONFIG_POSIX
-#include "sysemu/os-posix.h"
-#endif
-
-#ifndef O_LARGEFILE
-#define O_LARGEFILE 0
-#endif
-#ifndef O_BINARY
-#define O_BINARY 0
-#endif
-#ifndef MAP_ANONYMOUS
-#define MAP_ANONYMOUS MAP_ANON
-#endif
-#ifndef ENOMEDIUM
-#define ENOMEDIUM ENODEV
-#endif
-#if !defined(ENOTSUP)
-#define ENOTSUP 4096
-#endif
-#if !defined(ECANCELED)
-#define ECANCELED 4097
-#endif
-#if !defined(EMEDIUMTYPE)
-#define EMEDIUMTYPE 4098
-#endif
-#ifndef TIME_MAX
-#define TIME_MAX LONG_MAX
-#endif
-
-/* HOST_LONG_BITS is the size of a native pointer in bits. */
-#if UINTPTR_MAX == UINT32_MAX
-# define HOST_LONG_BITS 32
-#elif UINTPTR_MAX == UINT64_MAX
-# define HOST_LONG_BITS 64
-#else
-# error Unknown pointer size
-#endif
-
-typedef int (*fprintf_function)(FILE *f, const char *fmt, ...)
-    GCC_FMT_ATTR(2, 3);
-
-#ifdef _WIN32
-#define fsync _commit
-#if !defined(lseek)
-# define lseek _lseeki64
-#endif
-int qemu_ftruncate64(int, int64_t);
-#if !defined(ftruncate)
-# define ftruncate qemu_ftruncate64
-#endif
-
-static inline char *realpath(const char *path, char *resolved_path)
-{
-    _fullpath(resolved_path, path, _MAX_PATH);
-    return resolved_path;
-}
-#endif
-
-/* icount */
-void configure_icount(const char *option);
-extern int use_icount;
-
-#include "qemu/osdep.h"
-#include "qemu/bswap.h"
-
-/* FIXME: Remove NEED_CPU_H.  */
-#ifdef NEED_CPU_H
-#include "cpu.h"
-#endif /* !defined(NEED_CPU_H) */
-
-/* main function, renamed */
-#if defined(CONFIG_COCOA)
-int qemu_main(int argc, char **argv, char **envp);
-#endif
-
-void qemu_get_timedate(struct tm *tm, int offset);
-int qemu_timedate_diff(struct tm *tm);
-
-#if !GLIB_CHECK_VERSION(2, 20, 0)
-/*
- * Glib before 2.20.0 doesn't implement g_poll, so wrap it to compile properly
- * on older systems.
- */
-static inline gint g_poll(GPollFD *fds, guint nfds, gint timeout)
-{
-    GMainContext *ctx = g_main_context_default();
-    return g_main_context_get_poll_func(ctx)(fds, nfds, timeout);
-}
-#endif
-
-/**
- * is_help_option:
- * @s: string to test
- *
- * Check whether @s is one of the standard strings which indicate
- * that the user is asking for a list of the valid values for a
- * command option like -cpu or -M. The current accepted strings
- * are 'help' and '?'. '?' is deprecated (it is a shell wildcard
- * which makes it annoying to use in a reliable way) but provided
- * for backwards compatibility.
- *
- * Returns: true if @s is a request for a list.
- */
-static inline bool is_help_option(const char *s)
-{
-    return !strcmp(s, "?") || !strcmp(s, "help");
-}
-
-/* cutils.c */
-void pstrcpy(char *buf, int buf_size, const char *str);
-void strpadcpy(char *buf, int buf_size, const char *str, char pad);
-char *pstrcat(char *buf, int buf_size, const char *s);
-int strstart(const char *str, const char *val, const char **ptr);
-int stristart(const char *str, const char *val, const char **ptr);
-int qemu_strnlen(const char *s, int max_len);
-char *qemu_strsep(char **input, const char *delim);
-time_t mktimegm(struct tm *tm);
-int qemu_fls(int i);
-int qemu_fdatasync(int fd);
-int fcntl_setfl(int fd, int flag);
-int qemu_parse_fd(const char *param);
-
-int parse_uint(const char *s, unsigned long long *value, char **endptr,
-               int base);
-int parse_uint_full(const char *s, unsigned long long *value, int base);
-
-/*
- * strtosz() suffixes used to specify the default treatment of an
- * argument passed to strtosz() without an explicit suffix.
- * These should be defined using upper case characters in the range
- * A-Z, as strtosz() will use qemu_toupper() on the given argument
- * prior to comparison.
- */
-#define STRTOSZ_DEFSUFFIX_EB	'E'
-#define STRTOSZ_DEFSUFFIX_PB	'P'
-#define STRTOSZ_DEFSUFFIX_TB	'T'
-#define STRTOSZ_DEFSUFFIX_GB	'G'
-#define STRTOSZ_DEFSUFFIX_MB	'M'
-#define STRTOSZ_DEFSUFFIX_KB	'K'
-#define STRTOSZ_DEFSUFFIX_B	'B'
-int64_t strtosz(const char *nptr, char **end);
-int64_t strtosz_suffix(const char *nptr, char **end, const char default_suffix);
-int64_t strtosz_suffix_unit(const char *nptr, char **end,
-                            const char default_suffix, int64_t unit);
-
-/* path.c */
-void init_paths(const char *prefix);
-const char *path(const char *pathname);
-
-#define qemu_isalnum(c)		isalnum((unsigned char)(c))
-#define qemu_isalpha(c)		isalpha((unsigned char)(c))
-#define qemu_iscntrl(c)		iscntrl((unsigned char)(c))
-#define qemu_isdigit(c)		isdigit((unsigned char)(c))
-#define qemu_isgraph(c)		isgraph((unsigned char)(c))
-#define qemu_islower(c)		islower((unsigned char)(c))
-#define qemu_isprint(c)		isprint((unsigned char)(c))
-#define qemu_ispunct(c)		ispunct((unsigned char)(c))
-#define qemu_isspace(c)		isspace((unsigned char)(c))
-#define qemu_isupper(c)		isupper((unsigned char)(c))
-#define qemu_isxdigit(c)	isxdigit((unsigned char)(c))
-#define qemu_tolower(c)		tolower((unsigned char)(c))
-#define qemu_toupper(c)		toupper((unsigned char)(c))
-#define qemu_isascii(c)		isascii((unsigned char)(c))
-#define qemu_toascii(c)		toascii((unsigned char)(c))
-
-void *qemu_oom_check(void *ptr);
-
-ssize_t qemu_write_full(int fd, const void *buf, size_t count)
-    QEMU_WARN_UNUSED_RESULT;
-ssize_t qemu_send_full(int fd, const void *buf, size_t count, int flags)
-    QEMU_WARN_UNUSED_RESULT;
-ssize_t qemu_recv_full(int fd, void *buf, size_t count, int flags)
-    QEMU_WARN_UNUSED_RESULT;
-
-#ifndef _WIN32
-int qemu_pipe(int pipefd[2]);
-/* like openpty() but also makes it raw; return master fd */
-int qemu_openpty_raw(int *aslave, char *pty_name);
-#endif
-
-#ifdef _WIN32
-/* MinGW needs type casts for the 'buf' and 'optval' arguments. */
-#define qemu_getsockopt(sockfd, level, optname, optval, optlen) \
-    getsockopt(sockfd, level, optname, (void *)optval, optlen)
-#define qemu_setsockopt(sockfd, level, optname, optval, optlen) \
-    setsockopt(sockfd, level, optname, (const void *)optval, optlen)
-#define qemu_recv(sockfd, buf, len, flags) recv(sockfd, (void *)buf, len, flags)
-#define qemu_sendto(sockfd, buf, len, flags, destaddr, addrlen) \
-    sendto(sockfd, (const void *)buf, len, flags, destaddr, addrlen)
-#else
-#define qemu_getsockopt(sockfd, level, optname, optval, optlen) \
-    getsockopt(sockfd, level, optname, optval, optlen)
-#define qemu_setsockopt(sockfd, level, optname, optval, optlen) \
-    setsockopt(sockfd, level, optname, optval, optlen)
-#define qemu_recv(sockfd, buf, len, flags) recv(sockfd, buf, len, flags)
-#define qemu_sendto(sockfd, buf, len, flags, destaddr, addrlen) \
-    sendto(sockfd, buf, len, flags, destaddr, addrlen)
-#endif
-
-/* Error handling.  */
-
-void QEMU_NORETURN hw_error(const char *fmt, ...) GCC_FMT_ATTR(1, 2);
-
-struct ParallelIOArg {
-    void *buffer;
-    int count;
-};
-
-typedef int (*DMA_transfer_handler) (void *opaque, int nchan, int pos, int size);
-
-typedef uint64_t pcibus_t;
-
-typedef enum LostTickPolicy {
-    LOST_TICK_DISCARD,
-    LOST_TICK_DELAY,
-    LOST_TICK_MERGE,
-    LOST_TICK_SLEW,
-    LOST_TICK_MAX
-} LostTickPolicy;
-
-typedef struct PCIHostDeviceAddress {
-    unsigned int domain;
-    unsigned int bus;
-    unsigned int slot;
-    unsigned int function;
-} PCIHostDeviceAddress;
-
-void tcg_exec_init(unsigned long tb_size);
-bool tcg_enabled(void);
-
-void cpu_exec_init_all(void);
-
-/* CPU save/load.  */
-#ifdef CPU_SAVE_VERSION
-void cpu_save(QEMUFile *f, void *opaque);
-int cpu_load(QEMUFile *f, void *opaque, int version_id);
-#endif
-
-/* Unblock cpu */
-void qemu_cpu_kick_self(void);
-
-/* work queue */
-struct qemu_work_item {
-    struct qemu_work_item *next;
-    void (*func)(void *data);
-    void *data;
-    int done;
-    bool free;
-};
-
-
-/**
- * Sends a (part of) iovec down a socket, yielding when the socket is full, or
- * Receives data into a (part of) iovec from a socket,
- * yielding when there is no data in the socket.
- * The same interface as qemu_sendv_recvv(), with added yielding.
- * XXX should mark these as coroutine_fn
- */
-ssize_t qemu_co_sendv_recvv(int sockfd, struct iovec *iov, unsigned iov_cnt,
-                            size_t offset, size_t bytes, bool do_send);
-#define qemu_co_recvv(sockfd, iov, iov_cnt, offset, bytes) \
-  qemu_co_sendv_recvv(sockfd, iov, iov_cnt, offset, bytes, false)
-#define qemu_co_sendv(sockfd, iov, iov_cnt, offset, bytes) \
-  qemu_co_sendv_recvv(sockfd, iov, iov_cnt, offset, bytes, true)
-
-/**
- * The same as above, but with just a single buffer
- */
-ssize_t qemu_co_send_recv(int sockfd, void *buf, size_t bytes, bool do_send);
-#define qemu_co_recv(sockfd, buf, bytes) \
-  qemu_co_send_recv(sockfd, buf, bytes, false)
-#define qemu_co_send(sockfd, buf, bytes) \
-  qemu_co_send_recv(sockfd, buf, bytes, true)
-
-typedef struct QEMUIOVector {
-    struct iovec *iov;
-    int niov;
-    int nalloc;
-    size_t size;
-} QEMUIOVector;
-
-void qemu_iovec_init(QEMUIOVector *qiov, int alloc_hint);
-void qemu_iovec_init_external(QEMUIOVector *qiov, struct iovec *iov, int niov);
-void qemu_iovec_add(QEMUIOVector *qiov, void *base, size_t len);
-void qemu_iovec_concat(QEMUIOVector *dst,
-                       QEMUIOVector *src, size_t soffset, size_t sbytes);
-void qemu_iovec_concat_iov(QEMUIOVector *dst,
-                           struct iovec *src_iov, unsigned int src_cnt,
-                           size_t soffset, size_t sbytes);
-void qemu_iovec_destroy(QEMUIOVector *qiov);
-void qemu_iovec_reset(QEMUIOVector *qiov);
-size_t qemu_iovec_to_buf(QEMUIOVector *qiov, size_t offset,
-                         void *buf, size_t bytes);
-size_t qemu_iovec_from_buf(QEMUIOVector *qiov, size_t offset,
-                           const void *buf, size_t bytes);
-size_t qemu_iovec_memset(QEMUIOVector *qiov, size_t offset,
-                         int fillc, size_t bytes);
-
-bool buffer_is_zero(const void *buf, size_t len);
-
-void qemu_progress_init(int enabled, float min_skip);
-void qemu_progress_end(void);
-void qemu_progress_print(float delta, int max);
-const char *qemu_get_vm_name(void);
-
-#define QEMU_FILE_TYPE_BIOS   0
-#define QEMU_FILE_TYPE_KEYMAP 1
-char *qemu_find_file(int type, const char *name);
-
-/* OS specific functions */
-void os_setup_early_signal_handling(void);
-char *os_find_datadir(const char *argv0);
-void os_parse_cmd_args(int index, const char *optarg);
-void os_pidfile_error(void);
-
-/* Convert a byte between binary and BCD.  */
-static inline uint8_t to_bcd(uint8_t val)
-{
-    return ((val / 10) << 4) | (val % 10);
-}
-
-static inline uint8_t from_bcd(uint8_t val)
-{
-    return ((val >> 4) * 10) + (val & 0x0f);
-}
-
-/* compute with 96 bit intermediate result: (a*b)/c */
-static inline uint64_t muldiv64(uint64_t a, uint32_t b, uint32_t c)
-{
-    union {
-        uint64_t ll;
-        struct {
-#ifdef HOST_WORDS_BIGENDIAN
-            uint32_t high, low;
-#else
-            uint32_t low, high;
-#endif
-        } l;
-    } u, res;
-    uint64_t rl, rh;
-
-    u.ll = a;
-    rl = (uint64_t)u.l.low * (uint64_t)b;
-    rh = (uint64_t)u.l.high * (uint64_t)b;
-    rh += (rl >> 32);
-    res.l.high = rh / c;
-    res.l.low = (((rh % c) << 32) + (rl & 0xffffffff)) / c;
-    return res.ll;
-}
-
-/* Round number down to multiple */
-#define QEMU_ALIGN_DOWN(n, m) ((n) / (m) * (m))
-
-/* Round number up to multiple */
-#define QEMU_ALIGN_UP(n, m) QEMU_ALIGN_DOWN((n) + (m) - 1, (m))
-
-static inline bool is_power_of_2(uint64_t value)
-{
-    if (!value) {
-        return 0;
-    }
-
-    return !(value & (value - 1));
-}
-
-/* round down to the nearest power of 2*/
-int64_t pow2floor(int64_t value);
-
-#include "qemu/module.h"
-
-/*
- * Implementation of ULEB128 (http://en.wikipedia.org/wiki/LEB128)
- * Input is limited to 14-bit numbers
- */
-
-int uleb128_encode_small(uint8_t *out, uint32_t n);
-int uleb128_decode_small(const uint8_t *in, uint32_t *n);
-
-/* unicode.c */
-int mod_utf8_codepoint(const char *s, size_t n, char **end);
-
-/*
- * Hexdump a buffer to a file. An optional string prefix is added to every line
- */
-
-void qemu_hexdump(const char *buf, FILE *fp, const char *prefix, size_t size);
-
-/* vector definitions */
-#ifdef __ALTIVEC__
-#include <altivec.h>
-/* The altivec.h header says we're allowed to undef these for
- * C++ compatibility.  Here we don't care about C++, but we
- * undef them anyway to avoid namespace pollution.
- */
-#undef vector
-#undef pixel
-#undef bool
-#define VECTYPE        __vector unsigned char
-#define SPLAT(p)       vec_splat(vec_ld(0, p), 0)
-#define ALL_EQ(v1, v2) vec_all_eq(v1, v2)
-/* altivec.h may redefine the bool macro as vector type.
- * Reset it to POSIX semantics. */
-#define bool _Bool
-#elif defined __SSE2__
-#include <emmintrin.h>
-#define VECTYPE        __m128i
-#define SPLAT(p)       _mm_set1_epi8(*(p))
-#define ALL_EQ(v1, v2) (_mm_movemask_epi8(_mm_cmpeq_epi8(v1, v2)) == 0xFFFF)
-#else
-#define VECTYPE        unsigned long
-#define SPLAT(p)       (*(p) * (~0UL / 255))
-#define ALL_EQ(v1, v2) ((v1) == (v2))
-#endif
-
-#define BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR 8
-static inline bool
-can_use_buffer_find_nonzero_offset(const void *buf, size_t len)
-{
-    return (len % (BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR
-                   * sizeof(VECTYPE)) == 0
-            && ((uintptr_t) buf) % sizeof(VECTYPE) == 0);
-}
-size_t buffer_find_nonzero_offset(const void *buf, size_t len);
-
-/*
- * helper to parse debug environment variables
- */
-int parse_debug_env(const char *name, int max, int initial);
-
-#endif
diff --git a/contrib/qemu/include/qemu/aes.h b/contrib/qemu/include/qemu/aes.h
deleted file mode 100644
index e79c707436f..00000000000
--- a/contrib/qemu/include/qemu/aes.h
+++ /dev/null
@@ -1,45 +0,0 @@
-#ifndef QEMU_AES_H
-#define QEMU_AES_H
-
-#define AES_MAXNR 14
-#define AES_BLOCK_SIZE 16
-
-struct aes_key_st {
-    uint32_t rd_key[4 *(AES_MAXNR + 1)];
-    int rounds;
-};
-typedef struct aes_key_st AES_KEY;
-
-int AES_set_encrypt_key(const unsigned char *userKey, const int bits,
-	AES_KEY *key);
-int AES_set_decrypt_key(const unsigned char *userKey, const int bits,
-	AES_KEY *key);
-
-void AES_encrypt(const unsigned char *in, unsigned char *out,
-	const AES_KEY *key);
-void AES_decrypt(const unsigned char *in, unsigned char *out,
-	const AES_KEY *key);
-void AES_cbc_encrypt(const unsigned char *in, unsigned char *out,
-		     const unsigned long length, const AES_KEY *key,
-		     unsigned char *ivec, const int enc);
-
-/*
-AES_Te0[x] = S [x].[02, 01, 01, 03];
-AES_Te1[x] = S [x].[03, 02, 01, 01];
-AES_Te2[x] = S [x].[01, 03, 02, 01];
-AES_Te3[x] = S [x].[01, 01, 03, 02];
-AES_Te4[x] = S [x].[01, 01, 01, 01];
-
-AES_Td0[x] = Si[x].[0e, 09, 0d, 0b];
-AES_Td1[x] = Si[x].[0b, 0e, 09, 0d];
-AES_Td2[x] = Si[x].[0d, 0b, 0e, 09];
-AES_Td3[x] = Si[x].[09, 0d, 0b, 0e];
-AES_Td4[x] = Si[x].[01, 01, 01, 01];
-*/
-
-extern const uint32_t AES_Te0[256], AES_Te1[256], AES_Te2[256],
-                      AES_Te3[256], AES_Te4[256];
-extern const uint32_t AES_Td0[256], AES_Td1[256], AES_Td2[256],
-                      AES_Td3[256], AES_Td4[256];
-
-#endif
diff --git a/contrib/qemu/include/qemu/atomic.h b/contrib/qemu/include/qemu/atomic.h
deleted file mode 100644
index 0aa89133018..00000000000
--- a/contrib/qemu/include/qemu/atomic.h
+++ /dev/null
@@ -1,202 +0,0 @@
-/*
- * Simple interface for atomic operations.
- *
- * Copyright (C) 2013 Red Hat, Inc.
- *
- * Author: Paolo Bonzini <pbonzini@redhat.com>
- *
- * This work is licensed under the terms of the GNU GPL, version 2 or later.
- * See the COPYING file in the top-level directory.
- *
- */
-
-#ifndef __QEMU_ATOMIC_H
-#define __QEMU_ATOMIC_H 1
-
-#include "qemu/compiler.h"
-
-/* For C11 atomic ops */
-
-/* Compiler barrier */
-#define barrier()   ({ asm volatile("" ::: "memory"); (void)0; })
-
-#ifndef __ATOMIC_RELAXED
-
-/*
- * We use GCC builtin if it's available, as that can use mfence on
- * 32-bit as well, e.g. if built with -march=pentium-m. However, on
- * i386 the spec is buggy, and the implementation followed it until
- * 4.3 (http://gcc.gnu.org/bugzilla/show_bug.cgi?id=36793).
- */
-#if defined(__i386__) || defined(__x86_64__)
-#if !QEMU_GNUC_PREREQ(4, 4)
-#if defined __x86_64__
-#define smp_mb()    ({ asm volatile("mfence" ::: "memory"); (void)0; })
-#else
-#define smp_mb()    ({ asm volatile("lock; addl $0,0(%%esp) " ::: "memory"); (void)0; })
-#endif
-#endif
-#endif
-
-
-#ifdef __alpha__
-#define smp_read_barrier_depends()   asm volatile("mb":::"memory")
-#endif
-
-#if defined(__i386__) || defined(__x86_64__) || defined(__s390x__)
-
-/*
- * Because of the strongly ordered storage model, wmb() and rmb() are nops
- * here (a compiler barrier only).  QEMU doesn't do accesses to write-combining
- * qemu memory or non-temporal load/stores from C code.
- */
-#define smp_wmb()   barrier()
-#define smp_rmb()   barrier()
-
-/*
- * __sync_lock_test_and_set() is documented to be an acquire barrier only,
- * but it is a full barrier at the hardware level.  Add a compiler barrier
- * to make it a full barrier also at the compiler level.
- */
-#define atomic_xchg(ptr, i)    (barrier(), __sync_lock_test_and_set(ptr, i))
-
-/*
- * Load/store with Java volatile semantics.
- */
-#define atomic_mb_set(ptr, i)  ((void)atomic_xchg(ptr, i))
-
-#elif defined(_ARCH_PPC)
-
-/*
- * We use an eieio() for wmb() on powerpc.  This assumes we don't
- * need to order cacheable and non-cacheable stores with respect to
- * each other.
- *
- * smp_mb has the same problem as on x86 for not-very-new GCC
- * (http://patchwork.ozlabs.org/patch/126184/, Nov 2011).
- */
-#define smp_wmb()   ({ asm volatile("eieio" ::: "memory"); (void)0; })
-#if defined(__powerpc64__)
-#define smp_rmb()   ({ asm volatile("lwsync" ::: "memory"); (void)0; })
-#else
-#define smp_rmb()   ({ asm volatile("sync" ::: "memory"); (void)0; })
-#endif
-#define smp_mb()    ({ asm volatile("sync" ::: "memory"); (void)0; })
-
-#endif /* _ARCH_PPC */
-
-#endif /* C11 atomics */
-
-/*
- * For (host) platforms we don't have explicit barrier definitions
- * for, we use the gcc __sync_synchronize() primitive to generate a
- * full barrier.  This should be safe on all platforms, though it may
- * be overkill for smp_wmb() and smp_rmb().
- */
-#ifndef smp_mb
-#define smp_mb()    __sync_synchronize()
-#endif
-
-#ifndef smp_wmb
-#ifdef __ATOMIC_RELEASE
-#define smp_wmb()   __atomic_thread_fence(__ATOMIC_RELEASE)
-#else
-#define smp_wmb()   __sync_synchronize()
-#endif
-#endif
-
-#ifndef smp_rmb
-#ifdef __ATOMIC_ACQUIRE
-#define smp_rmb()   __atomic_thread_fence(__ATOMIC_ACQUIRE)
-#else
-#define smp_rmb()   __sync_synchronize()
-#endif
-#endif
-
-#ifndef smp_read_barrier_depends
-#ifdef __ATOMIC_CONSUME
-#define smp_read_barrier_depends()   __atomic_thread_fence(__ATOMIC_CONSUME)
-#else
-#define smp_read_barrier_depends()   barrier()
-#endif
-#endif
-
-#ifndef atomic_read
-#define atomic_read(ptr)       (*(__typeof__(*ptr) *volatile) (ptr))
-#endif
-
-#ifndef atomic_set
-#define atomic_set(ptr, i)     ((*(__typeof__(*ptr) *volatile) (ptr)) = (i))
-#endif
-
-/* These have the same semantics as Java volatile variables.
- * See http://gee.cs.oswego.edu/dl/jmm/cookbook.html:
- * "1. Issue a StoreStore barrier (wmb) before each volatile store."
- *  2. Issue a StoreLoad barrier after each volatile store.
- *     Note that you could instead issue one before each volatile load, but
- *     this would be slower for typical programs using volatiles in which
- *     reads greatly outnumber writes. Alternatively, if available, you
- *     can implement volatile store as an atomic instruction (for example
- *     XCHG on x86) and omit the barrier. This may be more efficient if
- *     atomic instructions are cheaper than StoreLoad barriers.
- *  3. Issue LoadLoad and LoadStore barriers after each volatile load."
- *
- * If you prefer to think in terms of "pairing" of memory barriers,
- * an atomic_mb_read pairs with an atomic_mb_set.
- *
- * And for the few ia64 lovers that exist, an atomic_mb_read is a ld.acq,
- * while an atomic_mb_set is a st.rel followed by a memory barrier.
- *
- * These are a bit weaker than __atomic_load/store with __ATOMIC_SEQ_CST
- * (see docs/atomics.txt), and I'm not sure that __ATOMIC_ACQ_REL is enough.
- * Just always use the barriers manually by the rules above.
- */
-#ifndef atomic_mb_read
-#define atomic_mb_read(ptr)    ({           \
-    typeof(*ptr) _val = atomic_read(ptr);   \
-    smp_rmb();                              \
-    _val;                                   \
-})
-#endif
-
-#ifndef atomic_mb_set
-#define atomic_mb_set(ptr, i)  do {         \
-    smp_wmb();                              \
-    atomic_set(ptr, i);                     \
-    smp_mb();                               \
-} while (0)
-#endif
-
-#ifndef atomic_xchg
-#ifdef __ATOMIC_SEQ_CST
-#define atomic_xchg(ptr, i)    ({                           \
-    typeof(*ptr) _new = (i), _old;                          \
-    __atomic_exchange(ptr, &_new, &_old, __ATOMIC_SEQ_CST); \
-    _old;                                                   \
-})
-#elif defined __clang__
-#define atomic_xchg(ptr, i)    __sync_exchange(ptr, i)
-#else
-/* __sync_lock_test_and_set() is documented to be an acquire barrier only.  */
-#define atomic_xchg(ptr, i)    (smp_mb(), __sync_lock_test_and_set(ptr, i))
-#endif
-#endif
-
-/* Provide shorter names for GCC atomic builtins.  */
-#define atomic_fetch_inc(ptr)  __sync_fetch_and_add(ptr, 1)
-#define atomic_fetch_dec(ptr)  __sync_fetch_and_add(ptr, -1)
-#define atomic_fetch_add       __sync_fetch_and_add
-#define atomic_fetch_sub       __sync_fetch_and_sub
-#define atomic_fetch_and       __sync_fetch_and_and
-#define atomic_fetch_or        __sync_fetch_and_or
-#define atomic_cmpxchg         __sync_val_compare_and_swap
-
-/* And even shorter names that return void.  */
-#define atomic_inc(ptr)        ((void) __sync_fetch_and_add(ptr, 1))
-#define atomic_dec(ptr)        ((void) __sync_fetch_and_add(ptr, -1))
-#define atomic_add(ptr, n)     ((void) __sync_fetch_and_add(ptr, n))
-#define atomic_sub(ptr, n)     ((void) __sync_fetch_and_sub(ptr, n))
-#define atomic_and(ptr, n)     ((void) __sync_fetch_and_and(ptr, n))
-#define atomic_or(ptr, n)      ((void) __sync_fetch_and_or(ptr, n))
-
-#endif
diff --git a/contrib/qemu/include/qemu/bitmap.h b/contrib/qemu/include/qemu/bitmap.h
deleted file mode 100644
index 308bbb71e9b..00000000000
--- a/contrib/qemu/include/qemu/bitmap.h
+++ /dev/null
@@ -1,222 +0,0 @@
-/*
- * Bitmap Module
- *
- * Copyright (C) 2010 Corentin Chary <corentin.chary@gmail.com>
- *
- * Mostly inspired by (stolen from) linux/bitmap.h and linux/bitops.h
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- */
-
-#ifndef BITMAP_H
-#define BITMAP_H
-
-#include "qemu-common.h"
-#include "qemu/bitops.h"
-
-/*
- * The available bitmap operations and their rough meaning in the
- * case that the bitmap is a single unsigned long are thus:
- *
- * Note that nbits should be always a compile time evaluable constant.
- * Otherwise many inlines will generate horrible code.
- *
- * bitmap_zero(dst, nbits)			*dst = 0UL
- * bitmap_fill(dst, nbits)			*dst = ~0UL
- * bitmap_copy(dst, src, nbits)			*dst = *src
- * bitmap_and(dst, src1, src2, nbits)		*dst = *src1 & *src2
- * bitmap_or(dst, src1, src2, nbits)		*dst = *src1 | *src2
- * bitmap_xor(dst, src1, src2, nbits)		*dst = *src1 ^ *src2
- * bitmap_andnot(dst, src1, src2, nbits)	*dst = *src1 & ~(*src2)
- * bitmap_complement(dst, src, nbits)		*dst = ~(*src)
- * bitmap_equal(src1, src2, nbits)		Are *src1 and *src2 equal?
- * bitmap_intersects(src1, src2, nbits) 	Do *src1 and *src2 overlap?
- * bitmap_empty(src, nbits)			Are all bits zero in *src?
- * bitmap_full(src, nbits)			Are all bits set in *src?
- * bitmap_set(dst, pos, nbits)			Set specified bit area
- * bitmap_clear(dst, pos, nbits)		Clear specified bit area
- * bitmap_find_next_zero_area(buf, len, pos, n, mask)	Find bit free area
- */
-
-/*
- * Also the following operations apply to bitmaps.
- *
- * set_bit(bit, addr)			*addr |= bit
- * clear_bit(bit, addr)			*addr &= ~bit
- * change_bit(bit, addr)		*addr ^= bit
- * test_bit(bit, addr)			Is bit set in *addr?
- * test_and_set_bit(bit, addr)		Set bit and return old value
- * test_and_clear_bit(bit, addr)	Clear bit and return old value
- * test_and_change_bit(bit, addr)	Change bit and return old value
- * find_first_zero_bit(addr, nbits)	Position first zero bit in *addr
- * find_first_bit(addr, nbits)		Position first set bit in *addr
- * find_next_zero_bit(addr, nbits, bit)	Position next zero bit in *addr >= bit
- * find_next_bit(addr, nbits, bit)	Position next set bit in *addr >= bit
- */
-
-#define BITMAP_LAST_WORD_MASK(nbits)                                    \
-    (                                                                   \
-        ((nbits) % BITS_PER_LONG) ?                                     \
-        (1UL<<((nbits) % BITS_PER_LONG))-1 : ~0UL                       \
-        )
-
-#define DECLARE_BITMAP(name,bits)                  \
-	unsigned long name[BITS_TO_LONGS(bits)]
-
-#define small_nbits(nbits)                      \
-	((nbits) <= BITS_PER_LONG)
-
-int slow_bitmap_empty(const unsigned long *bitmap, int bits);
-int slow_bitmap_full(const unsigned long *bitmap, int bits);
-int slow_bitmap_equal(const unsigned long *bitmap1,
-                   const unsigned long *bitmap2, int bits);
-void slow_bitmap_complement(unsigned long *dst, const unsigned long *src,
-                         int bits);
-void slow_bitmap_shift_right(unsigned long *dst,
-                          const unsigned long *src, int shift, int bits);
-void slow_bitmap_shift_left(unsigned long *dst,
-                         const unsigned long *src, int shift, int bits);
-int slow_bitmap_and(unsigned long *dst, const unsigned long *bitmap1,
-                 const unsigned long *bitmap2, int bits);
-void slow_bitmap_or(unsigned long *dst, const unsigned long *bitmap1,
-                 const unsigned long *bitmap2, int bits);
-void slow_bitmap_xor(unsigned long *dst, const unsigned long *bitmap1,
-                  const unsigned long *bitmap2, int bits);
-int slow_bitmap_andnot(unsigned long *dst, const unsigned long *bitmap1,
-                    const unsigned long *bitmap2, int bits);
-int slow_bitmap_intersects(const unsigned long *bitmap1,
-			const unsigned long *bitmap2, int bits);
-
-static inline unsigned long *bitmap_new(int nbits)
-{
-    int len = BITS_TO_LONGS(nbits) * sizeof(unsigned long);
-    return g_malloc0(len);
-}
-
-static inline void bitmap_zero(unsigned long *dst, int nbits)
-{
-    if (small_nbits(nbits)) {
-        *dst = 0UL;
-    } else {
-        int len = BITS_TO_LONGS(nbits) * sizeof(unsigned long);
-        memset(dst, 0, len);
-    }
-}
-
-static inline void bitmap_fill(unsigned long *dst, int nbits)
-{
-    size_t nlongs = BITS_TO_LONGS(nbits);
-    if (!small_nbits(nbits)) {
-        int len = (nlongs - 1) * sizeof(unsigned long);
-        memset(dst, 0xff,  len);
-    }
-    dst[nlongs - 1] = BITMAP_LAST_WORD_MASK(nbits);
-}
-
-static inline void bitmap_copy(unsigned long *dst, const unsigned long *src,
-                               int nbits)
-{
-    if (small_nbits(nbits)) {
-        *dst = *src;
-    } else {
-        int len = BITS_TO_LONGS(nbits) * sizeof(unsigned long);
-        memcpy(dst, src, len);
-    }
-}
-
-static inline int bitmap_and(unsigned long *dst, const unsigned long *src1,
-                             const unsigned long *src2, int nbits)
-{
-    if (small_nbits(nbits)) {
-        return (*dst = *src1 & *src2) != 0;
-    }
-    return slow_bitmap_and(dst, src1, src2, nbits);
-}
-
-static inline void bitmap_or(unsigned long *dst, const unsigned long *src1,
-			const unsigned long *src2, int nbits)
-{
-    if (small_nbits(nbits)) {
-        *dst = *src1 | *src2;
-    } else {
-        slow_bitmap_or(dst, src1, src2, nbits);
-    }
-}
-
-static inline void bitmap_xor(unsigned long *dst, const unsigned long *src1,
-			const unsigned long *src2, int nbits)
-{
-    if (small_nbits(nbits)) {
-        *dst = *src1 ^ *src2;
-    } else {
-        slow_bitmap_xor(dst, src1, src2, nbits);
-    }
-}
-
-static inline int bitmap_andnot(unsigned long *dst, const unsigned long *src1,
-			const unsigned long *src2, int nbits)
-{
-    if (small_nbits(nbits)) {
-        return (*dst = *src1 & ~(*src2)) != 0;
-    }
-    return slow_bitmap_andnot(dst, src1, src2, nbits);
-}
-
-static inline void bitmap_complement(unsigned long *dst, const unsigned long *src,
-			int nbits)
-{
-    if (small_nbits(nbits)) {
-        *dst = ~(*src) & BITMAP_LAST_WORD_MASK(nbits);
-    } else {
-        slow_bitmap_complement(dst, src, nbits);
-    }
-}
-
-static inline int bitmap_equal(const unsigned long *src1,
-			const unsigned long *src2, int nbits)
-{
-    if (small_nbits(nbits)) {
-        return ! ((*src1 ^ *src2) & BITMAP_LAST_WORD_MASK(nbits));
-    } else {
-        return slow_bitmap_equal(src1, src2, nbits);
-    }
-}
-
-static inline int bitmap_empty(const unsigned long *src, int nbits)
-{
-    if (small_nbits(nbits)) {
-        return ! (*src & BITMAP_LAST_WORD_MASK(nbits));
-    } else {
-        return slow_bitmap_empty(src, nbits);
-    }
-}
-
-static inline int bitmap_full(const unsigned long *src, int nbits)
-{
-    if (small_nbits(nbits)) {
-        return ! (~(*src) & BITMAP_LAST_WORD_MASK(nbits));
-    } else {
-        return slow_bitmap_full(src, nbits);
-    }
-}
-
-static inline int bitmap_intersects(const unsigned long *src1,
-			const unsigned long *src2, int nbits)
-{
-    if (small_nbits(nbits)) {
-        return ((*src1 & *src2) & BITMAP_LAST_WORD_MASK(nbits)) != 0;
-    } else {
-        return slow_bitmap_intersects(src1, src2, nbits);
-    }
-}
-
-void bitmap_set(unsigned long *map, int i, int len);
-void bitmap_clear(unsigned long *map, int start, int nr);
-unsigned long bitmap_find_next_zero_area(unsigned long *map,
-					 unsigned long size,
-					 unsigned long start,
-					 unsigned int nr,
-					 unsigned long align_mask);
-
-#endif /* BITMAP_H */
diff --git a/contrib/qemu/include/qemu/bitops.h b/contrib/qemu/include/qemu/bitops.h
deleted file mode 100644
index affcc969dcf..00000000000
--- a/contrib/qemu/include/qemu/bitops.h
+++ /dev/null
@@ -1,276 +0,0 @@
-/*
- * Bitops Module
- *
- * Copyright (C) 2010 Corentin Chary <corentin.chary@gmail.com>
- *
- * Mostly inspired by (stolen from) linux/bitmap.h and linux/bitops.h
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- */
-
-#ifndef BITOPS_H
-#define BITOPS_H
-
-#include "qemu-common.h"
-#include "host-utils.h"
-
-#define BITS_PER_BYTE           CHAR_BIT
-#define BITS_PER_LONG           (sizeof (unsigned long) * BITS_PER_BYTE)
-
-#define BIT(nr)			(1UL << (nr))
-#define BIT_MASK(nr)		(1UL << ((nr) % BITS_PER_LONG))
-#define BIT_WORD(nr)		((nr) / BITS_PER_LONG)
-#define BITS_TO_LONGS(nr)	DIV_ROUND_UP(nr, BITS_PER_BYTE * sizeof(long))
-
-/**
- * set_bit - Set a bit in memory
- * @nr: the bit to set
- * @addr: the address to start counting from
- */
-static inline void set_bit(int nr, unsigned long *addr)
-{
-	unsigned long mask = BIT_MASK(nr);
-        unsigned long *p = addr + BIT_WORD(nr);
-
-	*p  |= mask;
-}
-
-/**
- * clear_bit - Clears a bit in memory
- * @nr: Bit to clear
- * @addr: Address to start counting from
- */
-static inline void clear_bit(int nr, unsigned long *addr)
-{
-	unsigned long mask = BIT_MASK(nr);
-        unsigned long *p = addr + BIT_WORD(nr);
-
-	*p &= ~mask;
-}
-
-/**
- * change_bit - Toggle a bit in memory
- * @nr: Bit to change
- * @addr: Address to start counting from
- */
-static inline void change_bit(int nr, unsigned long *addr)
-{
-	unsigned long mask = BIT_MASK(nr);
-        unsigned long *p = addr + BIT_WORD(nr);
-
-	*p ^= mask;
-}
-
-/**
- * test_and_set_bit - Set a bit and return its old value
- * @nr: Bit to set
- * @addr: Address to count from
- */
-static inline int test_and_set_bit(int nr, unsigned long *addr)
-{
-	unsigned long mask = BIT_MASK(nr);
-        unsigned long *p = addr + BIT_WORD(nr);
-	unsigned long old = *p;
-
-	*p = old | mask;
-	return (old & mask) != 0;
-}
-
-/**
- * test_and_clear_bit - Clear a bit and return its old value
- * @nr: Bit to clear
- * @addr: Address to count from
- */
-static inline int test_and_clear_bit(int nr, unsigned long *addr)
-{
-	unsigned long mask = BIT_MASK(nr);
-        unsigned long *p = addr + BIT_WORD(nr);
-	unsigned long old = *p;
-
-	*p = old & ~mask;
-	return (old & mask) != 0;
-}
-
-/**
- * test_and_change_bit - Change a bit and return its old value
- * @nr: Bit to change
- * @addr: Address to count from
- */
-static inline int test_and_change_bit(int nr, unsigned long *addr)
-{
-	unsigned long mask = BIT_MASK(nr);
-        unsigned long *p = addr + BIT_WORD(nr);
-	unsigned long old = *p;
-
-	*p = old ^ mask;
-	return (old & mask) != 0;
-}
-
-/**
- * test_bit - Determine whether a bit is set
- * @nr: bit number to test
- * @addr: Address to start counting from
- */
-static inline int test_bit(int nr, const unsigned long *addr)
-{
-	return 1UL & (addr[BIT_WORD(nr)] >> (nr & (BITS_PER_LONG-1)));
-}
-
-/**
- * find_last_bit - find the last set bit in a memory region
- * @addr: The address to start the search at
- * @size: The maximum size to search
- *
- * Returns the bit number of the first set bit, or size.
- */
-unsigned long find_last_bit(const unsigned long *addr,
-                            unsigned long size);
-
-/**
- * find_next_bit - find the next set bit in a memory region
- * @addr: The address to base the search on
- * @offset: The bitnumber to start searching at
- * @size: The bitmap size in bits
- */
-unsigned long find_next_bit(const unsigned long *addr,
-				   unsigned long size, unsigned long offset);
-
-/**
- * find_next_zero_bit - find the next cleared bit in a memory region
- * @addr: The address to base the search on
- * @offset: The bitnumber to start searching at
- * @size: The bitmap size in bits
- */
-
-unsigned long find_next_zero_bit(const unsigned long *addr,
-                                 unsigned long size,
-                                 unsigned long offset);
-
-/**
- * find_first_bit - find the first set bit in a memory region
- * @addr: The address to start the search at
- * @size: The maximum size to search
- *
- * Returns the bit number of the first set bit.
- */
-static inline unsigned long find_first_bit(const unsigned long *addr,
-                                           unsigned long size)
-{
-    return find_next_bit(addr, size, 0);
-}
-
-/**
- * find_first_zero_bit - find the first cleared bit in a memory region
- * @addr: The address to start the search at
- * @size: The maximum size to search
- *
- * Returns the bit number of the first cleared bit.
- */
-static inline unsigned long find_first_zero_bit(const unsigned long *addr,
-                                                unsigned long size)
-{
-    return find_next_zero_bit(addr, size, 0);
-}
-
-static inline unsigned long hweight_long(unsigned long w)
-{
-    unsigned long count;
-
-    for (count = 0; w; w >>= 1) {
-        count += w & 1;
-    }
-    return count;
-}
-
-/**
- * extract32:
- * @value: the value to extract the bit field from
- * @start: the lowest bit in the bit field (numbered from 0)
- * @length: the length of the bit field
- *
- * Extract from the 32 bit input @value the bit field specified by the
- * @start and @length parameters, and return it. The bit field must
- * lie entirely within the 32 bit word. It is valid to request that
- * all 32 bits are returned (ie @length 32 and @start 0).
- *
- * Returns: the value of the bit field extracted from the input value.
- */
-static inline uint32_t extract32(uint32_t value, int start, int length)
-{
-    assert(start >= 0 && length > 0 && length <= 32 - start);
-    return (value >> start) & (~0U >> (32 - length));
-}
-
-/**
- * extract64:
- * @value: the value to extract the bit field from
- * @start: the lowest bit in the bit field (numbered from 0)
- * @length: the length of the bit field
- *
- * Extract from the 64 bit input @value the bit field specified by the
- * @start and @length parameters, and return it. The bit field must
- * lie entirely within the 64 bit word. It is valid to request that
- * all 64 bits are returned (ie @length 64 and @start 0).
- *
- * Returns: the value of the bit field extracted from the input value.
- */
-static inline uint64_t extract64(uint64_t value, int start, int length)
-{
-    assert(start >= 0 && length > 0 && length <= 64 - start);
-    return (value >> start) & (~0ULL >> (64 - length));
-}
-
-/**
- * deposit32:
- * @value: initial value to insert bit field into
- * @start: the lowest bit in the bit field (numbered from 0)
- * @length: the length of the bit field
- * @fieldval: the value to insert into the bit field
- *
- * Deposit @fieldval into the 32 bit @value at the bit field specified
- * by the @start and @length parameters, and return the modified
- * @value. Bits of @value outside the bit field are not modified.
- * Bits of @fieldval above the least significant @length bits are
- * ignored. The bit field must lie entirely within the 32 bit word.
- * It is valid to request that all 32 bits are modified (ie @length
- * 32 and @start 0).
- *
- * Returns: the modified @value.
- */
-static inline uint32_t deposit32(uint32_t value, int start, int length,
-                                 uint32_t fieldval)
-{
-    uint32_t mask;
-    assert(start >= 0 && length > 0 && length <= 32 - start);
-    mask = (~0U >> (32 - length)) << start;
-    return (value & ~mask) | ((fieldval << start) & mask);
-}
-
-/**
- * deposit64:
- * @value: initial value to insert bit field into
- * @start: the lowest bit in the bit field (numbered from 0)
- * @length: the length of the bit field
- * @fieldval: the value to insert into the bit field
- *
- * Deposit @fieldval into the 64 bit @value at the bit field specified
- * by the @start and @length parameters, and return the modified
- * @value. Bits of @value outside the bit field are not modified.
- * Bits of @fieldval above the least significant @length bits are
- * ignored. The bit field must lie entirely within the 64 bit word.
- * It is valid to request that all 64 bits are modified (ie @length
- * 64 and @start 0).
- *
- * Returns: the modified @value.
- */
-static inline uint64_t deposit64(uint64_t value, int start, int length,
-                                 uint64_t fieldval)
-{
-    uint64_t mask;
-    assert(start >= 0 && length > 0 && length <= 64 - start);
-    mask = (~0ULL >> (64 - length)) << start;
-    return (value & ~mask) | ((fieldval << start) & mask);
-}
-
-#endif
diff --git a/contrib/qemu/include/qemu/bswap.h b/contrib/qemu/include/qemu/bswap.h
deleted file mode 100644
index bf6457b8c46..00000000000
--- a/contrib/qemu/include/qemu/bswap.h
+++ /dev/null
@@ -1,487 +0,0 @@
-#ifndef BSWAP_H
-#define BSWAP_H
-
-#include "config-host.h"
-#include <inttypes.h>
-#include <limits.h>
-#include <string.h>
-#include "fpu/softfloat.h"
-
-#ifdef CONFIG_MACHINE_BSWAP_H
-# include <sys/endian.h>
-# include <sys/types.h>
-# include <machine/bswap.h>
-#elif defined(__FreeBSD__)
-# include <sys/endian.h>
-#elif defined(CONFIG_BYTESWAP_H)
-#ifdef GF_DARWIN_HOST_OS
-# include <libkern/OSByteOrder.h>
-#define bswap_16(x) OSSwapInt16(x)
-#define bswap_32(x) OSSwapInt32(x)
-#define bswap_64(x) OSSwapInt64(x)
-#else
-# include <byteswap.h>
-#endif
-
-static inline uint16_t bswap16(uint16_t x)
-{
-    return bswap_16(x);
-}
-
-static inline uint32_t bswap32(uint32_t x)
-{
-    return bswap_32(x);
-}
-
-static inline uint64_t bswap64(uint64_t x)
-{
-    return bswap_64(x);
-}
-# else
-static inline uint16_t bswap16(uint16_t x)
-{
-    return (((x & 0x00ff) << 8) |
-            ((x & 0xff00) >> 8));
-}
-
-static inline uint32_t bswap32(uint32_t x)
-{
-    return (((x & 0x000000ffU) << 24) |
-            ((x & 0x0000ff00U) <<  8) |
-            ((x & 0x00ff0000U) >>  8) |
-            ((x & 0xff000000U) >> 24));
-}
-
-static inline uint64_t bswap64(uint64_t x)
-{
-    return (((x & 0x00000000000000ffULL) << 56) |
-            ((x & 0x000000000000ff00ULL) << 40) |
-            ((x & 0x0000000000ff0000ULL) << 24) |
-            ((x & 0x00000000ff000000ULL) <<  8) |
-            ((x & 0x000000ff00000000ULL) >>  8) |
-            ((x & 0x0000ff0000000000ULL) >> 24) |
-            ((x & 0x00ff000000000000ULL) >> 40) |
-            ((x & 0xff00000000000000ULL) >> 56));
-}
-#endif /* ! CONFIG_MACHINE_BSWAP_H */
-
-static inline void bswap16s(uint16_t *s)
-{
-    *s = bswap16(*s);
-}
-
-static inline void bswap32s(uint32_t *s)
-{
-    *s = bswap32(*s);
-}
-
-static inline void bswap64s(uint64_t *s)
-{
-    *s = bswap64(*s);
-}
-
-#if defined(HOST_WORDS_BIGENDIAN)
-#define be_bswap(v, size) (v)
-#define le_bswap(v, size) glue(bswap, size)(v)
-#define be_bswaps(v, size)
-#define le_bswaps(p, size) do { *p = glue(bswap, size)(*p); } while(0)
-#else
-#define le_bswap(v, size) (v)
-#define be_bswap(v, size) glue(bswap, size)(v)
-#define le_bswaps(v, size)
-#define be_bswaps(p, size) do { *p = glue(bswap, size)(*p); } while(0)
-#endif
-
-#define CPU_CONVERT(endian, size, type)\
-static inline type endian ## size ## _to_cpu(type v)\
-{\
-    return glue(endian, _bswap)(v, size);\
-}\
-\
-static inline type cpu_to_ ## endian ## size(type v)\
-{\
-    return glue(endian, _bswap)(v, size);\
-}\
-\
-static inline void endian ## size ## _to_cpus(type *p)\
-{\
-    glue(endian, _bswaps)(p, size);\
-}\
-\
-static inline void cpu_to_ ## endian ## size ## s(type *p)\
-{\
-    glue(endian, _bswaps)(p, size);\
-}\
-\
-static inline type endian ## size ## _to_cpup(const type *p)\
-{\
-    return glue(glue(endian, size), _to_cpu)(*p);\
-}\
-\
-static inline void cpu_to_ ## endian ## size ## w(type *p, type v)\
-{\
-    *p = glue(glue(cpu_to_, endian), size)(v);\
-}
-
-CPU_CONVERT(be, 16, uint16_t)
-CPU_CONVERT(be, 32, uint32_t)
-CPU_CONVERT(be, 64, uint64_t)
-
-CPU_CONVERT(le, 16, uint16_t)
-CPU_CONVERT(le, 32, uint32_t)
-CPU_CONVERT(le, 64, uint64_t)
-
-/* len must be one of 1, 2, 4 */
-static inline uint32_t qemu_bswap_len(uint32_t value, int len)
-{
-    return bswap32(value) >> (32 - 8 * len);
-}
-
-/* Unions for reinterpreting between floats and integers.  */
-
-typedef union {
-    float32 f;
-    uint32_t l;
-} CPU_FloatU;
-
-typedef union {
-    float64 d;
-#if defined(HOST_WORDS_BIGENDIAN)
-    struct {
-        uint32_t upper;
-        uint32_t lower;
-    } l;
-#else
-    struct {
-        uint32_t lower;
-        uint32_t upper;
-    } l;
-#endif
-    uint64_t ll;
-} CPU_DoubleU;
-
-typedef union {
-     floatx80 d;
-     struct {
-         uint64_t lower;
-         uint16_t upper;
-     } l;
-} CPU_LDoubleU;
-
-typedef union {
-    float128 q;
-#if defined(HOST_WORDS_BIGENDIAN)
-    struct {
-        uint32_t upmost;
-        uint32_t upper;
-        uint32_t lower;
-        uint32_t lowest;
-    } l;
-    struct {
-        uint64_t upper;
-        uint64_t lower;
-    } ll;
-#else
-    struct {
-        uint32_t lowest;
-        uint32_t lower;
-        uint32_t upper;
-        uint32_t upmost;
-    } l;
-    struct {
-        uint64_t lower;
-        uint64_t upper;
-    } ll;
-#endif
-} CPU_QuadU;
-
-/* unaligned/endian-independent pointer access */
-
-/*
- * the generic syntax is:
- *
- * load: ld{type}{sign}{size}{endian}_p(ptr)
- *
- * store: st{type}{size}{endian}_p(ptr, val)
- *
- * Note there are small differences with the softmmu access API!
- *
- * type is:
- * (empty): integer access
- *   f    : float access
- *
- * sign is:
- * (empty): for floats or 32 bit size
- *   u    : unsigned
- *   s    : signed
- *
- * size is:
- *   b: 8 bits
- *   w: 16 bits
- *   l: 32 bits
- *   q: 64 bits
- *
- * endian is:
- * (empty): host endian
- *   be   : big endian
- *   le   : little endian
- */
-
-static inline int ldub_p(const void *ptr)
-{
-    return *(uint8_t *)ptr;
-}
-
-static inline int ldsb_p(const void *ptr)
-{
-    return *(int8_t *)ptr;
-}
-
-static inline void stb_p(void *ptr, int v)
-{
-    *(uint8_t *)ptr = v;
-}
-
-/* Any compiler worth its salt will turn these memcpy into native unaligned
-   operations.  Thus we don't need to play games with packed attributes, or
-   inline byte-by-byte stores.  */
-
-static inline int lduw_p(const void *ptr)
-{
-    uint16_t r;
-    memcpy(&r, ptr, sizeof(r));
-    return r;
-}
-
-static inline int ldsw_p(const void *ptr)
-{
-    int16_t r;
-    memcpy(&r, ptr, sizeof(r));
-    return r;
-}
-
-static inline void stw_p(void *ptr, uint16_t v)
-{
-    memcpy(ptr, &v, sizeof(v));
-}
-
-static inline int ldl_p(const void *ptr)
-{
-    int32_t r;
-    memcpy(&r, ptr, sizeof(r));
-    return r;
-}
-
-static inline void stl_p(void *ptr, uint32_t v)
-{
-    memcpy(ptr, &v, sizeof(v));
-}
-
-static inline uint64_t ldq_p(const void *ptr)
-{
-    uint64_t r;
-    memcpy(&r, ptr, sizeof(r));
-    return r;
-}
-
-static inline void stq_p(void *ptr, uint64_t v)
-{
-    memcpy(ptr, &v, sizeof(v));
-}
-
-static inline int lduw_le_p(const void *ptr)
-{
-    return (uint16_t)le_bswap(lduw_p(ptr), 16);
-}
-
-static inline int ldsw_le_p(const void *ptr)
-{
-    return (int16_t)le_bswap(lduw_p(ptr), 16);
-}
-
-static inline int ldl_le_p(const void *ptr)
-{
-    return le_bswap(ldl_p(ptr), 32);
-}
-
-static inline uint64_t ldq_le_p(const void *ptr)
-{
-    return le_bswap(ldq_p(ptr), 64);
-}
-
-static inline void stw_le_p(void *ptr, int v)
-{
-    stw_p(ptr, le_bswap(v, 16));
-}
-
-static inline void stl_le_p(void *ptr, int v)
-{
-    stl_p(ptr, le_bswap(v, 32));
-}
-
-static inline void stq_le_p(void *ptr, uint64_t v)
-{
-    stq_p(ptr, le_bswap(v, 64));
-}
-
-/* float access */
-
-static inline float32 ldfl_le_p(const void *ptr)
-{
-    CPU_FloatU u;
-    u.l = ldl_le_p(ptr);
-    return u.f;
-}
-
-static inline void stfl_le_p(void *ptr, float32 v)
-{
-    CPU_FloatU u;
-    u.f = v;
-    stl_le_p(ptr, u.l);
-}
-
-static inline float64 ldfq_le_p(const void *ptr)
-{
-    CPU_DoubleU u;
-    u.ll = ldq_le_p(ptr);
-    return u.d;
-}
-
-static inline void stfq_le_p(void *ptr, float64 v)
-{
-    CPU_DoubleU u;
-    u.d = v;
-    stq_le_p(ptr, u.ll);
-}
-
-static inline int lduw_be_p(const void *ptr)
-{
-    return (uint16_t)be_bswap(lduw_p(ptr), 16);
-}
-
-static inline int ldsw_be_p(const void *ptr)
-{
-    return (int16_t)be_bswap(lduw_p(ptr), 16);
-}
-
-static inline int ldl_be_p(const void *ptr)
-{
-    return be_bswap(ldl_p(ptr), 32);
-}
-
-static inline uint64_t ldq_be_p(const void *ptr)
-{
-    return be_bswap(ldq_p(ptr), 64);
-}
-
-static inline void stw_be_p(void *ptr, int v)
-{
-    stw_p(ptr, be_bswap(v, 16));
-}
-
-static inline void stl_be_p(void *ptr, int v)
-{
-    stl_p(ptr, be_bswap(v, 32));
-}
-
-static inline void stq_be_p(void *ptr, uint64_t v)
-{
-    stq_p(ptr, be_bswap(v, 64));
-}
-
-/* float access */
-
-static inline float32 ldfl_be_p(const void *ptr)
-{
-    CPU_FloatU u;
-    u.l = ldl_be_p(ptr);
-    return u.f;
-}
-
-static inline void stfl_be_p(void *ptr, float32 v)
-{
-    CPU_FloatU u;
-    u.f = v;
-    stl_be_p(ptr, u.l);
-}
-
-static inline float64 ldfq_be_p(const void *ptr)
-{
-    CPU_DoubleU u;
-    u.ll = ldq_be_p(ptr);
-    return u.d;
-}
-
-static inline void stfq_be_p(void *ptr, float64 v)
-{
-    CPU_DoubleU u;
-    u.d = v;
-    stq_be_p(ptr, u.ll);
-}
-
-/* Legacy unaligned versions.  Note that we never had a complete set.  */
-
-static inline void cpu_to_le16wu(uint16_t *p, uint16_t v)
-{
-    stw_le_p(p, v);
-}
-
-static inline void cpu_to_le32wu(uint32_t *p, uint32_t v)
-{
-    stl_le_p(p, v);
-}
-
-static inline uint16_t le16_to_cpupu(const uint16_t *p)
-{
-    return lduw_le_p(p);
-}
-
-static inline uint32_t le32_to_cpupu(const uint32_t *p)
-{
-    return ldl_le_p(p);
-}
-
-static inline uint32_t be32_to_cpupu(const uint32_t *p)
-{
-    return ldl_be_p(p);
-}
-
-static inline void cpu_to_be16wu(uint16_t *p, uint16_t v)
-{
-    stw_be_p(p, v);
-}
-
-static inline void cpu_to_be32wu(uint32_t *p, uint32_t v)
-{
-    stl_be_p(p, v);
-}
-
-static inline void cpu_to_be64wu(uint64_t *p, uint64_t v)
-{
-    stq_be_p(p, v);
-}
-
-static inline void cpu_to_32wu(uint32_t *p, uint32_t v)
-{
-    stl_p(p, v);
-}
-
-static inline unsigned long leul_to_cpu(unsigned long v)
-{
-    /* In order to break an include loop between here and
-       qemu-common.h, don't rely on HOST_LONG_BITS.  */
-#if ULONG_MAX == UINT32_MAX
-    return le_bswap(v, 32);
-#elif ULONG_MAX == UINT64_MAX
-    return le_bswap(v, 64);
-#else
-# error Unknown sizeof long
-#endif
-}
-
-#undef le_bswap
-#undef be_bswap
-#undef le_bswaps
-#undef be_bswaps
-
-#endif /* BSWAP_H */
diff --git a/contrib/qemu/include/qemu/compiler.h b/contrib/qemu/include/qemu/compiler.h
deleted file mode 100644
index 155b358964a..00000000000
--- a/contrib/qemu/include/qemu/compiler.h
+++ /dev/null
@@ -1,55 +0,0 @@
-/* public domain */
-
-#ifndef COMPILER_H
-#define COMPILER_H
-
-#include "config-host.h"
-
-/*----------------------------------------------------------------------------
-| The macro QEMU_GNUC_PREREQ tests for minimum version of the GNU C compiler.
-| The code is a copy of SOFTFLOAT_GNUC_PREREQ, see softfloat-macros.h.
-*----------------------------------------------------------------------------*/
-#if defined(__GNUC__) && defined(__GNUC_MINOR__)
-# define QEMU_GNUC_PREREQ(maj, min) \
-         ((__GNUC__ << 16) + __GNUC_MINOR__ >= ((maj) << 16) + (min))
-#else
-# define QEMU_GNUC_PREREQ(maj, min) 0
-#endif
-
-#define QEMU_NORETURN __attribute__ ((__noreturn__))
-
-#if QEMU_GNUC_PREREQ(3, 4)
-#define QEMU_WARN_UNUSED_RESULT __attribute__((warn_unused_result))
-#else
-#define QEMU_WARN_UNUSED_RESULT
-#endif
-
-#if defined(_WIN32)
-# define QEMU_PACKED __attribute__((gcc_struct, packed))
-#else
-# define QEMU_PACKED __attribute__((packed))
-#endif
-
-#define cat(x,y) x ## y
-#define cat2(x,y) cat(x,y)
-#define QEMU_BUILD_BUG_ON(x) \
-    typedef char cat2(qemu_build_bug_on__,__LINE__)[(x)?-1:1] __attribute__((unused));
-
-#if defined __GNUC__
-# if !QEMU_GNUC_PREREQ(4, 4)
-   /* gcc versions before 4.4.x don't support gnu_printf, so use printf. */
-#  define GCC_FMT_ATTR(n, m) __attribute__((format(printf, n, m)))
-# else
-   /* Use gnu_printf when supported (qemu uses standard format strings). */
-#  define GCC_FMT_ATTR(n, m) __attribute__((format(gnu_printf, n, m)))
-#  if defined(_WIN32)
-    /* Map __printf__ to __gnu_printf__ because we want standard format strings
-     * even when MinGW or GLib include files use __printf__. */
-#   define __printf__ __gnu_printf__
-#  endif
-# endif
-#else
-#define GCC_FMT_ATTR(n, m)
-#endif
-
-#endif /* COMPILER_H */
diff --git a/contrib/qemu/include/qemu/error-report.h b/contrib/qemu/include/qemu/error-report.h
deleted file mode 100644
index 3b098a91730..00000000000
--- a/contrib/qemu/include/qemu/error-report.h
+++ /dev/null
@@ -1,46 +0,0 @@
-/*
- * Error reporting
- *
- * Copyright (C) 2010 Red Hat Inc.
- *
- * Authors:
- *  Markus Armbruster <armbru@redhat.com>,
- *
- * This work is licensed under the terms of the GNU GPL, version 2 or later.
- * See the COPYING file in the top-level directory.
- */
-
-#ifndef QEMU_ERROR_H
-#define QEMU_ERROR_H
-
-#include <stdarg.h>
-#include <stdbool.h>
-#include "qemu/compiler.h"
-
-typedef struct Location {
-    /* all members are private to qemu-error.c */
-    enum { LOC_NONE, LOC_CMDLINE, LOC_FILE } kind;
-    int num;
-    const void *ptr;
-    struct Location *prev;
-} Location;
-
-Location *loc_push_restore(Location *loc);
-Location *loc_push_none(Location *loc);
-Location *loc_pop(Location *loc);
-Location *loc_save(Location *loc);
-void loc_restore(Location *loc);
-void loc_set_none(void);
-void loc_set_cmdline(char **argv, int idx, int cnt);
-void loc_set_file(const char *fname, int lno);
-
-void error_vprintf(const char *fmt, va_list ap) GCC_FMT_ATTR(1, 0);
-void error_printf(const char *fmt, ...) GCC_FMT_ATTR(1, 2);
-void error_printf_unless_qmp(const char *fmt, ...) GCC_FMT_ATTR(1, 2);
-void error_print_loc(void);
-void error_set_progname(const char *argv0);
-void error_report(const char *fmt, ...) GCC_FMT_ATTR(1, 2);
-const char *error_get_progname(void);
-extern bool enable_timestamp_msg;
-
-#endif
diff --git a/contrib/qemu/include/qemu/event_notifier.h b/contrib/qemu/include/qemu/event_notifier.h
deleted file mode 100644
index 88b57af7ce8..00000000000
--- a/contrib/qemu/include/qemu/event_notifier.h
+++ /dev/null
@@ -1,46 +0,0 @@
-/*
- * event notifier support
- *
- * Copyright Red Hat, Inc. 2010
- *
- * Authors:
- *  Michael S. Tsirkin <mst@redhat.com>
- *
- * This work is licensed under the terms of the GNU GPL, version 2 or later.
- * See the COPYING file in the top-level directory.
- */
-
-#ifndef QEMU_EVENT_NOTIFIER_H
-#define QEMU_EVENT_NOTIFIER_H
-
-#include "qemu-common.h"
-
-#ifdef _WIN32
-#include <windows.h>
-#endif
-
-struct EventNotifier {
-#ifdef _WIN32
-    HANDLE event;
-#else
-    int rfd;
-    int wfd;
-#endif
-};
-
-typedef void EventNotifierHandler(EventNotifier *);
-
-int event_notifier_init(EventNotifier *, int active);
-void event_notifier_cleanup(EventNotifier *);
-int event_notifier_set(EventNotifier *);
-int event_notifier_test_and_clear(EventNotifier *);
-int event_notifier_set_handler(EventNotifier *, EventNotifierHandler *);
-
-#ifdef CONFIG_POSIX
-void event_notifier_init_fd(EventNotifier *, int fd);
-int event_notifier_get_fd(EventNotifier *);
-#else
-HANDLE event_notifier_get_handle(EventNotifier *);
-#endif
-
-#endif
diff --git a/contrib/qemu/include/qemu/hbitmap.h b/contrib/qemu/include/qemu/hbitmap.h
deleted file mode 100644
index 550d7ce2c39..00000000000
--- a/contrib/qemu/include/qemu/hbitmap.h
+++ /dev/null
@@ -1,209 +0,0 @@
-/*
- * Hierarchical Bitmap Data Type
- *
- * Copyright Red Hat, Inc., 2012
- *
- * Author: Paolo Bonzini <pbonzini@redhat.com>
- *
- * This work is licensed under the terms of the GNU GPL, version 2 or
- * later.  See the COPYING file in the top-level directory.
- */
-
-#ifndef HBITMAP_H
-#define HBITMAP_H 1
-
-#include <limits.h>
-#include <stdint.h>
-#include <stdbool.h>
-#include "bitops.h"
-#include "host-utils.h"
-
-typedef struct HBitmap HBitmap;
-typedef struct HBitmapIter HBitmapIter;
-
-#define BITS_PER_LEVEL         (BITS_PER_LONG == 32 ? 5 : 6)
-
-/* For 32-bit, the largest that fits in a 4 GiB address space.
- * For 64-bit, the number of sectors in 1 PiB.  Good luck, in
- * either case... :)
- */
-#define HBITMAP_LOG_MAX_SIZE   (BITS_PER_LONG == 32 ? 34 : 41)
-
-/* We need to place a sentinel in level 0 to speed up iteration.  Thus,
- * we do this instead of HBITMAP_LOG_MAX_SIZE / BITS_PER_LEVEL.  The
- * difference is that it allocates an extra level when HBITMAP_LOG_MAX_SIZE
- * is an exact multiple of BITS_PER_LEVEL.
- */
-#define HBITMAP_LEVELS         ((HBITMAP_LOG_MAX_SIZE / BITS_PER_LEVEL) + 1)
-
-struct HBitmapIter {
-    const HBitmap *hb;
-
-    /* Copied from hb for access in the inline functions (hb is opaque).  */
-    int granularity;
-
-    /* Entry offset into the last-level array of longs.  */
-    size_t pos;
-
-    /* The currently-active path in the tree.  Each item of cur[i] stores
-     * the bits (i.e. the subtrees) yet to be processed under that node.
-     */
-    unsigned long cur[HBITMAP_LEVELS];
-};
-
-/**
- * hbitmap_alloc:
- * @size: Number of bits in the bitmap.
- * @granularity: Granularity of the bitmap.  Aligned groups of 2^@granularity
- * bits will be represented by a single bit.  Each operation on a
- * range of bits first rounds the bits to determine which group they land
- * in, and then affect the entire set; iteration will only visit the first
- * bit of each group.
- *
- * Allocate a new HBitmap.
- */
-HBitmap *hbitmap_alloc(uint64_t size, int granularity);
-
-/**
- * hbitmap_empty:
- * @hb: HBitmap to operate on.
- *
- * Return whether the bitmap is empty.
- */
-bool hbitmap_empty(const HBitmap *hb);
-
-/**
- * hbitmap_granularity:
- * @hb: HBitmap to operate on.
- *
- * Return the granularity of the HBitmap.
- */
-int hbitmap_granularity(const HBitmap *hb);
-
-/**
- * hbitmap_count:
- * @hb: HBitmap to operate on.
- *
- * Return the number of bits set in the HBitmap.
- */
-uint64_t hbitmap_count(const HBitmap *hb);
-
-/**
- * hbitmap_set:
- * @hb: HBitmap to operate on.
- * @start: First bit to set (0-based).
- * @count: Number of bits to set.
- *
- * Set a consecutive range of bits in an HBitmap.
- */
-void hbitmap_set(HBitmap *hb, uint64_t start, uint64_t count);
-
-/**
- * hbitmap_reset:
- * @hb: HBitmap to operate on.
- * @start: First bit to reset (0-based).
- * @count: Number of bits to reset.
- *
- * Reset a consecutive range of bits in an HBitmap.
- */
-void hbitmap_reset(HBitmap *hb, uint64_t start, uint64_t count);
-
-/**
- * hbitmap_get:
- * @hb: HBitmap to operate on.
- * @item: Bit to query (0-based).
- *
- * Return whether the @item-th bit in an HBitmap is set.
- */
-bool hbitmap_get(const HBitmap *hb, uint64_t item);
-
-/**
- * hbitmap_free:
- * @hb: HBitmap to operate on.
- *
- * Free an HBitmap and all of its associated memory.
- */
-void hbitmap_free(HBitmap *hb);
-
-/**
- * hbitmap_iter_init:
- * @hbi: HBitmapIter to initialize.
- * @hb: HBitmap to iterate on.
- * @first: First bit to visit (0-based, must be strictly less than the
- * size of the bitmap).
- *
- * Set up @hbi to iterate on the HBitmap @hb.  hbitmap_iter_next will return
- * the lowest-numbered bit that is set in @hb, starting at @first.
- *
- * Concurrent setting of bits is acceptable, and will at worst cause the
- * iteration to miss some of those bits.  Resetting bits before the current
- * position of the iterator is also okay.  However, concurrent resetting of
- * bits can lead to unexpected behavior if the iterator has not yet reached
- * those bits.
- */
-void hbitmap_iter_init(HBitmapIter *hbi, const HBitmap *hb, uint64_t first);
-
-/* hbitmap_iter_skip_words:
- * @hbi: HBitmapIter to operate on.
- *
- * Internal function used by hbitmap_iter_next and hbitmap_iter_next_word.
- */
-unsigned long hbitmap_iter_skip_words(HBitmapIter *hbi);
-
-/**
- * hbitmap_iter_next:
- * @hbi: HBitmapIter to operate on.
- *
- * Return the next bit that is set in @hbi's associated HBitmap,
- * or -1 if all remaining bits are zero.
- */
-static inline int64_t hbitmap_iter_next(HBitmapIter *hbi)
-{
-    unsigned long cur = hbi->cur[HBITMAP_LEVELS - 1];
-    int64_t item;
-
-    if (cur == 0) {
-        cur = hbitmap_iter_skip_words(hbi);
-        if (cur == 0) {
-            return -1;
-        }
-    }
-
-    /* The next call will resume work from the next bit.  */
-    hbi->cur[HBITMAP_LEVELS - 1] = cur & (cur - 1);
-    item = ((uint64_t)hbi->pos << BITS_PER_LEVEL) + ctzl(cur);
-
-    return item << hbi->granularity;
-}
-
-/**
- * hbitmap_iter_next_word:
- * @hbi: HBitmapIter to operate on.
- * @p_cur: Location where to store the next non-zero word.
- *
- * Return the index of the next nonzero word that is set in @hbi's
- * associated HBitmap, and set *p_cur to the content of that word
- * (bits before the index that was passed to hbitmap_iter_init are
- * trimmed on the first call).  Return -1, and set *p_cur to zero,
- * if all remaining words are zero.
- */
-static inline size_t hbitmap_iter_next_word(HBitmapIter *hbi, unsigned long *p_cur)
-{
-    unsigned long cur = hbi->cur[HBITMAP_LEVELS - 1];
-
-    if (cur == 0) {
-        cur = hbitmap_iter_skip_words(hbi);
-        if (cur == 0) {
-            *p_cur = 0;
-            return -1;
-        }
-    }
-
-    /* The next call will resume work from the next word.  */
-    hbi->cur[HBITMAP_LEVELS - 1] = 0;
-    *p_cur = cur;
-    return hbi->pos;
-}
-
-
-#endif
diff --git a/contrib/qemu/include/qemu/host-utils.h b/contrib/qemu/include/qemu/host-utils.h
deleted file mode 100644
index 0f688c1c00a..00000000000
--- a/contrib/qemu/include/qemu/host-utils.h
+++ /dev/null
@@ -1,322 +0,0 @@
-/*
- * Utility compute operations used by translated code.
- *
- * Copyright (c) 2007 Thiemo Seufer
- * Copyright (c) 2007 Jocelyn Mayer
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-#ifndef HOST_UTILS_H
-#define HOST_UTILS_H 1
-
-#include "qemu/compiler.h"   /* QEMU_GNUC_PREREQ */
-#include <limits.h>
-
-#ifdef CONFIG_INT128
-static inline void mulu64(uint64_t *plow, uint64_t *phigh,
-                          uint64_t a, uint64_t b)
-{
-    __uint128_t r = (__uint128_t)a * b;
-    *plow = r;
-    *phigh = r >> 64;
-}
-
-static inline void muls64(uint64_t *plow, uint64_t *phigh,
-                          int64_t a, int64_t b)
-{
-    __int128_t r = (__int128_t)a * b;
-    *plow = r;
-    *phigh = r >> 64;
-}
-#else
-void muls64(uint64_t *phigh, uint64_t *plow, int64_t a, int64_t b);
-void mulu64(uint64_t *phigh, uint64_t *plow, uint64_t a, uint64_t b);
-#endif
-
-/**
- * clz32 - count leading zeros in a 32-bit value.
- * @val: The value to search
- *
- * Returns 32 if the value is zero.  Note that the GCC builtin is
- * undefined if the value is zero.
- */
-static inline int clz32(uint32_t val)
-{
-#if QEMU_GNUC_PREREQ(3, 4)
-    return val ? __builtin_clz(val) : 32;
-#else
-    /* Binary search for the leading one bit.  */
-    int cnt = 0;
-
-    if (!(val & 0xFFFF0000U)) {
-        cnt += 16;
-        val <<= 16;
-    }
-    if (!(val & 0xFF000000U)) {
-        cnt += 8;
-        val <<= 8;
-    }
-    if (!(val & 0xF0000000U)) {
-        cnt += 4;
-        val <<= 4;
-    }
-    if (!(val & 0xC0000000U)) {
-        cnt += 2;
-        val <<= 2;
-    }
-    if (!(val & 0x80000000U)) {
-        cnt++;
-        val <<= 1;
-    }
-    if (!(val & 0x80000000U)) {
-        cnt++;
-    }
-    return cnt;
-#endif
-}
-
-/**
- * clo32 - count leading ones in a 32-bit value.
- * @val: The value to search
- *
- * Returns 32 if the value is -1.
- */
-static inline int clo32(uint32_t val)
-{
-    return clz32(~val);
-}
-
-/**
- * clz64 - count leading zeros in a 64-bit value.
- * @val: The value to search
- *
- * Returns 64 if the value is zero.  Note that the GCC builtin is
- * undefined if the value is zero.
- */
-static inline int clz64(uint64_t val)
-{
-#if QEMU_GNUC_PREREQ(3, 4)
-    return val ? __builtin_clzll(val) : 64;
-#else
-    int cnt = 0;
-
-    if (!(val >> 32)) {
-        cnt += 32;
-    } else {
-        val >>= 32;
-    }
-
-    return cnt + clz32(val);
-#endif
-}
-
-/**
- * clo64 - count leading ones in a 64-bit value.
- * @val: The value to search
- *
- * Returns 64 if the value is -1.
- */
-static inline int clo64(uint64_t val)
-{
-    return clz64(~val);
-}
-
-/**
- * ctz32 - count trailing zeros in a 32-bit value.
- * @val: The value to search
- *
- * Returns 32 if the value is zero.  Note that the GCC builtin is
- * undefined if the value is zero.
- */
-static inline int ctz32(uint32_t val)
-{
-#if QEMU_GNUC_PREREQ(3, 4)
-    return val ? __builtin_ctz(val) : 32;
-#else
-    /* Binary search for the trailing one bit.  */
-    int cnt;
-
-    cnt = 0;
-    if (!(val & 0x0000FFFFUL)) {
-        cnt += 16;
-        val >>= 16;
-    }
-    if (!(val & 0x000000FFUL)) {
-        cnt += 8;
-        val >>= 8;
-    }
-    if (!(val & 0x0000000FUL)) {
-        cnt += 4;
-        val >>= 4;
-    }
-    if (!(val & 0x00000003UL)) {
-        cnt += 2;
-        val >>= 2;
-    }
-    if (!(val & 0x00000001UL)) {
-        cnt++;
-        val >>= 1;
-    }
-    if (!(val & 0x00000001UL)) {
-        cnt++;
-    }
-
-    return cnt;
-#endif
-}
-
-/**
- * cto32 - count trailing ones in a 32-bit value.
- * @val: The value to search
- *
- * Returns 32 if the value is -1.
- */
-static inline int cto32(uint32_t val)
-{
-    return ctz32(~val);
-}
-
-/**
- * ctz64 - count trailing zeros in a 64-bit value.
- * @val: The value to search
- *
- * Returns 64 if the value is zero.  Note that the GCC builtin is
- * undefined if the value is zero.
- */
-static inline int ctz64(uint64_t val)
-{
-#if QEMU_GNUC_PREREQ(3, 4)
-    return val ? __builtin_ctzll(val) : 64;
-#else
-    int cnt;
-
-    cnt = 0;
-    if (!((uint32_t)val)) {
-        cnt += 32;
-        val >>= 32;
-    }
-
-    return cnt + ctz32(val);
-#endif
-}
-
-/**
- * ctz64 - count trailing ones in a 64-bit value.
- * @val: The value to search
- *
- * Returns 64 if the value is -1.
- */
-static inline int cto64(uint64_t val)
-{
-    return ctz64(~val);
-}
-
-/**
- * ctpop8 - count the population of one bits in an 8-bit value.
- * @val: The value to search
- */
-static inline int ctpop8(uint8_t val)
-{
-#if QEMU_GNUC_PREREQ(3, 4)
-    return __builtin_popcount(val);
-#else
-    val = (val & 0x55) + ((val >> 1) & 0x55);
-    val = (val & 0x33) + ((val >> 2) & 0x33);
-    val = (val & 0x0f) + ((val >> 4) & 0x0f);
-
-    return val;
-#endif
-}
-
-/**
- * ctpop16 - count the population of one bits in a 16-bit value.
- * @val: The value to search
- */
-static inline int ctpop16(uint16_t val)
-{
-#if QEMU_GNUC_PREREQ(3, 4)
-    return __builtin_popcount(val);
-#else
-    val = (val & 0x5555) + ((val >> 1) & 0x5555);
-    val = (val & 0x3333) + ((val >> 2) & 0x3333);
-    val = (val & 0x0f0f) + ((val >> 4) & 0x0f0f);
-    val = (val & 0x00ff) + ((val >> 8) & 0x00ff);
-
-    return val;
-#endif
-}
-
-/**
- * ctpop32 - count the population of one bits in a 32-bit value.
- * @val: The value to search
- */
-static inline int ctpop32(uint32_t val)
-{
-#if QEMU_GNUC_PREREQ(3, 4)
-    return __builtin_popcount(val);
-#else
-    val = (val & 0x55555555) + ((val >>  1) & 0x55555555);
-    val = (val & 0x33333333) + ((val >>  2) & 0x33333333);
-    val = (val & 0x0f0f0f0f) + ((val >>  4) & 0x0f0f0f0f);
-    val = (val & 0x00ff00ff) + ((val >>  8) & 0x00ff00ff);
-    val = (val & 0x0000ffff) + ((val >> 16) & 0x0000ffff);
-
-    return val;
-#endif
-}
-
-/**
- * ctpop64 - count the population of one bits in a 64-bit value.
- * @val: The value to search
- */
-static inline int ctpop64(uint64_t val)
-{
-#if QEMU_GNUC_PREREQ(3, 4)
-    return __builtin_popcountll(val);
-#else
-    val = (val & 0x5555555555555555ULL) + ((val >>  1) & 0x5555555555555555ULL);
-    val = (val & 0x3333333333333333ULL) + ((val >>  2) & 0x3333333333333333ULL);
-    val = (val & 0x0f0f0f0f0f0f0f0fULL) + ((val >>  4) & 0x0f0f0f0f0f0f0f0fULL);
-    val = (val & 0x00ff00ff00ff00ffULL) + ((val >>  8) & 0x00ff00ff00ff00ffULL);
-    val = (val & 0x0000ffff0000ffffULL) + ((val >> 16) & 0x0000ffff0000ffffULL);
-    val = (val & 0x00000000ffffffffULL) + ((val >> 32) & 0x00000000ffffffffULL);
-
-    return val;
-#endif
-}
-
-/* Host type specific sizes of these routines.  */
-
-#if ULONG_MAX == UINT32_MAX
-# define clzl   clz32
-# define ctzl   ctz32
-# define clol   clo32
-# define ctol   cto32
-# define ctpopl ctpop32
-#elif ULONG_MAX == UINT64_MAX
-# define clzl   clz64
-# define ctzl   ctz64
-# define clol   clo64
-# define ctol   cto64
-# define ctpopl ctpop64
-#else
-# error Unknown sizeof long
-#endif
-
-#endif
diff --git a/contrib/qemu/include/qemu/iov.h b/contrib/qemu/include/qemu/iov.h
deleted file mode 100644
index 68d25f29b76..00000000000
--- a/contrib/qemu/include/qemu/iov.h
+++ /dev/null
@@ -1,115 +0,0 @@
-/*
- * Helpers for using (partial) iovecs.
- *
- * Copyright (C) 2010 Red Hat, Inc.
- *
- * Author(s):
- *  Amit Shah <amit.shah@redhat.com>
- *  Michael Tokarev <mjt@tls.msk.ru>
- *
- * This work is licensed under the terms of the GNU GPL, version 2.  See
- * the COPYING file in the top-level directory.
- */
-
-#ifndef IOV_H
-#define IOV_H
-
-#include "qemu-common.h"
-
-/**
- * count and return data size, in bytes, of an iovec
- * starting at `iov' of `iov_cnt' number of elements.
- */
-size_t iov_size(const struct iovec *iov, const unsigned int iov_cnt);
-
-/**
- * Copy from single continuous buffer to scatter-gather vector of buffers
- * (iovec) and back like memcpy() between two continuous memory regions.
- * Data in single continuous buffer starting at address `buf' and
- * `bytes' bytes long will be copied to/from an iovec `iov' with
- * `iov_cnt' number of elements, starting at byte position `offset'
- * within the iovec.  If the iovec does not contain enough space,
- * only part of data will be copied, up to the end of the iovec.
- * Number of bytes actually copied will be returned, which is
- *  min(bytes, iov_size(iov)-offset)
- * `Offset' must point to the inside of iovec.
- * It is okay to use very large value for `bytes' since we're
- * limited by the size of the iovec anyway, provided that the
- * buffer pointed to by buf has enough space.  One possible
- * such "large" value is -1 (sinice size_t is unsigned),
- * so specifying `-1' as `bytes' means 'up to the end of iovec'.
- */
-size_t iov_from_buf(const struct iovec *iov, unsigned int iov_cnt,
-                    size_t offset, const void *buf, size_t bytes);
-size_t iov_to_buf(const struct iovec *iov, const unsigned int iov_cnt,
-                  size_t offset, void *buf, size_t bytes);
-
-/**
- * Set data bytes pointed out by iovec `iov' of size `iov_cnt' elements,
- * starting at byte offset `start', to value `fillc', repeating it
- * `bytes' number of times.  `Offset' must point to the inside of iovec.
- * If `bytes' is large enough, only last bytes portion of iovec,
- * up to the end of it, will be filled with the specified value.
- * Function return actual number of bytes processed, which is
- * min(size, iov_size(iov) - offset).
- * Again, it is okay to use large value for `bytes' to mean "up to the end".
- */
-size_t iov_memset(const struct iovec *iov, const unsigned int iov_cnt,
-                  size_t offset, int fillc, size_t bytes);
-
-/*
- * Send/recv data from/to iovec buffers directly
- *
- * `offset' bytes in the beginning of iovec buffer are skipped and
- * next `bytes' bytes are used, which must be within data of iovec.
- *
- *   r = iov_send_recv(sockfd, iov, iovcnt, offset, bytes, true);
- *
- * is logically equivalent to
- *
- *   char *buf = malloc(bytes);
- *   iov_to_buf(iov, iovcnt, offset, buf, bytes);
- *   r = send(sockfd, buf, bytes, 0);
- *   free(buf);
- *
- * For iov_send_recv() _whole_ area being sent or received
- * should be within the iovec, not only beginning of it.
- */
-ssize_t iov_send_recv(int sockfd, struct iovec *iov, unsigned iov_cnt,
-                      size_t offset, size_t bytes, bool do_send);
-#define iov_recv(sockfd, iov, iov_cnt, offset, bytes) \
-  iov_send_recv(sockfd, iov, iov_cnt, offset, bytes, false)
-#define iov_send(sockfd, iov, iov_cnt, offset, bytes) \
-  iov_send_recv(sockfd, iov, iov_cnt, offset, bytes, true)
-
-/**
- * Produce a text hexdump of iovec `iov' with `iov_cnt' number of elements
- * in file `fp', prefixing each line with `prefix' and processing not more
- * than `limit' data bytes.
- */
-void iov_hexdump(const struct iovec *iov, const unsigned int iov_cnt,
-                 FILE *fp, const char *prefix, size_t limit);
-
-/*
- * Partial copy of vector from iov to dst_iov (data is not copied).
- * dst_iov overlaps iov at a specified offset.
- * size of dst_iov is at most bytes. dst vector count is returned.
- */
-unsigned iov_copy(struct iovec *dst_iov, unsigned int dst_iov_cnt,
-                 const struct iovec *iov, unsigned int iov_cnt,
-                 size_t offset, size_t bytes);
-
-/*
- * Remove a given number of bytes from the front or back of a vector.
- * This may update iov and/or iov_cnt to exclude iovec elements that are
- * no longer required.
- *
- * The number of bytes actually discarded is returned.  This number may be
- * smaller than requested if the vector is too small.
- */
-size_t iov_discard_front(struct iovec **iov, unsigned int *iov_cnt,
-                         size_t bytes);
-size_t iov_discard_back(struct iovec *iov, unsigned int *iov_cnt,
-                        size_t bytes);
-
-#endif
diff --git a/contrib/qemu/include/qemu/main-loop.h b/contrib/qemu/include/qemu/main-loop.h
deleted file mode 100644
index 6f0200a7acc..00000000000
--- a/contrib/qemu/include/qemu/main-loop.h
+++ /dev/null
@@ -1,311 +0,0 @@
-/*
- * QEMU System Emulator
- *
- * Copyright (c) 2003-2008 Fabrice Bellard
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-
-#ifndef QEMU_MAIN_LOOP_H
-#define QEMU_MAIN_LOOP_H 1
-
-#include "block/aio.h"
-
-#define SIG_IPI SIGUSR1
-
-/**
- * qemu_init_main_loop: Set up the process so that it can run the main loop.
- *
- * This includes setting up signal handlers.  It should be called before
- * any other threads are created.  In addition, threads other than the
- * main one should block signals that are trapped by the main loop.
- * For simplicity, you can consider these signals to be safe: SIGUSR1,
- * SIGUSR2, thread signals (SIGFPE, SIGILL, SIGSEGV, SIGBUS) and real-time
- * signals if available.  Remember that Windows in practice does not have
- * signals, though.
- *
- * In the case of QEMU tools, this will also start/initialize timers.
- */
-int qemu_init_main_loop(void);
-
-/**
- * main_loop_wait: Run one iteration of the main loop.
- *
- * If @nonblocking is true, poll for events, otherwise suspend until
- * one actually occurs.  The main loop usually consists of a loop that
- * repeatedly calls main_loop_wait(false).
- *
- * Main loop services include file descriptor callbacks, bottom halves
- * and timers (defined in qemu-timer.h).  Bottom halves are similar to timers
- * that execute immediately, but have a lower overhead and scheduling them
- * is wait-free, thread-safe and signal-safe.
- *
- * It is sometimes useful to put a whole program in a coroutine.  In this
- * case, the coroutine actually should be started from within the main loop,
- * so that the main loop can run whenever the coroutine yields.  To do this,
- * you can use a bottom half to enter the coroutine as soon as the main loop
- * starts:
- *
- *     void enter_co_bh(void *opaque) {
- *         QEMUCoroutine *co = opaque;
- *         qemu_coroutine_enter(co, NULL);
- *     }
- *
- *     ...
- *     QEMUCoroutine *co = qemu_coroutine_create(coroutine_entry);
- *     QEMUBH *start_bh = qemu_bh_new(enter_co_bh, co);
- *     qemu_bh_schedule(start_bh);
- *     while (...) {
- *         main_loop_wait(false);
- *     }
- *
- * (In the future we may provide a wrapper for this).
- *
- * @nonblocking: Whether the caller should block until an event occurs.
- */
-int main_loop_wait(int nonblocking);
-
-/**
- * qemu_get_aio_context: Return the main loop's AioContext
- */
-AioContext *qemu_get_aio_context(void);
-
-/**
- * qemu_notify_event: Force processing of pending events.
- *
- * Similar to signaling a condition variable, qemu_notify_event forces
- * main_loop_wait to look at pending events and exit.  The caller of
- * main_loop_wait will usually call it again very soon, so qemu_notify_event
- * also has the side effect of recalculating the sets of file descriptors
- * that the main loop waits for.
- *
- * Calling qemu_notify_event is rarely necessary, because main loop
- * services (bottom halves and timers) call it themselves.  One notable
- * exception occurs when using qemu_set_fd_handler2 (see below).
- */
-void qemu_notify_event(void);
-
-#ifdef _WIN32
-/* return TRUE if no sleep should be done afterwards */
-typedef int PollingFunc(void *opaque);
-
-/**
- * qemu_add_polling_cb: Register a Windows-specific polling callback
- *
- * Currently, under Windows some events are polled rather than waited for.
- * Polling callbacks do not ensure that @func is called timely, because
- * the main loop might wait for an arbitrarily long time.  If possible,
- * you should instead create a separate thread that does a blocking poll
- * and set a Win32 event object.  The event can then be passed to
- * qemu_add_wait_object.
- *
- * Polling callbacks really have nothing Windows specific in them, but
- * as they are a hack and are currently not necessary under POSIX systems,
- * they are only available when QEMU is running under Windows.
- *
- * @func: The function that does the polling, and returns 1 to force
- * immediate completion of main_loop_wait.
- * @opaque: A pointer-size value that is passed to @func.
- */
-int qemu_add_polling_cb(PollingFunc *func, void *opaque);
-
-/**
- * qemu_del_polling_cb: Unregister a Windows-specific polling callback
- *
- * This function removes a callback that was registered with
- * qemu_add_polling_cb.
- *
- * @func: The function that was passed to qemu_add_polling_cb.
- * @opaque: A pointer-size value that was passed to qemu_add_polling_cb.
- */
-void qemu_del_polling_cb(PollingFunc *func, void *opaque);
-
-/* Wait objects handling */
-typedef void WaitObjectFunc(void *opaque);
-
-/**
- * qemu_add_wait_object: Register a callback for a Windows handle
- *
- * Under Windows, the iohandler mechanism can only be used with sockets.
- * QEMU must use the WaitForMultipleObjects API to wait on other handles.
- * This function registers a #HANDLE with QEMU, so that it will be included
- * in the main loop's calls to WaitForMultipleObjects.  When the handle
- * is in a signaled state, QEMU will call @func.
- *
- * @handle: The Windows handle to be observed.
- * @func: A function to be called when @handle is in a signaled state.
- * @opaque: A pointer-size value that is passed to @func.
- */
-int qemu_add_wait_object(HANDLE handle, WaitObjectFunc *func, void *opaque);
-
-/**
- * qemu_del_wait_object: Unregister a callback for a Windows handle
- *
- * This function removes a callback that was registered with
- * qemu_add_wait_object.
- *
- * @func: The function that was passed to qemu_add_wait_object.
- * @opaque: A pointer-size value that was passed to qemu_add_wait_object.
- */
-void qemu_del_wait_object(HANDLE handle, WaitObjectFunc *func, void *opaque);
-#endif
-
-/* async I/O support */
-
-typedef void IOReadHandler(void *opaque, const uint8_t *buf, int size);
-typedef int IOCanReadHandler(void *opaque);
-
-/**
- * qemu_set_fd_handler2: Register a file descriptor with the main loop
- *
- * This function tells the main loop to wake up whenever one of the
- * following conditions is true:
- *
- * 1) if @fd_write is not %NULL, when the file descriptor is writable;
- *
- * 2) if @fd_read is not %NULL, when the file descriptor is readable.
- *
- * @fd_read_poll can be used to disable the @fd_read callback temporarily.
- * This is useful to avoid calling qemu_set_fd_handler2 every time the
- * client becomes interested in reading (or dually, stops being interested).
- * A typical example is when @fd is a listening socket and you want to bound
- * the number of active clients.  Remember to call qemu_notify_event whenever
- * the condition may change from %false to %true.
- *
- * The callbacks that are set up by qemu_set_fd_handler2 are level-triggered.
- * If @fd_read does not read from @fd, or @fd_write does not write to @fd
- * until its buffers are full, they will be called again on the next
- * iteration.
- *
- * @fd: The file descriptor to be observed.  Under Windows it must be
- * a #SOCKET.
- *
- * @fd_read_poll: A function that returns 1 if the @fd_read callback
- * should be fired.  If the function returns 0, the main loop will not
- * end its iteration even if @fd becomes readable.
- *
- * @fd_read: A level-triggered callback that is fired if @fd is readable
- * at the beginning of a main loop iteration, or if it becomes readable
- * during one.
- *
- * @fd_write: A level-triggered callback that is fired when @fd is writable
- * at the beginning of a main loop iteration, or if it becomes writable
- * during one.
- *
- * @opaque: A pointer-sized value that is passed to @fd_read_poll,
- * @fd_read and @fd_write.
- */
-int qemu_set_fd_handler2(int fd,
-                         IOCanReadHandler *fd_read_poll,
-                         IOHandler *fd_read,
-                         IOHandler *fd_write,
-                         void *opaque);
-
-/**
- * qemu_set_fd_handler: Register a file descriptor with the main loop
- *
- * This function tells the main loop to wake up whenever one of the
- * following conditions is true:
- *
- * 1) if @fd_write is not %NULL, when the file descriptor is writable;
- *
- * 2) if @fd_read is not %NULL, when the file descriptor is readable.
- *
- * The callbacks that are set up by qemu_set_fd_handler are level-triggered.
- * If @fd_read does not read from @fd, or @fd_write does not write to @fd
- * until its buffers are full, they will be called again on the next
- * iteration.
- *
- * @fd: The file descriptor to be observed.  Under Windows it must be
- * a #SOCKET.
- *
- * @fd_read: A level-triggered callback that is fired if @fd is readable
- * at the beginning of a main loop iteration, or if it becomes readable
- * during one.
- *
- * @fd_write: A level-triggered callback that is fired when @fd is writable
- * at the beginning of a main loop iteration, or if it becomes writable
- * during one.
- *
- * @opaque: A pointer-sized value that is passed to @fd_read and @fd_write.
- */
-int qemu_set_fd_handler(int fd,
-                        IOHandler *fd_read,
-                        IOHandler *fd_write,
-                        void *opaque);
-
-#ifdef CONFIG_POSIX
-/**
- * qemu_add_child_watch: Register a child process for reaping.
- *
- * Under POSIX systems, a parent process must read the exit status of
- * its child processes using waitpid, or the operating system will not
- * free some of the resources attached to that process.
- *
- * This function directs the QEMU main loop to observe a child process
- * and call waitpid as soon as it exits; the watch is then removed
- * automatically.  It is useful whenever QEMU forks a child process
- * but will find out about its termination by other means such as a
- * "broken pipe".
- *
- * @pid: The pid that QEMU should observe.
- */
-int qemu_add_child_watch(pid_t pid);
-#endif
-
-/**
- * qemu_mutex_lock_iothread: Lock the main loop mutex.
- *
- * This function locks the main loop mutex.  The mutex is taken by
- * qemu_init_main_loop and always taken except while waiting on
- * external events (such as with select).  The mutex should be taken
- * by threads other than the main loop thread when calling
- * qemu_bh_new(), qemu_set_fd_handler() and basically all other
- * functions documented in this file.
- *
- * NOTE: tools currently are single-threaded and qemu_mutex_lock_iothread
- * is a no-op there.
- */
-void qemu_mutex_lock_iothread(void);
-
-/**
- * qemu_mutex_unlock_iothread: Unlock the main loop mutex.
- *
- * This function unlocks the main loop mutex.  The mutex is taken by
- * qemu_init_main_loop and always taken except while waiting on
- * external events (such as with select).  The mutex should be unlocked
- * as soon as possible by threads other than the main loop thread,
- * because it prevents the main loop from processing callbacks,
- * including timers and bottom halves.
- *
- * NOTE: tools currently are single-threaded and qemu_mutex_unlock_iothread
- * is a no-op there.
- */
-void qemu_mutex_unlock_iothread(void);
-
-/* internal interfaces */
-
-void qemu_fd_register(int fd);
-void qemu_iohandler_fill(GArray *pollfds);
-void qemu_iohandler_poll(GArray *pollfds, int rc);
-
-QEMUBH *qemu_bh_new(QEMUBHFunc *cb, void *opaque);
-void qemu_bh_schedule_idle(QEMUBH *bh);
-
-#endif
diff --git a/contrib/qemu/include/qemu/module.h b/contrib/qemu/include/qemu/module.h
deleted file mode 100644
index c4ccd571664..00000000000
--- a/contrib/qemu/include/qemu/module.h
+++ /dev/null
@@ -1,40 +0,0 @@
-/*
- * QEMU Module Infrastructure
- *
- * Copyright IBM, Corp. 2009
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU GPL, version 2.  See
- * the COPYING file in the top-level directory.
- *
- */
-
-#ifndef QEMU_MODULE_H
-#define QEMU_MODULE_H
-
-/* This should not be used directly.  Use block_init etc. instead.  */
-#define module_init(function, type)                                         \
-static void __attribute__((constructor)) do_qemu_init_ ## function(void) {  \
-    register_module_init(function, type);                                   \
-}
-
-typedef enum {
-    MODULE_INIT_BLOCK,
-    MODULE_INIT_MACHINE,
-    MODULE_INIT_QAPI,
-    MODULE_INIT_QOM,
-    MODULE_INIT_MAX
-} module_init_type;
-
-#define block_init(function) module_init(function, MODULE_INIT_BLOCK)
-#define machine_init(function) module_init(function, MODULE_INIT_MACHINE)
-#define qapi_init(function) module_init(function, MODULE_INIT_QAPI)
-#define type_init(function) module_init(function, MODULE_INIT_QOM)
-
-void register_module_init(void (*fn)(void), module_init_type type);
-
-void module_call_init(module_init_type type);
-
-#endif
diff --git a/contrib/qemu/include/qemu/notify.h b/contrib/qemu/include/qemu/notify.h
deleted file mode 100644
index a3d73e4bc76..00000000000
--- a/contrib/qemu/include/qemu/notify.h
+++ /dev/null
@@ -1,72 +0,0 @@
-/*
- * Notifier lists
- *
- * Copyright IBM, Corp. 2010
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU GPL, version 2.  See
- * the COPYING file in the top-level directory.
- *
- */
-
-#ifndef QEMU_NOTIFY_H
-#define QEMU_NOTIFY_H
-
-#include "qemu/queue.h"
-
-typedef struct Notifier Notifier;
-
-struct Notifier
-{
-    void (*notify)(Notifier *notifier, void *data);
-    QLIST_ENTRY(Notifier) node;
-};
-
-typedef struct NotifierList
-{
-    QLIST_HEAD(, Notifier) notifiers;
-} NotifierList;
-
-#define NOTIFIER_LIST_INITIALIZER(head) \
-    { QLIST_HEAD_INITIALIZER((head).notifiers) }
-
-void notifier_list_init(NotifierList *list);
-
-void notifier_list_add(NotifierList *list, Notifier *notifier);
-
-void notifier_remove(Notifier *notifier);
-
-void notifier_list_notify(NotifierList *list, void *data);
-
-/* Same as Notifier but allows .notify() to return errors */
-typedef struct NotifierWithReturn NotifierWithReturn;
-
-struct NotifierWithReturn {
-    /**
-     * Return 0 on success (next notifier will be invoked), otherwise
-     * notifier_with_return_list_notify() will stop and return the value.
-     */
-    int (*notify)(NotifierWithReturn *notifier, void *data);
-    QLIST_ENTRY(NotifierWithReturn) node;
-};
-
-typedef struct NotifierWithReturnList {
-    QLIST_HEAD(, NotifierWithReturn) notifiers;
-} NotifierWithReturnList;
-
-#define NOTIFIER_WITH_RETURN_LIST_INITIALIZER(head) \
-    { QLIST_HEAD_INITIALIZER((head).notifiers) }
-
-void notifier_with_return_list_init(NotifierWithReturnList *list);
-
-void notifier_with_return_list_add(NotifierWithReturnList *list,
-                                   NotifierWithReturn *notifier);
-
-void notifier_with_return_remove(NotifierWithReturn *notifier);
-
-int notifier_with_return_list_notify(NotifierWithReturnList *list,
-                                     void *data);
-
-#endif
diff --git a/contrib/qemu/include/qemu/option.h b/contrib/qemu/include/qemu/option.h
deleted file mode 100644
index a83c700323e..00000000000
--- a/contrib/qemu/include/qemu/option.h
+++ /dev/null
@@ -1,157 +0,0 @@
-/*
- * Commandline option parsing functions
- *
- * Copyright (c) 2003-2008 Fabrice Bellard
- * Copyright (c) 2009 Kevin Wolf <kwolf@redhat.com>
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-
-#ifndef QEMU_OPTIONS_H
-#define QEMU_OPTIONS_H
-
-#include <stdint.h>
-#include "qemu/queue.h"
-#include "qapi/error.h"
-#include "qapi/qmp/qdict.h"
-
-enum QEMUOptionParType {
-    OPT_FLAG,
-    OPT_NUMBER,
-    OPT_SIZE,
-    OPT_STRING,
-};
-
-typedef struct QEMUOptionParameter {
-    const char *name;
-    enum QEMUOptionParType type;
-    union {
-        uint64_t n;
-        char* s;
-    } value;
-    const char *help;
-} QEMUOptionParameter;
-
-
-const char *get_opt_name(char *buf, int buf_size, const char *p, char delim);
-const char *get_opt_value(char *buf, int buf_size, const char *p);
-int get_next_param_value(char *buf, int buf_size,
-                         const char *tag, const char **pstr);
-int get_param_value(char *buf, int buf_size,
-                    const char *tag, const char *str);
-
-
-/*
- * The following functions take a parameter list as input. This is a pointer to
- * the first element of a QEMUOptionParameter array which is terminated by an
- * entry with entry->name == NULL.
- */
-
-QEMUOptionParameter *get_option_parameter(QEMUOptionParameter *list,
-    const char *name);
-int set_option_parameter(QEMUOptionParameter *list, const char *name,
-    const char *value);
-int set_option_parameter_int(QEMUOptionParameter *list, const char *name,
-    uint64_t value);
-QEMUOptionParameter *append_option_parameters(QEMUOptionParameter *dest,
-    QEMUOptionParameter *list);
-QEMUOptionParameter *parse_option_parameters(const char *param,
-    QEMUOptionParameter *list, QEMUOptionParameter *dest);
-void free_option_parameters(QEMUOptionParameter *list);
-void print_option_parameters(QEMUOptionParameter *list);
-void print_option_help(QEMUOptionParameter *list);
-
-/* ------------------------------------------------------------------ */
-
-typedef struct QemuOpt QemuOpt;
-typedef struct QemuOpts QemuOpts;
-typedef struct QemuOptsList QemuOptsList;
-
-enum QemuOptType {
-    QEMU_OPT_STRING = 0,  /* no parsing (use string as-is)                        */
-    QEMU_OPT_BOOL,        /* on/off                                               */
-    QEMU_OPT_NUMBER,      /* simple number                                        */
-    QEMU_OPT_SIZE,        /* size, accepts (K)ilo, (M)ega, (G)iga, (T)era postfix */
-};
-
-typedef struct QemuOptDesc {
-    const char *name;
-    enum QemuOptType type;
-    const char *help;
-} QemuOptDesc;
-
-struct QemuOptsList {
-    const char *name;
-    const char *implied_opt_name;
-    bool merge_lists;  /* Merge multiple uses of option into a single list? */
-    QTAILQ_HEAD(, QemuOpts) head;
-    QemuOptDesc desc[];
-};
-
-const char *qemu_opt_get(QemuOpts *opts, const char *name);
-/**
- * qemu_opt_has_help_opt:
- * @opts: options to search for a help request
- *
- * Check whether the options specified by @opts include one of the
- * standard strings which indicate that the user is asking for a
- * list of the valid values for a command line option (as defined
- * by is_help_option()).
- *
- * Returns: true if @opts includes 'help' or equivalent.
- */
-bool qemu_opt_has_help_opt(QemuOpts *opts);
-bool qemu_opt_get_bool(QemuOpts *opts, const char *name, bool defval);
-uint64_t qemu_opt_get_number(QemuOpts *opts, const char *name, uint64_t defval);
-uint64_t qemu_opt_get_size(QemuOpts *opts, const char *name, uint64_t defval);
-int qemu_opt_set(QemuOpts *opts, const char *name, const char *value);
-void qemu_opt_set_err(QemuOpts *opts, const char *name, const char *value,
-                      Error **errp);
-int qemu_opt_set_bool(QemuOpts *opts, const char *name, bool val);
-int qemu_opt_set_number(QemuOpts *opts, const char *name, int64_t val);
-typedef int (*qemu_opt_loopfunc)(const char *name, const char *value, void *opaque);
-int qemu_opt_foreach(QemuOpts *opts, qemu_opt_loopfunc func, void *opaque,
-                     int abort_on_failure);
-
-QemuOpts *qemu_opts_find(QemuOptsList *list, const char *id);
-QemuOpts *qemu_opts_create(QemuOptsList *list, const char *id,
-                           int fail_if_exists, Error **errp);
-QemuOpts *qemu_opts_create_nofail(QemuOptsList *list);
-void qemu_opts_reset(QemuOptsList *list);
-void qemu_opts_loc_restore(QemuOpts *opts);
-int qemu_opts_set(QemuOptsList *list, const char *id,
-                  const char *name, const char *value);
-const char *qemu_opts_id(QemuOpts *opts);
-void qemu_opts_del(QemuOpts *opts);
-void qemu_opts_validate(QemuOpts *opts, const QemuOptDesc *desc, Error **errp);
-int qemu_opts_do_parse(QemuOpts *opts, const char *params, const char *firstname);
-QemuOpts *qemu_opts_parse(QemuOptsList *list, const char *params, int permit_abbrev);
-void qemu_opts_set_defaults(QemuOptsList *list, const char *params,
-                            int permit_abbrev);
-QemuOpts *qemu_opts_from_qdict(QemuOptsList *list, const QDict *qdict,
-                               Error **errp);
-QDict *qemu_opts_to_qdict(QemuOpts *opts, QDict *qdict);
-void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp);
-
-typedef int (*qemu_opts_loopfunc)(QemuOpts *opts, void *opaque);
-int qemu_opts_print(QemuOpts *opts, void *dummy);
-int qemu_opts_foreach(QemuOptsList *list, qemu_opts_loopfunc func, void *opaque,
-                      int abort_on_failure);
-
-#endif
diff --git a/contrib/qemu/include/qemu/option_int.h b/contrib/qemu/include/qemu/option_int.h
deleted file mode 100644
index 8212fa4a485..00000000000
--- a/contrib/qemu/include/qemu/option_int.h
+++ /dev/null
@@ -1,54 +0,0 @@
-/*
- * Commandline option parsing functions
- *
- * Copyright (c) 2003-2008 Fabrice Bellard
- * Copyright (c) 2009 Kevin Wolf <kwolf@redhat.com>
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-
-#ifndef QEMU_OPTIONS_INTERNAL_H
-#define QEMU_OPTIONS_INTERNAL_H
-
-#include "qemu/option.h"
-#include "qemu/error-report.h"
-
-struct QemuOpt {
-    const char   *name;
-    const char   *str;
-
-    const QemuOptDesc *desc;
-    union {
-        bool boolean;
-        uint64_t uint;
-    } value;
-
-    QemuOpts     *opts;
-    QTAILQ_ENTRY(QemuOpt) next;
-};
-
-struct QemuOpts {
-    char *id;
-    QemuOptsList *list;
-    Location loc;
-    QTAILQ_HEAD(QemuOptHead, QemuOpt) head;
-    QTAILQ_ENTRY(QemuOpts) next;
-};
-
-#endif
diff --git a/contrib/qemu/include/qemu/osdep.h b/contrib/qemu/include/qemu/osdep.h
deleted file mode 100644
index 26136f16ecd..00000000000
--- a/contrib/qemu/include/qemu/osdep.h
+++ /dev/null
@@ -1,218 +0,0 @@
-#ifndef QEMU_OSDEP_H
-#define QEMU_OSDEP_H
-
-#include "config-host.h"
-#include <stdarg.h>
-#include <stddef.h>
-#include <stdbool.h>
-#include <sys/types.h>
-#ifdef __OpenBSD__
-#include <sys/signal.h>
-#endif
-
-#ifndef _WIN32
-#include <sys/wait.h>
-#else
-#define WIFEXITED(x)   1
-#define WEXITSTATUS(x) (x)
-#endif
-
-#include <sys/time.h>
-
-#if defined(CONFIG_SOLARIS) && CONFIG_SOLARIS_VERSION < 10
-/* [u]int_fast*_t not in <sys/int_types.h> */
-typedef unsigned char           uint_fast8_t;
-typedef unsigned int            uint_fast16_t;
-typedef signed int              int_fast16_t;
-#endif
-
-#ifndef glue
-#define xglue(x, y) x ## y
-#define glue(x, y) xglue(x, y)
-#define stringify(s)	tostring(s)
-#define tostring(s)	#s
-#endif
-
-#ifndef likely
-#if __GNUC__ < 3
-#define __builtin_expect(x, n) (x)
-#endif
-
-#define likely(x)   __builtin_expect(!!(x), 1)
-#define unlikely(x)   __builtin_expect(!!(x), 0)
-#endif
-
-#ifndef container_of
-#define container_of(ptr, type, member) ({                      \
-        const typeof(((type *) 0)->member) *__mptr = (ptr);     \
-        (type *) ((char *) __mptr - offsetof(type, member));})
-#endif
-
-/* Convert from a base type to a parent type, with compile time checking.  */
-#ifdef __GNUC__
-#define DO_UPCAST(type, field, dev) ( __extension__ ( { \
-    char __attribute__((unused)) offset_must_be_zero[ \
-        -offsetof(type, field)]; \
-    container_of(dev, type, field);}))
-#else
-#define DO_UPCAST(type, field, dev) container_of(dev, type, field)
-#endif
-
-#define typeof_field(type, field) typeof(((type *)0)->field)
-#define type_check(t1,t2) ((t1*)0 - (t2*)0)
-
-#ifndef MIN
-#define MIN(a, b) (((a) < (b)) ? (a) : (b))
-#endif
-#ifndef MAX
-#define MAX(a, b) (((a) > (b)) ? (a) : (b))
-#endif
-
-#ifndef ROUND_UP
-#define ROUND_UP(n,d) (((n) + (d) - 1) & -(d))
-#endif
-
-#ifndef DIV_ROUND_UP
-#define DIV_ROUND_UP(n,d) (((n) + (d) - 1) / (d))
-#endif
-
-#ifndef ARRAY_SIZE
-#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))
-#endif
-
-#ifndef always_inline
-#if !((__GNUC__ < 3) || defined(__APPLE__))
-#ifdef __OPTIMIZE__
-#undef inline
-#define inline __attribute__ (( always_inline )) __inline__
-#endif
-#endif
-#else
-#undef inline
-#define inline always_inline
-#endif
-
-#define qemu_printf printf
-
-int qemu_daemon(int nochdir, int noclose);
-void *qemu_memalign(size_t alignment, size_t size);
-void *qemu_anon_ram_alloc(size_t size);
-void qemu_vfree(void *ptr);
-void qemu_anon_ram_free(void *ptr, size_t size);
-
-#define QEMU_MADV_INVALID -1
-
-#if defined(CONFIG_MADVISE)
-
-#define QEMU_MADV_WILLNEED  MADV_WILLNEED
-#define QEMU_MADV_DONTNEED  MADV_DONTNEED
-#ifdef MADV_DONTFORK
-#define QEMU_MADV_DONTFORK  MADV_DONTFORK
-#else
-#define QEMU_MADV_DONTFORK  QEMU_MADV_INVALID
-#endif
-#ifdef MADV_MERGEABLE
-#define QEMU_MADV_MERGEABLE MADV_MERGEABLE
-#else
-#define QEMU_MADV_MERGEABLE QEMU_MADV_INVALID
-#endif
-#ifdef MADV_DONTDUMP
-#define QEMU_MADV_DONTDUMP MADV_DONTDUMP
-#else
-#define QEMU_MADV_DONTDUMP QEMU_MADV_INVALID
-#endif
-#ifdef MADV_HUGEPAGE
-#define QEMU_MADV_HUGEPAGE MADV_HUGEPAGE
-#else
-#define QEMU_MADV_HUGEPAGE QEMU_MADV_INVALID
-#endif
-
-#elif defined(CONFIG_POSIX_MADVISE)
-
-#define QEMU_MADV_WILLNEED  POSIX_MADV_WILLNEED
-#define QEMU_MADV_DONTNEED  POSIX_MADV_DONTNEED
-#define QEMU_MADV_DONTFORK  QEMU_MADV_INVALID
-#define QEMU_MADV_MERGEABLE QEMU_MADV_INVALID
-#define QEMU_MADV_DONTDUMP QEMU_MADV_INVALID
-#define QEMU_MADV_HUGEPAGE  QEMU_MADV_INVALID
-
-#else /* no-op */
-
-#define QEMU_MADV_WILLNEED  QEMU_MADV_INVALID
-#define QEMU_MADV_DONTNEED  QEMU_MADV_INVALID
-#define QEMU_MADV_DONTFORK  QEMU_MADV_INVALID
-#define QEMU_MADV_MERGEABLE QEMU_MADV_INVALID
-#define QEMU_MADV_DONTDUMP QEMU_MADV_INVALID
-#define QEMU_MADV_HUGEPAGE  QEMU_MADV_INVALID
-
-#endif
-
-int qemu_madvise(void *addr, size_t len, int advice);
-
-int qemu_open(const char *name, int flags, ...);
-int qemu_close(int fd);
-
-#if defined(__HAIKU__) && defined(__i386__)
-#define FMT_pid "%ld"
-#elif defined(WIN64)
-#define FMT_pid "%" PRId64
-#else
-#define FMT_pid "%d"
-#endif
-
-int qemu_create_pidfile(const char *filename);
-int qemu_get_thread_id(void);
-
-#ifndef CONFIG_IOVEC
-struct iovec {
-    void *iov_base;
-    size_t iov_len;
-};
-/*
- * Use the same value as Linux for now.
- */
-#define IOV_MAX 1024
-
-ssize_t readv(int fd, const struct iovec *iov, int iov_cnt);
-ssize_t writev(int fd, const struct iovec *iov, int iov_cnt);
-#else
-#include <sys/uio.h>
-#endif
-
-#ifdef _WIN32
-static inline void qemu_timersub(const struct timeval *val1,
-                                 const struct timeval *val2,
-                                 struct timeval *res)
-{
-    res->tv_sec = val1->tv_sec - val2->tv_sec;
-    if (val1->tv_usec < val2->tv_usec) {
-        res->tv_sec--;
-        res->tv_usec = val1->tv_usec - val2->tv_usec + 1000 * 1000;
-    } else {
-        res->tv_usec = val1->tv_usec - val2->tv_usec;
-    }
-}
-#else
-#define qemu_timersub timersub
-#endif
-
-void qemu_set_cloexec(int fd);
-
-void qemu_set_version(const char *);
-const char *qemu_get_version(void);
-
-void fips_set_state(bool requested);
-bool fips_get_state(void);
-
-/* Return a dynamically allocated pathname denoting a file or directory that is
- * appropriate for storing local state.
- *
- * @relative_pathname need not start with a directory separator; one will be
- * added automatically.
- *
- * The caller is responsible for releasing the value returned with g_free()
- * after use.
- */
-char *qemu_get_local_state_pathname(const char *relative_pathname);
-
-#endif
diff --git a/contrib/qemu/include/qemu/queue.h b/contrib/qemu/include/qemu/queue.h
deleted file mode 100644
index d433b9017ce..00000000000
--- a/contrib/qemu/include/qemu/queue.h
+++ /dev/null
@@ -1,414 +0,0 @@
-/*      $NetBSD: queue.h,v 1.52 2009/04/20 09:56:08 mschuett Exp $ */
-
-/*
- * QEMU version: Copy from netbsd, removed debug code, removed some of
- * the implementations.  Left in singly-linked lists, lists, simple
- * queues, and tail queues.
- */
-
-/*
- * Copyright (c) 1991, 1993
- *      The Regents of the University of California.  All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- * 3. Neither the name of the University nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- *      @(#)queue.h     8.5 (Berkeley) 8/20/94
- */
-
-#ifndef QEMU_SYS_QUEUE_H_
-#define QEMU_SYS_QUEUE_H_
-
-/*
- * This file defines four types of data structures: singly-linked lists,
- * lists, simple queues, and tail queues.
- *
- * A singly-linked list is headed by a single forward pointer. The
- * elements are singly linked for minimum space and pointer manipulation
- * overhead at the expense of O(n) removal for arbitrary elements. New
- * elements can be added to the list after an existing element or at the
- * head of the list.  Elements being removed from the head of the list
- * should use the explicit macro for this purpose for optimum
- * efficiency. A singly-linked list may only be traversed in the forward
- * direction.  Singly-linked lists are ideal for applications with large
- * datasets and few or no removals or for implementing a LIFO queue.
- *
- * A list is headed by a single forward pointer (or an array of forward
- * pointers for a hash table header). The elements are doubly linked
- * so that an arbitrary element can be removed without a need to
- * traverse the list. New elements can be added to the list before
- * or after an existing element or at the head of the list. A list
- * may only be traversed in the forward direction.
- *
- * A simple queue is headed by a pair of pointers, one the head of the
- * list and the other to the tail of the list. The elements are singly
- * linked to save space, so elements can only be removed from the
- * head of the list. New elements can be added to the list after
- * an existing element, at the head of the list, or at the end of the
- * list. A simple queue may only be traversed in the forward direction.
- *
- * A tail queue is headed by a pair of pointers, one to the head of the
- * list and the other to the tail of the list. The elements are doubly
- * linked so that an arbitrary element can be removed without a need to
- * traverse the list. New elements can be added to the list before or
- * after an existing element, at the head of the list, or at the end of
- * the list. A tail queue may be traversed in either direction.
- *
- * For details on the use of these macros, see the queue(3) manual page.
- */
-
-#include "qemu/atomic.h" /* for smp_wmb() */
-
-/*
- * List definitions.
- */
-#define QLIST_HEAD(name, type)                                          \
-struct name {                                                           \
-        struct type *lh_first;  /* first element */                     \
-}
-
-#define QLIST_HEAD_INITIALIZER(head)                                    \
-        { NULL }
-
-#define QLIST_ENTRY(type)                                               \
-struct {                                                                \
-        struct type *le_next;   /* next element */                      \
-        struct type **le_prev;  /* address of previous next element */  \
-}
-
-/*
- * List functions.
- */
-#define QLIST_INIT(head) do {                                           \
-        (head)->lh_first = NULL;                                        \
-} while (/*CONSTCOND*/0)
-
-#define QLIST_INSERT_AFTER(listelm, elm, field) do {                    \
-        if (((elm)->field.le_next = (listelm)->field.le_next) != NULL)  \
-                (listelm)->field.le_next->field.le_prev =               \
-                    &(elm)->field.le_next;                              \
-        (listelm)->field.le_next = (elm);                               \
-        (elm)->field.le_prev = &(listelm)->field.le_next;               \
-} while (/*CONSTCOND*/0)
-
-#define QLIST_INSERT_BEFORE(listelm, elm, field) do {                   \
-        (elm)->field.le_prev = (listelm)->field.le_prev;                \
-        (elm)->field.le_next = (listelm);                               \
-        *(listelm)->field.le_prev = (elm);                              \
-        (listelm)->field.le_prev = &(elm)->field.le_next;               \
-} while (/*CONSTCOND*/0)
-
-#define QLIST_INSERT_HEAD(head, elm, field) do {                        \
-        if (((elm)->field.le_next = (head)->lh_first) != NULL)          \
-                (head)->lh_first->field.le_prev = &(elm)->field.le_next;\
-        (head)->lh_first = (elm);                                       \
-        (elm)->field.le_prev = &(head)->lh_first;                       \
-} while (/*CONSTCOND*/0)
-
-#define QLIST_INSERT_HEAD_RCU(head, elm, field) do {                    \
-        (elm)->field.le_prev = &(head)->lh_first;                       \
-        (elm)->field.le_next = (head)->lh_first;                        \
-        smp_wmb(); /* fill elm before linking it */                     \
-        if ((head)->lh_first != NULL)  {                                \
-            (head)->lh_first->field.le_prev = &(elm)->field.le_next;    \
-        }                                                               \
-        (head)->lh_first = (elm);                                       \
-        smp_wmb();                                                      \
-} while (/* CONSTCOND*/0)
-
-#define QLIST_REMOVE(elm, field) do {                                   \
-        if ((elm)->field.le_next != NULL)                               \
-                (elm)->field.le_next->field.le_prev =                   \
-                    (elm)->field.le_prev;                               \
-        *(elm)->field.le_prev = (elm)->field.le_next;                   \
-} while (/*CONSTCOND*/0)
-
-#define QLIST_FOREACH(var, head, field)                                 \
-        for ((var) = ((head)->lh_first);                                \
-                (var);                                                  \
-                (var) = ((var)->field.le_next))
-
-#define QLIST_FOREACH_SAFE(var, head, field, next_var)                  \
-        for ((var) = ((head)->lh_first);                                \
-                (var) && ((next_var) = ((var)->field.le_next), 1);      \
-                (var) = (next_var))
-
-/*
- * List access methods.
- */
-#define QLIST_EMPTY(head)                ((head)->lh_first == NULL)
-#define QLIST_FIRST(head)                ((head)->lh_first)
-#define QLIST_NEXT(elm, field)           ((elm)->field.le_next)
-
-
-/*
- * Singly-linked List definitions.
- */
-#define QSLIST_HEAD(name, type)                                          \
-struct name {                                                           \
-        struct type *slh_first; /* first element */                     \
-}
-
-#define QSLIST_HEAD_INITIALIZER(head)                                    \
-        { NULL }
-
-#define QSLIST_ENTRY(type)                                               \
-struct {                                                                \
-        struct type *sle_next;  /* next element */                      \
-}
-
-/*
- * Singly-linked List functions.
- */
-#define QSLIST_INIT(head) do {                                           \
-        (head)->slh_first = NULL;                                       \
-} while (/*CONSTCOND*/0)
-
-#define QSLIST_INSERT_AFTER(slistelm, elm, field) do {                   \
-        (elm)->field.sle_next = (slistelm)->field.sle_next;             \
-        (slistelm)->field.sle_next = (elm);                             \
-} while (/*CONSTCOND*/0)
-
-#define QSLIST_INSERT_HEAD(head, elm, field) do {                        \
-        (elm)->field.sle_next = (head)->slh_first;                      \
-        (head)->slh_first = (elm);                                      \
-} while (/*CONSTCOND*/0)
-
-#define QSLIST_REMOVE_HEAD(head, field) do {                             \
-        (head)->slh_first = (head)->slh_first->field.sle_next;          \
-} while (/*CONSTCOND*/0)
-
-#define QSLIST_REMOVE_AFTER(slistelm, field) do {                        \
-        (slistelm)->field.sle_next =                                    \
-            QSLIST_NEXT(QSLIST_NEXT((slistelm), field), field);           \
-} while (/*CONSTCOND*/0)
-
-#define QSLIST_FOREACH(var, head, field)                                 \
-        for((var) = (head)->slh_first; (var); (var) = (var)->field.sle_next)
-
-#define QSLIST_FOREACH_SAFE(var, head, field, tvar)                      \
-        for ((var) = QSLIST_FIRST((head));                               \
-            (var) && ((tvar) = QSLIST_NEXT((var), field), 1);            \
-            (var) = (tvar))
-
-/*
- * Singly-linked List access methods.
- */
-#define QSLIST_EMPTY(head)       ((head)->slh_first == NULL)
-#define QSLIST_FIRST(head)       ((head)->slh_first)
-#define QSLIST_NEXT(elm, field)  ((elm)->field.sle_next)
-
-
-/*
- * Simple queue definitions.
- */
-#define QSIMPLEQ_HEAD(name, type)                                       \
-struct name {                                                           \
-    struct type *sqh_first;    /* first element */                      \
-    struct type **sqh_last;    /* addr of last next element */          \
-}
-
-#define QSIMPLEQ_HEAD_INITIALIZER(head)                                 \
-    { NULL, &(head).sqh_first }
-
-#define QSIMPLEQ_ENTRY(type)                                            \
-struct {                                                                \
-    struct type *sqe_next;    /* next element */                        \
-}
-
-/*
- * Simple queue functions.
- */
-#define QSIMPLEQ_INIT(head) do {                                        \
-    (head)->sqh_first = NULL;                                           \
-    (head)->sqh_last = &(head)->sqh_first;                              \
-} while (/*CONSTCOND*/0)
-
-#define QSIMPLEQ_INSERT_HEAD(head, elm, field) do {                     \
-    if (((elm)->field.sqe_next = (head)->sqh_first) == NULL)            \
-        (head)->sqh_last = &(elm)->field.sqe_next;                      \
-    (head)->sqh_first = (elm);                                          \
-} while (/*CONSTCOND*/0)
-
-#define QSIMPLEQ_INSERT_TAIL(head, elm, field) do {                     \
-    (elm)->field.sqe_next = NULL;                                       \
-    *(head)->sqh_last = (elm);                                          \
-    (head)->sqh_last = &(elm)->field.sqe_next;                          \
-} while (/*CONSTCOND*/0)
-
-#define QSIMPLEQ_INSERT_AFTER(head, listelm, elm, field) do {           \
-    if (((elm)->field.sqe_next = (listelm)->field.sqe_next) == NULL)    \
-        (head)->sqh_last = &(elm)->field.sqe_next;                      \
-    (listelm)->field.sqe_next = (elm);                                  \
-} while (/*CONSTCOND*/0)
-
-#define QSIMPLEQ_REMOVE_HEAD(head, field) do {                          \
-    if (((head)->sqh_first = (head)->sqh_first->field.sqe_next) == NULL)\
-        (head)->sqh_last = &(head)->sqh_first;                          \
-} while (/*CONSTCOND*/0)
-
-#define QSIMPLEQ_REMOVE(head, elm, type, field) do {                    \
-    if ((head)->sqh_first == (elm)) {                                   \
-        QSIMPLEQ_REMOVE_HEAD((head), field);                            \
-    } else {                                                            \
-        struct type *curelm = (head)->sqh_first;                        \
-        while (curelm->field.sqe_next != (elm))                         \
-            curelm = curelm->field.sqe_next;                            \
-        if ((curelm->field.sqe_next =                                   \
-            curelm->field.sqe_next->field.sqe_next) == NULL)            \
-                (head)->sqh_last = &(curelm)->field.sqe_next;           \
-    }                                                                   \
-} while (/*CONSTCOND*/0)
-
-#define QSIMPLEQ_FOREACH(var, head, field)                              \
-    for ((var) = ((head)->sqh_first);                                   \
-        (var);                                                          \
-        (var) = ((var)->field.sqe_next))
-
-#define QSIMPLEQ_FOREACH_SAFE(var, head, field, next)                   \
-    for ((var) = ((head)->sqh_first);                                   \
-        (var) && ((next = ((var)->field.sqe_next)), 1);                 \
-        (var) = (next))
-
-#define QSIMPLEQ_CONCAT(head1, head2) do {                              \
-    if (!QSIMPLEQ_EMPTY((head2))) {                                     \
-        *(head1)->sqh_last = (head2)->sqh_first;                        \
-        (head1)->sqh_last = (head2)->sqh_last;                          \
-        QSIMPLEQ_INIT((head2));                                         \
-    }                                                                   \
-} while (/*CONSTCOND*/0)
-
-#define QSIMPLEQ_LAST(head, type, field)                                \
-    (QSIMPLEQ_EMPTY((head)) ?                                           \
-        NULL :                                                          \
-            ((struct type *)(void *)                                    \
-        ((char *)((head)->sqh_last) - offsetof(struct type, field))))
-
-/*
- * Simple queue access methods.
- */
-#define QSIMPLEQ_EMPTY(head)        ((head)->sqh_first == NULL)
-#define QSIMPLEQ_FIRST(head)        ((head)->sqh_first)
-#define QSIMPLEQ_NEXT(elm, field)   ((elm)->field.sqe_next)
-
-
-/*
- * Tail queue definitions.
- */
-#define Q_TAILQ_HEAD(name, type, qual)                                  \
-struct name {                                                           \
-        qual type *tqh_first;           /* first element */             \
-        qual type *qual *tqh_last;      /* addr of last next element */ \
-}
-#define QTAILQ_HEAD(name, type)  Q_TAILQ_HEAD(name, struct type,)
-
-#define QTAILQ_HEAD_INITIALIZER(head)                                   \
-        { NULL, &(head).tqh_first }
-
-#define Q_TAILQ_ENTRY(type, qual)                                       \
-struct {                                                                \
-        qual type *tqe_next;            /* next element */              \
-        qual type *qual *tqe_prev;      /* address of previous next element */\
-}
-#define QTAILQ_ENTRY(type)       Q_TAILQ_ENTRY(struct type,)
-
-/*
- * Tail queue functions.
- */
-#define QTAILQ_INIT(head) do {                                          \
-        (head)->tqh_first = NULL;                                       \
-        (head)->tqh_last = &(head)->tqh_first;                          \
-} while (/*CONSTCOND*/0)
-
-#define QTAILQ_INSERT_HEAD(head, elm, field) do {                       \
-        if (((elm)->field.tqe_next = (head)->tqh_first) != NULL)        \
-                (head)->tqh_first->field.tqe_prev =                     \
-                    &(elm)->field.tqe_next;                             \
-        else                                                            \
-                (head)->tqh_last = &(elm)->field.tqe_next;              \
-        (head)->tqh_first = (elm);                                      \
-        (elm)->field.tqe_prev = &(head)->tqh_first;                     \
-} while (/*CONSTCOND*/0)
-
-#define QTAILQ_INSERT_TAIL(head, elm, field) do {                       \
-        (elm)->field.tqe_next = NULL;                                   \
-        (elm)->field.tqe_prev = (head)->tqh_last;                       \
-        *(head)->tqh_last = (elm);                                      \
-        (head)->tqh_last = &(elm)->field.tqe_next;                      \
-} while (/*CONSTCOND*/0)
-
-#define QTAILQ_INSERT_AFTER(head, listelm, elm, field) do {             \
-        if (((elm)->field.tqe_next = (listelm)->field.tqe_next) != NULL)\
-                (elm)->field.tqe_next->field.tqe_prev =                 \
-                    &(elm)->field.tqe_next;                             \
-        else                                                            \
-                (head)->tqh_last = &(elm)->field.tqe_next;              \
-        (listelm)->field.tqe_next = (elm);                              \
-        (elm)->field.tqe_prev = &(listelm)->field.tqe_next;             \
-} while (/*CONSTCOND*/0)
-
-#define QTAILQ_INSERT_BEFORE(listelm, elm, field) do {                  \
-        (elm)->field.tqe_prev = (listelm)->field.tqe_prev;              \
-        (elm)->field.tqe_next = (listelm);                              \
-        *(listelm)->field.tqe_prev = (elm);                             \
-        (listelm)->field.tqe_prev = &(elm)->field.tqe_next;             \
-} while (/*CONSTCOND*/0)
-
-#define QTAILQ_REMOVE(head, elm, field) do {                            \
-        if (((elm)->field.tqe_next) != NULL)                            \
-                (elm)->field.tqe_next->field.tqe_prev =                 \
-                    (elm)->field.tqe_prev;                              \
-        else                                                            \
-                (head)->tqh_last = (elm)->field.tqe_prev;               \
-        *(elm)->field.tqe_prev = (elm)->field.tqe_next;                 \
-} while (/*CONSTCOND*/0)
-
-#define QTAILQ_FOREACH(var, head, field)                                \
-        for ((var) = ((head)->tqh_first);                               \
-                (var);                                                  \
-                (var) = ((var)->field.tqe_next))
-
-#define QTAILQ_FOREACH_SAFE(var, head, field, next_var)                 \
-        for ((var) = ((head)->tqh_first);                               \
-                (var) && ((next_var) = ((var)->field.tqe_next), 1);     \
-                (var) = (next_var))
-
-#define QTAILQ_FOREACH_REVERSE(var, head, headname, field)              \
-        for ((var) = (*(((struct headname *)((head)->tqh_last))->tqh_last));    \
-                (var);                                                  \
-                (var) = (*(((struct headname *)((var)->field.tqe_prev))->tqh_last)))
-
-/*
- * Tail queue access methods.
- */
-#define QTAILQ_EMPTY(head)               ((head)->tqh_first == NULL)
-#define QTAILQ_FIRST(head)               ((head)->tqh_first)
-#define QTAILQ_NEXT(elm, field)          ((elm)->field.tqe_next)
-
-#define QTAILQ_LAST(head, headname) \
-        (*(((struct headname *)((head)->tqh_last))->tqh_last))
-#define QTAILQ_PREV(elm, headname, field) \
-        (*(((struct headname *)((elm)->field.tqe_prev))->tqh_last))
-
-#endif  /* !QEMU_SYS_QUEUE_H_ */
diff --git a/contrib/qemu/include/qemu/sockets.h b/contrib/qemu/include/qemu/sockets.h
deleted file mode 100644
index c5174d76a70..00000000000
--- a/contrib/qemu/include/qemu/sockets.h
+++ /dev/null
@@ -1,83 +0,0 @@
-/* headers to use the BSD sockets */
-#ifndef QEMU_SOCKET_H
-#define QEMU_SOCKET_H
-
-#ifdef _WIN32
-#include <windows.h>
-#include <winsock2.h>
-#include <ws2tcpip.h>
-
-#define socket_error() WSAGetLastError()
-
-int inet_aton(const char *cp, struct in_addr *ia);
-
-#else
-
-#include <sys/types.h>
-#include <sys/socket.h>
-#include <netinet/in.h>
-#include <netinet/tcp.h>
-#include <arpa/inet.h>
-#include <netdb.h>
-#include <sys/un.h>
-
-#define socket_error() errno
-#define closesocket(s) close(s)
-
-#endif /* !_WIN32 */
-
-#include "qemu/option.h"
-#include "qapi/error.h"
-#include "qapi/qmp/qerror.h"
-
-extern QemuOptsList socket_optslist;
-
-/* misc helpers */
-int qemu_socket(int domain, int type, int protocol);
-int qemu_accept(int s, struct sockaddr *addr, socklen_t *addrlen);
-int socket_set_cork(int fd, int v);
-int socket_set_nodelay(int fd);
-void qemu_set_block(int fd);
-void qemu_set_nonblock(int fd);
-int send_all(int fd, const void *buf, int len1);
-int recv_all(int fd, void *buf, int len1, bool single_read);
-
-/* callback function for nonblocking connect
- * valid fd on success, negative error code on failure
- */
-typedef void NonBlockingConnectHandler(int fd, void *opaque);
-
-InetSocketAddress *inet_parse(const char *str, Error **errp);
-int inet_listen_opts(QemuOpts *opts, int port_offset, Error **errp);
-int inet_listen(const char *str, char *ostr, int olen,
-                int socktype, int port_offset, Error **errp);
-int inet_connect_opts(QemuOpts *opts, Error **errp,
-                      NonBlockingConnectHandler *callback, void *opaque);
-int inet_connect(const char *str, Error **errp);
-int inet_nonblocking_connect(const char *str,
-                             NonBlockingConnectHandler *callback,
-                             void *opaque, Error **errp);
-
-int inet_dgram_opts(QemuOpts *opts, Error **errp);
-const char *inet_strfamily(int family);
-
-int unix_listen_opts(QemuOpts *opts, Error **errp);
-int unix_listen(const char *path, char *ostr, int olen, Error **errp);
-int unix_connect_opts(QemuOpts *opts, Error **errp,
-                      NonBlockingConnectHandler *callback, void *opaque);
-int unix_connect(const char *path, Error **errp);
-int unix_nonblocking_connect(const char *str,
-                             NonBlockingConnectHandler *callback,
-                             void *opaque, Error **errp);
-
-SocketAddress *socket_parse(const char *str, Error **errp);
-int socket_connect(SocketAddress *addr, Error **errp,
-                   NonBlockingConnectHandler *callback, void *opaque);
-int socket_listen(SocketAddress *addr, Error **errp);
-int socket_dgram(SocketAddress *remote, SocketAddress *local, Error **errp);
-
-/* Old, ipv4 only bits.  Don't use for new code. */
-int parse_host_port(struct sockaddr_in *saddr, const char *str);
-int socket_init(void);
-
-#endif /* QEMU_SOCKET_H */
diff --git a/contrib/qemu/include/qemu/thread-posix.h b/contrib/qemu/include/qemu/thread-posix.h
deleted file mode 100644
index 0f30dccb53c..00000000000
--- a/contrib/qemu/include/qemu/thread-posix.h
+++ /dev/null
@@ -1,28 +0,0 @@
-#ifndef __QEMU_THREAD_POSIX_H
-#define __QEMU_THREAD_POSIX_H 1
-#include "pthread.h"
-#include <semaphore.h>
-
-struct QemuMutex {
-    pthread_mutex_t lock;
-};
-
-struct QemuCond {
-    pthread_cond_t cond;
-};
-
-struct QemuSemaphore {
-#if defined(__APPLE__) || defined(__NetBSD__)
-    pthread_mutex_t lock;
-    pthread_cond_t cond;
-    int count;
-#else
-    sem_t sem;
-#endif
-};
-
-struct QemuThread {
-    pthread_t thread;
-};
-
-#endif
diff --git a/contrib/qemu/include/qemu/thread.h b/contrib/qemu/include/qemu/thread.h
deleted file mode 100644
index c02404b9fbf..00000000000
--- a/contrib/qemu/include/qemu/thread.h
+++ /dev/null
@@ -1,56 +0,0 @@
-#ifndef __QEMU_THREAD_H
-#define __QEMU_THREAD_H 1
-
-#include <inttypes.h>
-#include <stdbool.h>
-
-typedef struct QemuMutex QemuMutex;
-typedef struct QemuCond QemuCond;
-typedef struct QemuSemaphore QemuSemaphore;
-typedef struct QemuThread QemuThread;
-
-#ifdef _WIN32
-#include "qemu/thread-win32.h"
-#else
-#include "qemu/thread-posix.h"
-#endif
-
-#define QEMU_THREAD_JOINABLE 0
-#define QEMU_THREAD_DETACHED 1
-
-void qemu_mutex_init(QemuMutex *mutex);
-void qemu_mutex_destroy(QemuMutex *mutex);
-void qemu_mutex_lock(QemuMutex *mutex);
-int qemu_mutex_trylock(QemuMutex *mutex);
-void qemu_mutex_unlock(QemuMutex *mutex);
-
-#define rcu_read_lock() do { } while (0)
-#define rcu_read_unlock() do { } while (0)
-
-void qemu_cond_init(QemuCond *cond);
-void qemu_cond_destroy(QemuCond *cond);
-
-/*
- * IMPORTANT: The implementation does not guarantee that pthread_cond_signal
- * and pthread_cond_broadcast can be called except while the same mutex is
- * held as in the corresponding pthread_cond_wait calls!
- */
-void qemu_cond_signal(QemuCond *cond);
-void qemu_cond_broadcast(QemuCond *cond);
-void qemu_cond_wait(QemuCond *cond, QemuMutex *mutex);
-
-void qemu_sem_init(QemuSemaphore *sem, int init);
-void qemu_sem_post(QemuSemaphore *sem);
-void qemu_sem_wait(QemuSemaphore *sem);
-int qemu_sem_timedwait(QemuSemaphore *sem, int ms);
-void qemu_sem_destroy(QemuSemaphore *sem);
-
-void qemu_thread_create(QemuThread *thread,
-                        void *(*start_routine)(void *),
-                        void *arg, int mode);
-void *qemu_thread_join(QemuThread *thread);
-void qemu_thread_get_self(QemuThread *thread);
-bool qemu_thread_is_self(QemuThread *thread);
-void qemu_thread_exit(void *retval);
-
-#endif
diff --git a/contrib/qemu/include/qemu/timer.h b/contrib/qemu/include/qemu/timer.h
deleted file mode 100644
index 9dd206ce7f4..00000000000
--- a/contrib/qemu/include/qemu/timer.h
+++ /dev/null
@@ -1,305 +0,0 @@
-#ifndef QEMU_TIMER_H
-#define QEMU_TIMER_H
-
-#include "qemu-common.h"
-#include "qemu/main-loop.h"
-#include "qemu/notify.h"
-
-/* timers */
-
-#define SCALE_MS 1000000
-#define SCALE_US 1000
-#define SCALE_NS 1
-
-typedef struct QEMUClock QEMUClock;
-typedef void QEMUTimerCB(void *opaque);
-
-/* The real time clock should be used only for stuff which does not
-   change the virtual machine state, as it is run even if the virtual
-   machine is stopped. The real time clock has a frequency of 1000
-   Hz. */
-extern QEMUClock *rt_clock;
-
-/* The virtual clock is only run during the emulation. It is stopped
-   when the virtual machine is stopped. Virtual timers use a high
-   precision clock, usually cpu cycles (use ticks_per_sec). */
-extern QEMUClock *vm_clock;
-
-/* The host clock should be use for device models that emulate accurate
-   real time sources. It will continue to run when the virtual machine
-   is suspended, and it will reflect system time changes the host may
-   undergo (e.g. due to NTP). The host clock has the same precision as
-   the virtual clock. */
-extern QEMUClock *host_clock;
-
-int64_t qemu_get_clock_ns(QEMUClock *clock);
-int64_t qemu_clock_has_timers(QEMUClock *clock);
-int64_t qemu_clock_expired(QEMUClock *clock);
-int64_t qemu_clock_deadline(QEMUClock *clock);
-void qemu_clock_enable(QEMUClock *clock, bool enabled);
-void qemu_clock_warp(QEMUClock *clock);
-
-void qemu_register_clock_reset_notifier(QEMUClock *clock, Notifier *notifier);
-void qemu_unregister_clock_reset_notifier(QEMUClock *clock,
-                                          Notifier *notifier);
-
-QEMUTimer *qemu_new_timer(QEMUClock *clock, int scale,
-                          QEMUTimerCB *cb, void *opaque);
-void qemu_free_timer(QEMUTimer *ts);
-void qemu_del_timer(QEMUTimer *ts);
-void qemu_mod_timer_ns(QEMUTimer *ts, int64_t expire_time);
-void qemu_mod_timer(QEMUTimer *ts, int64_t expire_time);
-bool qemu_timer_pending(QEMUTimer *ts);
-bool qemu_timer_expired(QEMUTimer *timer_head, int64_t current_time);
-uint64_t qemu_timer_expire_time_ns(QEMUTimer *ts);
-
-void qemu_run_timers(QEMUClock *clock);
-void qemu_run_all_timers(void);
-void configure_alarms(char const *opt);
-void init_clocks(void);
-int init_timer_alarm(void);
-
-int64_t cpu_get_ticks(void);
-void cpu_enable_ticks(void);
-void cpu_disable_ticks(void);
-
-static inline QEMUTimer *qemu_new_timer_ns(QEMUClock *clock, QEMUTimerCB *cb,
-                                           void *opaque)
-{
-    return qemu_new_timer(clock, SCALE_NS, cb, opaque);
-}
-
-static inline QEMUTimer *qemu_new_timer_ms(QEMUClock *clock, QEMUTimerCB *cb,
-                                           void *opaque)
-{
-    return qemu_new_timer(clock, SCALE_MS, cb, opaque);
-}
-
-static inline int64_t qemu_get_clock_ms(QEMUClock *clock)
-{
-    return qemu_get_clock_ns(clock) / SCALE_MS;
-}
-
-static inline int64_t get_ticks_per_sec(void)
-{
-    return 1000000000LL;
-}
-
-/* real time host monotonic timer */
-static inline int64_t get_clock_realtime(void)
-{
-    struct timeval tv;
-
-    gettimeofday(&tv, NULL);
-    return tv.tv_sec * 1000000000LL + (tv.tv_usec * 1000);
-}
-
-/* Warning: don't insert tracepoints into these functions, they are
-   also used by simpletrace backend and tracepoints would cause
-   an infinite recursion! */
-#ifdef _WIN32
-extern int64_t clock_freq;
-
-static inline int64_t get_clock(void)
-{
-    LARGE_INTEGER ti;
-    QueryPerformanceCounter(&ti);
-    return muldiv64(ti.QuadPart, get_ticks_per_sec(), clock_freq);
-}
-
-#else
-
-extern int use_rt_clock;
-
-static inline int64_t get_clock(void)
-{
-#ifdef CLOCK_MONOTONIC
-    if (use_rt_clock) {
-        struct timespec ts;
-        clock_gettime(CLOCK_MONOTONIC, &ts);
-        return ts.tv_sec * 1000000000LL + ts.tv_nsec;
-    } else
-#endif
-    {
-        /* XXX: using gettimeofday leads to problems if the date
-           changes, so it should be avoided. */
-        return get_clock_realtime();
-    }
-}
-#endif
-
-void qemu_get_timer(QEMUFile *f, QEMUTimer *ts);
-void qemu_put_timer(QEMUFile *f, QEMUTimer *ts);
-
-/* icount */
-int64_t cpu_get_icount(void);
-int64_t cpu_get_clock(void);
-
-/*******************************************/
-/* host CPU ticks (if available) */
-
-#if defined(_ARCH_PPC)
-
-static inline int64_t cpu_get_real_ticks(void)
-{
-    int64_t retval;
-#ifdef _ARCH_PPC64
-    /* This reads timebase in one 64bit go and includes Cell workaround from:
-       http://ozlabs.org/pipermail/linuxppc-dev/2006-October/027052.html
-    */
-    __asm__ __volatile__ ("mftb    %0\n\t"
-                          "cmpwi   %0,0\n\t"
-                          "beq-    $-8"
-                          : "=r" (retval));
-#else
-    /* http://ozlabs.org/pipermail/linuxppc-dev/1999-October/003889.html */
-    unsigned long junk;
-    __asm__ __volatile__ ("mfspr   %1,269\n\t"  /* mftbu */
-                          "mfspr   %L0,268\n\t" /* mftb */
-                          "mfspr   %0,269\n\t"  /* mftbu */
-                          "cmpw    %0,%1\n\t"
-                          "bne     $-16"
-                          : "=r" (retval), "=r" (junk));
-#endif
-    return retval;
-}
-
-#elif defined(__i386__)
-
-static inline int64_t cpu_get_real_ticks(void)
-{
-    int64_t val;
-    asm volatile ("rdtsc" : "=A" (val));
-    return val;
-}
-
-#elif defined(__x86_64__)
-
-static inline int64_t cpu_get_real_ticks(void)
-{
-    uint32_t low,high;
-    int64_t val;
-    asm volatile("rdtsc" : "=a" (low), "=d" (high));
-    val = high;
-    val <<= 32;
-    val |= low;
-    return val;
-}
-
-#elif defined(__hppa__)
-
-static inline int64_t cpu_get_real_ticks(void)
-{
-    int val;
-    asm volatile ("mfctl %%cr16, %0" : "=r"(val));
-    return val;
-}
-
-#elif defined(__ia64)
-
-static inline int64_t cpu_get_real_ticks(void)
-{
-    int64_t val;
-    asm volatile ("mov %0 = ar.itc" : "=r"(val) :: "memory");
-    return val;
-}
-
-#elif defined(__s390__)
-
-static inline int64_t cpu_get_real_ticks(void)
-{
-    int64_t val;
-    asm volatile("stck 0(%1)" : "=m" (val) : "a" (&val) : "cc");
-    return val;
-}
-
-#elif defined(__sparc__)
-
-static inline int64_t cpu_get_real_ticks (void)
-{
-#if defined(_LP64)
-    uint64_t        rval;
-    asm volatile("rd %%tick,%0" : "=r"(rval));
-    return rval;
-#else
-    /* We need an %o or %g register for this.  For recent enough gcc
-       there is an "h" constraint for that.  Don't bother with that.  */
-    union {
-        uint64_t i64;
-        struct {
-            uint32_t high;
-            uint32_t low;
-        }       i32;
-    } rval;
-    asm volatile("rd %%tick,%%g1; srlx %%g1,32,%0; mov %%g1,%1"
-                 : "=r"(rval.i32.high), "=r"(rval.i32.low) : : "g1");
-    return rval.i64;
-#endif
-}
-
-#elif defined(__mips__) && \
-    ((defined(__mips_isa_rev) && __mips_isa_rev >= 2) || defined(__linux__))
-/*
- * binutils wants to use rdhwr only on mips32r2
- * but as linux kernel emulate it, it's fine
- * to use it.
- *
- */
-#define MIPS_RDHWR(rd, value) {                         \
-        __asm__ __volatile__ (".set   push\n\t"         \
-                              ".set mips32r2\n\t"       \
-                              "rdhwr  %0, "rd"\n\t"     \
-                              ".set   pop"              \
-                              : "=r" (value));          \
-    }
-
-static inline int64_t cpu_get_real_ticks(void)
-{
-    /* On kernels >= 2.6.25 rdhwr <reg>, $2 and $3 are emulated */
-    uint32_t count;
-    static uint32_t cyc_per_count = 0;
-
-    if (!cyc_per_count) {
-        MIPS_RDHWR("$3", cyc_per_count);
-    }
-
-    MIPS_RDHWR("$2", count);
-    return (int64_t)(count * cyc_per_count);
-}
-
-#elif defined(__alpha__)
-
-static inline int64_t cpu_get_real_ticks(void)
-{
-    uint64_t cc;
-    uint32_t cur, ofs;
-
-    asm volatile("rpcc %0" : "=r"(cc));
-    cur = cc;
-    ofs = cc >> 32;
-    return cur - ofs;
-}
-
-#else
-/* The host CPU doesn't have an easily accessible cycle counter.
-   Just return a monotonically increasing value.  This will be
-   totally wrong, but hopefully better than nothing.  */
-static inline int64_t cpu_get_real_ticks (void)
-{
-    static int64_t ticks = 0;
-    return ticks++;
-}
-#endif
-
-#ifdef CONFIG_PROFILER
-static inline int64_t profile_getclock(void)
-{
-    return cpu_get_real_ticks();
-}
-
-extern int64_t qemu_time, qemu_time_start;
-extern int64_t tlb_flush_time;
-extern int64_t dev_time;
-#endif
-
-#endif
diff --git a/contrib/qemu/include/qemu/typedefs.h b/contrib/qemu/include/qemu/typedefs.h
deleted file mode 100644
index ac9f8d41a35..00000000000
--- a/contrib/qemu/include/qemu/typedefs.h
+++ /dev/null
@@ -1,69 +0,0 @@
-#ifndef QEMU_TYPEDEFS_H
-#define QEMU_TYPEDEFS_H
-
-/* A load of opaque types so that device init declarations don't have to
-   pull in all the real definitions.  */
-typedef struct QEMUTimer QEMUTimer;
-typedef struct QEMUFile QEMUFile;
-typedef struct QEMUBH QEMUBH;
-
-struct Monitor;
-typedef struct Monitor Monitor;
-typedef struct MigrationParams MigrationParams;
-
-typedef struct Property Property;
-typedef struct PropertyInfo PropertyInfo;
-typedef struct CompatProperty CompatProperty;
-typedef struct DeviceState DeviceState;
-typedef struct BusState BusState;
-typedef struct BusClass BusClass;
-
-typedef struct AddressSpace AddressSpace;
-typedef struct MemoryRegion MemoryRegion;
-typedef struct MemoryRegionSection MemoryRegionSection;
-
-typedef struct MemoryMappingList MemoryMappingList;
-
-typedef struct NICInfo NICInfo;
-typedef struct HCIInfo HCIInfo;
-typedef struct AudioState AudioState;
-typedef struct BlockDriverState BlockDriverState;
-typedef struct DriveInfo DriveInfo;
-typedef struct DisplayState DisplayState;
-typedef struct DisplayChangeListener DisplayChangeListener;
-typedef struct DisplaySurface DisplaySurface;
-typedef struct PixelFormat PixelFormat;
-typedef struct QemuConsole QemuConsole;
-typedef struct CharDriverState CharDriverState;
-typedef struct MACAddr MACAddr;
-typedef struct NetClientState NetClientState;
-typedef struct i2c_bus i2c_bus;
-typedef struct ISABus ISABus;
-typedef struct ISADevice ISADevice;
-typedef struct SMBusDevice SMBusDevice;
-typedef struct PCIHostState PCIHostState;
-typedef struct PCIExpressHost PCIExpressHost;
-typedef struct PCIBus PCIBus;
-typedef struct PCIDevice PCIDevice;
-typedef struct PCIExpressDevice PCIExpressDevice;
-typedef struct PCIBridge PCIBridge;
-typedef struct PCIEAERMsg PCIEAERMsg;
-typedef struct PCIEAERLog PCIEAERLog;
-typedef struct PCIEAERErr PCIEAERErr;
-typedef struct PCIEPort PCIEPort;
-typedef struct PCIESlot PCIESlot;
-typedef struct MSIMessage MSIMessage;
-typedef struct SerialState SerialState;
-typedef struct PCMCIACardState PCMCIACardState;
-typedef struct MouseTransformInfo MouseTransformInfo;
-typedef struct uWireSlave uWireSlave;
-typedef struct I2SCodec I2SCodec;
-typedef struct SSIBus SSIBus;
-typedef struct EventNotifier EventNotifier;
-typedef struct VirtIODevice VirtIODevice;
-typedef struct QEMUSGList QEMUSGList;
-typedef struct SHPCDevice SHPCDevice;
-typedef struct FWCfgState FWCfgState;
-typedef struct PcGuestInfo PcGuestInfo;
-
-#endif /* QEMU_TYPEDEFS_H */
diff --git a/contrib/qemu/include/sysemu/os-posix.h b/contrib/qemu/include/sysemu/os-posix.h
deleted file mode 100644
index ac4a80e6d8c..00000000000
--- a/contrib/qemu/include/sysemu/os-posix.h
+++ /dev/null
@@ -1,52 +0,0 @@
-/*
- * posix specific declarations
- *
- * Copyright (c) 2003-2008 Fabrice Bellard
- * Copyright (c) 2010 Jes Sorensen <Jes.Sorensen@redhat.com>
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-
-#ifndef QEMU_OS_POSIX_H
-#define QEMU_OS_POSIX_H
-
-void os_set_line_buffering(void);
-void os_set_proc_name(const char *s);
-void os_setup_signal_handling(void);
-void os_daemonize(void);
-void os_setup_post(void);
-int os_mlock(void);
-
-typedef struct timeval qemu_timeval;
-#define qemu_gettimeofday(tp) gettimeofday(tp, NULL)
-
-#if !defined(CONFIG_UTIMENSAT) || defined(__FreeBSD__) || defined(__APPLE__)
-#ifndef UTIME_NOW
-# define UTIME_NOW     ((1l << 30) - 1l)
-#endif
-#ifndef UTIME_OMIT
-# define UTIME_OMIT    ((1l << 30) - 2l)
-#endif
-#endif
-typedef struct timespec qemu_timespec;
-int qemu_utimens(const char *path, const qemu_timespec *times);
-
-bool is_daemonized(void);
-
-#endif
diff --git a/contrib/qemu/include/sysemu/sysemu.h b/contrib/qemu/include/sysemu/sysemu.h
deleted file mode 100644
index 3caeb66eb2f..00000000000
--- a/contrib/qemu/include/sysemu/sysemu.h
+++ /dev/null
@@ -1,200 +0,0 @@
-#ifndef SYSEMU_H
-#define SYSEMU_H
-/* Misc. things related to the system emulator.  */
-
-#include "qemu/typedefs.h"
-#include "qemu/option.h"
-#include "qemu/queue.h"
-#include "qemu/timer.h"
-#include "qapi-types.h"
-#include "qemu/notify.h"
-#include "qemu/main-loop.h"
-
-/* vl.c */
-
-extern const char *bios_name;
-
-extern const char *qemu_name;
-extern uint8_t qemu_uuid[];
-int qemu_uuid_parse(const char *str, uint8_t *uuid);
-#define UUID_FMT "%02hhx%02hhx%02hhx%02hhx-%02hhx%02hhx-%02hhx%02hhx-%02hhx%02hhx-%02hhx%02hhx%02hhx%02hhx%02hhx%02hhx"
-
-bool runstate_check(RunState state);
-void runstate_set(RunState new_state);
-int runstate_is_running(void);
-bool runstate_needs_reset(void);
-typedef struct vm_change_state_entry VMChangeStateEntry;
-typedef void VMChangeStateHandler(void *opaque, int running, RunState state);
-
-VMChangeStateEntry *qemu_add_vm_change_state_handler(VMChangeStateHandler *cb,
-                                                     void *opaque);
-void qemu_del_vm_change_state_handler(VMChangeStateEntry *e);
-void vm_state_notify(int running, RunState state);
-
-#define VMRESET_SILENT   false
-#define VMRESET_REPORT   true
-
-void vm_start(void);
-int vm_stop(RunState state);
-int vm_stop_force_state(RunState state);
-
-typedef enum WakeupReason {
-    QEMU_WAKEUP_REASON_OTHER = 0,
-    QEMU_WAKEUP_REASON_RTC,
-    QEMU_WAKEUP_REASON_PMTIMER,
-} WakeupReason;
-
-void qemu_system_reset_request(void);
-void qemu_system_suspend_request(void);
-void qemu_register_suspend_notifier(Notifier *notifier);
-void qemu_system_wakeup_request(WakeupReason reason);
-void qemu_system_wakeup_enable(WakeupReason reason, bool enabled);
-void qemu_register_wakeup_notifier(Notifier *notifier);
-void qemu_system_shutdown_request(void);
-void qemu_system_powerdown_request(void);
-void qemu_register_powerdown_notifier(Notifier *notifier);
-void qemu_system_debug_request(void);
-void qemu_system_vmstop_request(RunState reason);
-int qemu_shutdown_requested_get(void);
-int qemu_reset_requested_get(void);
-void qemu_system_killed(int signal, pid_t pid);
-void qemu_devices_reset(void);
-void qemu_system_reset(bool report);
-
-void qemu_add_exit_notifier(Notifier *notify);
-void qemu_remove_exit_notifier(Notifier *notify);
-
-void qemu_add_machine_init_done_notifier(Notifier *notify);
-
-void do_savevm(Monitor *mon, const QDict *qdict);
-int load_vmstate(const char *name);
-void do_delvm(Monitor *mon, const QDict *qdict);
-void do_info_snapshots(Monitor *mon, const QDict *qdict);
-
-void qemu_announce_self(void);
-
-bool qemu_savevm_state_blocked(Error **errp);
-void qemu_savevm_state_begin(QEMUFile *f,
-                             const MigrationParams *params);
-int qemu_savevm_state_iterate(QEMUFile *f);
-void qemu_savevm_state_complete(QEMUFile *f);
-void qemu_savevm_state_cancel(void);
-uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
-int qemu_loadvm_state(QEMUFile *f);
-
-/* SLIRP */
-void do_info_slirp(Monitor *mon);
-
-typedef enum DisplayType
-{
-    DT_DEFAULT,
-    DT_CURSES,
-    DT_SDL,
-    DT_GTK,
-    DT_NOGRAPHIC,
-    DT_NONE,
-} DisplayType;
-
-extern int autostart;
-
-typedef enum {
-    VGA_NONE, VGA_STD, VGA_CIRRUS, VGA_VMWARE, VGA_XENFB, VGA_QXL,
-} VGAInterfaceType;
-
-extern int vga_interface_type;
-#define xenfb_enabled (vga_interface_type == VGA_XENFB)
-#define qxl_enabled (vga_interface_type == VGA_QXL)
-
-extern int graphic_width;
-extern int graphic_height;
-extern int graphic_depth;
-extern DisplayType display_type;
-extern const char *keyboard_layout;
-extern int win2k_install_hack;
-extern int alt_grab;
-extern int ctrl_grab;
-extern int smp_cpus;
-extern int max_cpus;
-extern int cursor_hide;
-extern int graphic_rotate;
-extern int no_quit;
-extern int no_shutdown;
-extern int semihosting_enabled;
-extern int old_param;
-extern int boot_menu;
-extern uint8_t *boot_splash_filedata;
-extern size_t boot_splash_filedata_size;
-extern uint8_t qemu_extra_params_fw[2];
-extern QEMUClock *rtc_clock;
-
-#define MAX_NODES 64
-#define MAX_CPUMASK_BITS 255
-extern int nb_numa_nodes;
-extern uint64_t node_mem[MAX_NODES];
-extern unsigned long *node_cpumask[MAX_NODES];
-
-#define MAX_OPTION_ROMS 16
-typedef struct QEMUOptionRom {
-    const char *name;
-    int32_t bootindex;
-} QEMUOptionRom;
-extern QEMUOptionRom option_rom[MAX_OPTION_ROMS];
-extern int nb_option_roms;
-
-#define MAX_PROM_ENVS 128
-extern const char *prom_envs[MAX_PROM_ENVS];
-extern unsigned int nb_prom_envs;
-
-/* pci-hotplug */
-void pci_device_hot_add(Monitor *mon, const QDict *qdict);
-int pci_drive_hot_add(Monitor *mon, const QDict *qdict, DriveInfo *dinfo);
-void do_pci_device_hot_remove(Monitor *mon, const QDict *qdict);
-
-/* generic hotplug */
-void drive_hot_add(Monitor *mon, const QDict *qdict);
-
-/* CPU hotplug */
-void qemu_register_cpu_added_notifier(Notifier *notifier);
-
-/* pcie aer error injection */
-void pcie_aer_inject_error_print(Monitor *mon, const QObject *data);
-int do_pcie_aer_inject_error(Monitor *mon,
-                             const QDict *qdict, QObject **ret_data);
-
-/* serial ports */
-
-#define MAX_SERIAL_PORTS 4
-
-extern CharDriverState *serial_hds[MAX_SERIAL_PORTS];
-
-/* parallel ports */
-
-#define MAX_PARALLEL_PORTS 3
-
-extern CharDriverState *parallel_hds[MAX_PARALLEL_PORTS];
-
-void do_usb_add(Monitor *mon, const QDict *qdict);
-void do_usb_del(Monitor *mon, const QDict *qdict);
-void usb_info(Monitor *mon, const QDict *qdict);
-
-void rtc_change_mon_event(struct tm *tm);
-
-void add_boot_device_path(int32_t bootindex, DeviceState *dev,
-                          const char *suffix);
-char *get_boot_devices_list(size_t *size);
-
-DeviceState *get_boot_device(uint32_t position);
-
-QemuOpts *qemu_get_machine_opts(void);
-
-bool usb_enabled(bool default_usb);
-
-extern QemuOptsList qemu_drive_opts;
-extern QemuOptsList qemu_chardev_opts;
-extern QemuOptsList qemu_device_opts;
-extern QemuOptsList qemu_netdev_opts;
-extern QemuOptsList qemu_net_opts;
-extern QemuOptsList qemu_global_opts;
-extern QemuOptsList qemu_mon_opts;
-
-#endif
diff --git a/contrib/qemu/include/trace.h b/contrib/qemu/include/trace.h
deleted file mode 100644
index c15f4981280..00000000000
--- a/contrib/qemu/include/trace.h
+++ /dev/null
@@ -1,6 +0,0 @@
-#ifndef TRACE_H
-#define TRACE_H
-
-#include "trace/generated-tracers.h"
-
-#endif  /* TRACE_H */
diff --git a/contrib/qemu/nop-symbols.c b/contrib/qemu/nop-symbols.c
deleted file mode 100644
index ae93a3d3bef..00000000000
--- a/contrib/qemu/nop-symbols.c
+++ /dev/null
@@ -1,12 +0,0 @@
-int notifier_with_return_list_init () { return 0; }
-int notifier_with_return_list_notify () { return 0; }
-int notifier_with_return_list_add () { return 0; }
-int notifier_list_init () { return 0; }
-int notifier_list_notify () { return 0; }
-int notifier_list_add () { return 0; }
-int monitor_protocol_event () { return 0; }
-int block_job_cancel_sync () { return 0; }
-int block_job_iostatus_reset () { return 0; }
-int vm_stop () { return 0; }
-int qemu_get_aio_context () { return 0; }
-
diff --git a/contrib/qemu/qapi-types.h b/contrib/qemu/qapi-types.h
deleted file mode 100644
index 082b06d1c2b..00000000000
--- a/contrib/qemu/qapi-types.h
+++ /dev/null
@@ -1,2746 +0,0 @@
-/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
-
-/*
- * schema-defined QAPI types
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#ifndef QAPI_TYPES_H
-#define QAPI_TYPES_H
-
-#include <stdbool.h>
-#include <stdint.h>
-
-
-#ifndef QAPI_TYPES_BUILTIN_STRUCT_DECL_H
-#define QAPI_TYPES_BUILTIN_STRUCT_DECL_H
-
-
-typedef struct strList
-{
-    union {
-        char * value;
-        uint64_t padding;
-    };
-    struct strList *next;
-} strList;
-
-typedef struct intList
-{
-    union {
-        int64_t value;
-        uint64_t padding;
-    };
-    struct intList *next;
-} intList;
-
-typedef struct numberList
-{
-    union {
-        double value;
-        uint64_t padding;
-    };
-    struct numberList *next;
-} numberList;
-
-typedef struct boolList
-{
-    union {
-        bool value;
-        uint64_t padding;
-    };
-    struct boolList *next;
-} boolList;
-
-typedef struct int8List
-{
-    union {
-        int8_t value;
-        uint64_t padding;
-    };
-    struct int8List *next;
-} int8List;
-
-typedef struct int16List
-{
-    union {
-        int16_t value;
-        uint64_t padding;
-    };
-    struct int16List *next;
-} int16List;
-
-typedef struct int32List
-{
-    union {
-        int32_t value;
-        uint64_t padding;
-    };
-    struct int32List *next;
-} int32List;
-
-typedef struct int64List
-{
-    union {
-        int64_t value;
-        uint64_t padding;
-    };
-    struct int64List *next;
-} int64List;
-
-typedef struct uint8List
-{
-    union {
-        uint8_t value;
-        uint64_t padding;
-    };
-    struct uint8List *next;
-} uint8List;
-
-typedef struct uint16List
-{
-    union {
-        uint16_t value;
-        uint64_t padding;
-    };
-    struct uint16List *next;
-} uint16List;
-
-typedef struct uint32List
-{
-    union {
-        uint32_t value;
-        uint64_t padding;
-    };
-    struct uint32List *next;
-} uint32List;
-
-typedef struct uint64List
-{
-    union {
-        uint64_t value;
-        uint64_t padding;
-    };
-    struct uint64List *next;
-} uint64List;
-
-#endif /* QAPI_TYPES_BUILTIN_STRUCT_DECL_H */
-
-
-extern const char *ErrorClass_lookup[];
-typedef enum ErrorClass
-{
-    ERROR_CLASS_GENERIC_ERROR = 0,
-    ERROR_CLASS_COMMAND_NOT_FOUND = 1,
-    ERROR_CLASS_DEVICE_ENCRYPTED = 2,
-    ERROR_CLASS_DEVICE_NOT_ACTIVE = 3,
-    ERROR_CLASS_DEVICE_NOT_FOUND = 4,
-    ERROR_CLASS_K_V_M_MISSING_CAP = 5,
-    ERROR_CLASS_MAX = 6,
-} ErrorClass;
-
-typedef struct ErrorClassList
-{
-    ErrorClass value;
-    struct ErrorClassList *next;
-} ErrorClassList;
-
-
-typedef struct NameInfo NameInfo;
-
-typedef struct NameInfoList
-{
-    union {
-        NameInfo *value;
-        uint64_t padding;
-    };
-    struct NameInfoList *next;
-} NameInfoList;
-
-
-typedef struct VersionInfo VersionInfo;
-
-typedef struct VersionInfoList
-{
-    union {
-        VersionInfo *value;
-        uint64_t padding;
-    };
-    struct VersionInfoList *next;
-} VersionInfoList;
-
-
-typedef struct KvmInfo KvmInfo;
-
-typedef struct KvmInfoList
-{
-    union {
-        KvmInfo *value;
-        uint64_t padding;
-    };
-    struct KvmInfoList *next;
-} KvmInfoList;
-
-extern const char *RunState_lookup[];
-typedef enum RunState
-{
-    RUN_STATE_DEBUG = 0,
-    RUN_STATE_INMIGRATE = 1,
-    RUN_STATE_INTERNAL_ERROR = 2,
-    RUN_STATE_IO_ERROR = 3,
-    RUN_STATE_PAUSED = 4,
-    RUN_STATE_POSTMIGRATE = 5,
-    RUN_STATE_PRELAUNCH = 6,
-    RUN_STATE_FINISH_MIGRATE = 7,
-    RUN_STATE_RESTORE_VM = 8,
-    RUN_STATE_RUNNING = 9,
-    RUN_STATE_SAVE_VM = 10,
-    RUN_STATE_SHUTDOWN = 11,
-    RUN_STATE_SUSPENDED = 12,
-    RUN_STATE_WATCHDOG = 13,
-    RUN_STATE_GUEST_PANICKED = 14,
-    RUN_STATE_MAX = 15,
-} RunState;
-
-typedef struct RunStateList
-{
-    RunState value;
-    struct RunStateList *next;
-} RunStateList;
-
-
-typedef struct SnapshotInfo SnapshotInfo;
-
-typedef struct SnapshotInfoList
-{
-    union {
-        SnapshotInfo *value;
-        uint64_t padding;
-    };
-    struct SnapshotInfoList *next;
-} SnapshotInfoList;
-
-
-typedef struct ImageInfo ImageInfo;
-
-typedef struct ImageInfoList
-{
-    union {
-        ImageInfo *value;
-        uint64_t padding;
-    };
-    struct ImageInfoList *next;
-} ImageInfoList;
-
-
-typedef struct ImageCheck ImageCheck;
-
-typedef struct ImageCheckList
-{
-    union {
-        ImageCheck *value;
-        uint64_t padding;
-    };
-    struct ImageCheckList *next;
-} ImageCheckList;
-
-
-typedef struct StatusInfo StatusInfo;
-
-typedef struct StatusInfoList
-{
-    union {
-        StatusInfo *value;
-        uint64_t padding;
-    };
-    struct StatusInfoList *next;
-} StatusInfoList;
-
-
-typedef struct UuidInfo UuidInfo;
-
-typedef struct UuidInfoList
-{
-    union {
-        UuidInfo *value;
-        uint64_t padding;
-    };
-    struct UuidInfoList *next;
-} UuidInfoList;
-
-
-typedef struct ChardevInfo ChardevInfo;
-
-typedef struct ChardevInfoList
-{
-    union {
-        ChardevInfo *value;
-        uint64_t padding;
-    };
-    struct ChardevInfoList *next;
-} ChardevInfoList;
-
-extern const char *DataFormat_lookup[];
-typedef enum DataFormat
-{
-    DATA_FORMAT_UTF8 = 0,
-    DATA_FORMAT_BASE64 = 1,
-    DATA_FORMAT_MAX = 2,
-} DataFormat;
-
-typedef struct DataFormatList
-{
-    DataFormat value;
-    struct DataFormatList *next;
-} DataFormatList;
-
-
-typedef struct CommandInfo CommandInfo;
-
-typedef struct CommandInfoList
-{
-    union {
-        CommandInfo *value;
-        uint64_t padding;
-    };
-    struct CommandInfoList *next;
-} CommandInfoList;
-
-
-typedef struct EventInfo EventInfo;
-
-typedef struct EventInfoList
-{
-    union {
-        EventInfo *value;
-        uint64_t padding;
-    };
-    struct EventInfoList *next;
-} EventInfoList;
-
-
-typedef struct MigrationStats MigrationStats;
-
-typedef struct MigrationStatsList
-{
-    union {
-        MigrationStats *value;
-        uint64_t padding;
-    };
-    struct MigrationStatsList *next;
-} MigrationStatsList;
-
-
-typedef struct XBZRLECacheStats XBZRLECacheStats;
-
-typedef struct XBZRLECacheStatsList
-{
-    union {
-        XBZRLECacheStats *value;
-        uint64_t padding;
-    };
-    struct XBZRLECacheStatsList *next;
-} XBZRLECacheStatsList;
-
-
-typedef struct MigrationInfo MigrationInfo;
-
-typedef struct MigrationInfoList
-{
-    union {
-        MigrationInfo *value;
-        uint64_t padding;
-    };
-    struct MigrationInfoList *next;
-} MigrationInfoList;
-
-extern const char *MigrationCapability_lookup[];
-typedef enum MigrationCapability
-{
-    MIGRATION_CAPABILITY_XBZRLE = 0,
-    MIGRATION_CAPABILITY_X_RDMA_PIN_ALL = 1,
-    MIGRATION_CAPABILITY_AUTO_CONVERGE = 2,
-    MIGRATION_CAPABILITY_MAX = 3,
-} MigrationCapability;
-
-typedef struct MigrationCapabilityList
-{
-    MigrationCapability value;
-    struct MigrationCapabilityList *next;
-} MigrationCapabilityList;
-
-
-typedef struct MigrationCapabilityStatus MigrationCapabilityStatus;
-
-typedef struct MigrationCapabilityStatusList
-{
-    union {
-        MigrationCapabilityStatus *value;
-        uint64_t padding;
-    };
-    struct MigrationCapabilityStatusList *next;
-} MigrationCapabilityStatusList;
-
-
-typedef struct MouseInfo MouseInfo;
-
-typedef struct MouseInfoList
-{
-    union {
-        MouseInfo *value;
-        uint64_t padding;
-    };
-    struct MouseInfoList *next;
-} MouseInfoList;
-
-
-typedef struct CpuInfo CpuInfo;
-
-typedef struct CpuInfoList
-{
-    union {
-        CpuInfo *value;
-        uint64_t padding;
-    };
-    struct CpuInfoList *next;
-} CpuInfoList;
-
-
-typedef struct BlockDeviceInfo BlockDeviceInfo;
-
-typedef struct BlockDeviceInfoList
-{
-    union {
-        BlockDeviceInfo *value;
-        uint64_t padding;
-    };
-    struct BlockDeviceInfoList *next;
-} BlockDeviceInfoList;
-
-extern const char *BlockDeviceIoStatus_lookup[];
-typedef enum BlockDeviceIoStatus
-{
-    BLOCK_DEVICE_IO_STATUS_OK = 0,
-    BLOCK_DEVICE_IO_STATUS_FAILED = 1,
-    BLOCK_DEVICE_IO_STATUS_NOSPACE = 2,
-    BLOCK_DEVICE_IO_STATUS_MAX = 3,
-} BlockDeviceIoStatus;
-
-typedef struct BlockDeviceIoStatusList
-{
-    BlockDeviceIoStatus value;
-    struct BlockDeviceIoStatusList *next;
-} BlockDeviceIoStatusList;
-
-
-typedef struct BlockDirtyInfo BlockDirtyInfo;
-
-typedef struct BlockDirtyInfoList
-{
-    union {
-        BlockDirtyInfo *value;
-        uint64_t padding;
-    };
-    struct BlockDirtyInfoList *next;
-} BlockDirtyInfoList;
-
-
-typedef struct BlockInfo BlockInfo;
-
-typedef struct BlockInfoList
-{
-    union {
-        BlockInfo *value;
-        uint64_t padding;
-    };
-    struct BlockInfoList *next;
-} BlockInfoList;
-
-
-typedef struct BlockDeviceStats BlockDeviceStats;
-
-typedef struct BlockDeviceStatsList
-{
-    union {
-        BlockDeviceStats *value;
-        uint64_t padding;
-    };
-    struct BlockDeviceStatsList *next;
-} BlockDeviceStatsList;
-
-
-typedef struct BlockStats BlockStats;
-
-typedef struct BlockStatsList
-{
-    union {
-        BlockStats *value;
-        uint64_t padding;
-    };
-    struct BlockStatsList *next;
-} BlockStatsList;
-
-
-typedef struct VncClientInfo VncClientInfo;
-
-typedef struct VncClientInfoList
-{
-    union {
-        VncClientInfo *value;
-        uint64_t padding;
-    };
-    struct VncClientInfoList *next;
-} VncClientInfoList;
-
-
-typedef struct VncInfo VncInfo;
-
-typedef struct VncInfoList
-{
-    union {
-        VncInfo *value;
-        uint64_t padding;
-    };
-    struct VncInfoList *next;
-} VncInfoList;
-
-
-typedef struct SpiceChannel SpiceChannel;
-
-typedef struct SpiceChannelList
-{
-    union {
-        SpiceChannel *value;
-        uint64_t padding;
-    };
-    struct SpiceChannelList *next;
-} SpiceChannelList;
-
-extern const char *SpiceQueryMouseMode_lookup[];
-typedef enum SpiceQueryMouseMode
-{
-    SPICE_QUERY_MOUSE_MODE_CLIENT = 0,
-    SPICE_QUERY_MOUSE_MODE_SERVER = 1,
-    SPICE_QUERY_MOUSE_MODE_UNKNOWN = 2,
-    SPICE_QUERY_MOUSE_MODE_MAX = 3,
-} SpiceQueryMouseMode;
-
-typedef struct SpiceQueryMouseModeList
-{
-    SpiceQueryMouseMode value;
-    struct SpiceQueryMouseModeList *next;
-} SpiceQueryMouseModeList;
-
-
-typedef struct SpiceInfo SpiceInfo;
-
-typedef struct SpiceInfoList
-{
-    union {
-        SpiceInfo *value;
-        uint64_t padding;
-    };
-    struct SpiceInfoList *next;
-} SpiceInfoList;
-
-
-typedef struct BalloonInfo BalloonInfo;
-
-typedef struct BalloonInfoList
-{
-    union {
-        BalloonInfo *value;
-        uint64_t padding;
-    };
-    struct BalloonInfoList *next;
-} BalloonInfoList;
-
-
-typedef struct PciMemoryRange PciMemoryRange;
-
-typedef struct PciMemoryRangeList
-{
-    union {
-        PciMemoryRange *value;
-        uint64_t padding;
-    };
-    struct PciMemoryRangeList *next;
-} PciMemoryRangeList;
-
-
-typedef struct PciMemoryRegion PciMemoryRegion;
-
-typedef struct PciMemoryRegionList
-{
-    union {
-        PciMemoryRegion *value;
-        uint64_t padding;
-    };
-    struct PciMemoryRegionList *next;
-} PciMemoryRegionList;
-
-
-typedef struct PciBridgeInfo PciBridgeInfo;
-
-typedef struct PciBridgeInfoList
-{
-    union {
-        PciBridgeInfo *value;
-        uint64_t padding;
-    };
-    struct PciBridgeInfoList *next;
-} PciBridgeInfoList;
-
-
-typedef struct PciDeviceInfo PciDeviceInfo;
-
-typedef struct PciDeviceInfoList
-{
-    union {
-        PciDeviceInfo *value;
-        uint64_t padding;
-    };
-    struct PciDeviceInfoList *next;
-} PciDeviceInfoList;
-
-
-typedef struct PciInfo PciInfo;
-
-typedef struct PciInfoList
-{
-    union {
-        PciInfo *value;
-        uint64_t padding;
-    };
-    struct PciInfoList *next;
-} PciInfoList;
-
-extern const char *BlockdevOnError_lookup[];
-typedef enum BlockdevOnError
-{
-    BLOCKDEV_ON_ERROR_REPORT = 0,
-    BLOCKDEV_ON_ERROR_IGNORE = 1,
-    BLOCKDEV_ON_ERROR_ENOSPC = 2,
-    BLOCKDEV_ON_ERROR_STOP = 3,
-    BLOCKDEV_ON_ERROR_MAX = 4,
-} BlockdevOnError;
-
-typedef struct BlockdevOnErrorList
-{
-    BlockdevOnError value;
-    struct BlockdevOnErrorList *next;
-} BlockdevOnErrorList;
-
-extern const char *MirrorSyncMode_lookup[];
-typedef enum MirrorSyncMode
-{
-    MIRROR_SYNC_MODE_TOP = 0,
-    MIRROR_SYNC_MODE_FULL = 1,
-    MIRROR_SYNC_MODE_NONE = 2,
-    MIRROR_SYNC_MODE_MAX = 3,
-} MirrorSyncMode;
-
-typedef struct MirrorSyncModeList
-{
-    MirrorSyncMode value;
-    struct MirrorSyncModeList *next;
-} MirrorSyncModeList;
-
-
-typedef struct BlockJobInfo BlockJobInfo;
-
-typedef struct BlockJobInfoList
-{
-    union {
-        BlockJobInfo *value;
-        uint64_t padding;
-    };
-    struct BlockJobInfoList *next;
-} BlockJobInfoList;
-
-extern const char *NewImageMode_lookup[];
-typedef enum NewImageMode
-{
-    NEW_IMAGE_MODE_EXISTING = 0,
-    NEW_IMAGE_MODE_ABSOLUTE_PATHS = 1,
-    NEW_IMAGE_MODE_MAX = 2,
-} NewImageMode;
-
-typedef struct NewImageModeList
-{
-    NewImageMode value;
-    struct NewImageModeList *next;
-} NewImageModeList;
-
-
-typedef struct BlockdevSnapshot BlockdevSnapshot;
-
-typedef struct BlockdevSnapshotList
-{
-    union {
-        BlockdevSnapshot *value;
-        uint64_t padding;
-    };
-    struct BlockdevSnapshotList *next;
-} BlockdevSnapshotList;
-
-
-typedef struct DriveBackup DriveBackup;
-
-typedef struct DriveBackupList
-{
-    union {
-        DriveBackup *value;
-        uint64_t padding;
-    };
-    struct DriveBackupList *next;
-} DriveBackupList;
-
-
-typedef struct Abort Abort;
-
-typedef struct AbortList
-{
-    union {
-        Abort *value;
-        uint64_t padding;
-    };
-    struct AbortList *next;
-} AbortList;
-
-
-typedef struct TransactionAction TransactionAction;
-
-typedef struct TransactionActionList
-{
-    union {
-        TransactionAction *value;
-        uint64_t padding;
-    };
-    struct TransactionActionList *next;
-} TransactionActionList;
-
-extern const char *TransactionActionKind_lookup[];
-typedef enum TransactionActionKind
-{
-    TRANSACTION_ACTION_KIND_BLOCKDEV_SNAPSHOT_SYNC = 0,
-    TRANSACTION_ACTION_KIND_DRIVE_BACKUP = 1,
-    TRANSACTION_ACTION_KIND_ABORT = 2,
-    TRANSACTION_ACTION_KIND_MAX = 3,
-} TransactionActionKind;
-
-
-typedef struct ObjectPropertyInfo ObjectPropertyInfo;
-
-typedef struct ObjectPropertyInfoList
-{
-    union {
-        ObjectPropertyInfo *value;
-        uint64_t padding;
-    };
-    struct ObjectPropertyInfoList *next;
-} ObjectPropertyInfoList;
-
-
-typedef struct ObjectTypeInfo ObjectTypeInfo;
-
-typedef struct ObjectTypeInfoList
-{
-    union {
-        ObjectTypeInfo *value;
-        uint64_t padding;
-    };
-    struct ObjectTypeInfoList *next;
-} ObjectTypeInfoList;
-
-
-typedef struct DevicePropertyInfo DevicePropertyInfo;
-
-typedef struct DevicePropertyInfoList
-{
-    union {
-        DevicePropertyInfo *value;
-        uint64_t padding;
-    };
-    struct DevicePropertyInfoList *next;
-} DevicePropertyInfoList;
-
-
-typedef struct NetdevNoneOptions NetdevNoneOptions;
-
-typedef struct NetdevNoneOptionsList
-{
-    union {
-        NetdevNoneOptions *value;
-        uint64_t padding;
-    };
-    struct NetdevNoneOptionsList *next;
-} NetdevNoneOptionsList;
-
-
-typedef struct NetLegacyNicOptions NetLegacyNicOptions;
-
-typedef struct NetLegacyNicOptionsList
-{
-    union {
-        NetLegacyNicOptions *value;
-        uint64_t padding;
-    };
-    struct NetLegacyNicOptionsList *next;
-} NetLegacyNicOptionsList;
-
-
-typedef struct String String;
-
-typedef struct StringList
-{
-    union {
-        String *value;
-        uint64_t padding;
-    };
-    struct StringList *next;
-} StringList;
-
-
-typedef struct NetdevUserOptions NetdevUserOptions;
-
-typedef struct NetdevUserOptionsList
-{
-    union {
-        NetdevUserOptions *value;
-        uint64_t padding;
-    };
-    struct NetdevUserOptionsList *next;
-} NetdevUserOptionsList;
-
-
-typedef struct NetdevTapOptions NetdevTapOptions;
-
-typedef struct NetdevTapOptionsList
-{
-    union {
-        NetdevTapOptions *value;
-        uint64_t padding;
-    };
-    struct NetdevTapOptionsList *next;
-} NetdevTapOptionsList;
-
-
-typedef struct NetdevSocketOptions NetdevSocketOptions;
-
-typedef struct NetdevSocketOptionsList
-{
-    union {
-        NetdevSocketOptions *value;
-        uint64_t padding;
-    };
-    struct NetdevSocketOptionsList *next;
-} NetdevSocketOptionsList;
-
-
-typedef struct NetdevVdeOptions NetdevVdeOptions;
-
-typedef struct NetdevVdeOptionsList
-{
-    union {
-        NetdevVdeOptions *value;
-        uint64_t padding;
-    };
-    struct NetdevVdeOptionsList *next;
-} NetdevVdeOptionsList;
-
-
-typedef struct NetdevDumpOptions NetdevDumpOptions;
-
-typedef struct NetdevDumpOptionsList
-{
-    union {
-        NetdevDumpOptions *value;
-        uint64_t padding;
-    };
-    struct NetdevDumpOptionsList *next;
-} NetdevDumpOptionsList;
-
-
-typedef struct NetdevBridgeOptions NetdevBridgeOptions;
-
-typedef struct NetdevBridgeOptionsList
-{
-    union {
-        NetdevBridgeOptions *value;
-        uint64_t padding;
-    };
-    struct NetdevBridgeOptionsList *next;
-} NetdevBridgeOptionsList;
-
-
-typedef struct NetdevHubPortOptions NetdevHubPortOptions;
-
-typedef struct NetdevHubPortOptionsList
-{
-    union {
-        NetdevHubPortOptions *value;
-        uint64_t padding;
-    };
-    struct NetdevHubPortOptionsList *next;
-} NetdevHubPortOptionsList;
-
-
-typedef struct NetClientOptions NetClientOptions;
-
-typedef struct NetClientOptionsList
-{
-    union {
-        NetClientOptions *value;
-        uint64_t padding;
-    };
-    struct NetClientOptionsList *next;
-} NetClientOptionsList;
-
-extern const char *NetClientOptionsKind_lookup[];
-typedef enum NetClientOptionsKind
-{
-    NET_CLIENT_OPTIONS_KIND_NONE = 0,
-    NET_CLIENT_OPTIONS_KIND_NIC = 1,
-    NET_CLIENT_OPTIONS_KIND_USER = 2,
-    NET_CLIENT_OPTIONS_KIND_TAP = 3,
-    NET_CLIENT_OPTIONS_KIND_SOCKET = 4,
-    NET_CLIENT_OPTIONS_KIND_VDE = 5,
-    NET_CLIENT_OPTIONS_KIND_DUMP = 6,
-    NET_CLIENT_OPTIONS_KIND_BRIDGE = 7,
-    NET_CLIENT_OPTIONS_KIND_HUBPORT = 8,
-    NET_CLIENT_OPTIONS_KIND_MAX = 9,
-} NetClientOptionsKind;
-
-
-typedef struct NetLegacy NetLegacy;
-
-typedef struct NetLegacyList
-{
-    union {
-        NetLegacy *value;
-        uint64_t padding;
-    };
-    struct NetLegacyList *next;
-} NetLegacyList;
-
-
-typedef struct Netdev Netdev;
-
-typedef struct NetdevList
-{
-    union {
-        Netdev *value;
-        uint64_t padding;
-    };
-    struct NetdevList *next;
-} NetdevList;
-
-
-typedef struct InetSocketAddress InetSocketAddress;
-
-typedef struct InetSocketAddressList
-{
-    union {
-        InetSocketAddress *value;
-        uint64_t padding;
-    };
-    struct InetSocketAddressList *next;
-} InetSocketAddressList;
-
-
-typedef struct UnixSocketAddress UnixSocketAddress;
-
-typedef struct UnixSocketAddressList
-{
-    union {
-        UnixSocketAddress *value;
-        uint64_t padding;
-    };
-    struct UnixSocketAddressList *next;
-} UnixSocketAddressList;
-
-
-typedef struct SocketAddress SocketAddress;
-
-typedef struct SocketAddressList
-{
-    union {
-        SocketAddress *value;
-        uint64_t padding;
-    };
-    struct SocketAddressList *next;
-} SocketAddressList;
-
-extern const char *SocketAddressKind_lookup[];
-typedef enum SocketAddressKind
-{
-    SOCKET_ADDRESS_KIND_INET = 0,
-    SOCKET_ADDRESS_KIND_UNIX = 1,
-    SOCKET_ADDRESS_KIND_FD = 2,
-    SOCKET_ADDRESS_KIND_MAX = 3,
-} SocketAddressKind;
-
-
-typedef struct MachineInfo MachineInfo;
-
-typedef struct MachineInfoList
-{
-    union {
-        MachineInfo *value;
-        uint64_t padding;
-    };
-    struct MachineInfoList *next;
-} MachineInfoList;
-
-
-typedef struct CpuDefinitionInfo CpuDefinitionInfo;
-
-typedef struct CpuDefinitionInfoList
-{
-    union {
-        CpuDefinitionInfo *value;
-        uint64_t padding;
-    };
-    struct CpuDefinitionInfoList *next;
-} CpuDefinitionInfoList;
-
-
-typedef struct AddfdInfo AddfdInfo;
-
-typedef struct AddfdInfoList
-{
-    union {
-        AddfdInfo *value;
-        uint64_t padding;
-    };
-    struct AddfdInfoList *next;
-} AddfdInfoList;
-
-
-typedef struct FdsetFdInfo FdsetFdInfo;
-
-typedef struct FdsetFdInfoList
-{
-    union {
-        FdsetFdInfo *value;
-        uint64_t padding;
-    };
-    struct FdsetFdInfoList *next;
-} FdsetFdInfoList;
-
-
-typedef struct FdsetInfo FdsetInfo;
-
-typedef struct FdsetInfoList
-{
-    union {
-        FdsetInfo *value;
-        uint64_t padding;
-    };
-    struct FdsetInfoList *next;
-} FdsetInfoList;
-
-
-typedef struct TargetInfo TargetInfo;
-
-typedef struct TargetInfoList
-{
-    union {
-        TargetInfo *value;
-        uint64_t padding;
-    };
-    struct TargetInfoList *next;
-} TargetInfoList;
-
-extern const char *QKeyCode_lookup[];
-typedef enum QKeyCode
-{
-    Q_KEY_CODE_SHIFT = 0,
-    Q_KEY_CODE_SHIFT_R = 1,
-    Q_KEY_CODE_ALT = 2,
-    Q_KEY_CODE_ALT_R = 3,
-    Q_KEY_CODE_ALTGR = 4,
-    Q_KEY_CODE_ALTGR_R = 5,
-    Q_KEY_CODE_CTRL = 6,
-    Q_KEY_CODE_CTRL_R = 7,
-    Q_KEY_CODE_MENU = 8,
-    Q_KEY_CODE_ESC = 9,
-    Q_KEY_CODE_1 = 10,
-    Q_KEY_CODE_2 = 11,
-    Q_KEY_CODE_3 = 12,
-    Q_KEY_CODE_4 = 13,
-    Q_KEY_CODE_5 = 14,
-    Q_KEY_CODE_6 = 15,
-    Q_KEY_CODE_7 = 16,
-    Q_KEY_CODE_8 = 17,
-    Q_KEY_CODE_9 = 18,
-    Q_KEY_CODE_0 = 19,
-    Q_KEY_CODE_MINUS = 20,
-    Q_KEY_CODE_EQUAL = 21,
-    Q_KEY_CODE_BACKSPACE = 22,
-    Q_KEY_CODE_TAB = 23,
-    Q_KEY_CODE_Q = 24,
-    Q_KEY_CODE_W = 25,
-    Q_KEY_CODE_E = 26,
-    Q_KEY_CODE_R = 27,
-    Q_KEY_CODE_T = 28,
-    Q_KEY_CODE_Y = 29,
-    Q_KEY_CODE_U = 30,
-    Q_KEY_CODE_I = 31,
-    Q_KEY_CODE_O = 32,
-    Q_KEY_CODE_P = 33,
-    Q_KEY_CODE_BRACKET_LEFT = 34,
-    Q_KEY_CODE_BRACKET_RIGHT = 35,
-    Q_KEY_CODE_RET = 36,
-    Q_KEY_CODE_A = 37,
-    Q_KEY_CODE_S = 38,
-    Q_KEY_CODE_D = 39,
-    Q_KEY_CODE_F = 40,
-    Q_KEY_CODE_G = 41,
-    Q_KEY_CODE_H = 42,
-    Q_KEY_CODE_J = 43,
-    Q_KEY_CODE_K = 44,
-    Q_KEY_CODE_L = 45,
-    Q_KEY_CODE_SEMICOLON = 46,
-    Q_KEY_CODE_APOSTROPHE = 47,
-    Q_KEY_CODE_GRAVE_ACCENT = 48,
-    Q_KEY_CODE_BACKSLASH = 49,
-    Q_KEY_CODE_Z = 50,
-    Q_KEY_CODE_X = 51,
-    Q_KEY_CODE_C = 52,
-    Q_KEY_CODE_V = 53,
-    Q_KEY_CODE_B = 54,
-    Q_KEY_CODE_N = 55,
-    Q_KEY_CODE_M = 56,
-    Q_KEY_CODE_COMMA = 57,
-    Q_KEY_CODE_DOT = 58,
-    Q_KEY_CODE_SLASH = 59,
-    Q_KEY_CODE_ASTERISK = 60,
-    Q_KEY_CODE_SPC = 61,
-    Q_KEY_CODE_CAPS_LOCK = 62,
-    Q_KEY_CODE_F1 = 63,
-    Q_KEY_CODE_F2 = 64,
-    Q_KEY_CODE_F3 = 65,
-    Q_KEY_CODE_F4 = 66,
-    Q_KEY_CODE_F5 = 67,
-    Q_KEY_CODE_F6 = 68,
-    Q_KEY_CODE_F7 = 69,
-    Q_KEY_CODE_F8 = 70,
-    Q_KEY_CODE_F9 = 71,
-    Q_KEY_CODE_F10 = 72,
-    Q_KEY_CODE_NUM_LOCK = 73,
-    Q_KEY_CODE_SCROLL_LOCK = 74,
-    Q_KEY_CODE_KP_DIVIDE = 75,
-    Q_KEY_CODE_KP_MULTIPLY = 76,
-    Q_KEY_CODE_KP_SUBTRACT = 77,
-    Q_KEY_CODE_KP_ADD = 78,
-    Q_KEY_CODE_KP_ENTER = 79,
-    Q_KEY_CODE_KP_DECIMAL = 80,
-    Q_KEY_CODE_SYSRQ = 81,
-    Q_KEY_CODE_KP_0 = 82,
-    Q_KEY_CODE_KP_1 = 83,
-    Q_KEY_CODE_KP_2 = 84,
-    Q_KEY_CODE_KP_3 = 85,
-    Q_KEY_CODE_KP_4 = 86,
-    Q_KEY_CODE_KP_5 = 87,
-    Q_KEY_CODE_KP_6 = 88,
-    Q_KEY_CODE_KP_7 = 89,
-    Q_KEY_CODE_KP_8 = 90,
-    Q_KEY_CODE_KP_9 = 91,
-    Q_KEY_CODE_LESS = 92,
-    Q_KEY_CODE_F11 = 93,
-    Q_KEY_CODE_F12 = 94,
-    Q_KEY_CODE_PRINT = 95,
-    Q_KEY_CODE_HOME = 96,
-    Q_KEY_CODE_PGUP = 97,
-    Q_KEY_CODE_PGDN = 98,
-    Q_KEY_CODE_END = 99,
-    Q_KEY_CODE_LEFT = 100,
-    Q_KEY_CODE_UP = 101,
-    Q_KEY_CODE_DOWN = 102,
-    Q_KEY_CODE_RIGHT = 103,
-    Q_KEY_CODE_INSERT = 104,
-    Q_KEY_CODE_DELETE = 105,
-    Q_KEY_CODE_STOP = 106,
-    Q_KEY_CODE_AGAIN = 107,
-    Q_KEY_CODE_PROPS = 108,
-    Q_KEY_CODE_UNDO = 109,
-    Q_KEY_CODE_FRONT = 110,
-    Q_KEY_CODE_COPY = 111,
-    Q_KEY_CODE_OPEN = 112,
-    Q_KEY_CODE_PASTE = 113,
-    Q_KEY_CODE_FIND = 114,
-    Q_KEY_CODE_CUT = 115,
-    Q_KEY_CODE_LF = 116,
-    Q_KEY_CODE_HELP = 117,
-    Q_KEY_CODE_META_L = 118,
-    Q_KEY_CODE_META_R = 119,
-    Q_KEY_CODE_COMPOSE = 120,
-    Q_KEY_CODE_MAX = 121,
-} QKeyCode;
-
-typedef struct QKeyCodeList
-{
-    QKeyCode value;
-    struct QKeyCodeList *next;
-} QKeyCodeList;
-
-
-typedef struct KeyValue KeyValue;
-
-typedef struct KeyValueList
-{
-    union {
-        KeyValue *value;
-        uint64_t padding;
-    };
-    struct KeyValueList *next;
-} KeyValueList;
-
-extern const char *KeyValueKind_lookup[];
-typedef enum KeyValueKind
-{
-    KEY_VALUE_KIND_NUMBER = 0,
-    KEY_VALUE_KIND_QCODE = 1,
-    KEY_VALUE_KIND_MAX = 2,
-} KeyValueKind;
-
-
-typedef struct ChardevFile ChardevFile;
-
-typedef struct ChardevFileList
-{
-    union {
-        ChardevFile *value;
-        uint64_t padding;
-    };
-    struct ChardevFileList *next;
-} ChardevFileList;
-
-
-typedef struct ChardevHostdev ChardevHostdev;
-
-typedef struct ChardevHostdevList
-{
-    union {
-        ChardevHostdev *value;
-        uint64_t padding;
-    };
-    struct ChardevHostdevList *next;
-} ChardevHostdevList;
-
-
-typedef struct ChardevSocket ChardevSocket;
-
-typedef struct ChardevSocketList
-{
-    union {
-        ChardevSocket *value;
-        uint64_t padding;
-    };
-    struct ChardevSocketList *next;
-} ChardevSocketList;
-
-
-typedef struct ChardevUdp ChardevUdp;
-
-typedef struct ChardevUdpList
-{
-    union {
-        ChardevUdp *value;
-        uint64_t padding;
-    };
-    struct ChardevUdpList *next;
-} ChardevUdpList;
-
-
-typedef struct ChardevMux ChardevMux;
-
-typedef struct ChardevMuxList
-{
-    union {
-        ChardevMux *value;
-        uint64_t padding;
-    };
-    struct ChardevMuxList *next;
-} ChardevMuxList;
-
-
-typedef struct ChardevStdio ChardevStdio;
-
-typedef struct ChardevStdioList
-{
-    union {
-        ChardevStdio *value;
-        uint64_t padding;
-    };
-    struct ChardevStdioList *next;
-} ChardevStdioList;
-
-
-typedef struct ChardevSpiceChannel ChardevSpiceChannel;
-
-typedef struct ChardevSpiceChannelList
-{
-    union {
-        ChardevSpiceChannel *value;
-        uint64_t padding;
-    };
-    struct ChardevSpiceChannelList *next;
-} ChardevSpiceChannelList;
-
-
-typedef struct ChardevSpicePort ChardevSpicePort;
-
-typedef struct ChardevSpicePortList
-{
-    union {
-        ChardevSpicePort *value;
-        uint64_t padding;
-    };
-    struct ChardevSpicePortList *next;
-} ChardevSpicePortList;
-
-
-typedef struct ChardevVC ChardevVC;
-
-typedef struct ChardevVCList
-{
-    union {
-        ChardevVC *value;
-        uint64_t padding;
-    };
-    struct ChardevVCList *next;
-} ChardevVCList;
-
-
-typedef struct ChardevMemory ChardevMemory;
-
-typedef struct ChardevMemoryList
-{
-    union {
-        ChardevMemory *value;
-        uint64_t padding;
-    };
-    struct ChardevMemoryList *next;
-} ChardevMemoryList;
-
-
-typedef struct ChardevDummy ChardevDummy;
-
-typedef struct ChardevDummyList
-{
-    union {
-        ChardevDummy *value;
-        uint64_t padding;
-    };
-    struct ChardevDummyList *next;
-} ChardevDummyList;
-
-
-typedef struct ChardevBackend ChardevBackend;
-
-typedef struct ChardevBackendList
-{
-    union {
-        ChardevBackend *value;
-        uint64_t padding;
-    };
-    struct ChardevBackendList *next;
-} ChardevBackendList;
-
-extern const char *ChardevBackendKind_lookup[];
-typedef enum ChardevBackendKind
-{
-    CHARDEV_BACKEND_KIND_FILE = 0,
-    CHARDEV_BACKEND_KIND_SERIAL = 1,
-    CHARDEV_BACKEND_KIND_PARALLEL = 2,
-    CHARDEV_BACKEND_KIND_PIPE = 3,
-    CHARDEV_BACKEND_KIND_SOCKET = 4,
-    CHARDEV_BACKEND_KIND_UDP = 5,
-    CHARDEV_BACKEND_KIND_PTY = 6,
-    CHARDEV_BACKEND_KIND_NULL = 7,
-    CHARDEV_BACKEND_KIND_MUX = 8,
-    CHARDEV_BACKEND_KIND_MSMOUSE = 9,
-    CHARDEV_BACKEND_KIND_BRAILLE = 10,
-    CHARDEV_BACKEND_KIND_STDIO = 11,
-    CHARDEV_BACKEND_KIND_CONSOLE = 12,
-    CHARDEV_BACKEND_KIND_SPICEVMC = 13,
-    CHARDEV_BACKEND_KIND_SPICEPORT = 14,
-    CHARDEV_BACKEND_KIND_VC = 15,
-    CHARDEV_BACKEND_KIND_MEMORY = 16,
-    CHARDEV_BACKEND_KIND_MAX = 17,
-} ChardevBackendKind;
-
-
-typedef struct ChardevReturn ChardevReturn;
-
-typedef struct ChardevReturnList
-{
-    union {
-        ChardevReturn *value;
-        uint64_t padding;
-    };
-    struct ChardevReturnList *next;
-} ChardevReturnList;
-
-extern const char *TpmModel_lookup[];
-typedef enum TpmModel
-{
-    TPM_MODEL_TPM_TIS = 0,
-    TPM_MODEL_MAX = 1,
-} TpmModel;
-
-typedef struct TpmModelList
-{
-    TpmModel value;
-    struct TpmModelList *next;
-} TpmModelList;
-
-extern const char *TpmType_lookup[];
-typedef enum TpmType
-{
-    TPM_TYPE_PASSTHROUGH = 0,
-    TPM_TYPE_MAX = 1,
-} TpmType;
-
-typedef struct TpmTypeList
-{
-    TpmType value;
-    struct TpmTypeList *next;
-} TpmTypeList;
-
-
-typedef struct TPMPassthroughOptions TPMPassthroughOptions;
-
-typedef struct TPMPassthroughOptionsList
-{
-    union {
-        TPMPassthroughOptions *value;
-        uint64_t padding;
-    };
-    struct TPMPassthroughOptionsList *next;
-} TPMPassthroughOptionsList;
-
-
-typedef struct TpmTypeOptions TpmTypeOptions;
-
-typedef struct TpmTypeOptionsList
-{
-    union {
-        TpmTypeOptions *value;
-        uint64_t padding;
-    };
-    struct TpmTypeOptionsList *next;
-} TpmTypeOptionsList;
-
-extern const char *TpmTypeOptionsKind_lookup[];
-typedef enum TpmTypeOptionsKind
-{
-    TPM_TYPE_OPTIONS_KIND_PASSTHROUGH = 0,
-    TPM_TYPE_OPTIONS_KIND_MAX = 1,
-} TpmTypeOptionsKind;
-
-
-typedef struct TPMInfo TPMInfo;
-
-typedef struct TPMInfoList
-{
-    union {
-        TPMInfo *value;
-        uint64_t padding;
-    };
-    struct TPMInfoList *next;
-} TPMInfoList;
-
-
-typedef struct AcpiTableOptions AcpiTableOptions;
-
-typedef struct AcpiTableOptionsList
-{
-    union {
-        AcpiTableOptions *value;
-        uint64_t padding;
-    };
-    struct AcpiTableOptionsList *next;
-} AcpiTableOptionsList;
-
-extern const char *CommandLineParameterType_lookup[];
-typedef enum CommandLineParameterType
-{
-    COMMAND_LINE_PARAMETER_TYPE_STRING = 0,
-    COMMAND_LINE_PARAMETER_TYPE_BOOLEAN = 1,
-    COMMAND_LINE_PARAMETER_TYPE_NUMBER = 2,
-    COMMAND_LINE_PARAMETER_TYPE_SIZE = 3,
-    COMMAND_LINE_PARAMETER_TYPE_MAX = 4,
-} CommandLineParameterType;
-
-typedef struct CommandLineParameterTypeList
-{
-    CommandLineParameterType value;
-    struct CommandLineParameterTypeList *next;
-} CommandLineParameterTypeList;
-
-
-typedef struct CommandLineParameterInfo CommandLineParameterInfo;
-
-typedef struct CommandLineParameterInfoList
-{
-    union {
-        CommandLineParameterInfo *value;
-        uint64_t padding;
-    };
-    struct CommandLineParameterInfoList *next;
-} CommandLineParameterInfoList;
-
-
-typedef struct CommandLineOptionInfo CommandLineOptionInfo;
-
-typedef struct CommandLineOptionInfoList
-{
-    union {
-        CommandLineOptionInfo *value;
-        uint64_t padding;
-    };
-    struct CommandLineOptionInfoList *next;
-} CommandLineOptionInfoList;
-
-extern const char *X86CPURegister32_lookup[];
-typedef enum X86CPURegister32
-{
-    X86_C_P_U_REGISTER32_EAX = 0,
-    X86_C_P_U_REGISTER32_EBX = 1,
-    X86_C_P_U_REGISTER32_ECX = 2,
-    X86_C_P_U_REGISTER32_EDX = 3,
-    X86_C_P_U_REGISTER32_ESP = 4,
-    X86_C_P_U_REGISTER32_EBP = 5,
-    X86_C_P_U_REGISTER32_ESI = 6,
-    X86_C_P_U_REGISTER32_EDI = 7,
-    X86_C_P_U_REGISTER32_MAX = 8,
-} X86CPURegister32;
-
-typedef struct X86CPURegister32List
-{
-    X86CPURegister32 value;
-    struct X86CPURegister32List *next;
-} X86CPURegister32List;
-
-
-typedef struct X86CPUFeatureWordInfo X86CPUFeatureWordInfo;
-
-typedef struct X86CPUFeatureWordInfoList
-{
-    union {
-        X86CPUFeatureWordInfo *value;
-        uint64_t padding;
-    };
-    struct X86CPUFeatureWordInfoList *next;
-} X86CPUFeatureWordInfoList;
-
-extern const char *RxState_lookup[];
-typedef enum RxState
-{
-    RX_STATE_NORMAL = 0,
-    RX_STATE_NONE = 1,
-    RX_STATE_ALL = 2,
-    RX_STATE_MAX = 3,
-} RxState;
-
-typedef struct RxStateList
-{
-    RxState value;
-    struct RxStateList *next;
-} RxStateList;
-
-
-typedef struct RxFilterInfo RxFilterInfo;
-
-typedef struct RxFilterInfoList
-{
-    union {
-        RxFilterInfo *value;
-        uint64_t padding;
-    };
-    struct RxFilterInfoList *next;
-} RxFilterInfoList;
-
-#ifndef QAPI_TYPES_BUILTIN_CLEANUP_DECL_H
-#define QAPI_TYPES_BUILTIN_CLEANUP_DECL_H
-
-void qapi_free_strList(strList * obj);
-void qapi_free_intList(intList * obj);
-void qapi_free_numberList(numberList * obj);
-void qapi_free_boolList(boolList * obj);
-void qapi_free_int8List(int8List * obj);
-void qapi_free_int16List(int16List * obj);
-void qapi_free_int32List(int32List * obj);
-void qapi_free_int64List(int64List * obj);
-void qapi_free_uint8List(uint8List * obj);
-void qapi_free_uint16List(uint16List * obj);
-void qapi_free_uint32List(uint32List * obj);
-void qapi_free_uint64List(uint64List * obj);
-
-#endif /* QAPI_TYPES_BUILTIN_CLEANUP_DECL_H */
-
-
-void qapi_free_ErrorClassList(ErrorClassList * obj);
-
-struct NameInfo
-{
-    bool has_name;
-    char * name;
-};
-
-void qapi_free_NameInfoList(NameInfoList * obj);
-void qapi_free_NameInfo(NameInfo * obj);
-
-struct VersionInfo
-{
-    struct 
-    {
-        int64_t major;
-        int64_t minor;
-        int64_t micro;
-    } qemu;
-    char * package;
-};
-
-void qapi_free_VersionInfoList(VersionInfoList * obj);
-void qapi_free_VersionInfo(VersionInfo * obj);
-
-struct KvmInfo
-{
-    bool enabled;
-    bool present;
-};
-
-void qapi_free_KvmInfoList(KvmInfoList * obj);
-void qapi_free_KvmInfo(KvmInfo * obj);
-
-void qapi_free_RunStateList(RunStateList * obj);
-
-struct SnapshotInfo
-{
-    char * id;
-    char * name;
-    int64_t vm_state_size;
-    int64_t date_sec;
-    int64_t date_nsec;
-    int64_t vm_clock_sec;
-    int64_t vm_clock_nsec;
-};
-
-void qapi_free_SnapshotInfoList(SnapshotInfoList * obj);
-void qapi_free_SnapshotInfo(SnapshotInfo * obj);
-
-struct ImageInfo
-{
-    char * filename;
-    char * format;
-    bool has_dirty_flag;
-    bool dirty_flag;
-    bool has_actual_size;
-    int64_t actual_size;
-    int64_t virtual_size;
-    bool has_cluster_size;
-    int64_t cluster_size;
-    bool has_encrypted;
-    bool encrypted;
-    bool has_backing_filename;
-    char * backing_filename;
-    bool has_full_backing_filename;
-    char * full_backing_filename;
-    bool has_backing_filename_format;
-    char * backing_filename_format;
-    bool has_snapshots;
-    SnapshotInfoList * snapshots;
-    bool has_backing_image;
-    ImageInfo * backing_image;
-};
-
-void qapi_free_ImageInfoList(ImageInfoList * obj);
-void qapi_free_ImageInfo(ImageInfo * obj);
-
-struct ImageCheck
-{
-    char * filename;
-    char * format;
-    int64_t check_errors;
-    bool has_image_end_offset;
-    int64_t image_end_offset;
-    bool has_corruptions;
-    int64_t corruptions;
-    bool has_leaks;
-    int64_t leaks;
-    bool has_corruptions_fixed;
-    int64_t corruptions_fixed;
-    bool has_leaks_fixed;
-    int64_t leaks_fixed;
-    bool has_total_clusters;
-    int64_t total_clusters;
-    bool has_allocated_clusters;
-    int64_t allocated_clusters;
-    bool has_fragmented_clusters;
-    int64_t fragmented_clusters;
-    bool has_compressed_clusters;
-    int64_t compressed_clusters;
-};
-
-void qapi_free_ImageCheckList(ImageCheckList * obj);
-void qapi_free_ImageCheck(ImageCheck * obj);
-
-struct StatusInfo
-{
-    bool running;
-    bool singlestep;
-    RunState status;
-};
-
-void qapi_free_StatusInfoList(StatusInfoList * obj);
-void qapi_free_StatusInfo(StatusInfo * obj);
-
-struct UuidInfo
-{
-    char * UUID;
-};
-
-void qapi_free_UuidInfoList(UuidInfoList * obj);
-void qapi_free_UuidInfo(UuidInfo * obj);
-
-struct ChardevInfo
-{
-    char * label;
-    char * filename;
-};
-
-void qapi_free_ChardevInfoList(ChardevInfoList * obj);
-void qapi_free_ChardevInfo(ChardevInfo * obj);
-
-void qapi_free_DataFormatList(DataFormatList * obj);
-
-struct CommandInfo
-{
-    char * name;
-};
-
-void qapi_free_CommandInfoList(CommandInfoList * obj);
-void qapi_free_CommandInfo(CommandInfo * obj);
-
-struct EventInfo
-{
-    char * name;
-};
-
-void qapi_free_EventInfoList(EventInfoList * obj);
-void qapi_free_EventInfo(EventInfo * obj);
-
-struct MigrationStats
-{
-    int64_t transferred;
-    int64_t remaining;
-    int64_t total;
-    int64_t duplicate;
-    int64_t skipped;
-    int64_t normal;
-    int64_t normal_bytes;
-    int64_t dirty_pages_rate;
-    double mbps;
-};
-
-void qapi_free_MigrationStatsList(MigrationStatsList * obj);
-void qapi_free_MigrationStats(MigrationStats * obj);
-
-struct XBZRLECacheStats
-{
-    int64_t cache_size;
-    int64_t bytes;
-    int64_t pages;
-    int64_t cache_miss;
-    int64_t overflow;
-};
-
-void qapi_free_XBZRLECacheStatsList(XBZRLECacheStatsList * obj);
-void qapi_free_XBZRLECacheStats(XBZRLECacheStats * obj);
-
-struct MigrationInfo
-{
-    bool has_status;
-    char * status;
-    bool has_ram;
-    MigrationStats * ram;
-    bool has_disk;
-    MigrationStats * disk;
-    bool has_xbzrle_cache;
-    XBZRLECacheStats * xbzrle_cache;
-    bool has_total_time;
-    int64_t total_time;
-    bool has_expected_downtime;
-    int64_t expected_downtime;
-    bool has_downtime;
-    int64_t downtime;
-};
-
-void qapi_free_MigrationInfoList(MigrationInfoList * obj);
-void qapi_free_MigrationInfo(MigrationInfo * obj);
-
-void qapi_free_MigrationCapabilityList(MigrationCapabilityList * obj);
-
-struct MigrationCapabilityStatus
-{
-    MigrationCapability capability;
-    bool state;
-};
-
-void qapi_free_MigrationCapabilityStatusList(MigrationCapabilityStatusList * obj);
-void qapi_free_MigrationCapabilityStatus(MigrationCapabilityStatus * obj);
-
-struct MouseInfo
-{
-    char * name;
-    int64_t index;
-    bool current;
-    bool absolute;
-};
-
-void qapi_free_MouseInfoList(MouseInfoList * obj);
-void qapi_free_MouseInfo(MouseInfo * obj);
-
-struct CpuInfo
-{
-    int64_t CPU;
-    bool current;
-    bool halted;
-    bool has_pc;
-    int64_t pc;
-    bool has_nip;
-    int64_t nip;
-    bool has_npc;
-    int64_t npc;
-    bool has_PC;
-    int64_t PC;
-    int64_t thread_id;
-};
-
-void qapi_free_CpuInfoList(CpuInfoList * obj);
-void qapi_free_CpuInfo(CpuInfo * obj);
-
-struct BlockDeviceInfo
-{
-    char * file;
-    bool ro;
-    char * drv;
-    bool has_backing_file;
-    char * backing_file;
-    int64_t backing_file_depth;
-    bool encrypted;
-    bool encryption_key_missing;
-    int64_t bps;
-    int64_t bps_rd;
-    int64_t bps_wr;
-    int64_t iops;
-    int64_t iops_rd;
-    int64_t iops_wr;
-    ImageInfo * image;
-};
-
-void qapi_free_BlockDeviceInfoList(BlockDeviceInfoList * obj);
-void qapi_free_BlockDeviceInfo(BlockDeviceInfo * obj);
-
-void qapi_free_BlockDeviceIoStatusList(BlockDeviceIoStatusList * obj);
-
-struct BlockDirtyInfo
-{
-    int64_t count;
-    int64_t granularity;
-};
-
-void qapi_free_BlockDirtyInfoList(BlockDirtyInfoList * obj);
-void qapi_free_BlockDirtyInfo(BlockDirtyInfo * obj);
-
-struct BlockInfo
-{
-    char * device;
-    char * type;
-    bool removable;
-    bool locked;
-    bool has_inserted;
-    BlockDeviceInfo * inserted;
-    bool has_tray_open;
-    bool tray_open;
-    bool has_io_status;
-    BlockDeviceIoStatus io_status;
-    bool has_dirty;
-    BlockDirtyInfo * dirty;
-};
-
-void qapi_free_BlockInfoList(BlockInfoList * obj);
-void qapi_free_BlockInfo(BlockInfo * obj);
-
-struct BlockDeviceStats
-{
-    int64_t rd_bytes;
-    int64_t wr_bytes;
-    int64_t rd_operations;
-    int64_t wr_operations;
-    int64_t flush_operations;
-    int64_t flush_total_time_ns;
-    int64_t wr_total_time_ns;
-    int64_t rd_total_time_ns;
-    int64_t wr_highest_offset;
-};
-
-void qapi_free_BlockDeviceStatsList(BlockDeviceStatsList * obj);
-void qapi_free_BlockDeviceStats(BlockDeviceStats * obj);
-
-struct BlockStats
-{
-    bool has_device;
-    char * device;
-    BlockDeviceStats * stats;
-    bool has_parent;
-    BlockStats * parent;
-};
-
-void qapi_free_BlockStatsList(BlockStatsList * obj);
-void qapi_free_BlockStats(BlockStats * obj);
-
-struct VncClientInfo
-{
-    char * host;
-    char * family;
-    char * service;
-    bool has_x509_dname;
-    char * x509_dname;
-    bool has_sasl_username;
-    char * sasl_username;
-};
-
-void qapi_free_VncClientInfoList(VncClientInfoList * obj);
-void qapi_free_VncClientInfo(VncClientInfo * obj);
-
-struct VncInfo
-{
-    bool enabled;
-    bool has_host;
-    char * host;
-    bool has_family;
-    char * family;
-    bool has_service;
-    char * service;
-    bool has_auth;
-    char * auth;
-    bool has_clients;
-    VncClientInfoList * clients;
-};
-
-void qapi_free_VncInfoList(VncInfoList * obj);
-void qapi_free_VncInfo(VncInfo * obj);
-
-struct SpiceChannel
-{
-    char * host;
-    char * family;
-    char * port;
-    int64_t connection_id;
-    int64_t channel_type;
-    int64_t channel_id;
-    bool tls;
-};
-
-void qapi_free_SpiceChannelList(SpiceChannelList * obj);
-void qapi_free_SpiceChannel(SpiceChannel * obj);
-
-void qapi_free_SpiceQueryMouseModeList(SpiceQueryMouseModeList * obj);
-
-struct SpiceInfo
-{
-    bool enabled;
-    bool migrated;
-    bool has_host;
-    char * host;
-    bool has_port;
-    int64_t port;
-    bool has_tls_port;
-    int64_t tls_port;
-    bool has_auth;
-    char * auth;
-    bool has_compiled_version;
-    char * compiled_version;
-    SpiceQueryMouseMode mouse_mode;
-    bool has_channels;
-    SpiceChannelList * channels;
-};
-
-void qapi_free_SpiceInfoList(SpiceInfoList * obj);
-void qapi_free_SpiceInfo(SpiceInfo * obj);
-
-struct BalloonInfo
-{
-    int64_t actual;
-};
-
-void qapi_free_BalloonInfoList(BalloonInfoList * obj);
-void qapi_free_BalloonInfo(BalloonInfo * obj);
-
-struct PciMemoryRange
-{
-    int64_t base;
-    int64_t limit;
-};
-
-void qapi_free_PciMemoryRangeList(PciMemoryRangeList * obj);
-void qapi_free_PciMemoryRange(PciMemoryRange * obj);
-
-struct PciMemoryRegion
-{
-    int64_t bar;
-    char * type;
-    int64_t address;
-    int64_t size;
-    bool has_prefetch;
-    bool prefetch;
-    bool has_mem_type_64;
-    bool mem_type_64;
-};
-
-void qapi_free_PciMemoryRegionList(PciMemoryRegionList * obj);
-void qapi_free_PciMemoryRegion(PciMemoryRegion * obj);
-
-struct PciBridgeInfo
-{
-    struct 
-    {
-        int64_t number;
-        int64_t secondary;
-        int64_t subordinate;
-        PciMemoryRange * io_range;
-        PciMemoryRange * memory_range;
-        PciMemoryRange * prefetchable_range;
-    } bus;
-    bool has_devices;
-    PciDeviceInfoList * devices;
-};
-
-void qapi_free_PciBridgeInfoList(PciBridgeInfoList * obj);
-void qapi_free_PciBridgeInfo(PciBridgeInfo * obj);
-
-struct PciDeviceInfo
-{
-    int64_t bus;
-    int64_t slot;
-    int64_t function;
-    struct 
-    {
-        bool has_desc;
-        char * desc;
-        int64_t class;
-    } class_info;
-    struct 
-    {
-        int64_t device;
-        int64_t vendor;
-    } id;
-    bool has_irq;
-    int64_t irq;
-    char * qdev_id;
-    bool has_pci_bridge;
-    PciBridgeInfo * pci_bridge;
-    PciMemoryRegionList * regions;
-};
-
-void qapi_free_PciDeviceInfoList(PciDeviceInfoList * obj);
-void qapi_free_PciDeviceInfo(PciDeviceInfo * obj);
-
-struct PciInfo
-{
-    int64_t bus;
-    PciDeviceInfoList * devices;
-};
-
-void qapi_free_PciInfoList(PciInfoList * obj);
-void qapi_free_PciInfo(PciInfo * obj);
-
-void qapi_free_BlockdevOnErrorList(BlockdevOnErrorList * obj);
-
-void qapi_free_MirrorSyncModeList(MirrorSyncModeList * obj);
-
-struct BlockJobInfo
-{
-    char * type;
-    char * device;
-    int64_t len;
-    int64_t offset;
-    bool busy;
-    bool paused;
-    int64_t speed;
-    BlockDeviceIoStatus io_status;
-};
-
-void qapi_free_BlockJobInfoList(BlockJobInfoList * obj);
-void qapi_free_BlockJobInfo(BlockJobInfo * obj);
-
-void qapi_free_NewImageModeList(NewImageModeList * obj);
-
-struct BlockdevSnapshot
-{
-    char * device;
-    char * snapshot_file;
-    bool has_format;
-    char * format;
-    bool has_mode;
-    NewImageMode mode;
-};
-
-void qapi_free_BlockdevSnapshotList(BlockdevSnapshotList * obj);
-void qapi_free_BlockdevSnapshot(BlockdevSnapshot * obj);
-
-struct DriveBackup
-{
-    char * device;
-    char * target;
-    bool has_format;
-    char * format;
-    MirrorSyncMode sync;
-    bool has_mode;
-    NewImageMode mode;
-    bool has_speed;
-    int64_t speed;
-    bool has_on_source_error;
-    BlockdevOnError on_source_error;
-    bool has_on_target_error;
-    BlockdevOnError on_target_error;
-};
-
-void qapi_free_DriveBackupList(DriveBackupList * obj);
-void qapi_free_DriveBackup(DriveBackup * obj);
-
-struct Abort
-{
-};
-
-void qapi_free_AbortList(AbortList * obj);
-void qapi_free_Abort(Abort * obj);
-
-struct TransactionAction
-{
-    TransactionActionKind kind;
-    union {
-        void *data;
-        BlockdevSnapshot * blockdev_snapshot_sync;
-        DriveBackup * drive_backup;
-        Abort * abort;
-    };
-};
-void qapi_free_TransactionActionList(TransactionActionList * obj);
-void qapi_free_TransactionAction(TransactionAction * obj);
-
-struct ObjectPropertyInfo
-{
-    char * name;
-    char * type;
-};
-
-void qapi_free_ObjectPropertyInfoList(ObjectPropertyInfoList * obj);
-void qapi_free_ObjectPropertyInfo(ObjectPropertyInfo * obj);
-
-struct ObjectTypeInfo
-{
-    char * name;
-};
-
-void qapi_free_ObjectTypeInfoList(ObjectTypeInfoList * obj);
-void qapi_free_ObjectTypeInfo(ObjectTypeInfo * obj);
-
-struct DevicePropertyInfo
-{
-    char * name;
-    char * type;
-};
-
-void qapi_free_DevicePropertyInfoList(DevicePropertyInfoList * obj);
-void qapi_free_DevicePropertyInfo(DevicePropertyInfo * obj);
-
-struct NetdevNoneOptions
-{
-};
-
-void qapi_free_NetdevNoneOptionsList(NetdevNoneOptionsList * obj);
-void qapi_free_NetdevNoneOptions(NetdevNoneOptions * obj);
-
-struct NetLegacyNicOptions
-{
-    bool has_netdev;
-    char * netdev;
-    bool has_macaddr;
-    char * macaddr;
-    bool has_model;
-    char * model;
-    bool has_addr;
-    char * addr;
-    bool has_vectors;
-    uint32_t vectors;
-};
-
-void qapi_free_NetLegacyNicOptionsList(NetLegacyNicOptionsList * obj);
-void qapi_free_NetLegacyNicOptions(NetLegacyNicOptions * obj);
-
-struct String
-{
-    char * str;
-};
-
-void qapi_free_StringList(StringList * obj);
-void qapi_free_String(String * obj);
-
-struct NetdevUserOptions
-{
-    bool has_hostname;
-    char * hostname;
-    bool has_q_restrict;
-    bool q_restrict;
-    bool has_ip;
-    char * ip;
-    bool has_net;
-    char * net;
-    bool has_host;
-    char * host;
-    bool has_tftp;
-    char * tftp;
-    bool has_bootfile;
-    char * bootfile;
-    bool has_dhcpstart;
-    char * dhcpstart;
-    bool has_dns;
-    char * dns;
-    bool has_dnssearch;
-    StringList * dnssearch;
-    bool has_smb;
-    char * smb;
-    bool has_smbserver;
-    char * smbserver;
-    bool has_hostfwd;
-    StringList * hostfwd;
-    bool has_guestfwd;
-    StringList * guestfwd;
-};
-
-void qapi_free_NetdevUserOptionsList(NetdevUserOptionsList * obj);
-void qapi_free_NetdevUserOptions(NetdevUserOptions * obj);
-
-struct NetdevTapOptions
-{
-    bool has_ifname;
-    char * ifname;
-    bool has_fd;
-    char * fd;
-    bool has_fds;
-    char * fds;
-    bool has_script;
-    char * script;
-    bool has_downscript;
-    char * downscript;
-    bool has_helper;
-    char * helper;
-    bool has_sndbuf;
-    uint64_t sndbuf;
-    bool has_vnet_hdr;
-    bool vnet_hdr;
-    bool has_vhost;
-    bool vhost;
-    bool has_vhostfd;
-    char * vhostfd;
-    bool has_vhostfds;
-    char * vhostfds;
-    bool has_vhostforce;
-    bool vhostforce;
-    bool has_queues;
-    uint32_t queues;
-};
-
-void qapi_free_NetdevTapOptionsList(NetdevTapOptionsList * obj);
-void qapi_free_NetdevTapOptions(NetdevTapOptions * obj);
-
-struct NetdevSocketOptions
-{
-    bool has_fd;
-    char * fd;
-    bool has_listen;
-    char * listen;
-    bool has_connect;
-    char * connect;
-    bool has_mcast;
-    char * mcast;
-    bool has_localaddr;
-    char * localaddr;
-    bool has_udp;
-    char * udp;
-};
-
-void qapi_free_NetdevSocketOptionsList(NetdevSocketOptionsList * obj);
-void qapi_free_NetdevSocketOptions(NetdevSocketOptions * obj);
-
-struct NetdevVdeOptions
-{
-    bool has_sock;
-    char * sock;
-    bool has_port;
-    uint16_t port;
-    bool has_group;
-    char * group;
-    bool has_mode;
-    uint16_t mode;
-};
-
-void qapi_free_NetdevVdeOptionsList(NetdevVdeOptionsList * obj);
-void qapi_free_NetdevVdeOptions(NetdevVdeOptions * obj);
-
-struct NetdevDumpOptions
-{
-    bool has_len;
-    uint64_t len;
-    bool has_file;
-    char * file;
-};
-
-void qapi_free_NetdevDumpOptionsList(NetdevDumpOptionsList * obj);
-void qapi_free_NetdevDumpOptions(NetdevDumpOptions * obj);
-
-struct NetdevBridgeOptions
-{
-    bool has_br;
-    char * br;
-    bool has_helper;
-    char * helper;
-};
-
-void qapi_free_NetdevBridgeOptionsList(NetdevBridgeOptionsList * obj);
-void qapi_free_NetdevBridgeOptions(NetdevBridgeOptions * obj);
-
-struct NetdevHubPortOptions
-{
-    int32_t hubid;
-};
-
-void qapi_free_NetdevHubPortOptionsList(NetdevHubPortOptionsList * obj);
-void qapi_free_NetdevHubPortOptions(NetdevHubPortOptions * obj);
-
-struct NetClientOptions
-{
-    NetClientOptionsKind kind;
-    union {
-        void *data;
-        NetdevNoneOptions * none;
-        NetLegacyNicOptions * nic;
-        NetdevUserOptions * user;
-        NetdevTapOptions * tap;
-        NetdevSocketOptions * socket;
-        NetdevVdeOptions * vde;
-        NetdevDumpOptions * dump;
-        NetdevBridgeOptions * bridge;
-        NetdevHubPortOptions * hubport;
-    };
-};
-void qapi_free_NetClientOptionsList(NetClientOptionsList * obj);
-void qapi_free_NetClientOptions(NetClientOptions * obj);
-
-struct NetLegacy
-{
-    bool has_vlan;
-    int32_t vlan;
-    bool has_id;
-    char * id;
-    bool has_name;
-    char * name;
-    NetClientOptions * opts;
-};
-
-void qapi_free_NetLegacyList(NetLegacyList * obj);
-void qapi_free_NetLegacy(NetLegacy * obj);
-
-struct Netdev
-{
-    char * id;
-    NetClientOptions * opts;
-};
-
-void qapi_free_NetdevList(NetdevList * obj);
-void qapi_free_Netdev(Netdev * obj);
-
-struct InetSocketAddress
-{
-    char * host;
-    char * port;
-    bool has_to;
-    uint16_t to;
-    bool has_ipv4;
-    bool ipv4;
-    bool has_ipv6;
-    bool ipv6;
-};
-
-void qapi_free_InetSocketAddressList(InetSocketAddressList * obj);
-void qapi_free_InetSocketAddress(InetSocketAddress * obj);
-
-struct UnixSocketAddress
-{
-    char * path;
-};
-
-void qapi_free_UnixSocketAddressList(UnixSocketAddressList * obj);
-void qapi_free_UnixSocketAddress(UnixSocketAddress * obj);
-
-struct SocketAddress
-{
-    SocketAddressKind kind;
-    union {
-        void *data;
-        InetSocketAddress * inet;
-        UnixSocketAddress * q_unix;
-        String * fd;
-    };
-};
-void qapi_free_SocketAddressList(SocketAddressList * obj);
-void qapi_free_SocketAddress(SocketAddress * obj);
-
-struct MachineInfo
-{
-    char * name;
-    bool has_alias;
-    char * alias;
-    bool has_is_default;
-    bool is_default;
-    int64_t cpu_max;
-};
-
-void qapi_free_MachineInfoList(MachineInfoList * obj);
-void qapi_free_MachineInfo(MachineInfo * obj);
-
-struct CpuDefinitionInfo
-{
-    char * name;
-};
-
-void qapi_free_CpuDefinitionInfoList(CpuDefinitionInfoList * obj);
-void qapi_free_CpuDefinitionInfo(CpuDefinitionInfo * obj);
-
-struct AddfdInfo
-{
-    int64_t fdset_id;
-    int64_t fd;
-};
-
-void qapi_free_AddfdInfoList(AddfdInfoList * obj);
-void qapi_free_AddfdInfo(AddfdInfo * obj);
-
-struct FdsetFdInfo
-{
-    int64_t fd;
-    bool has_opaque;
-    char * opaque;
-};
-
-void qapi_free_FdsetFdInfoList(FdsetFdInfoList * obj);
-void qapi_free_FdsetFdInfo(FdsetFdInfo * obj);
-
-struct FdsetInfo
-{
-    int64_t fdset_id;
-    FdsetFdInfoList * fds;
-};
-
-void qapi_free_FdsetInfoList(FdsetInfoList * obj);
-void qapi_free_FdsetInfo(FdsetInfo * obj);
-
-struct TargetInfo
-{
-    char * arch;
-};
-
-void qapi_free_TargetInfoList(TargetInfoList * obj);
-void qapi_free_TargetInfo(TargetInfo * obj);
-
-void qapi_free_QKeyCodeList(QKeyCodeList * obj);
-
-struct KeyValue
-{
-    KeyValueKind kind;
-    union {
-        void *data;
-        int64_t number;
-        QKeyCode qcode;
-    };
-};
-void qapi_free_KeyValueList(KeyValueList * obj);
-void qapi_free_KeyValue(KeyValue * obj);
-
-struct ChardevFile
-{
-    bool has_in;
-    char * in;
-    char * out;
-};
-
-void qapi_free_ChardevFileList(ChardevFileList * obj);
-void qapi_free_ChardevFile(ChardevFile * obj);
-
-struct ChardevHostdev
-{
-    char * device;
-};
-
-void qapi_free_ChardevHostdevList(ChardevHostdevList * obj);
-void qapi_free_ChardevHostdev(ChardevHostdev * obj);
-
-struct ChardevSocket
-{
-    SocketAddress * addr;
-    bool has_server;
-    bool server;
-    bool has_wait;
-    bool wait;
-    bool has_nodelay;
-    bool nodelay;
-    bool has_telnet;
-    bool telnet;
-};
-
-void qapi_free_ChardevSocketList(ChardevSocketList * obj);
-void qapi_free_ChardevSocket(ChardevSocket * obj);
-
-struct ChardevUdp
-{
-    SocketAddress * remote;
-    bool has_local;
-    SocketAddress * local;
-};
-
-void qapi_free_ChardevUdpList(ChardevUdpList * obj);
-void qapi_free_ChardevUdp(ChardevUdp * obj);
-
-struct ChardevMux
-{
-    char * chardev;
-};
-
-void qapi_free_ChardevMuxList(ChardevMuxList * obj);
-void qapi_free_ChardevMux(ChardevMux * obj);
-
-struct ChardevStdio
-{
-    bool has_signal;
-    bool signal;
-};
-
-void qapi_free_ChardevStdioList(ChardevStdioList * obj);
-void qapi_free_ChardevStdio(ChardevStdio * obj);
-
-struct ChardevSpiceChannel
-{
-    char * type;
-};
-
-void qapi_free_ChardevSpiceChannelList(ChardevSpiceChannelList * obj);
-void qapi_free_ChardevSpiceChannel(ChardevSpiceChannel * obj);
-
-struct ChardevSpicePort
-{
-    char * fqdn;
-};
-
-void qapi_free_ChardevSpicePortList(ChardevSpicePortList * obj);
-void qapi_free_ChardevSpicePort(ChardevSpicePort * obj);
-
-struct ChardevVC
-{
-    bool has_width;
-    int64_t width;
-    bool has_height;
-    int64_t height;
-    bool has_cols;
-    int64_t cols;
-    bool has_rows;
-    int64_t rows;
-};
-
-void qapi_free_ChardevVCList(ChardevVCList * obj);
-void qapi_free_ChardevVC(ChardevVC * obj);
-
-struct ChardevMemory
-{
-    bool has_size;
-    int64_t size;
-};
-
-void qapi_free_ChardevMemoryList(ChardevMemoryList * obj);
-void qapi_free_ChardevMemory(ChardevMemory * obj);
-
-struct ChardevDummy
-{
-};
-
-void qapi_free_ChardevDummyList(ChardevDummyList * obj);
-void qapi_free_ChardevDummy(ChardevDummy * obj);
-
-struct ChardevBackend
-{
-    ChardevBackendKind kind;
-    union {
-        void *data;
-        ChardevFile * file;
-        ChardevHostdev * serial;
-        ChardevHostdev * parallel;
-        ChardevHostdev * pipe;
-        ChardevSocket * socket;
-        ChardevUdp * udp;
-        ChardevDummy * pty;
-        ChardevDummy * null;
-        ChardevMux * mux;
-        ChardevDummy * msmouse;
-        ChardevDummy * braille;
-        ChardevStdio * stdio;
-        ChardevDummy * console;
-        ChardevSpiceChannel * spicevmc;
-        ChardevSpicePort * spiceport;
-        ChardevVC * vc;
-        ChardevMemory * memory;
-    };
-};
-void qapi_free_ChardevBackendList(ChardevBackendList * obj);
-void qapi_free_ChardevBackend(ChardevBackend * obj);
-
-struct ChardevReturn
-{
-    bool has_pty;
-    char * pty;
-};
-
-void qapi_free_ChardevReturnList(ChardevReturnList * obj);
-void qapi_free_ChardevReturn(ChardevReturn * obj);
-
-void qapi_free_TpmModelList(TpmModelList * obj);
-
-void qapi_free_TpmTypeList(TpmTypeList * obj);
-
-struct TPMPassthroughOptions
-{
-    bool has_path;
-    char * path;
-    bool has_cancel_path;
-    char * cancel_path;
-};
-
-void qapi_free_TPMPassthroughOptionsList(TPMPassthroughOptionsList * obj);
-void qapi_free_TPMPassthroughOptions(TPMPassthroughOptions * obj);
-
-struct TpmTypeOptions
-{
-    TpmTypeOptionsKind kind;
-    union {
-        void *data;
-        TPMPassthroughOptions * passthrough;
-    };
-};
-void qapi_free_TpmTypeOptionsList(TpmTypeOptionsList * obj);
-void qapi_free_TpmTypeOptions(TpmTypeOptions * obj);
-
-struct TPMInfo
-{
-    char * id;
-    TpmModel model;
-    TpmTypeOptions * options;
-};
-
-void qapi_free_TPMInfoList(TPMInfoList * obj);
-void qapi_free_TPMInfo(TPMInfo * obj);
-
-struct AcpiTableOptions
-{
-    bool has_sig;
-    char * sig;
-    bool has_rev;
-    uint8_t rev;
-    bool has_oem_id;
-    char * oem_id;
-    bool has_oem_table_id;
-    char * oem_table_id;
-    bool has_oem_rev;
-    uint32_t oem_rev;
-    bool has_asl_compiler_id;
-    char * asl_compiler_id;
-    bool has_asl_compiler_rev;
-    uint32_t asl_compiler_rev;
-    bool has_file;
-    char * file;
-    bool has_data;
-    char * data;
-};
-
-void qapi_free_AcpiTableOptionsList(AcpiTableOptionsList * obj);
-void qapi_free_AcpiTableOptions(AcpiTableOptions * obj);
-
-void qapi_free_CommandLineParameterTypeList(CommandLineParameterTypeList * obj);
-
-struct CommandLineParameterInfo
-{
-    char * name;
-    CommandLineParameterType type;
-    bool has_help;
-    char * help;
-};
-
-void qapi_free_CommandLineParameterInfoList(CommandLineParameterInfoList * obj);
-void qapi_free_CommandLineParameterInfo(CommandLineParameterInfo * obj);
-
-struct CommandLineOptionInfo
-{
-    char * option;
-    CommandLineParameterInfoList * parameters;
-};
-
-void qapi_free_CommandLineOptionInfoList(CommandLineOptionInfoList * obj);
-void qapi_free_CommandLineOptionInfo(CommandLineOptionInfo * obj);
-
-void qapi_free_X86CPURegister32List(X86CPURegister32List * obj);
-
-struct X86CPUFeatureWordInfo
-{
-    int64_t cpuid_input_eax;
-    bool has_cpuid_input_ecx;
-    int64_t cpuid_input_ecx;
-    X86CPURegister32 cpuid_register;
-    int64_t features;
-};
-
-void qapi_free_X86CPUFeatureWordInfoList(X86CPUFeatureWordInfoList * obj);
-void qapi_free_X86CPUFeatureWordInfo(X86CPUFeatureWordInfo * obj);
-
-void qapi_free_RxStateList(RxStateList * obj);
-
-struct RxFilterInfo
-{
-    char * name;
-    bool promiscuous;
-    RxState multicast;
-    RxState unicast;
-    bool broadcast_allowed;
-    bool multicast_overflow;
-    bool unicast_overflow;
-    char * main_mac;
-    intList * vlan_table;
-    strList * unicast_table;
-    strList * multicast_table;
-};
-
-void qapi_free_RxFilterInfoList(RxFilterInfoList * obj);
-void qapi_free_RxFilterInfo(RxFilterInfo * obj);
-
-#endif
diff --git a/contrib/qemu/qemu-coroutine-lock.c b/contrib/qemu/qemu-coroutine-lock.c
deleted file mode 100644
index d9fea4989d4..00000000000
--- a/contrib/qemu/qemu-coroutine-lock.c
+++ /dev/null
@@ -1,178 +0,0 @@
-/*
- * coroutine queues and locks
- *
- * Copyright (c) 2011 Kevin Wolf <kwolf@redhat.com>
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-
-#include "qemu-common.h"
-#include "block/coroutine.h"
-#include "block/coroutine_int.h"
-#include "qemu/queue.h"
-#include "trace.h"
-
-void qemu_co_queue_init(CoQueue *queue)
-{
-    QTAILQ_INIT(&queue->entries);
-}
-
-void coroutine_fn qemu_co_queue_wait(CoQueue *queue)
-{
-    Coroutine *self = qemu_coroutine_self();
-    QTAILQ_INSERT_TAIL(&queue->entries, self, co_queue_next);
-    qemu_coroutine_yield();
-    assert(qemu_in_coroutine());
-}
-
-void coroutine_fn qemu_co_queue_wait_insert_head(CoQueue *queue)
-{
-    Coroutine *self = qemu_coroutine_self();
-    QTAILQ_INSERT_HEAD(&queue->entries, self, co_queue_next);
-    qemu_coroutine_yield();
-    assert(qemu_in_coroutine());
-}
-
-/**
- * qemu_co_queue_run_restart:
- *
- * Enter each coroutine that was previously marked for restart by
- * qemu_co_queue_next() or qemu_co_queue_restart_all().  This function is
- * invoked by the core coroutine code when the current coroutine yields or
- * terminates.
- */
-void qemu_co_queue_run_restart(Coroutine *co)
-{
-    Coroutine *next;
-
-    trace_qemu_co_queue_run_restart(co);
-    while ((next = QTAILQ_FIRST(&co->co_queue_wakeup))) {
-        QTAILQ_REMOVE(&co->co_queue_wakeup, next, co_queue_next);
-        qemu_coroutine_enter(next, NULL);
-    }
-}
-
-static bool qemu_co_queue_do_restart(CoQueue *queue, bool single)
-{
-    Coroutine *self = qemu_coroutine_self();
-    Coroutine *next;
-
-    if (QTAILQ_EMPTY(&queue->entries)) {
-        return false;
-    }
-
-    while ((next = QTAILQ_FIRST(&queue->entries)) != NULL) {
-        QTAILQ_REMOVE(&queue->entries, next, co_queue_next);
-        QTAILQ_INSERT_TAIL(&self->co_queue_wakeup, next, co_queue_next);
-        trace_qemu_co_queue_next(next);
-        if (single) {
-            break;
-        }
-    }
-    return true;
-}
-
-bool qemu_co_queue_next(CoQueue *queue)
-{
-    return qemu_co_queue_do_restart(queue, true);
-}
-
-void qemu_co_queue_restart_all(CoQueue *queue)
-{
-    qemu_co_queue_do_restart(queue, false);
-}
-
-bool qemu_co_queue_empty(CoQueue *queue)
-{
-    return (QTAILQ_FIRST(&queue->entries) == NULL);
-}
-
-void qemu_co_mutex_init(CoMutex *mutex)
-{
-    memset(mutex, 0, sizeof(*mutex));
-    qemu_co_queue_init(&mutex->queue);
-}
-
-void coroutine_fn qemu_co_mutex_lock(CoMutex *mutex)
-{
-    Coroutine *self = qemu_coroutine_self();
-
-    trace_qemu_co_mutex_lock_entry(mutex, self);
-
-    while (mutex->locked) {
-        qemu_co_queue_wait(&mutex->queue);
-    }
-
-    mutex->locked = true;
-
-    trace_qemu_co_mutex_lock_return(mutex, self);
-}
-
-void coroutine_fn qemu_co_mutex_unlock(CoMutex *mutex)
-{
-    Coroutine *self = qemu_coroutine_self();
-
-    trace_qemu_co_mutex_unlock_entry(mutex, self);
-
-    assert(mutex->locked == true);
-    assert(qemu_in_coroutine());
-
-    mutex->locked = false;
-    qemu_co_queue_next(&mutex->queue);
-
-    trace_qemu_co_mutex_unlock_return(mutex, self);
-}
-
-void qemu_co_rwlock_init(CoRwlock *lock)
-{
-    memset(lock, 0, sizeof(*lock));
-    qemu_co_queue_init(&lock->queue);
-}
-
-void qemu_co_rwlock_rdlock(CoRwlock *lock)
-{
-    while (lock->writer) {
-        qemu_co_queue_wait(&lock->queue);
-    }
-    lock->reader++;
-}
-
-void qemu_co_rwlock_unlock(CoRwlock *lock)
-{
-    assert(qemu_in_coroutine());
-    if (lock->writer) {
-        lock->writer = false;
-        qemu_co_queue_restart_all(&lock->queue);
-    } else {
-        lock->reader--;
-        assert(lock->reader >= 0);
-        /* Wakeup only one waiting writer */
-        if (!lock->reader) {
-            qemu_co_queue_next(&lock->queue);
-        }
-    }
-}
-
-void qemu_co_rwlock_wrlock(CoRwlock *lock)
-{
-    while (lock->writer || lock->reader) {
-        qemu_co_queue_wait(&lock->queue);
-    }
-    lock->writer = true;
-}
diff --git a/contrib/qemu/qemu-coroutine-sleep.c b/contrib/qemu/qemu-coroutine-sleep.c
deleted file mode 100644
index 169ce5ccc92..00000000000
--- a/contrib/qemu/qemu-coroutine-sleep.c
+++ /dev/null
@@ -1,39 +0,0 @@
-/*
- * QEMU coroutine sleep
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Stefan Hajnoczi    <stefanha@linux.vnet.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#include "block/coroutine.h"
-#include "qemu/timer.h"
-
-typedef struct CoSleepCB {
-    QEMUTimer *ts;
-    Coroutine *co;
-} CoSleepCB;
-
-static void co_sleep_cb(void *opaque)
-{
-    CoSleepCB *sleep_cb = opaque;
-
-    qemu_coroutine_enter(sleep_cb->co, NULL);
-}
-
-void coroutine_fn co_sleep_ns(QEMUClock *clock, int64_t ns)
-{
-    CoSleepCB sleep_cb = {
-        .co = qemu_coroutine_self(),
-    };
-    sleep_cb.ts = qemu_new_timer(clock, SCALE_NS, co_sleep_cb, &sleep_cb);
-    qemu_mod_timer(sleep_cb.ts, qemu_get_clock_ns(clock) + ns);
-    qemu_coroutine_yield();
-    qemu_del_timer(sleep_cb.ts);
-    qemu_free_timer(sleep_cb.ts);
-}
diff --git a/contrib/qemu/qemu-coroutine.c b/contrib/qemu/qemu-coroutine.c
deleted file mode 100644
index 5e19307eeda..00000000000
--- a/contrib/qemu/qemu-coroutine.c
+++ /dev/null
@@ -1,135 +0,0 @@
-/*
- * QEMU coroutines
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Stefan Hajnoczi    <stefanha@linux.vnet.ibm.com>
- *  Kevin Wolf         <kwolf@redhat.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#include "trace.h"
-#include "qemu-common.h"
-#include "qemu/thread.h"
-#include "block/coroutine.h"
-#include "block/coroutine_int.h"
-
-enum {
-    /* Maximum free pool size prevents holding too many freed coroutines */
-    POOL_MAX_SIZE = 0,
-};
-
-/** Free list to speed up creation */
-static QemuMutex pool_lock;
-static QSLIST_HEAD(, Coroutine) pool = QSLIST_HEAD_INITIALIZER(pool);
-static unsigned int pool_size;
-
-Coroutine *qemu_coroutine_create(CoroutineEntry *entry)
-{
-    Coroutine *co;
-
-    qemu_mutex_lock(&pool_lock);
-    co = QSLIST_FIRST(&pool);
-    if (co) {
-        QSLIST_REMOVE_HEAD(&pool, pool_next);
-        pool_size--;
-    }
-    qemu_mutex_unlock(&pool_lock);
-
-    if (!co) {
-        co = qemu_coroutine_new();
-    }
-
-    co->entry = entry;
-    QTAILQ_INIT(&co->co_queue_wakeup);
-    return co;
-}
-
-static void coroutine_delete(Coroutine *co)
-{
-    qemu_mutex_lock(&pool_lock);
-    if (pool_size < POOL_MAX_SIZE) {
-        QSLIST_INSERT_HEAD(&pool, co, pool_next);
-        co->caller = NULL;
-        pool_size++;
-        qemu_mutex_unlock(&pool_lock);
-        return;
-    }
-    qemu_mutex_unlock(&pool_lock);
-
-    qemu_coroutine_delete(co);
-}
-
-static void __attribute__((constructor)) coroutine_pool_init(void)
-{
-    qemu_mutex_init(&pool_lock);
-}
-
-static void __attribute__((destructor)) coroutine_pool_cleanup(void)
-{
-    Coroutine *co;
-    Coroutine *tmp;
-
-    QSLIST_FOREACH_SAFE(co, &pool, pool_next, tmp) {
-        QSLIST_REMOVE_HEAD(&pool, pool_next);
-        qemu_coroutine_delete(co);
-    }
-
-    qemu_mutex_destroy(&pool_lock);
-}
-
-static void coroutine_swap(Coroutine *from, Coroutine *to)
-{
-    CoroutineAction ret;
-
-    ret = qemu_coroutine_switch(from, to, COROUTINE_YIELD);
-
-    qemu_co_queue_run_restart(to);
-
-    switch (ret) {
-    case COROUTINE_YIELD:
-        return;
-    case COROUTINE_TERMINATE:
-        trace_qemu_coroutine_terminate(to);
-        coroutine_delete(to);
-        return;
-    default:
-        abort();
-    }
-}
-
-void qemu_coroutine_enter(Coroutine *co, void *opaque)
-{
-    Coroutine *self = qemu_coroutine_self();
-
-    trace_qemu_coroutine_enter(self, co, opaque);
-
-    if (co->caller) {
-        fprintf(stderr, "Co-routine re-entered recursively\n");
-        abort();
-    }
-
-    co->caller = self;
-    co->entry_arg = opaque;
-    coroutine_swap(self, co);
-}
-
-void coroutine_fn qemu_coroutine_yield(void)
-{
-    Coroutine *self = qemu_coroutine_self();
-    Coroutine *to = self->caller;
-
-    trace_qemu_coroutine_yield(self, to);
-
-    if (!to) {
-        fprintf(stderr, "Co-routine is yielding to no one\n");
-        abort();
-    }
-
-    self->caller = NULL;
-    coroutine_swap(self, to);
-}
diff --git a/contrib/qemu/qmp-commands.h b/contrib/qemu/qmp-commands.h
deleted file mode 100644
index fcc0ff0f7f0..00000000000
--- a/contrib/qemu/qmp-commands.h
+++ /dev/null
@@ -1,204 +0,0 @@
-/* THIS FILE IS AUTOMATICALLY GENERATED, DO NOT MODIFY */
-
-/*
- * schema-defined QAPI function prototypes
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#ifndef QMP_COMMANDS_H
-#define QMP_COMMANDS_H
-
-#include "qapi-types.h"
-#include "qapi/qmp/qdict.h"
-#include "qapi/error.h"
-
-void qmp_add_client(const char * protocol, const char * fdname, bool has_skipauth, bool skipauth, bool has_tls, bool tls, Error **errp);
-int qmp_marshal_input_add_client(Monitor *mon, const QDict *qdict, QObject **ret);
-NameInfo * qmp_query_name(Error **errp);
-int qmp_marshal_input_query_name(Monitor *mon, const QDict *qdict, QObject **ret);
-VersionInfo * qmp_query_version(Error **errp);
-int qmp_marshal_input_query_version(Monitor *mon, const QDict *qdict, QObject **ret);
-KvmInfo * qmp_query_kvm(Error **errp);
-int qmp_marshal_input_query_kvm(Monitor *mon, const QDict *qdict, QObject **ret);
-StatusInfo * qmp_query_status(Error **errp);
-int qmp_marshal_input_query_status(Monitor *mon, const QDict *qdict, QObject **ret);
-UuidInfo * qmp_query_uuid(Error **errp);
-int qmp_marshal_input_query_uuid(Monitor *mon, const QDict *qdict, QObject **ret);
-ChardevInfoList * qmp_query_chardev(Error **errp);
-int qmp_marshal_input_query_chardev(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_ringbuf_write(const char * device, const char * data, bool has_format, DataFormat format, Error **errp);
-int qmp_marshal_input_ringbuf_write(Monitor *mon, const QDict *qdict, QObject **ret);
-char * qmp_ringbuf_read(const char * device, int64_t size, bool has_format, DataFormat format, Error **errp);
-int qmp_marshal_input_ringbuf_read(Monitor *mon, const QDict *qdict, QObject **ret);
-CommandInfoList * qmp_query_commands(Error **errp);
-int qmp_marshal_input_query_commands(Monitor *mon, const QDict *qdict, QObject **ret);
-EventInfoList * qmp_query_events(Error **errp);
-int qmp_marshal_input_query_events(Monitor *mon, const QDict *qdict, QObject **ret);
-MigrationInfo * qmp_query_migrate(Error **errp);
-int qmp_marshal_input_query_migrate(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_migrate_set_capabilities(MigrationCapabilityStatusList * capabilities, Error **errp);
-int qmp_marshal_input_migrate_set_capabilities(Monitor *mon, const QDict *qdict, QObject **ret);
-MigrationCapabilityStatusList * qmp_query_migrate_capabilities(Error **errp);
-int qmp_marshal_input_query_migrate_capabilities(Monitor *mon, const QDict *qdict, QObject **ret);
-MouseInfoList * qmp_query_mice(Error **errp);
-int qmp_marshal_input_query_mice(Monitor *mon, const QDict *qdict, QObject **ret);
-CpuInfoList * qmp_query_cpus(Error **errp);
-int qmp_marshal_input_query_cpus(Monitor *mon, const QDict *qdict, QObject **ret);
-BlockInfoList * qmp_query_block(Error **errp);
-int qmp_marshal_input_query_block(Monitor *mon, const QDict *qdict, QObject **ret);
-BlockStatsList * qmp_query_blockstats(Error **errp);
-int qmp_marshal_input_query_blockstats(Monitor *mon, const QDict *qdict, QObject **ret);
-VncInfo * qmp_query_vnc(Error **errp);
-int qmp_marshal_input_query_vnc(Monitor *mon, const QDict *qdict, QObject **ret);
-SpiceInfo * qmp_query_spice(Error **errp);
-int qmp_marshal_input_query_spice(Monitor *mon, const QDict *qdict, QObject **ret);
-BalloonInfo * qmp_query_balloon(Error **errp);
-int qmp_marshal_input_query_balloon(Monitor *mon, const QDict *qdict, QObject **ret);
-PciInfoList * qmp_query_pci(Error **errp);
-int qmp_marshal_input_query_pci(Monitor *mon, const QDict *qdict, QObject **ret);
-BlockJobInfoList * qmp_query_block_jobs(Error **errp);
-int qmp_marshal_input_query_block_jobs(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_quit(Error **errp);
-int qmp_marshal_input_quit(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_stop(Error **errp);
-int qmp_marshal_input_stop(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_system_reset(Error **errp);
-int qmp_marshal_input_system_reset(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_system_powerdown(Error **errp);
-int qmp_marshal_input_system_powerdown(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_cpu(int64_t index, Error **errp);
-int qmp_marshal_input_cpu(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_cpu_add(int64_t id, Error **errp);
-int qmp_marshal_input_cpu_add(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_memsave(int64_t val, int64_t size, const char * filename, bool has_cpu_index, int64_t cpu_index, Error **errp);
-int qmp_marshal_input_memsave(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_pmemsave(int64_t val, int64_t size, const char * filename, Error **errp);
-int qmp_marshal_input_pmemsave(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_cont(Error **errp);
-int qmp_marshal_input_cont(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_system_wakeup(Error **errp);
-int qmp_marshal_input_system_wakeup(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_inject_nmi(Error **errp);
-int qmp_marshal_input_inject_nmi(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_set_link(const char * name, bool up, Error **errp);
-int qmp_marshal_input_set_link(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_block_passwd(const char * device, const char * password, Error **errp);
-int qmp_marshal_input_block_passwd(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_balloon(int64_t value, Error **errp);
-int qmp_marshal_input_balloon(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_block_resize(const char * device, int64_t size, Error **errp);
-int qmp_marshal_input_block_resize(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_transaction(TransactionActionList * actions, Error **errp);
-int qmp_marshal_input_transaction(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_blockdev_snapshot_sync(const char * device, const char * snapshot_file, bool has_format, const char * format, bool has_mode, NewImageMode mode, Error **errp);
-int qmp_marshal_input_blockdev_snapshot_sync(Monitor *mon, const QDict *qdict, QObject **ret);
-char * qmp_human_monitor_command(const char * command_line, bool has_cpu_index, int64_t cpu_index, Error **errp);
-int qmp_marshal_input_human_monitor_command(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_block_commit(const char * device, bool has_base, const char * base, const char * top, bool has_speed, int64_t speed, Error **errp);
-int qmp_marshal_input_block_commit(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_drive_backup(const char * device, const char * target, bool has_format, const char * format, MirrorSyncMode sync, bool has_mode, NewImageMode mode, bool has_speed, int64_t speed, bool has_on_source_error, BlockdevOnError on_source_error, bool has_on_target_error, BlockdevOnError on_target_error, Error **errp);
-int qmp_marshal_input_drive_backup(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_drive_mirror(const char * device, const char * target, bool has_format, const char * format, MirrorSyncMode sync, bool has_mode, NewImageMode mode, bool has_speed, int64_t speed, bool has_granularity, uint32_t granularity, bool has_buf_size, int64_t buf_size, bool has_on_source_error, BlockdevOnError on_source_error, bool has_on_target_error, BlockdevOnError on_target_error, Error **errp);
-int qmp_marshal_input_drive_mirror(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_migrate_cancel(Error **errp);
-int qmp_marshal_input_migrate_cancel(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_migrate_set_downtime(double value, Error **errp);
-int qmp_marshal_input_migrate_set_downtime(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_migrate_set_speed(int64_t value, Error **errp);
-int qmp_marshal_input_migrate_set_speed(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_migrate_set_cache_size(int64_t value, Error **errp);
-int qmp_marshal_input_migrate_set_cache_size(Monitor *mon, const QDict *qdict, QObject **ret);
-int64_t qmp_query_migrate_cache_size(Error **errp);
-int qmp_marshal_input_query_migrate_cache_size(Monitor *mon, const QDict *qdict, QObject **ret);
-ObjectPropertyInfoList * qmp_qom_list(const char * path, Error **errp);
-int qmp_marshal_input_qom_list(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_set_password(const char * protocol, const char * password, bool has_connected, const char * connected, Error **errp);
-int qmp_marshal_input_set_password(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_expire_password(const char * protocol, const char * time, Error **errp);
-int qmp_marshal_input_expire_password(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_eject(const char * device, bool has_force, bool force, Error **errp);
-int qmp_marshal_input_eject(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_change_vnc_password(const char * password, Error **errp);
-int qmp_marshal_input_change_vnc_password(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_change(const char * device, const char * target, bool has_arg, const char * arg, Error **errp);
-int qmp_marshal_input_change(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_block_set_io_throttle(const char * device, int64_t bps, int64_t bps_rd, int64_t bps_wr, int64_t iops, int64_t iops_rd, int64_t iops_wr, Error **errp);
-int qmp_marshal_input_block_set_io_throttle(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_block_stream(const char * device, bool has_base, const char * base, bool has_speed, int64_t speed, bool has_on_error, BlockdevOnError on_error, Error **errp);
-int qmp_marshal_input_block_stream(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_block_job_set_speed(const char * device, int64_t speed, Error **errp);
-int qmp_marshal_input_block_job_set_speed(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_block_job_cancel(const char * device, bool has_force, bool force, Error **errp);
-int qmp_marshal_input_block_job_cancel(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_block_job_pause(const char * device, Error **errp);
-int qmp_marshal_input_block_job_pause(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_block_job_resume(const char * device, Error **errp);
-int qmp_marshal_input_block_job_resume(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_block_job_complete(const char * device, Error **errp);
-int qmp_marshal_input_block_job_complete(Monitor *mon, const QDict *qdict, QObject **ret);
-ObjectTypeInfoList * qmp_qom_list_types(bool has_implements, const char * implements, bool has_abstract, bool abstract, Error **errp);
-int qmp_marshal_input_qom_list_types(Monitor *mon, const QDict *qdict, QObject **ret);
-DevicePropertyInfoList * qmp_device_list_properties(const char * typename, Error **errp);
-int qmp_marshal_input_device_list_properties(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_migrate(const char * uri, bool has_blk, bool blk, bool has_inc, bool inc, bool has_detach, bool detach, Error **errp);
-int qmp_marshal_input_migrate(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_xen_save_devices_state(const char * filename, Error **errp);
-int qmp_marshal_input_xen_save_devices_state(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_xen_set_global_dirty_log(bool enable, Error **errp);
-int qmp_marshal_input_xen_set_global_dirty_log(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_device_del(const char * id, Error **errp);
-int qmp_marshal_input_device_del(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_dump_guest_memory(bool paging, const char * protocol, bool has_begin, int64_t begin, bool has_length, int64_t length, Error **errp);
-int qmp_marshal_input_dump_guest_memory(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_netdev_del(const char * id, Error **errp);
-int qmp_marshal_input_netdev_del(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_getfd(const char * fdname, Error **errp);
-int qmp_marshal_input_getfd(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_closefd(const char * fdname, Error **errp);
-int qmp_marshal_input_closefd(Monitor *mon, const QDict *qdict, QObject **ret);
-MachineInfoList * qmp_query_machines(Error **errp);
-int qmp_marshal_input_query_machines(Monitor *mon, const QDict *qdict, QObject **ret);
-CpuDefinitionInfoList * qmp_query_cpu_definitions(Error **errp);
-int qmp_marshal_input_query_cpu_definitions(Monitor *mon, const QDict *qdict, QObject **ret);
-AddfdInfo * qmp_add_fd(bool has_fdset_id, int64_t fdset_id, bool has_opaque, const char * opaque, Error **errp);
-int qmp_marshal_input_add_fd(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_remove_fd(int64_t fdset_id, bool has_fd, int64_t fd, Error **errp);
-int qmp_marshal_input_remove_fd(Monitor *mon, const QDict *qdict, QObject **ret);
-FdsetInfoList * qmp_query_fdsets(Error **errp);
-int qmp_marshal_input_query_fdsets(Monitor *mon, const QDict *qdict, QObject **ret);
-TargetInfo * qmp_query_target(Error **errp);
-int qmp_marshal_input_query_target(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_send_key(KeyValueList * keys, bool has_hold_time, int64_t hold_time, Error **errp);
-int qmp_marshal_input_send_key(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_screendump(const char * filename, Error **errp);
-int qmp_marshal_input_screendump(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_nbd_server_start(SocketAddress * addr, Error **errp);
-int qmp_marshal_input_nbd_server_start(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_nbd_server_add(const char * device, bool has_writable, bool writable, Error **errp);
-int qmp_marshal_input_nbd_server_add(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_nbd_server_stop(Error **errp);
-int qmp_marshal_input_nbd_server_stop(Monitor *mon, const QDict *qdict, QObject **ret);
-ChardevReturn * qmp_chardev_add(const char * id, ChardevBackend * backend, Error **errp);
-int qmp_marshal_input_chardev_add(Monitor *mon, const QDict *qdict, QObject **ret);
-void qmp_chardev_remove(const char * id, Error **errp);
-int qmp_marshal_input_chardev_remove(Monitor *mon, const QDict *qdict, QObject **ret);
-TpmModelList * qmp_query_tpm_models(Error **errp);
-int qmp_marshal_input_query_tpm_models(Monitor *mon, const QDict *qdict, QObject **ret);
-TpmTypeList * qmp_query_tpm_types(Error **errp);
-int qmp_marshal_input_query_tpm_types(Monitor *mon, const QDict *qdict, QObject **ret);
-TPMInfoList * qmp_query_tpm(Error **errp);
-int qmp_marshal_input_query_tpm(Monitor *mon, const QDict *qdict, QObject **ret);
-CommandLineOptionInfoList * qmp_query_command_line_options(bool has_option, const char * option, Error **errp);
-int qmp_marshal_input_query_command_line_options(Monitor *mon, const QDict *qdict, QObject **ret);
-RxFilterInfoList * qmp_query_rx_filter(bool has_name, const char * name, Error **errp);
-int qmp_marshal_input_query_rx_filter(Monitor *mon, const QDict *qdict, QObject **ret);
-
-#endif
diff --git a/contrib/qemu/qobject/json-lexer.c b/contrib/qemu/qobject/json-lexer.c
deleted file mode 100644
index 440df60392b..00000000000
--- a/contrib/qemu/qobject/json-lexer.c
+++ /dev/null
@@ -1,373 +0,0 @@
-/*
- * JSON lexer
- *
- * Copyright IBM, Corp. 2009
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#include "qapi/qmp/qstring.h"
-#include "qapi/qmp/qlist.h"
-#include "qapi/qmp/qdict.h"
-#include "qapi/qmp/qint.h"
-#include "qemu-common.h"
-#include "qapi/qmp/json-lexer.h"
-
-#define MAX_TOKEN_SIZE (64ULL << 20)
-
-/*
- * \"([^\\\"]|(\\\"\\'\\\\\\/\\b\\f\\n\\r\\t\\u[0-9a-fA-F][0-9a-fA-F][0-9a-fA-F][0-9a-fA-F]))*\"
- * '([^\\']|(\\\"\\'\\\\\\/\\b\\f\\n\\r\\t\\u[0-9a-fA-F][0-9a-fA-F][0-9a-fA-F][0-9a-fA-F]))*'
- * 0|([1-9][0-9]*(.[0-9]+)?([eE]([-+])?[0-9]+))
- * [{}\[\],:]
- * [a-z]+
- *
- */
-
-enum json_lexer_state {
-    IN_ERROR = 0,
-    IN_DQ_UCODE3,
-    IN_DQ_UCODE2,
-    IN_DQ_UCODE1,
-    IN_DQ_UCODE0,
-    IN_DQ_STRING_ESCAPE,
-    IN_DQ_STRING,
-    IN_SQ_UCODE3,
-    IN_SQ_UCODE2,
-    IN_SQ_UCODE1,
-    IN_SQ_UCODE0,
-    IN_SQ_STRING_ESCAPE,
-    IN_SQ_STRING,
-    IN_ZERO,
-    IN_DIGITS,
-    IN_DIGIT,
-    IN_EXP_E,
-    IN_MANTISSA,
-    IN_MANTISSA_DIGITS,
-    IN_NONZERO_NUMBER,
-    IN_NEG_NONZERO_NUMBER,
-    IN_KEYWORD,
-    IN_ESCAPE,
-    IN_ESCAPE_L,
-    IN_ESCAPE_LL,
-    IN_ESCAPE_I,
-    IN_ESCAPE_I6,
-    IN_ESCAPE_I64,
-    IN_WHITESPACE,
-    IN_START,
-};
-
-#define TERMINAL(state) [0 ... 0x7F] = (state)
-
-/* Return whether TERMINAL is a terminal state and the transition to it
-   from OLD_STATE required lookahead.  This happens whenever the table
-   below uses the TERMINAL macro.  */
-#define TERMINAL_NEEDED_LOOKAHEAD(old_state, terminal) \
-            (json_lexer[(old_state)][0] == (terminal))
-
-static const uint8_t json_lexer[][256] =  {
-    /* double quote string */
-    [IN_DQ_UCODE3] = {
-        ['0' ... '9'] = IN_DQ_STRING,
-        ['a' ... 'f'] = IN_DQ_STRING,
-        ['A' ... 'F'] = IN_DQ_STRING,
-    },
-    [IN_DQ_UCODE2] = {
-        ['0' ... '9'] = IN_DQ_UCODE3,
-        ['a' ... 'f'] = IN_DQ_UCODE3,
-        ['A' ... 'F'] = IN_DQ_UCODE3,
-    },
-    [IN_DQ_UCODE1] = {
-        ['0' ... '9'] = IN_DQ_UCODE2,
-        ['a' ... 'f'] = IN_DQ_UCODE2,
-        ['A' ... 'F'] = IN_DQ_UCODE2,
-    },
-    [IN_DQ_UCODE0] = {
-        ['0' ... '9'] = IN_DQ_UCODE1,
-        ['a' ... 'f'] = IN_DQ_UCODE1,
-        ['A' ... 'F'] = IN_DQ_UCODE1,
-    },
-    [IN_DQ_STRING_ESCAPE] = {
-        ['b'] = IN_DQ_STRING,
-        ['f'] =  IN_DQ_STRING,
-        ['n'] =  IN_DQ_STRING,
-        ['r'] =  IN_DQ_STRING,
-        ['t'] =  IN_DQ_STRING,
-        ['/'] = IN_DQ_STRING,
-        ['\\'] = IN_DQ_STRING,
-        ['\''] = IN_DQ_STRING,
-        ['\"'] = IN_DQ_STRING,
-        ['u'] = IN_DQ_UCODE0,
-    },
-    [IN_DQ_STRING] = {
-        [1 ... 0xBF] = IN_DQ_STRING,
-        [0xC2 ... 0xF4] = IN_DQ_STRING,
-        ['\\'] = IN_DQ_STRING_ESCAPE,
-        ['"'] = JSON_STRING,
-    },
-
-    /* single quote string */
-    [IN_SQ_UCODE3] = {
-        ['0' ... '9'] = IN_SQ_STRING,
-        ['a' ... 'f'] = IN_SQ_STRING,
-        ['A' ... 'F'] = IN_SQ_STRING,
-    },
-    [IN_SQ_UCODE2] = {
-        ['0' ... '9'] = IN_SQ_UCODE3,
-        ['a' ... 'f'] = IN_SQ_UCODE3,
-        ['A' ... 'F'] = IN_SQ_UCODE3,
-    },
-    [IN_SQ_UCODE1] = {
-        ['0' ... '9'] = IN_SQ_UCODE2,
-        ['a' ... 'f'] = IN_SQ_UCODE2,
-        ['A' ... 'F'] = IN_SQ_UCODE2,
-    },
-    [IN_SQ_UCODE0] = {
-        ['0' ... '9'] = IN_SQ_UCODE1,
-        ['a' ... 'f'] = IN_SQ_UCODE1,
-        ['A' ... 'F'] = IN_SQ_UCODE1,
-    },
-    [IN_SQ_STRING_ESCAPE] = {
-        ['b'] = IN_SQ_STRING,
-        ['f'] =  IN_SQ_STRING,
-        ['n'] =  IN_SQ_STRING,
-        ['r'] =  IN_SQ_STRING,
-        ['t'] =  IN_SQ_STRING,
-        ['/'] = IN_DQ_STRING,
-        ['\\'] = IN_DQ_STRING,
-        ['\''] = IN_SQ_STRING,
-        ['\"'] = IN_SQ_STRING,
-        ['u'] = IN_SQ_UCODE0,
-    },
-    [IN_SQ_STRING] = {
-        [1 ... 0xBF] = IN_SQ_STRING,
-        [0xC2 ... 0xF4] = IN_SQ_STRING,
-        ['\\'] = IN_SQ_STRING_ESCAPE,
-        ['\''] = JSON_STRING,
-    },
-
-    /* Zero */
-    [IN_ZERO] = {
-        TERMINAL(JSON_INTEGER),
-        ['0' ... '9'] = IN_ERROR,
-        ['.'] = IN_MANTISSA,
-    },
-
-    /* Float */
-    [IN_DIGITS] = {
-        TERMINAL(JSON_FLOAT),
-        ['0' ... '9'] = IN_DIGITS,
-    },
-
-    [IN_DIGIT] = {
-        ['0' ... '9'] = IN_DIGITS,
-    },
-
-    [IN_EXP_E] = {
-        ['-'] = IN_DIGIT,
-        ['+'] = IN_DIGIT,
-        ['0' ... '9'] = IN_DIGITS,
-    },
-
-    [IN_MANTISSA_DIGITS] = {
-        TERMINAL(JSON_FLOAT),
-        ['0' ... '9'] = IN_MANTISSA_DIGITS,
-        ['e'] = IN_EXP_E,
-        ['E'] = IN_EXP_E,
-    },
-
-    [IN_MANTISSA] = {
-        ['0' ... '9'] = IN_MANTISSA_DIGITS,
-    },
-
-    /* Number */
-    [IN_NONZERO_NUMBER] = {
-        TERMINAL(JSON_INTEGER),
-        ['0' ... '9'] = IN_NONZERO_NUMBER,
-        ['e'] = IN_EXP_E,
-        ['E'] = IN_EXP_E,
-        ['.'] = IN_MANTISSA,
-    },
-
-    [IN_NEG_NONZERO_NUMBER] = {
-        ['0'] = IN_ZERO,
-        ['1' ... '9'] = IN_NONZERO_NUMBER,
-    },
-
-    /* keywords */
-    [IN_KEYWORD] = {
-        TERMINAL(JSON_KEYWORD),
-        ['a' ... 'z'] = IN_KEYWORD,
-    },
-
-    /* whitespace */
-    [IN_WHITESPACE] = {
-        TERMINAL(JSON_SKIP),
-        [' '] = IN_WHITESPACE,
-        ['\t'] = IN_WHITESPACE,
-        ['\r'] = IN_WHITESPACE,
-        ['\n'] = IN_WHITESPACE,
-    },        
-
-    /* escape */
-    [IN_ESCAPE_LL] = {
-        ['d'] = JSON_ESCAPE,
-    },
-
-    [IN_ESCAPE_L] = {
-        ['d'] = JSON_ESCAPE,
-        ['l'] = IN_ESCAPE_LL,
-    },
-
-    [IN_ESCAPE_I64] = {
-        ['d'] = JSON_ESCAPE,
-    },
-
-    [IN_ESCAPE_I6] = {
-        ['4'] = IN_ESCAPE_I64,
-    },
-
-    [IN_ESCAPE_I] = {
-        ['6'] = IN_ESCAPE_I6,
-    },
-
-    [IN_ESCAPE] = {
-        ['d'] = JSON_ESCAPE,
-        ['i'] = JSON_ESCAPE,
-        ['p'] = JSON_ESCAPE,
-        ['s'] = JSON_ESCAPE,
-        ['f'] = JSON_ESCAPE,
-        ['l'] = IN_ESCAPE_L,
-        ['I'] = IN_ESCAPE_I,
-    },
-
-    /* top level rule */
-    [IN_START] = {
-        ['"'] = IN_DQ_STRING,
-        ['\''] = IN_SQ_STRING,
-        ['0'] = IN_ZERO,
-        ['1' ... '9'] = IN_NONZERO_NUMBER,
-        ['-'] = IN_NEG_NONZERO_NUMBER,
-        ['{'] = JSON_OPERATOR,
-        ['}'] = JSON_OPERATOR,
-        ['['] = JSON_OPERATOR,
-        [']'] = JSON_OPERATOR,
-        [','] = JSON_OPERATOR,
-        [':'] = JSON_OPERATOR,
-        ['a' ... 'z'] = IN_KEYWORD,
-        ['%'] = IN_ESCAPE,
-        [' '] = IN_WHITESPACE,
-        ['\t'] = IN_WHITESPACE,
-        ['\r'] = IN_WHITESPACE,
-        ['\n'] = IN_WHITESPACE,
-    },
-};
-
-void json_lexer_init(JSONLexer *lexer, JSONLexerEmitter func)
-{
-    lexer->emit = func;
-    lexer->state = IN_START;
-    lexer->token = qstring_new();
-    lexer->x = lexer->y = 0;
-}
-
-static int json_lexer_feed_char(JSONLexer *lexer, char ch, bool flush)
-{
-    int char_consumed, new_state;
-
-    lexer->x++;
-    if (ch == '\n') {
-        lexer->x = 0;
-        lexer->y++;
-    }
-
-    do {
-        new_state = json_lexer[lexer->state][(uint8_t)ch];
-        char_consumed = !TERMINAL_NEEDED_LOOKAHEAD(lexer->state, new_state);
-        if (char_consumed) {
-            qstring_append_chr(lexer->token, ch);
-        }
-
-        switch (new_state) {
-        case JSON_OPERATOR:
-        case JSON_ESCAPE:
-        case JSON_INTEGER:
-        case JSON_FLOAT:
-        case JSON_KEYWORD:
-        case JSON_STRING:
-            lexer->emit(lexer, lexer->token, new_state, lexer->x, lexer->y);
-            /* fall through */
-        case JSON_SKIP:
-            QDECREF(lexer->token);
-            lexer->token = qstring_new();
-            new_state = IN_START;
-            break;
-        case IN_ERROR:
-            /* XXX: To avoid having previous bad input leaving the parser in an
-             * unresponsive state where we consume unpredictable amounts of
-             * subsequent "good" input, percolate this error state up to the
-             * tokenizer/parser by forcing a NULL object to be emitted, then
-             * reset state.
-             *
-             * Also note that this handling is required for reliable channel
-             * negotiation between QMP and the guest agent, since chr(0xFF)
-             * is placed at the beginning of certain events to ensure proper
-             * delivery when the channel is in an unknown state. chr(0xFF) is
-             * never a valid ASCII/UTF-8 sequence, so this should reliably
-             * induce an error/flush state.
-             */
-            lexer->emit(lexer, lexer->token, JSON_ERROR, lexer->x, lexer->y);
-            QDECREF(lexer->token);
-            lexer->token = qstring_new();
-            new_state = IN_START;
-            lexer->state = new_state;
-            return 0;
-        default:
-            break;
-        }
-        lexer->state = new_state;
-    } while (!char_consumed && !flush);
-
-    /* Do not let a single token grow to an arbitrarily large size,
-     * this is a security consideration.
-     */
-    if (lexer->token->length > MAX_TOKEN_SIZE) {
-        lexer->emit(lexer, lexer->token, lexer->state, lexer->x, lexer->y);
-        QDECREF(lexer->token);
-        lexer->token = qstring_new();
-        lexer->state = IN_START;
-    }
-
-    return 0;
-}
-
-int json_lexer_feed(JSONLexer *lexer, const char *buffer, size_t size)
-{
-    size_t i;
-
-    for (i = 0; i < size; i++) {
-        int err;
-
-        err = json_lexer_feed_char(lexer, buffer[i], false);
-        if (err < 0) {
-            return err;
-        }
-    }
-
-    return 0;
-}
-
-int json_lexer_flush(JSONLexer *lexer)
-{
-    return lexer->state == IN_START ? 0 : json_lexer_feed_char(lexer, 0, true);
-}
-
-void json_lexer_destroy(JSONLexer *lexer)
-{
-    QDECREF(lexer->token);
-}
diff --git a/contrib/qemu/qobject/json-parser.c b/contrib/qemu/qobject/json-parser.c
deleted file mode 100644
index e7947b340c1..00000000000
--- a/contrib/qemu/qobject/json-parser.c
+++ /dev/null
@@ -1,724 +0,0 @@
-/*
- * JSON Parser 
- *
- * Copyright IBM, Corp. 2009
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#include <stdarg.h>
-
-#include "qemu-common.h"
-#include "qapi/qmp/qstring.h"
-#include "qapi/qmp/qint.h"
-#include "qapi/qmp/qdict.h"
-#include "qapi/qmp/qlist.h"
-#include "qapi/qmp/qfloat.h"
-#include "qapi/qmp/qbool.h"
-#include "qapi/qmp/json-parser.h"
-#include "qapi/qmp/json-lexer.h"
-#include "qapi/qmp/qerror.h"
-
-typedef struct JSONParserContext
-{
-    Error *err;
-    struct {
-        QObject **buf;
-        size_t pos;
-        size_t count;
-    } tokens;
-} JSONParserContext;
-
-#define BUG_ON(cond) assert(!(cond))
-
-/**
- * TODO
- *
- * 0) make errors meaningful again
- * 1) add geometry information to tokens
- * 3) should we return a parsed size?
- * 4) deal with premature EOI
- */
-
-static QObject *parse_value(JSONParserContext *ctxt, va_list *ap);
-
-/**
- * Token manipulators
- *
- * tokens are dictionaries that contain a type, a string value, and geometry information
- * about a token identified by the lexer.  These are routines that make working with
- * these objects a bit easier.
- */
-static const char *token_get_value(QObject *obj)
-{
-    return qdict_get_str(qobject_to_qdict(obj), "token");
-}
-
-static JSONTokenType token_get_type(QObject *obj)
-{
-    return qdict_get_int(qobject_to_qdict(obj), "type");
-}
-
-static int token_is_operator(QObject *obj, char op)
-{
-    const char *val;
-
-    if (token_get_type(obj) != JSON_OPERATOR) {
-        return 0;
-    }
-
-    val = token_get_value(obj);
-
-    return (val[0] == op) && (val[1] == 0);
-}
-
-static int token_is_keyword(QObject *obj, const char *value)
-{
-    if (token_get_type(obj) != JSON_KEYWORD) {
-        return 0;
-    }
-
-    return strcmp(token_get_value(obj), value) == 0;
-}
-
-static int token_is_escape(QObject *obj, const char *value)
-{
-    if (token_get_type(obj) != JSON_ESCAPE) {
-        return 0;
-    }
-
-    return (strcmp(token_get_value(obj), value) == 0);
-}
-
-/**
- * Error handler
- */
-static void GCC_FMT_ATTR(3, 4) parse_error(JSONParserContext *ctxt,
-                                           QObject *token, const char *msg, ...)
-{
-    va_list ap;
-    char message[1024];
-    va_start(ap, msg);
-    vsnprintf(message, sizeof(message), msg, ap);
-    va_end(ap);
-    if (ctxt->err) {
-        error_free(ctxt->err);
-        ctxt->err = NULL;
-    }
-    error_set(&ctxt->err, QERR_JSON_PARSE_ERROR, message);
-}
-
-/**
- * String helpers
- *
- * These helpers are used to unescape strings.
- */
-static void wchar_to_utf8(uint16_t wchar, char *buffer, size_t buffer_length)
-{
-    if (wchar <= 0x007F) {
-        BUG_ON(buffer_length < 2);
-
-        buffer[0] = wchar & 0x7F;
-        buffer[1] = 0;
-    } else if (wchar <= 0x07FF) {
-        BUG_ON(buffer_length < 3);
-
-        buffer[0] = 0xC0 | ((wchar >> 6) & 0x1F);
-        buffer[1] = 0x80 | (wchar & 0x3F);
-        buffer[2] = 0;
-    } else {
-        BUG_ON(buffer_length < 4);
-
-        buffer[0] = 0xE0 | ((wchar >> 12) & 0x0F);
-        buffer[1] = 0x80 | ((wchar >> 6) & 0x3F);
-        buffer[2] = 0x80 | (wchar & 0x3F);
-        buffer[3] = 0;
-    }
-}
-
-static int hex2decimal(char ch)
-{
-    if (ch >= '0' && ch <= '9') {
-        return (ch - '0');
-    } else if (ch >= 'a' && ch <= 'f') {
-        return 10 + (ch - 'a');
-    } else if (ch >= 'A' && ch <= 'F') {
-        return 10 + (ch - 'A');
-    }
-
-    return -1;
-}
-
-/**
- * parse_string(): Parse a json string and return a QObject
- *
- *  string
- *      ""
- *      " chars "
- *  chars
- *      char
- *      char chars
- *  char
- *      any-Unicode-character-
- *          except-"-or-\-or-
- *          control-character
- *      \"
- *      \\
- *      \/
- *      \b
- *      \f
- *      \n
- *      \r
- *      \t
- *      \u four-hex-digits 
- */
-static QString *qstring_from_escaped_str(JSONParserContext *ctxt, QObject *token)
-{
-    const char *ptr = token_get_value(token);
-    QString *str;
-    int double_quote = 1;
-
-    if (*ptr == '"') {
-        double_quote = 1;
-    } else {
-        double_quote = 0;
-    }
-    ptr++;
-
-    str = qstring_new();
-    while (*ptr && 
-           ((double_quote && *ptr != '"') || (!double_quote && *ptr != '\''))) {
-        if (*ptr == '\\') {
-            ptr++;
-
-            switch (*ptr) {
-            case '"':
-                qstring_append(str, "\"");
-                ptr++;
-                break;
-            case '\'':
-                qstring_append(str, "'");
-                ptr++;
-                break;
-            case '\\':
-                qstring_append(str, "\\");
-                ptr++;
-                break;
-            case '/':
-                qstring_append(str, "/");
-                ptr++;
-                break;
-            case 'b':
-                qstring_append(str, "\b");
-                ptr++;
-                break;
-            case 'f':
-                qstring_append(str, "\f");
-                ptr++;
-                break;
-            case 'n':
-                qstring_append(str, "\n");
-                ptr++;
-                break;
-            case 'r':
-                qstring_append(str, "\r");
-                ptr++;
-                break;
-            case 't':
-                qstring_append(str, "\t");
-                ptr++;
-                break;
-            case 'u': {
-                uint16_t unicode_char = 0;
-                char utf8_char[4];
-                int i = 0;
-
-                ptr++;
-
-                for (i = 0; i < 4; i++) {
-                    if (qemu_isxdigit(*ptr)) {
-                        unicode_char |= hex2decimal(*ptr) << ((3 - i) * 4);
-                    } else {
-                        parse_error(ctxt, token,
-                                    "invalid hex escape sequence in string");
-                        goto out;
-                    }
-                    ptr++;
-                }
-
-                wchar_to_utf8(unicode_char, utf8_char, sizeof(utf8_char));
-                qstring_append(str, utf8_char);
-            }   break;
-            default:
-                parse_error(ctxt, token, "invalid escape sequence in string");
-                goto out;
-            }
-        } else {
-            char dummy[2];
-
-            dummy[0] = *ptr++;
-            dummy[1] = 0;
-
-            qstring_append(str, dummy);
-        }
-    }
-
-    return str;
-
-out:
-    QDECREF(str);
-    return NULL;
-}
-
-static QObject *parser_context_pop_token(JSONParserContext *ctxt)
-{
-    QObject *token;
-    g_assert(ctxt->tokens.pos < ctxt->tokens.count);
-    token = ctxt->tokens.buf[ctxt->tokens.pos];
-    ctxt->tokens.pos++;
-    return token;
-}
-
-/* Note: parser_context_{peek|pop}_token do not increment the
- * token object's refcount. In both cases the references will continue
- * to be tracked and cleaned up in parser_context_free(), so do not
- * attempt to free the token object.
- */
-static QObject *parser_context_peek_token(JSONParserContext *ctxt)
-{
-    QObject *token;
-    g_assert(ctxt->tokens.pos < ctxt->tokens.count);
-    token = ctxt->tokens.buf[ctxt->tokens.pos];
-    return token;
-}
-
-static JSONParserContext parser_context_save(JSONParserContext *ctxt)
-{
-    JSONParserContext saved_ctxt = {0};
-    saved_ctxt.tokens.pos = ctxt->tokens.pos;
-    saved_ctxt.tokens.count = ctxt->tokens.count;
-    saved_ctxt.tokens.buf = ctxt->tokens.buf;
-    return saved_ctxt;
-}
-
-static void parser_context_restore(JSONParserContext *ctxt,
-                                   JSONParserContext saved_ctxt)
-{
-    ctxt->tokens.pos = saved_ctxt.tokens.pos;
-    ctxt->tokens.count = saved_ctxt.tokens.count;
-    ctxt->tokens.buf = saved_ctxt.tokens.buf;
-}
-
-static void tokens_append_from_iter(QObject *obj, void *opaque)
-{
-    JSONParserContext *ctxt = opaque;
-    g_assert(ctxt->tokens.pos < ctxt->tokens.count);
-    ctxt->tokens.buf[ctxt->tokens.pos++] = obj;
-    qobject_incref(obj);
-}
-
-static JSONParserContext *parser_context_new(QList *tokens)
-{
-    JSONParserContext *ctxt;
-    size_t count;
-
-    if (!tokens) {
-        return NULL;
-    }
-
-    count = qlist_size(tokens);
-    if (count == 0) {
-        return NULL;
-    }
-
-    ctxt = g_malloc0(sizeof(JSONParserContext));
-    ctxt->tokens.pos = 0;
-    ctxt->tokens.count = count;
-    ctxt->tokens.buf = g_malloc(count * sizeof(QObject *));
-    qlist_iter(tokens, tokens_append_from_iter, ctxt);
-    ctxt->tokens.pos = 0;
-
-    return ctxt;
-}
-
-/* to support error propagation, ctxt->err must be freed separately */
-static void parser_context_free(JSONParserContext *ctxt)
-{
-    int i;
-    if (ctxt) {
-        for (i = 0; i < ctxt->tokens.count; i++) {
-            qobject_decref(ctxt->tokens.buf[i]);
-        }
-        g_free(ctxt->tokens.buf);
-        g_free(ctxt);
-    }
-}
-
-/**
- * Parsing rules
- */
-static int parse_pair(JSONParserContext *ctxt, QDict *dict, va_list *ap)
-{
-    QObject *key = NULL, *token = NULL, *value, *peek;
-    JSONParserContext saved_ctxt = parser_context_save(ctxt);
-
-    peek = parser_context_peek_token(ctxt);
-    if (peek == NULL) {
-        parse_error(ctxt, NULL, "premature EOI");
-        goto out;
-    }
-
-    key = parse_value(ctxt, ap);
-    if (!key || qobject_type(key) != QTYPE_QSTRING) {
-        parse_error(ctxt, peek, "key is not a string in object");
-        goto out;
-    }
-
-    token = parser_context_pop_token(ctxt);
-    if (token == NULL) {
-        parse_error(ctxt, NULL, "premature EOI");
-        goto out;
-    }
-
-    if (!token_is_operator(token, ':')) {
-        parse_error(ctxt, token, "missing : in object pair");
-        goto out;
-    }
-
-    value = parse_value(ctxt, ap);
-    if (value == NULL) {
-        parse_error(ctxt, token, "Missing value in dict");
-        goto out;
-    }
-
-    qdict_put_obj(dict, qstring_get_str(qobject_to_qstring(key)), value);
-
-    qobject_decref(key);
-
-    return 0;
-
-out:
-    parser_context_restore(ctxt, saved_ctxt);
-    qobject_decref(key);
-
-    return -1;
-}
-
-static QObject *parse_object(JSONParserContext *ctxt, va_list *ap)
-{
-    QDict *dict = NULL;
-    QObject *token, *peek;
-    JSONParserContext saved_ctxt = parser_context_save(ctxt);
-
-    token = parser_context_pop_token(ctxt);
-    if (token == NULL) {
-        goto out;
-    }
-
-    if (!token_is_operator(token, '{')) {
-        goto out;
-    }
-    token = NULL;
-
-    dict = qdict_new();
-
-    peek = parser_context_peek_token(ctxt);
-    if (peek == NULL) {
-        parse_error(ctxt, NULL, "premature EOI");
-        goto out;
-    }
-
-    if (!token_is_operator(peek, '}')) {
-        if (parse_pair(ctxt, dict, ap) == -1) {
-            goto out;
-        }
-
-        token = parser_context_pop_token(ctxt);
-        if (token == NULL) {
-            parse_error(ctxt, NULL, "premature EOI");
-            goto out;
-        }
-
-        while (!token_is_operator(token, '}')) {
-            if (!token_is_operator(token, ',')) {
-                parse_error(ctxt, token, "expected separator in dict");
-                goto out;
-            }
-            token = NULL;
-
-            if (parse_pair(ctxt, dict, ap) == -1) {
-                goto out;
-            }
-
-            token = parser_context_pop_token(ctxt);
-            if (token == NULL) {
-                parse_error(ctxt, NULL, "premature EOI");
-                goto out;
-            }
-        }
-        token = NULL;
-    } else {
-        token = parser_context_pop_token(ctxt);
-        token = NULL;
-    }
-
-    return QOBJECT(dict);
-
-out:
-    parser_context_restore(ctxt, saved_ctxt);
-    QDECREF(dict);
-    return NULL;
-}
-
-static QObject *parse_array(JSONParserContext *ctxt, va_list *ap)
-{
-    QList *list = NULL;
-    QObject *token, *peek;
-    JSONParserContext saved_ctxt = parser_context_save(ctxt);
-
-    token = parser_context_pop_token(ctxt);
-    if (token == NULL) {
-        goto out;
-    }
-
-    if (!token_is_operator(token, '[')) {
-        token = NULL;
-        goto out;
-    }
-    token = NULL;
-
-    list = qlist_new();
-
-    peek = parser_context_peek_token(ctxt);
-    if (peek == NULL) {
-        parse_error(ctxt, NULL, "premature EOI");
-        goto out;
-    }
-
-    if (!token_is_operator(peek, ']')) {
-        QObject *obj;
-
-        obj = parse_value(ctxt, ap);
-        if (obj == NULL) {
-            parse_error(ctxt, token, "expecting value");
-            goto out;
-        }
-
-        qlist_append_obj(list, obj);
-
-        token = parser_context_pop_token(ctxt);
-        if (token == NULL) {
-            parse_error(ctxt, NULL, "premature EOI");
-            goto out;
-        }
-
-        while (!token_is_operator(token, ']')) {
-            if (!token_is_operator(token, ',')) {
-                parse_error(ctxt, token, "expected separator in list");
-                goto out;
-            }
-
-            token = NULL;
-
-            obj = parse_value(ctxt, ap);
-            if (obj == NULL) {
-                parse_error(ctxt, token, "expecting value");
-                goto out;
-            }
-
-            qlist_append_obj(list, obj);
-
-            token = parser_context_pop_token(ctxt);
-            if (token == NULL) {
-                parse_error(ctxt, NULL, "premature EOI");
-                goto out;
-            }
-        }
-
-        token = NULL;
-    } else {
-        token = parser_context_pop_token(ctxt);
-        token = NULL;
-    }
-
-    return QOBJECT(list);
-
-out:
-    parser_context_restore(ctxt, saved_ctxt);
-    QDECREF(list);
-    return NULL;
-}
-
-static QObject *parse_keyword(JSONParserContext *ctxt)
-{
-    QObject *token, *ret;
-    JSONParserContext saved_ctxt = parser_context_save(ctxt);
-
-    token = parser_context_pop_token(ctxt);
-    if (token == NULL) {
-        goto out;
-    }
-
-    if (token_get_type(token) != JSON_KEYWORD) {
-        goto out;
-    }
-
-    if (token_is_keyword(token, "true")) {
-        ret = QOBJECT(qbool_from_int(true));
-    } else if (token_is_keyword(token, "false")) {
-        ret = QOBJECT(qbool_from_int(false));
-    } else {
-        parse_error(ctxt, token, "invalid keyword `%s'", token_get_value(token));
-        goto out;
-    }
-
-    return ret;
-
-out: 
-    parser_context_restore(ctxt, saved_ctxt);
-
-    return NULL;
-}
-
-static QObject *parse_escape(JSONParserContext *ctxt, va_list *ap)
-{
-    QObject *token = NULL, *obj;
-    JSONParserContext saved_ctxt = parser_context_save(ctxt);
-
-    if (ap == NULL) {
-        goto out;
-    }
-
-    token = parser_context_pop_token(ctxt);
-    if (token == NULL) {
-        goto out;
-    }
-
-    if (token_is_escape(token, "%p")) {
-        obj = va_arg(*ap, QObject *);
-    } else if (token_is_escape(token, "%i")) {
-        obj = QOBJECT(qbool_from_int(va_arg(*ap, int)));
-    } else if (token_is_escape(token, "%d")) {
-        obj = QOBJECT(qint_from_int(va_arg(*ap, int)));
-    } else if (token_is_escape(token, "%ld")) {
-        obj = QOBJECT(qint_from_int(va_arg(*ap, long)));
-    } else if (token_is_escape(token, "%lld") ||
-               token_is_escape(token, "%I64d")) {
-        obj = QOBJECT(qint_from_int(va_arg(*ap, long long)));
-    } else if (token_is_escape(token, "%s")) {
-        obj = QOBJECT(qstring_from_str(va_arg(*ap, const char *)));
-    } else if (token_is_escape(token, "%f")) {
-        obj = QOBJECT(qfloat_from_double(va_arg(*ap, double)));
-    } else {
-        goto out;
-    }
-
-    return obj;
-
-out:
-    parser_context_restore(ctxt, saved_ctxt);
-
-    return NULL;
-}
-
-static QObject *parse_literal(JSONParserContext *ctxt)
-{
-    QObject *token, *obj;
-    JSONParserContext saved_ctxt = parser_context_save(ctxt);
-
-    token = parser_context_pop_token(ctxt);
-    if (token == NULL) {
-        goto out;
-    }
-
-    switch (token_get_type(token)) {
-    case JSON_STRING:
-        obj = QOBJECT(qstring_from_escaped_str(ctxt, token));
-        break;
-    case JSON_INTEGER: {
-        /* A possibility exists that this is a whole-valued float where the
-         * fractional part was left out due to being 0 (.0). It's not a big
-         * deal to treat these as ints in the parser, so long as users of the
-         * resulting QObject know to expect a QInt in place of a QFloat in
-         * cases like these.
-         *
-         * However, in some cases these values will overflow/underflow a
-         * QInt/int64 container, thus we should assume these are to be handled
-         * as QFloats/doubles rather than silently changing their values.
-         *
-         * strtoll() indicates these instances by setting errno to ERANGE
-         */
-        int64_t value;
-
-        errno = 0; /* strtoll doesn't set errno on success */
-        value = strtoll(token_get_value(token), NULL, 10);
-        if (errno != ERANGE) {
-            obj = QOBJECT(qint_from_int(value));
-            break;
-        }
-        /* fall through to JSON_FLOAT */
-    }
-    case JSON_FLOAT:
-        /* FIXME dependent on locale */
-        obj = QOBJECT(qfloat_from_double(strtod(token_get_value(token), NULL)));
-        break;
-    default:
-        goto out;
-    }
-
-    return obj;
-
-out:
-    parser_context_restore(ctxt, saved_ctxt);
-
-    return NULL;
-}
-
-static QObject *parse_value(JSONParserContext *ctxt, va_list *ap)
-{
-    QObject *obj;
-
-    obj = parse_object(ctxt, ap);
-    if (obj == NULL) {
-        obj = parse_array(ctxt, ap);
-    }
-    if (obj == NULL) {
-        obj = parse_escape(ctxt, ap);
-    }
-    if (obj == NULL) {
-        obj = parse_keyword(ctxt);
-    } 
-    if (obj == NULL) {
-        obj = parse_literal(ctxt);
-    }
-
-    return obj;
-}
-
-QObject *json_parser_parse(QList *tokens, va_list *ap)
-{
-    return json_parser_parse_err(tokens, ap, NULL);
-}
-
-QObject *json_parser_parse_err(QList *tokens, va_list *ap, Error **errp)
-{
-    JSONParserContext *ctxt = parser_context_new(tokens);
-    QObject *result;
-
-    if (!ctxt) {
-        return NULL;
-    }
-
-    result = parse_value(ctxt, ap);
-
-    error_propagate(errp, ctxt->err);
-
-    parser_context_free(ctxt);
-
-    return result;
-}
diff --git a/contrib/qemu/qobject/json-streamer.c b/contrib/qemu/qobject/json-streamer.c
deleted file mode 100644
index 1b2f9b1d107..00000000000
--- a/contrib/qemu/qobject/json-streamer.c
+++ /dev/null
@@ -1,122 +0,0 @@
-/*
- * JSON streaming support
- *
- * Copyright IBM, Corp. 2009
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#include "qapi/qmp/qlist.h"
-#include "qapi/qmp/qint.h"
-#include "qapi/qmp/qdict.h"
-#include "qemu-common.h"
-#include "qapi/qmp/json-lexer.h"
-#include "qapi/qmp/json-streamer.h"
-
-#define MAX_TOKEN_SIZE (64ULL << 20)
-#define MAX_NESTING (1ULL << 10)
-
-static void json_message_process_token(JSONLexer *lexer, QString *token, JSONTokenType type, int x, int y)
-{
-    JSONMessageParser *parser = container_of(lexer, JSONMessageParser, lexer);
-    QDict *dict;
-
-    if (type == JSON_OPERATOR) {
-        switch (qstring_get_str(token)[0]) {
-        case '{':
-            parser->brace_count++;
-            break;
-        case '}':
-            parser->brace_count--;
-            break;
-        case '[':
-            parser->bracket_count++;
-            break;
-        case ']':
-            parser->bracket_count--;
-            break;
-        default:
-            break;
-        }
-    }
-
-    dict = qdict_new();
-    qdict_put(dict, "type", qint_from_int(type));
-    QINCREF(token);
-    qdict_put(dict, "token", token);
-    qdict_put(dict, "x", qint_from_int(x));
-    qdict_put(dict, "y", qint_from_int(y));
-
-    parser->token_size += token->length;
-
-    qlist_append(parser->tokens, dict);
-
-    if (type == JSON_ERROR) {
-        goto out_emit_bad;
-    } else if (parser->brace_count < 0 ||
-        parser->bracket_count < 0 ||
-        (parser->brace_count == 0 &&
-         parser->bracket_count == 0)) {
-        goto out_emit;
-    } else if (parser->token_size > MAX_TOKEN_SIZE ||
-               parser->bracket_count > MAX_NESTING ||
-               parser->brace_count > MAX_NESTING) {
-        /* Security consideration, we limit total memory allocated per object
-         * and the maximum recursion depth that a message can force.
-         */
-        goto out_emit;
-    }
-
-    return;
-
-out_emit_bad:
-    /* clear out token list and tell the parser to emit and error
-     * indication by passing it a NULL list
-     */
-    QDECREF(parser->tokens);
-    parser->tokens = NULL;
-out_emit:
-    /* send current list of tokens to parser and reset tokenizer */
-    parser->brace_count = 0;
-    parser->bracket_count = 0;
-    parser->emit(parser, parser->tokens);
-    if (parser->tokens) {
-        QDECREF(parser->tokens);
-    }
-    parser->tokens = qlist_new();
-    parser->token_size = 0;
-}
-
-void json_message_parser_init(JSONMessageParser *parser,
-                              void (*func)(JSONMessageParser *, QList *))
-{
-    parser->emit = func;
-    parser->brace_count = 0;
-    parser->bracket_count = 0;
-    parser->tokens = qlist_new();
-    parser->token_size = 0;
-
-    json_lexer_init(&parser->lexer, json_message_process_token);
-}
-
-int json_message_parser_feed(JSONMessageParser *parser,
-                             const char *buffer, size_t size)
-{
-    return json_lexer_feed(&parser->lexer, buffer, size);
-}
-
-int json_message_parser_flush(JSONMessageParser *parser)
-{
-    return json_lexer_flush(&parser->lexer);
-}
-
-void json_message_parser_destroy(JSONMessageParser *parser)
-{
-    json_lexer_destroy(&parser->lexer);
-    QDECREF(parser->tokens);
-}
diff --git a/contrib/qemu/qobject/qbool.c b/contrib/qemu/qobject/qbool.c
deleted file mode 100644
index a3d2afa827f..00000000000
--- a/contrib/qemu/qobject/qbool.c
+++ /dev/null
@@ -1,68 +0,0 @@
-/*
- * QBool Module
- *
- * Copyright IBM, Corp. 2009
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#include "qapi/qmp/qbool.h"
-#include "qapi/qmp/qobject.h"
-#include "qemu-common.h"
-
-static void qbool_destroy_obj(QObject *obj);
-
-static const QType qbool_type = {
-    .code = QTYPE_QBOOL,
-    .destroy = qbool_destroy_obj,
-};
-
-/**
- * qbool_from_int(): Create a new QBool from an int
- *
- * Return strong reference.
- */
-QBool *qbool_from_int(int value)
-{
-    QBool *qb;
-
-    qb = g_malloc(sizeof(*qb));
-    qb->value = value;
-    QOBJECT_INIT(qb, &qbool_type);
-
-    return qb;
-}
-
-/**
- * qbool_get_int(): Get the stored int
- */
-int qbool_get_int(const QBool *qb)
-{
-    return qb->value;
-}
-
-/**
- * qobject_to_qbool(): Convert a QObject into a QBool
- */
-QBool *qobject_to_qbool(const QObject *obj)
-{
-    if (qobject_type(obj) != QTYPE_QBOOL)
-        return NULL;
-
-    return container_of(obj, QBool, base);
-}
-
-/**
- * qbool_destroy_obj(): Free all memory allocated by a
- * QBool object
- */
-static void qbool_destroy_obj(QObject *obj)
-{
-    assert(obj != NULL);
-    g_free(qobject_to_qbool(obj));
-}
diff --git a/contrib/qemu/qobject/qdict.c b/contrib/qemu/qobject/qdict.c
deleted file mode 100644
index ed381f9a507..00000000000
--- a/contrib/qemu/qobject/qdict.c
+++ /dev/null
@@ -1,478 +0,0 @@
-/*
- * QDict Module
- *
- * Copyright (C) 2009 Red Hat Inc.
- *
- * Authors:
- *  Luiz Capitulino <lcapitulino@redhat.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- */
-
-#include "qapi/qmp/qint.h"
-#include "qapi/qmp/qfloat.h"
-#include "qapi/qmp/qdict.h"
-#include "qapi/qmp/qbool.h"
-#include "qapi/qmp/qstring.h"
-#include "qapi/qmp/qobject.h"
-#include "qemu/queue.h"
-#include "qemu-common.h"
-
-static void qdict_destroy_obj(QObject *obj);
-
-static const QType qdict_type = {
-    .code = QTYPE_QDICT,
-    .destroy = qdict_destroy_obj,
-};
-
-/**
- * qdict_new(): Create a new QDict
- *
- * Return strong reference.
- */
-QDict *qdict_new(void)
-{
-    QDict *qdict;
-
-    qdict = g_malloc0(sizeof(*qdict));
-    QOBJECT_INIT(qdict, &qdict_type);
-
-    return qdict;
-}
-
-/**
- * qobject_to_qdict(): Convert a QObject into a QDict
- */
-QDict *qobject_to_qdict(const QObject *obj)
-{
-    if (qobject_type(obj) != QTYPE_QDICT)
-        return NULL;
-
-    return container_of(obj, QDict, base);
-}
-
-/**
- * tdb_hash(): based on the hash agorithm from gdbm, via tdb
- * (from module-init-tools)
- */
-static unsigned int tdb_hash(const char *name)
-{
-    unsigned value;	/* Used to compute the hash value.  */
-    unsigned   i;	/* Used to cycle through random values. */
-
-    /* Set the initial value from the key size. */
-    for (value = 0x238F13AF * strlen(name), i=0; name[i]; i++)
-        value = (value + (((const unsigned char *)name)[i] << (i*5 % 24)));
-
-    return (1103515243 * value + 12345);
-}
-
-/**
- * alloc_entry(): allocate a new QDictEntry
- */
-static QDictEntry *alloc_entry(const char *key, QObject *value)
-{
-    QDictEntry *entry;
-
-    entry = g_malloc0(sizeof(*entry));
-    entry->key = g_strdup(key);
-    entry->value = value;
-
-    return entry;
-}
-
-/**
- * qdict_entry_value(): Return qdict entry value
- *
- * Return weak reference.
- */
-QObject *qdict_entry_value(const QDictEntry *entry)
-{
-    return entry->value;
-}
-
-/**
- * qdict_entry_key(): Return qdict entry key
- *
- * Return a *pointer* to the string, it has to be duplicated before being
- * stored.
- */
-const char *qdict_entry_key(const QDictEntry *entry)
-{
-    return entry->key;
-}
-
-/**
- * qdict_find(): List lookup function
- */
-static QDictEntry *qdict_find(const QDict *qdict,
-                              const char *key, unsigned int bucket)
-{
-    QDictEntry *entry;
-
-    QLIST_FOREACH(entry, &qdict->table[bucket], next)
-        if (!strcmp(entry->key, key))
-            return entry;
-
-    return NULL;
-}
-
-/**
- * qdict_put_obj(): Put a new QObject into the dictionary
- *
- * Insert the pair 'key:value' into 'qdict', if 'key' already exists
- * its 'value' will be replaced.
- *
- * This is done by freeing the reference to the stored QObject and
- * storing the new one in the same entry.
- *
- * NOTE: ownership of 'value' is transferred to the QDict
- */
-void qdict_put_obj(QDict *qdict, const char *key, QObject *value)
-{
-    unsigned int bucket;
-    QDictEntry *entry;
-
-    bucket = tdb_hash(key) % QDICT_BUCKET_MAX;
-    entry = qdict_find(qdict, key, bucket);
-    if (entry) {
-        /* replace key's value */
-        qobject_decref(entry->value);
-        entry->value = value;
-    } else {
-        /* allocate a new entry */
-        entry = alloc_entry(key, value);
-        QLIST_INSERT_HEAD(&qdict->table[bucket], entry, next);
-        qdict->size++;
-    }
-}
-
-/**
- * qdict_get(): Lookup for a given 'key'
- *
- * Return a weak reference to the QObject associated with 'key' if
- * 'key' is present in the dictionary, NULL otherwise.
- */
-QObject *qdict_get(const QDict *qdict, const char *key)
-{
-    QDictEntry *entry;
-
-    entry = qdict_find(qdict, key, tdb_hash(key) % QDICT_BUCKET_MAX);
-    return (entry == NULL ? NULL : entry->value);
-}
-
-/**
- * qdict_haskey(): Check if 'key' exists
- *
- * Return 1 if 'key' exists in the dict, 0 otherwise
- */
-int qdict_haskey(const QDict *qdict, const char *key)
-{
-    unsigned int bucket = tdb_hash(key) % QDICT_BUCKET_MAX;
-    return (qdict_find(qdict, key, bucket) == NULL ? 0 : 1);
-}
-
-/**
- * qdict_size(): Return the size of the dictionary
- */
-size_t qdict_size(const QDict *qdict)
-{
-    return qdict->size;
-}
-
-/**
- * qdict_get_obj(): Get a QObject of a specific type
- */
-static QObject *qdict_get_obj(const QDict *qdict, const char *key,
-                              qtype_code type)
-{
-    QObject *obj;
-
-    obj = qdict_get(qdict, key);
-    assert(obj != NULL);
-    assert(qobject_type(obj) == type);
-
-    return obj;
-}
-
-/**
- * qdict_get_double(): Get an number mapped by 'key'
- *
- * This function assumes that 'key' exists and it stores a
- * QFloat or QInt object.
- *
- * Return number mapped by 'key'.
- */
-double qdict_get_double(const QDict *qdict, const char *key)
-{
-    QObject *obj = qdict_get(qdict, key);
-
-    assert(obj);
-    switch (qobject_type(obj)) {
-    case QTYPE_QFLOAT:
-        return qfloat_get_double(qobject_to_qfloat(obj));
-    case QTYPE_QINT:
-        return qint_get_int(qobject_to_qint(obj));
-    default:
-        abort();
-    }
-}
-
-/**
- * qdict_get_int(): Get an integer mapped by 'key'
- *
- * This function assumes that 'key' exists and it stores a
- * QInt object.
- *
- * Return integer mapped by 'key'.
- */
-int64_t qdict_get_int(const QDict *qdict, const char *key)
-{
-    QObject *obj = qdict_get_obj(qdict, key, QTYPE_QINT);
-    return qint_get_int(qobject_to_qint(obj));
-}
-
-/**
- * qdict_get_bool(): Get a bool mapped by 'key'
- *
- * This function assumes that 'key' exists and it stores a
- * QBool object.
- *
- * Return bool mapped by 'key'.
- */
-int qdict_get_bool(const QDict *qdict, const char *key)
-{
-    QObject *obj = qdict_get_obj(qdict, key, QTYPE_QBOOL);
-    return qbool_get_int(qobject_to_qbool(obj));
-}
-
-/**
- * qdict_get_qlist(): Get the QList mapped by 'key'
- *
- * This function assumes that 'key' exists and it stores a
- * QList object.
- *
- * Return QList mapped by 'key'.
- */
-QList *qdict_get_qlist(const QDict *qdict, const char *key)
-{
-    return qobject_to_qlist(qdict_get_obj(qdict, key, QTYPE_QLIST));
-}
-
-/**
- * qdict_get_qdict(): Get the QDict mapped by 'key'
- *
- * This function assumes that 'key' exists and it stores a
- * QDict object.
- *
- * Return QDict mapped by 'key'.
- */
-QDict *qdict_get_qdict(const QDict *qdict, const char *key)
-{
-    return qobject_to_qdict(qdict_get_obj(qdict, key, QTYPE_QDICT));
-}
-
-/**
- * qdict_get_str(): Get a pointer to the stored string mapped
- * by 'key'
- *
- * This function assumes that 'key' exists and it stores a
- * QString object.
- *
- * Return pointer to the string mapped by 'key'.
- */
-const char *qdict_get_str(const QDict *qdict, const char *key)
-{
-    QObject *obj = qdict_get_obj(qdict, key, QTYPE_QSTRING);
-    return qstring_get_str(qobject_to_qstring(obj));
-}
-
-/**
- * qdict_get_try_int(): Try to get integer mapped by 'key'
- *
- * Return integer mapped by 'key', if it is not present in
- * the dictionary or if the stored object is not of QInt type
- * 'def_value' will be returned.
- */
-int64_t qdict_get_try_int(const QDict *qdict, const char *key,
-                          int64_t def_value)
-{
-    QObject *obj;
-
-    obj = qdict_get(qdict, key);
-    if (!obj || qobject_type(obj) != QTYPE_QINT)
-        return def_value;
-
-    return qint_get_int(qobject_to_qint(obj));
-}
-
-/**
- * qdict_get_try_bool(): Try to get a bool mapped by 'key'
- *
- * Return bool mapped by 'key', if it is not present in the
- * dictionary or if the stored object is not of QBool type
- * 'def_value' will be returned.
- */
-int qdict_get_try_bool(const QDict *qdict, const char *key, int def_value)
-{
-    QObject *obj;
-
-    obj = qdict_get(qdict, key);
-    if (!obj || qobject_type(obj) != QTYPE_QBOOL)
-        return def_value;
-
-    return qbool_get_int(qobject_to_qbool(obj));
-}
-
-/**
- * qdict_get_try_str(): Try to get a pointer to the stored string
- * mapped by 'key'
- *
- * Return a pointer to the string mapped by 'key', if it is not present
- * in the dictionary or if the stored object is not of QString type
- * NULL will be returned.
- */
-const char *qdict_get_try_str(const QDict *qdict, const char *key)
-{
-    QObject *obj;
-
-    obj = qdict_get(qdict, key);
-    if (!obj || qobject_type(obj) != QTYPE_QSTRING)
-        return NULL;
-
-    return qstring_get_str(qobject_to_qstring(obj));
-}
-
-/**
- * qdict_iter(): Iterate over all the dictionary's stored values.
- *
- * This function allows the user to provide an iterator, which will be
- * called for each stored value in the dictionary.
- */
-void qdict_iter(const QDict *qdict,
-                void (*iter)(const char *key, QObject *obj, void *opaque),
-                void *opaque)
-{
-    int i;
-    QDictEntry *entry;
-
-    for (i = 0; i < QDICT_BUCKET_MAX; i++) {
-        QLIST_FOREACH(entry, &qdict->table[i], next)
-            iter(entry->key, entry->value, opaque);
-    }
-}
-
-static QDictEntry *qdict_next_entry(const QDict *qdict, int first_bucket)
-{
-    int i;
-
-    for (i = first_bucket; i < QDICT_BUCKET_MAX; i++) {
-        if (!QLIST_EMPTY(&qdict->table[i])) {
-            return QLIST_FIRST(&qdict->table[i]);
-        }
-    }
-
-    return NULL;
-}
-
-/**
- * qdict_first(): Return first qdict entry for iteration.
- */
-const QDictEntry *qdict_first(const QDict *qdict)
-{
-    return qdict_next_entry(qdict, 0);
-}
-
-/**
- * qdict_next(): Return next qdict entry in an iteration.
- */
-const QDictEntry *qdict_next(const QDict *qdict, const QDictEntry *entry)
-{
-    QDictEntry *ret;
-
-    ret = QLIST_NEXT(entry, next);
-    if (!ret) {
-        unsigned int bucket = tdb_hash(entry->key) % QDICT_BUCKET_MAX;
-        ret = qdict_next_entry(qdict, bucket + 1);
-    }
-
-    return ret;
-}
-
-/**
- * qdict_clone_shallow(): Clones a given QDict. Its entries are not copied, but
- * another reference is added.
- */
-QDict *qdict_clone_shallow(const QDict *src)
-{
-    QDict *dest;
-    QDictEntry *entry;
-    int i;
-
-    dest = qdict_new();
-
-    for (i = 0; i < QDICT_BUCKET_MAX; i++) {
-        QLIST_FOREACH(entry, &src->table[i], next) {
-            qobject_incref(entry->value);
-            qdict_put_obj(dest, entry->key, entry->value);
-        }
-    }
-
-    return dest;
-}
-
-/**
- * qentry_destroy(): Free all the memory allocated by a QDictEntry
- */
-static void qentry_destroy(QDictEntry *e)
-{
-    assert(e != NULL);
-    assert(e->key != NULL);
-    assert(e->value != NULL);
-
-    qobject_decref(e->value);
-    g_free(e->key);
-    g_free(e);
-}
-
-/**
- * qdict_del(): Delete a 'key:value' pair from the dictionary
- *
- * This will destroy all data allocated by this entry.
- */
-void qdict_del(QDict *qdict, const char *key)
-{
-    QDictEntry *entry;
-
-    entry = qdict_find(qdict, key, tdb_hash(key) % QDICT_BUCKET_MAX);
-    if (entry) {
-        QLIST_REMOVE(entry, next);
-        qentry_destroy(entry);
-        qdict->size--;
-    }
-}
-
-/**
- * qdict_destroy_obj(): Free all the memory allocated by a QDict
- */
-static void qdict_destroy_obj(QObject *obj)
-{
-    int i;
-    QDict *qdict;
-
-    assert(obj != NULL);
-    qdict = qobject_to_qdict(obj);
-
-    for (i = 0; i < QDICT_BUCKET_MAX; i++) {
-        QDictEntry *entry = QLIST_FIRST(&qdict->table[i]);
-        while (entry) {
-            QDictEntry *tmp = QLIST_NEXT(entry, next);
-            QLIST_REMOVE(entry, next);
-            qentry_destroy(entry);
-            entry = tmp;
-        }
-    }
-
-    g_free(qdict);
-}
diff --git a/contrib/qemu/qobject/qerror.c b/contrib/qemu/qobject/qerror.c
deleted file mode 100644
index 3aee1cf6a69..00000000000
--- a/contrib/qemu/qobject/qerror.c
+++ /dev/null
@@ -1,156 +0,0 @@
-/*
- * QError Module
- *
- * Copyright (C) 2009 Red Hat Inc.
- *
- * Authors:
- *  Luiz Capitulino <lcapitulino@redhat.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- */
-
-#include "monitor/monitor.h"
-#include "qapi/qmp/qjson.h"
-#include "qapi/qmp/qerror.h"
-#include "qemu-common.h"
-
-static void qerror_destroy_obj(QObject *obj);
-
-static const QType qerror_type = {
-    .code = QTYPE_QERROR,
-    .destroy = qerror_destroy_obj,
-};
-
-/**
- * qerror_new(): Create a new QError
- *
- * Return strong reference.
- */
-static QError *qerror_new(void)
-{
-    QError *qerr;
-
-    qerr = g_malloc0(sizeof(*qerr));
-    QOBJECT_INIT(qerr, &qerror_type);
-
-    return qerr;
-}
-
-/**
- * qerror_from_info(): Create a new QError from error information
- *
- * Return strong reference.
- */
-static QError *qerror_from_info(ErrorClass err_class, const char *fmt,
-                                va_list *va)
-{
-    QError *qerr;
-
-    qerr = qerror_new();
-    loc_save(&qerr->loc);
-
-    qerr->err_msg = g_strdup_vprintf(fmt, *va);
-    qerr->err_class = err_class;
-
-    return qerr;
-}
-
-/**
- * qerror_human(): Format QError data into human-readable string.
- */
-QString *qerror_human(const QError *qerror)
-{
-    return qstring_from_str(qerror->err_msg);
-}
-
-/**
- * qerror_print(): Print QError data
- *
- * This function will print the member 'desc' of the specified QError object,
- * it uses error_report() for this, so that the output is routed to the right
- * place (ie. stderr or Monitor's device).
- */
-static void qerror_print(QError *qerror)
-{
-    QString *qstring = qerror_human(qerror);
-    loc_push_restore(&qerror->loc);
-    error_report("%s", qstring_get_str(qstring));
-    loc_pop(&qerror->loc);
-    QDECREF(qstring);
-}
-
-void qerror_report(ErrorClass eclass, const char *fmt, ...)
-{
-    va_list va;
-    QError *qerror;
-
-    va_start(va, fmt);
-    qerror = qerror_from_info(eclass, fmt, &va);
-    va_end(va);
-
-    if (monitor_cur_is_qmp()) {
-        monitor_set_error(cur_mon, qerror);
-    } else {
-        qerror_print(qerror);
-        QDECREF(qerror);
-    }
-}
-
-/* Evil... */
-struct Error
-{
-    char *msg;
-    ErrorClass err_class;
-};
-
-void qerror_report_err(Error *err)
-{
-    QError *qerr;
-
-    qerr = qerror_new();
-    loc_save(&qerr->loc);
-    qerr->err_msg = g_strdup(err->msg);
-    qerr->err_class = err->err_class;
-
-    if (monitor_cur_is_qmp()) {
-        monitor_set_error(cur_mon, qerr);
-    } else {
-        qerror_print(qerr);
-        QDECREF(qerr);
-    }
-}
-
-void assert_no_error(Error *err)
-{
-    if (err) {
-        qerror_report_err(err);
-        abort();
-    }
-}
-
-/**
- * qobject_to_qerror(): Convert a QObject into a QError
- */
-static QError *qobject_to_qerror(const QObject *obj)
-{
-    if (qobject_type(obj) != QTYPE_QERROR) {
-        return NULL;
-    }
-
-    return container_of(obj, QError, base);
-}
-
-/**
- * qerror_destroy_obj(): Free all memory allocated by a QError
- */
-static void qerror_destroy_obj(QObject *obj)
-{
-    QError *qerr;
-
-    assert(obj != NULL);
-    qerr = qobject_to_qerror(obj);
-
-    g_free(qerr->err_msg);
-    g_free(qerr);
-}
diff --git a/contrib/qemu/qobject/qfloat.c b/contrib/qemu/qobject/qfloat.c
deleted file mode 100644
index 7de0992dba7..00000000000
--- a/contrib/qemu/qobject/qfloat.c
+++ /dev/null
@@ -1,68 +0,0 @@
-/*
- * QFloat Module
- *
- * Copyright IBM, Corp. 2009
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#include "qapi/qmp/qfloat.h"
-#include "qapi/qmp/qobject.h"
-#include "qemu-common.h"
-
-static void qfloat_destroy_obj(QObject *obj);
-
-static const QType qfloat_type = {
-    .code = QTYPE_QFLOAT,
-    .destroy = qfloat_destroy_obj,
-};
-
-/**
- * qfloat_from_int(): Create a new QFloat from a float
- *
- * Return strong reference.
- */
-QFloat *qfloat_from_double(double value)
-{
-    QFloat *qf;
-
-    qf = g_malloc(sizeof(*qf));
-    qf->value = value;
-    QOBJECT_INIT(qf, &qfloat_type);
-
-    return qf;
-}
-
-/**
- * qfloat_get_double(): Get the stored float
- */
-double qfloat_get_double(const QFloat *qf)
-{
-    return qf->value;
-}
-
-/**
- * qobject_to_qfloat(): Convert a QObject into a QFloat
- */
-QFloat *qobject_to_qfloat(const QObject *obj)
-{
-    if (qobject_type(obj) != QTYPE_QFLOAT)
-        return NULL;
-
-    return container_of(obj, QFloat, base);
-}
-
-/**
- * qfloat_destroy_obj(): Free all memory allocated by a
- * QFloat object
- */
-static void qfloat_destroy_obj(QObject *obj)
-{
-    assert(obj != NULL);
-    g_free(qobject_to_qfloat(obj));
-}
diff --git a/contrib/qemu/qobject/qint.c b/contrib/qemu/qobject/qint.c
deleted file mode 100644
index 86b9b04f0b6..00000000000
--- a/contrib/qemu/qobject/qint.c
+++ /dev/null
@@ -1,67 +0,0 @@
-/*
- * QInt Module
- *
- * Copyright (C) 2009 Red Hat Inc.
- *
- * Authors:
- *  Luiz Capitulino <lcapitulino@redhat.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- */
-
-#include "qapi/qmp/qint.h"
-#include "qapi/qmp/qobject.h"
-#include "qemu-common.h"
-
-static void qint_destroy_obj(QObject *obj);
-
-static const QType qint_type = {
-    .code = QTYPE_QINT,
-    .destroy = qint_destroy_obj,
-};
-
-/**
- * qint_from_int(): Create a new QInt from an int64_t
- *
- * Return strong reference.
- */
-QInt *qint_from_int(int64_t value)
-{
-    QInt *qi;
-
-    qi = g_malloc(sizeof(*qi));
-    qi->value = value;
-    QOBJECT_INIT(qi, &qint_type);
-
-    return qi;
-}
-
-/**
- * qint_get_int(): Get the stored integer
- */
-int64_t qint_get_int(const QInt *qi)
-{
-    return qi->value;
-}
-
-/**
- * qobject_to_qint(): Convert a QObject into a QInt
- */
-QInt *qobject_to_qint(const QObject *obj)
-{
-    if (qobject_type(obj) != QTYPE_QINT)
-        return NULL;
-
-    return container_of(obj, QInt, base);
-}
-
-/**
- * qint_destroy_obj(): Free all memory allocated by a
- * QInt object
- */
-static void qint_destroy_obj(QObject *obj)
-{
-    assert(obj != NULL);
-    g_free(qobject_to_qint(obj));
-}
diff --git a/contrib/qemu/qobject/qjson.c b/contrib/qemu/qobject/qjson.c
deleted file mode 100644
index 19085a1bb7f..00000000000
--- a/contrib/qemu/qobject/qjson.c
+++ /dev/null
@@ -1,282 +0,0 @@
-/*
- * QObject JSON integration
- *
- * Copyright IBM, Corp. 2009
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#include "qapi/qmp/json-lexer.h"
-#include "qapi/qmp/json-parser.h"
-#include "qapi/qmp/json-streamer.h"
-#include "qapi/qmp/qjson.h"
-#include "qapi/qmp/qint.h"
-#include "qapi/qmp/qlist.h"
-#include "qapi/qmp/qbool.h"
-#include "qapi/qmp/qfloat.h"
-#include "qapi/qmp/qdict.h"
-
-typedef struct JSONParsingState
-{
-    JSONMessageParser parser;
-    va_list *ap;
-    QObject *result;
-} JSONParsingState;
-
-static void parse_json(JSONMessageParser *parser, QList *tokens)
-{
-    JSONParsingState *s = container_of(parser, JSONParsingState, parser);
-    s->result = json_parser_parse(tokens, s->ap);
-}
-
-QObject *qobject_from_jsonv(const char *string, va_list *ap)
-{
-    JSONParsingState state = {};
-
-    state.ap = ap;
-
-    json_message_parser_init(&state.parser, parse_json);
-    json_message_parser_feed(&state.parser, string, strlen(string));
-    json_message_parser_flush(&state.parser);
-    json_message_parser_destroy(&state.parser);
-
-    return state.result;
-}
-
-QObject *qobject_from_json(const char *string)
-{
-    return qobject_from_jsonv(string, NULL);
-}
-
-/*
- * IMPORTANT: This function aborts on error, thus it must not
- * be used with untrusted arguments.
- */
-QObject *qobject_from_jsonf(const char *string, ...)
-{
-    QObject *obj;
-    va_list ap;
-
-    va_start(ap, string);
-    obj = qobject_from_jsonv(string, &ap);
-    va_end(ap);
-
-    assert(obj != NULL);
-    return obj;
-}
-
-typedef struct ToJsonIterState
-{
-    int indent;
-    int pretty;
-    int count;
-    QString *str;
-} ToJsonIterState;
-
-static void to_json(const QObject *obj, QString *str, int pretty, int indent);
-
-static void to_json_dict_iter(const char *key, QObject *obj, void *opaque)
-{
-    ToJsonIterState *s = opaque;
-    QString *qkey;
-    int j;
-
-    if (s->count)
-        qstring_append(s->str, ", ");
-
-    if (s->pretty) {
-        qstring_append(s->str, "\n");
-        for (j = 0 ; j < s->indent ; j++)
-            qstring_append(s->str, "    ");
-    }
-
-    qkey = qstring_from_str(key);
-    to_json(QOBJECT(qkey), s->str, s->pretty, s->indent);
-    QDECREF(qkey);
-
-    qstring_append(s->str, ": ");
-    to_json(obj, s->str, s->pretty, s->indent);
-    s->count++;
-}
-
-static void to_json_list_iter(QObject *obj, void *opaque)
-{
-    ToJsonIterState *s = opaque;
-    int j;
-
-    if (s->count)
-        qstring_append(s->str, ", ");
-
-    if (s->pretty) {
-        qstring_append(s->str, "\n");
-        for (j = 0 ; j < s->indent ; j++)
-            qstring_append(s->str, "    ");
-    }
-
-    to_json(obj, s->str, s->pretty, s->indent);
-    s->count++;
-}
-
-static void to_json(const QObject *obj, QString *str, int pretty, int indent)
-{
-    switch (qobject_type(obj)) {
-    case QTYPE_QINT: {
-        QInt *val = qobject_to_qint(obj);
-        char buffer[1024];
-
-        snprintf(buffer, sizeof(buffer), "%" PRId64, qint_get_int(val));
-        qstring_append(str, buffer);
-        break;
-    }
-    case QTYPE_QSTRING: {
-        QString *val = qobject_to_qstring(obj);
-        const char *ptr;
-        int cp;
-        char buf[16];
-        char *end;
-
-        ptr = qstring_get_str(val);
-        qstring_append(str, "\"");
-
-        for (; *ptr; ptr = end) {
-            cp = mod_utf8_codepoint(ptr, 6, &end);
-            switch (cp) {
-            case '\"':
-                qstring_append(str, "\\\"");
-                break;
-            case '\\':
-                qstring_append(str, "\\\\");
-                break;
-            case '\b':
-                qstring_append(str, "\\b");
-                break;
-            case '\f':
-                qstring_append(str, "\\f");
-                break;
-            case '\n':
-                qstring_append(str, "\\n");
-                break;
-            case '\r':
-                qstring_append(str, "\\r");
-                break;
-            case '\t':
-                qstring_append(str, "\\t");
-                break;
-            default:
-                if (cp < 0) {
-                    cp = 0xFFFD; /* replacement character */
-                }
-                if (cp > 0xFFFF) {
-                    /* beyond BMP; need a surrogate pair */
-                    snprintf(buf, sizeof(buf), "\\u%04X\\u%04X",
-                             0xD800 + ((cp - 0x10000) >> 10),
-                             0xDC00 + ((cp - 0x10000) & 0x3FF));
-                } else if (cp < 0x20 || cp >= 0x7F) {
-                    snprintf(buf, sizeof(buf), "\\u%04X", cp);
-                } else {
-                    buf[0] = cp;
-                    buf[1] = 0;
-                }
-                qstring_append(str, buf);
-            }
-        };
-
-        qstring_append(str, "\"");
-        break;
-    }
-    case QTYPE_QDICT: {
-        ToJsonIterState s;
-        QDict *val = qobject_to_qdict(obj);
-
-        s.count = 0;
-        s.str = str;
-        s.indent = indent + 1;
-        s.pretty = pretty;
-        qstring_append(str, "{");
-        qdict_iter(val, to_json_dict_iter, &s);
-        if (pretty) {
-            int j;
-            qstring_append(str, "\n");
-            for (j = 0 ; j < indent ; j++)
-                qstring_append(str, "    ");
-        }
-        qstring_append(str, "}");
-        break;
-    }
-    case QTYPE_QLIST: {
-        ToJsonIterState s;
-        QList *val = qobject_to_qlist(obj);
-
-        s.count = 0;
-        s.str = str;
-        s.indent = indent + 1;
-        s.pretty = pretty;
-        qstring_append(str, "[");
-        qlist_iter(val, (void *)to_json_list_iter, &s);
-        if (pretty) {
-            int j;
-            qstring_append(str, "\n");
-            for (j = 0 ; j < indent ; j++)
-                qstring_append(str, "    ");
-        }
-        qstring_append(str, "]");
-        break;
-    }
-    case QTYPE_QFLOAT: {
-        QFloat *val = qobject_to_qfloat(obj);
-        char buffer[1024];
-        int len;
-
-        len = snprintf(buffer, sizeof(buffer), "%f", qfloat_get_double(val));
-        while (len > 0 && buffer[len - 1] == '0') {
-            len--;
-        }
-
-        if (len && buffer[len - 1] == '.') {
-            buffer[len - 1] = 0;
-        } else {
-            buffer[len] = 0;
-        }
-        
-        qstring_append(str, buffer);
-        break;
-    }
-    case QTYPE_QBOOL: {
-        QBool *val = qobject_to_qbool(obj);
-
-        if (qbool_get_int(val)) {
-            qstring_append(str, "true");
-        } else {
-            qstring_append(str, "false");
-        }
-        break;
-    }
-    case QTYPE_QERROR:
-        /* XXX: should QError be emitted? */
-    case QTYPE_NONE:
-        break;
-    }
-}
-
-QString *qobject_to_json(const QObject *obj)
-{
-    QString *str = qstring_new();
-
-    to_json(obj, str, 0, 0);
-
-    return str;
-}
-
-QString *qobject_to_json_pretty(const QObject *obj)
-{
-    QString *str = qstring_new();
-
-    to_json(obj, str, 1, 0);
-
-    return str;
-}
diff --git a/contrib/qemu/qobject/qlist.c b/contrib/qemu/qobject/qlist.c
deleted file mode 100644
index 1ced0de58e2..00000000000
--- a/contrib/qemu/qobject/qlist.c
+++ /dev/null
@@ -1,170 +0,0 @@
-/*
- * QList Module
- *
- * Copyright (C) 2009 Red Hat Inc.
- *
- * Authors:
- *  Luiz Capitulino <lcapitulino@redhat.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- */
-
-#include "qapi/qmp/qlist.h"
-#include "qapi/qmp/qobject.h"
-#include "qemu/queue.h"
-#include "qemu-common.h"
-
-static void qlist_destroy_obj(QObject *obj);
-
-static const QType qlist_type = {
-    .code = QTYPE_QLIST,
-    .destroy = qlist_destroy_obj,
-};
- 
-/**
- * qlist_new(): Create a new QList
- *
- * Return strong reference.
- */
-QList *qlist_new(void)
-{
-    QList *qlist;
-
-    qlist = g_malloc(sizeof(*qlist));
-    QTAILQ_INIT(&qlist->head);
-    QOBJECT_INIT(qlist, &qlist_type);
-
-    return qlist;
-}
-
-static void qlist_copy_elem(QObject *obj, void *opaque)
-{
-    QList *dst = opaque;
-
-    qobject_incref(obj);
-    qlist_append_obj(dst, obj);
-}
-
-QList *qlist_copy(QList *src)
-{
-    QList *dst = qlist_new();
-
-    qlist_iter(src, qlist_copy_elem, dst);
-
-    return dst;
-}
-
-/**
- * qlist_append_obj(): Append an QObject into QList
- *
- * NOTE: ownership of 'value' is transferred to the QList
- */
-void qlist_append_obj(QList *qlist, QObject *value)
-{
-    QListEntry *entry;
-
-    entry = g_malloc(sizeof(*entry));
-    entry->value = value;
-
-    QTAILQ_INSERT_TAIL(&qlist->head, entry, next);
-}
-
-/**
- * qlist_iter(): Iterate over all the list's stored values.
- *
- * This function allows the user to provide an iterator, which will be
- * called for each stored value in the list.
- */
-void qlist_iter(const QList *qlist,
-                void (*iter)(QObject *obj, void *opaque), void *opaque)
-{
-    QListEntry *entry;
-
-    QTAILQ_FOREACH(entry, &qlist->head, next)
-        iter(entry->value, opaque);
-}
-
-QObject *qlist_pop(QList *qlist)
-{
-    QListEntry *entry;
-    QObject *ret;
-
-    if (qlist == NULL || QTAILQ_EMPTY(&qlist->head)) {
-        return NULL;
-    }
-
-    entry = QTAILQ_FIRST(&qlist->head);
-    QTAILQ_REMOVE(&qlist->head, entry, next);
-
-    ret = entry->value;
-    g_free(entry);
-
-    return ret;
-}
-
-QObject *qlist_peek(QList *qlist)
-{
-    QListEntry *entry;
-    QObject *ret;
-
-    if (qlist == NULL || QTAILQ_EMPTY(&qlist->head)) {
-        return NULL;
-    }
-
-    entry = QTAILQ_FIRST(&qlist->head);
-
-    ret = entry->value;
-
-    return ret;
-}
-
-int qlist_empty(const QList *qlist)
-{
-    return QTAILQ_EMPTY(&qlist->head);
-}
-
-static void qlist_size_iter(QObject *obj, void *opaque)
-{
-    size_t *count = opaque;
-    (*count)++;
-}
-
-size_t qlist_size(const QList *qlist)
-{
-    size_t count = 0;
-    qlist_iter(qlist, qlist_size_iter, &count);
-    return count;
-}
-
-/**
- * qobject_to_qlist(): Convert a QObject into a QList
- */
-QList *qobject_to_qlist(const QObject *obj)
-{
-    if (qobject_type(obj) != QTYPE_QLIST) {
-        return NULL;
-    }
-
-    return container_of(obj, QList, base);
-}
-
-/**
- * qlist_destroy_obj(): Free all the memory allocated by a QList
- */
-static void qlist_destroy_obj(QObject *obj)
-{
-    QList *qlist;
-    QListEntry *entry, *next_entry;
-
-    assert(obj != NULL);
-    qlist = qobject_to_qlist(obj);
-
-    QTAILQ_FOREACH_SAFE(entry, &qlist->head, next, next_entry) {
-        QTAILQ_REMOVE(&qlist->head, entry, next);
-        qobject_decref(entry->value);
-        g_free(entry);
-    }
-
-    g_free(qlist);
-}
diff --git a/contrib/qemu/qobject/qstring.c b/contrib/qemu/qobject/qstring.c
deleted file mode 100644
index 607b7a142c6..00000000000
--- a/contrib/qemu/qobject/qstring.c
+++ /dev/null
@@ -1,149 +0,0 @@
-/*
- * QString Module
- *
- * Copyright (C) 2009 Red Hat Inc.
- *
- * Authors:
- *  Luiz Capitulino <lcapitulino@redhat.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- */
-
-#include "qapi/qmp/qobject.h"
-#include "qapi/qmp/qstring.h"
-#include "qemu-common.h"
-
-static void qstring_destroy_obj(QObject *obj);
-
-static const QType qstring_type = {
-    .code = QTYPE_QSTRING,
-    .destroy = qstring_destroy_obj,
-};
-
-/**
- * qstring_new(): Create a new empty QString
- *
- * Return strong reference.
- */
-QString *qstring_new(void)
-{
-    return qstring_from_str("");
-}
-
-/**
- * qstring_get_length(): Get the length of a QString
- */
-size_t qstring_get_length(const QString *qstring)
-{
-    return qstring->length;
-}
-
-/**
- * qstring_from_substr(): Create a new QString from a C string substring
- *
- * Return string reference
- */
-QString *qstring_from_substr(const char *str, int start, int end)
-{
-    QString *qstring;
-
-    qstring = g_malloc(sizeof(*qstring));
-
-    qstring->length = end - start + 1;
-    qstring->capacity = qstring->length;
-
-    qstring->string = g_malloc(qstring->capacity + 1);
-    memcpy(qstring->string, str + start, qstring->length);
-    qstring->string[qstring->length] = 0;
-
-    QOBJECT_INIT(qstring, &qstring_type);
-
-    return qstring;
-}
-
-/**
- * qstring_from_str(): Create a new QString from a regular C string
- *
- * Return strong reference.
- */
-QString *qstring_from_str(const char *str)
-{
-    return qstring_from_substr(str, 0, strlen(str) - 1);
-}
-
-static void capacity_increase(QString *qstring, size_t len)
-{
-    if (qstring->capacity < (qstring->length + len)) {
-        qstring->capacity += len;
-        qstring->capacity *= 2; /* use exponential growth */
-
-        qstring->string = g_realloc(qstring->string, qstring->capacity + 1);
-    }
-}
-
-/* qstring_append(): Append a C string to a QString
- */
-void qstring_append(QString *qstring, const char *str)
-{
-    size_t len = strlen(str);
-
-    capacity_increase(qstring, len);
-    memcpy(qstring->string + qstring->length, str, len);
-    qstring->length += len;
-    qstring->string[qstring->length] = 0;
-}
-
-void qstring_append_int(QString *qstring, int64_t value)
-{
-    char num[32];
-
-    snprintf(num, sizeof(num), "%" PRId64, value);
-    qstring_append(qstring, num);
-}
-
-/**
- * qstring_append_chr(): Append a C char to a QString
- */
-void qstring_append_chr(QString *qstring, int c)
-{
-    capacity_increase(qstring, 1);
-    qstring->string[qstring->length++] = c;
-    qstring->string[qstring->length] = 0;
-}
-
-/**
- * qobject_to_qstring(): Convert a QObject to a QString
- */
-QString *qobject_to_qstring(const QObject *obj)
-{
-    if (qobject_type(obj) != QTYPE_QSTRING)
-        return NULL;
-
-    return container_of(obj, QString, base);
-}
-
-/**
- * qstring_get_str(): Return a pointer to the stored string
- *
- * NOTE: Should be used with caution, if the object is deallocated
- * this pointer becomes invalid.
- */
-const char *qstring_get_str(const QString *qstring)
-{
-    return qstring->string;
-}
-
-/**
- * qstring_destroy_obj(): Free all memory allocated by a QString
- * object
- */
-static void qstring_destroy_obj(QObject *obj)
-{
-    QString *qs;
-
-    assert(obj != NULL);
-    qs = qobject_to_qstring(obj);
-    g_free(qs->string);
-    g_free(qs);
-}
diff --git a/contrib/qemu/trace/generated-tracers.h b/contrib/qemu/trace/generated-tracers.h
deleted file mode 100644
index b512660f358..00000000000
--- a/contrib/qemu/trace/generated-tracers.h
+++ /dev/null
@@ -1,3759 +0,0 @@
-/* This file is autogenerated by tracetool, do not edit. */
-
-#ifndef TRACE__GENERATED_TRACERS_H
-#define TRACE__GENERATED_TRACERS_H
-
-#include "qemu-common.h"
-
-static inline void trace_qxl_interface_set_mm_time(int qid, uint32_t mm_time)
-{
-}
-
-static inline void trace_qxl_io_write_vga(int qid, const char * mode, uint32_t addr, uint32_t val)
-{
-}
-
-static inline void trace_g_malloc(size_t size, void * ptr)
-{
-}
-
-static inline void trace_g_realloc(void * ptr, size_t size, void * newptr)
-{
-}
-
-static inline void trace_g_free(void * ptr)
-{
-}
-
-static inline void trace_qemu_memalign(size_t alignment, size_t size, void * ptr)
-{
-}
-
-static inline void trace_qemu_anon_ram_alloc(size_t size, void * ptr)
-{
-}
-
-static inline void trace_qemu_vfree(void * ptr)
-{
-}
-
-static inline void trace_qemu_anon_ram_free(void * ptr, size_t size)
-{
-}
-
-static inline void trace_virtqueue_fill(void * vq, const void * elem, unsigned int len, unsigned int idx)
-{
-}
-
-static inline void trace_virtqueue_flush(void * vq, unsigned int count)
-{
-}
-
-static inline void trace_virtqueue_pop(void * vq, void * elem, unsigned int in_num, unsigned int out_num)
-{
-}
-
-static inline void trace_virtio_queue_notify(void * vdev, int n, void * vq)
-{
-}
-
-static inline void trace_virtio_irq(void * vq)
-{
-}
-
-static inline void trace_virtio_notify(void * vdev, void * vq)
-{
-}
-
-static inline void trace_virtio_set_status(void * vdev, uint8_t val)
-{
-}
-
-static inline void trace_virtio_serial_send_control_event(unsigned int port, uint16_t event, uint16_t value)
-{
-}
-
-static inline void trace_virtio_serial_throttle_port(unsigned int port, bool throttle)
-{
-}
-
-static inline void trace_virtio_serial_handle_control_message(uint16_t event, uint16_t value)
-{
-}
-
-static inline void trace_virtio_serial_handle_control_message_port(unsigned int port)
-{
-}
-
-static inline void trace_virtio_console_flush_buf(unsigned int port, size_t len, ssize_t ret)
-{
-}
-
-static inline void trace_virtio_console_chr_read(unsigned int port, int size)
-{
-}
-
-static inline void trace_virtio_console_chr_event(unsigned int port, int event)
-{
-}
-
-static inline void trace_bdrv_open_common(void * bs, const char * filename, int flags, const char * format_name)
-{
-}
-
-static inline void trace_multiwrite_cb(void * mcb, int ret)
-{
-}
-
-static inline void trace_bdrv_aio_multiwrite(void * mcb, int num_callbacks, int num_reqs)
-{
-}
-
-static inline void trace_bdrv_aio_discard(void * bs, int64_t sector_num, int nb_sectors, void * opaque)
-{
-}
-
-static inline void trace_bdrv_aio_flush(void * bs, void * opaque)
-{
-}
-
-static inline void trace_bdrv_aio_readv(void * bs, int64_t sector_num, int nb_sectors, void * opaque)
-{
-}
-
-static inline void trace_bdrv_aio_writev(void * bs, int64_t sector_num, int nb_sectors, void * opaque)
-{
-}
-
-static inline void trace_bdrv_lock_medium(void * bs, bool locked)
-{
-}
-
-static inline void trace_bdrv_co_readv(void * bs, int64_t sector_num, int nb_sector)
-{
-}
-
-static inline void trace_bdrv_co_copy_on_readv(void * bs, int64_t sector_num, int nb_sector)
-{
-}
-
-static inline void trace_bdrv_co_writev(void * bs, int64_t sector_num, int nb_sector)
-{
-}
-
-static inline void trace_bdrv_co_write_zeroes(void * bs, int64_t sector_num, int nb_sector)
-{
-}
-
-static inline void trace_bdrv_co_io_em(void * bs, int64_t sector_num, int nb_sectors, int is_write, void * acb)
-{
-}
-
-static inline void trace_bdrv_co_do_copy_on_readv(void * bs, int64_t sector_num, int nb_sectors, int64_t cluster_sector_num, int cluster_nb_sectors)
-{
-}
-
-static inline void trace_stream_one_iteration(void * s, int64_t sector_num, int nb_sectors, int is_allocated)
-{
-}
-
-static inline void trace_stream_start(void * bs, void * base, void * s, void * co, void * opaque)
-{
-}
-
-static inline void trace_commit_one_iteration(void * s, int64_t sector_num, int nb_sectors, int is_allocated)
-{
-}
-
-static inline void trace_commit_start(void * bs, void * base, void * top, void * s, void * co, void * opaque)
-{
-}
-
-static inline void trace_mirror_start(void * bs, void * s, void * co, void * opaque)
-{
-}
-
-static inline void trace_mirror_restart_iter(void * s, int64_t cnt)
-{
-}
-
-static inline void trace_mirror_before_flush(void * s)
-{
-}
-
-static inline void trace_mirror_before_drain(void * s, int64_t cnt)
-{
-}
-
-static inline void trace_mirror_before_sleep(void * s, int64_t cnt, int synced)
-{
-}
-
-static inline void trace_mirror_one_iteration(void * s, int64_t sector_num, int nb_sectors)
-{
-}
-
-static inline void trace_mirror_cow(void * s, int64_t sector_num)
-{
-}
-
-static inline void trace_mirror_iteration_done(void * s, int64_t sector_num, int nb_sectors, int ret)
-{
-}
-
-static inline void trace_mirror_yield(void * s, int64_t cnt, int buf_free_count, int in_flight)
-{
-}
-
-static inline void trace_mirror_yield_in_flight(void * s, int64_t sector_num, int in_flight)
-{
-}
-
-static inline void trace_mirror_yield_buf_busy(void * s, int nb_chunks, int in_flight)
-{
-}
-
-static inline void trace_mirror_break_buf_busy(void * s, int nb_chunks, int in_flight)
-{
-}
-
-static inline void trace_backup_do_cow_enter(void * job, int64_t start, int64_t sector_num, int nb_sectors)
-{
-}
-
-static inline void trace_backup_do_cow_return(void * job, int64_t sector_num, int nb_sectors, int ret)
-{
-}
-
-static inline void trace_backup_do_cow_skip(void * job, int64_t start)
-{
-}
-
-static inline void trace_backup_do_cow_process(void * job, int64_t start)
-{
-}
-
-static inline void trace_backup_do_cow_read_fail(void * job, int64_t start, int ret)
-{
-}
-
-static inline void trace_backup_do_cow_write_fail(void * job, int64_t start, int ret)
-{
-}
-
-static inline void trace_qmp_block_job_cancel(void * job)
-{
-}
-
-static inline void trace_qmp_block_job_pause(void * job)
-{
-}
-
-static inline void trace_qmp_block_job_resume(void * job)
-{
-}
-
-static inline void trace_qmp_block_job_complete(void * job)
-{
-}
-
-static inline void trace_block_job_cb(void * bs, void * job, int ret)
-{
-}
-
-static inline void trace_qmp_block_stream(void * bs, void * job)
-{
-}
-
-static inline void trace_virtio_blk_req_complete(void * req, int status)
-{
-}
-
-static inline void trace_virtio_blk_rw_complete(void * req, int ret)
-{
-}
-
-static inline void trace_virtio_blk_handle_write(void * req, uint64_t sector, size_t nsectors)
-{
-}
-
-static inline void trace_virtio_blk_handle_read(void * req, uint64_t sector, size_t nsectors)
-{
-}
-
-static inline void trace_virtio_blk_data_plane_start(void * s)
-{
-}
-
-static inline void trace_virtio_blk_data_plane_stop(void * s)
-{
-}
-
-static inline void trace_virtio_blk_data_plane_process_request(void * s, unsigned int out_num, unsigned int in_num, unsigned int head)
-{
-}
-
-static inline void trace_virtio_blk_data_plane_complete_request(void * s, unsigned int head, int ret)
-{
-}
-
-static inline void trace_vring_setup(uint64_t physical, void * desc, void * avail, void * used)
-{
-}
-
-static inline void trace_thread_pool_submit(void * pool, void * req, void * opaque)
-{
-}
-
-static inline void trace_thread_pool_complete(void * pool, void * req, void * opaque, int ret)
-{
-}
-
-static inline void trace_thread_pool_cancel(void * req, void * opaque)
-{
-}
-
-static inline void trace_paio_submit(void * acb, void * opaque, int64_t sector_num, int nb_sectors, int type)
-{
-}
-
-static inline void trace_paio_complete(void * acb, void * opaque, int ret)
-{
-}
-
-static inline void trace_paio_cancel(void * acb, void * opaque)
-{
-}
-
-static inline void trace_cpu_in(unsigned int addr, unsigned int val)
-{
-}
-
-static inline void trace_cpu_out(unsigned int addr, unsigned int val)
-{
-}
-
-static inline void trace_balloon_event(void * opaque, unsigned long addr)
-{
-}
-
-static inline void trace_apic_local_deliver(int vector, uint32_t lvt)
-{
-}
-
-static inline void trace_apic_deliver_irq(uint8_t dest, uint8_t dest_mode, uint8_t delivery_mode, uint8_t vector_num, uint8_t trigger_mode)
-{
-}
-
-static inline void trace_cpu_set_apic_base(uint64_t val)
-{
-}
-
-static inline void trace_cpu_get_apic_base(uint64_t val)
-{
-}
-
-static inline void trace_apic_mem_readl(uint64_t addr, uint32_t val)
-{
-}
-
-static inline void trace_apic_mem_writel(uint64_t addr, uint32_t val)
-{
-}
-
-static inline void trace_apic_report_irq_delivered(int apic_irq_delivered)
-{
-}
-
-static inline void trace_apic_reset_irq_delivered(int apic_irq_delivered)
-{
-}
-
-static inline void trace_apic_get_irq_delivered(int apic_irq_delivered)
-{
-}
-
-static inline void trace_cs4231_mem_readl_dreg(uint32_t reg, uint32_t ret)
-{
-}
-
-static inline void trace_cs4231_mem_readl_reg(uint32_t reg, uint32_t ret)
-{
-}
-
-static inline void trace_cs4231_mem_writel_reg(uint32_t reg, uint32_t old, uint32_t val)
-{
-}
-
-static inline void trace_cs4231_mem_writel_dreg(uint32_t reg, uint32_t old, uint32_t val)
-{
-}
-
-static inline void trace_nvram_read(uint32_t addr, uint32_t ret)
-{
-}
-
-static inline void trace_nvram_write(uint32_t addr, uint32_t old, uint32_t val)
-{
-}
-
-static inline void trace_ecc_mem_writel_mer(uint32_t val)
-{
-}
-
-static inline void trace_ecc_mem_writel_mdr(uint32_t val)
-{
-}
-
-static inline void trace_ecc_mem_writel_mfsr(uint32_t val)
-{
-}
-
-static inline void trace_ecc_mem_writel_vcr(uint32_t val)
-{
-}
-
-static inline void trace_ecc_mem_writel_dr(uint32_t val)
-{
-}
-
-static inline void trace_ecc_mem_writel_ecr0(uint32_t val)
-{
-}
-
-static inline void trace_ecc_mem_writel_ecr1(uint32_t val)
-{
-}
-
-static inline void trace_ecc_mem_readl_mer(uint32_t ret)
-{
-}
-
-static inline void trace_ecc_mem_readl_mdr(uint32_t ret)
-{
-}
-
-static inline void trace_ecc_mem_readl_mfsr(uint32_t ret)
-{
-}
-
-static inline void trace_ecc_mem_readl_vcr(uint32_t ret)
-{
-}
-
-static inline void trace_ecc_mem_readl_mfar0(uint32_t ret)
-{
-}
-
-static inline void trace_ecc_mem_readl_mfar1(uint32_t ret)
-{
-}
-
-static inline void trace_ecc_mem_readl_dr(uint32_t ret)
-{
-}
-
-static inline void trace_ecc_mem_readl_ecr0(uint32_t ret)
-{
-}
-
-static inline void trace_ecc_mem_readl_ecr1(uint32_t ret)
-{
-}
-
-static inline void trace_ecc_diag_mem_writeb(uint64_t addr, uint32_t val)
-{
-}
-
-static inline void trace_ecc_diag_mem_readb(uint64_t addr, uint32_t ret)
-{
-}
-
-static inline void trace_fw_cfg_write(void * s, uint8_t value)
-{
-}
-
-static inline void trace_fw_cfg_select(void * s, uint16_t key, int ret)
-{
-}
-
-static inline void trace_fw_cfg_read(void * s, uint8_t ret)
-{
-}
-
-static inline void trace_fw_cfg_add_file_dupe(void * s, char * name)
-{
-}
-
-static inline void trace_fw_cfg_add_file(void * s, int index, char * name, size_t len)
-{
-}
-
-static inline void trace_hd_geometry_lchs_guess(void * bs, int cyls, int heads, int secs)
-{
-}
-
-static inline void trace_hd_geometry_guess(void * bs, uint32_t cyls, uint32_t heads, uint32_t secs, int trans)
-{
-}
-
-static inline void trace_jazz_led_read(uint64_t addr, uint8_t val)
-{
-}
-
-static inline void trace_jazz_led_write(uint64_t addr, uint8_t new)
-{
-}
-
-static inline void trace_lance_mem_readw(uint64_t addr, uint32_t ret)
-{
-}
-
-static inline void trace_lance_mem_writew(uint64_t addr, uint32_t val)
-{
-}
-
-static inline void trace_slavio_intctl_mem_readl(uint32_t cpu, uint64_t addr, uint32_t ret)
-{
-}
-
-static inline void trace_slavio_intctl_mem_writel(uint32_t cpu, uint64_t addr, uint32_t val)
-{
-}
-
-static inline void trace_slavio_intctl_mem_writel_clear(uint32_t cpu, uint32_t val, uint32_t intreg_pending)
-{
-}
-
-static inline void trace_slavio_intctl_mem_writel_set(uint32_t cpu, uint32_t val, uint32_t intreg_pending)
-{
-}
-
-static inline void trace_slavio_intctlm_mem_readl(uint64_t addr, uint32_t ret)
-{
-}
-
-static inline void trace_slavio_intctlm_mem_writel(uint64_t addr, uint32_t val)
-{
-}
-
-static inline void trace_slavio_intctlm_mem_writel_enable(uint32_t val, uint32_t intregm_disabled)
-{
-}
-
-static inline void trace_slavio_intctlm_mem_writel_disable(uint32_t val, uint32_t intregm_disabled)
-{
-}
-
-static inline void trace_slavio_intctlm_mem_writel_target(uint32_t cpu)
-{
-}
-
-static inline void trace_slavio_check_interrupts(uint32_t pending, uint32_t intregm_disabled)
-{
-}
-
-static inline void trace_slavio_set_irq(uint32_t target_cpu, int irq, uint32_t pil, int level)
-{
-}
-
-static inline void trace_slavio_set_timer_irq_cpu(int cpu, int level)
-{
-}
-
-static inline void trace_slavio_misc_update_irq_raise(void)
-{
-}
-
-static inline void trace_slavio_misc_update_irq_lower(void)
-{
-}
-
-static inline void trace_slavio_set_power_fail(int power_failing, uint8_t config)
-{
-}
-
-static inline void trace_slavio_cfg_mem_writeb(uint32_t val)
-{
-}
-
-static inline void trace_slavio_cfg_mem_readb(uint32_t ret)
-{
-}
-
-static inline void trace_slavio_diag_mem_writeb(uint32_t val)
-{
-}
-
-static inline void trace_slavio_diag_mem_readb(uint32_t ret)
-{
-}
-
-static inline void trace_slavio_mdm_mem_writeb(uint32_t val)
-{
-}
-
-static inline void trace_slavio_mdm_mem_readb(uint32_t ret)
-{
-}
-
-static inline void trace_slavio_aux1_mem_writeb(uint32_t val)
-{
-}
-
-static inline void trace_slavio_aux1_mem_readb(uint32_t ret)
-{
-}
-
-static inline void trace_slavio_aux2_mem_writeb(uint32_t val)
-{
-}
-
-static inline void trace_slavio_aux2_mem_readb(uint32_t ret)
-{
-}
-
-static inline void trace_apc_mem_writeb(uint32_t val)
-{
-}
-
-static inline void trace_apc_mem_readb(uint32_t ret)
-{
-}
-
-static inline void trace_slavio_sysctrl_mem_writel(uint32_t val)
-{
-}
-
-static inline void trace_slavio_sysctrl_mem_readl(uint32_t ret)
-{
-}
-
-static inline void trace_slavio_led_mem_writew(uint32_t val)
-{
-}
-
-static inline void trace_slavio_led_mem_readw(uint32_t ret)
-{
-}
-
-static inline void trace_slavio_timer_get_out(uint64_t limit, uint32_t counthigh, uint32_t count)
-{
-}
-
-static inline void trace_slavio_timer_irq(uint32_t counthigh, uint32_t count)
-{
-}
-
-static inline void trace_slavio_timer_mem_readl_invalid(uint64_t addr)
-{
-}
-
-static inline void trace_slavio_timer_mem_readl(uint64_t addr, uint32_t ret)
-{
-}
-
-static inline void trace_slavio_timer_mem_writel(uint64_t addr, uint32_t val)
-{
-}
-
-static inline void trace_slavio_timer_mem_writel_limit(unsigned int timer_index, uint64_t count)
-{
-}
-
-static inline void trace_slavio_timer_mem_writel_counter_invalid(void)
-{
-}
-
-static inline void trace_slavio_timer_mem_writel_status_start(unsigned int timer_index)
-{
-}
-
-static inline void trace_slavio_timer_mem_writel_status_stop(unsigned int timer_index)
-{
-}
-
-static inline void trace_slavio_timer_mem_writel_mode_user(unsigned int timer_index)
-{
-}
-
-static inline void trace_slavio_timer_mem_writel_mode_counter(unsigned int timer_index)
-{
-}
-
-static inline void trace_slavio_timer_mem_writel_mode_invalid(void)
-{
-}
-
-static inline void trace_slavio_timer_mem_writel_invalid(uint64_t addr)
-{
-}
-
-static inline void trace_ledma_memory_read(uint64_t addr)
-{
-}
-
-static inline void trace_ledma_memory_write(uint64_t addr)
-{
-}
-
-static inline void trace_sparc32_dma_set_irq_raise(void)
-{
-}
-
-static inline void trace_sparc32_dma_set_irq_lower(void)
-{
-}
-
-static inline void trace_espdma_memory_read(uint32_t addr)
-{
-}
-
-static inline void trace_espdma_memory_write(uint32_t addr)
-{
-}
-
-static inline void trace_sparc32_dma_mem_readl(uint64_t addr, uint32_t ret)
-{
-}
-
-static inline void trace_sparc32_dma_mem_writel(uint64_t addr, uint32_t old, uint32_t val)
-{
-}
-
-static inline void trace_sparc32_dma_enable_raise(void)
-{
-}
-
-static inline void trace_sparc32_dma_enable_lower(void)
-{
-}
-
-static inline void trace_sun4m_cpu_interrupt(unsigned int level)
-{
-}
-
-static inline void trace_sun4m_cpu_reset_interrupt(unsigned int level)
-{
-}
-
-static inline void trace_sun4m_cpu_set_irq_raise(int level)
-{
-}
-
-static inline void trace_sun4m_cpu_set_irq_lower(int level)
-{
-}
-
-static inline void trace_sun4m_iommu_mem_readl(uint64_t addr, uint32_t ret)
-{
-}
-
-static inline void trace_sun4m_iommu_mem_writel(uint64_t addr, uint32_t val)
-{
-}
-
-static inline void trace_sun4m_iommu_mem_writel_ctrl(uint64_t iostart)
-{
-}
-
-static inline void trace_sun4m_iommu_mem_writel_tlbflush(uint32_t val)
-{
-}
-
-static inline void trace_sun4m_iommu_mem_writel_pgflush(uint32_t val)
-{
-}
-
-static inline void trace_sun4m_iommu_page_get_flags(uint64_t pa, uint64_t iopte, uint32_t ret)
-{
-}
-
-static inline void trace_sun4m_iommu_translate_pa(uint64_t addr, uint64_t pa, uint32_t iopte)
-{
-}
-
-static inline void trace_sun4m_iommu_bad_addr(uint64_t addr)
-{
-}
-
-static inline void trace_usb_packet_state_change(int bus, const char * port, int ep, void * p, const char * o, const char * n)
-{
-}
-
-static inline void trace_usb_packet_state_fault(int bus, const char * port, int ep, void * p, const char * o, const char * n)
-{
-}
-
-static inline void trace_usb_port_claim(int bus, const char * port)
-{
-}
-
-static inline void trace_usb_port_attach(int bus, const char * port, const char * devspeed, const char * portspeed)
-{
-}
-
-static inline void trace_usb_port_detach(int bus, const char * port)
-{
-}
-
-static inline void trace_usb_port_release(int bus, const char * port)
-{
-}
-
-static inline void trace_usb_ehci_reset(void)
-{
-}
-
-static inline void trace_usb_ehci_opreg_read(uint32_t addr, const char * str, uint32_t val)
-{
-}
-
-static inline void trace_usb_ehci_opreg_write(uint32_t addr, const char * str, uint32_t val)
-{
-}
-
-static inline void trace_usb_ehci_opreg_change(uint32_t addr, const char * str, uint32_t new, uint32_t old)
-{
-}
-
-static inline void trace_usb_ehci_portsc_read(uint32_t addr, uint32_t port, uint32_t val)
-{
-}
-
-static inline void trace_usb_ehci_portsc_write(uint32_t addr, uint32_t port, uint32_t val)
-{
-}
-
-static inline void trace_usb_ehci_portsc_change(uint32_t addr, uint32_t port, uint32_t new, uint32_t old)
-{
-}
-
-static inline void trace_usb_ehci_usbsts(const char * sts, int state)
-{
-}
-
-static inline void trace_usb_ehci_state(const char * schedule, const char * state)
-{
-}
-
-static inline void trace_usb_ehci_qh_ptrs(void * q, uint32_t addr, uint32_t nxt, uint32_t c_qtd, uint32_t n_qtd, uint32_t a_qtd)
-{
-}
-
-static inline void trace_usb_ehci_qh_fields(uint32_t addr, int rl, int mplen, int eps, int ep, int devaddr)
-{
-}
-
-static inline void trace_usb_ehci_qh_bits(uint32_t addr, int c, int h, int dtc, int i)
-{
-}
-
-static inline void trace_usb_ehci_qtd_ptrs(void * q, uint32_t addr, uint32_t nxt, uint32_t altnext)
-{
-}
-
-static inline void trace_usb_ehci_qtd_fields(uint32_t addr, int tbytes, int cpage, int cerr, int pid)
-{
-}
-
-static inline void trace_usb_ehci_qtd_bits(uint32_t addr, int ioc, int active, int halt, int babble, int xacterr)
-{
-}
-
-static inline void trace_usb_ehci_itd(uint32_t addr, uint32_t nxt, uint32_t mplen, uint32_t mult, uint32_t ep, uint32_t devaddr)
-{
-}
-
-static inline void trace_usb_ehci_sitd(uint32_t addr, uint32_t nxt, uint32_t active)
-{
-}
-
-static inline void trace_usb_ehci_port_attach(uint32_t port, const char * owner, const char * device)
-{
-}
-
-static inline void trace_usb_ehci_port_detach(uint32_t port, const char * owner)
-{
-}
-
-static inline void trace_usb_ehci_port_reset(uint32_t port, int enable)
-{
-}
-
-static inline void trace_usb_ehci_data(int rw, uint32_t cpage, uint32_t offset, uint32_t addr, uint32_t len, uint32_t bufpos)
-{
-}
-
-static inline void trace_usb_ehci_queue_action(void * q, const char * action)
-{
-}
-
-static inline void trace_usb_ehci_packet_action(void * q, void * p, const char * action)
-{
-}
-
-static inline void trace_usb_ehci_irq(uint32_t level, uint32_t frindex, uint32_t sts, uint32_t mask)
-{
-}
-
-static inline void trace_usb_ehci_guest_bug(const char * reason)
-{
-}
-
-static inline void trace_usb_ehci_doorbell_ring(void)
-{
-}
-
-static inline void trace_usb_ehci_doorbell_ack(void)
-{
-}
-
-static inline void trace_usb_ehci_dma_error(void)
-{
-}
-
-static inline void trace_usb_uhci_reset(void)
-{
-}
-
-static inline void trace_usb_uhci_schedule_start(void)
-{
-}
-
-static inline void trace_usb_uhci_schedule_stop(void)
-{
-}
-
-static inline void trace_usb_uhci_frame_start(uint32_t num)
-{
-}
-
-static inline void trace_usb_uhci_frame_stop_bandwidth(void)
-{
-}
-
-static inline void trace_usb_uhci_frame_loop_stop_idle(void)
-{
-}
-
-static inline void trace_usb_uhci_frame_loop_continue(void)
-{
-}
-
-static inline void trace_usb_uhci_mmio_readw(uint32_t addr, uint32_t val)
-{
-}
-
-static inline void trace_usb_uhci_mmio_writew(uint32_t addr, uint32_t val)
-{
-}
-
-static inline void trace_usb_uhci_queue_add(uint32_t token)
-{
-}
-
-static inline void trace_usb_uhci_queue_del(uint32_t token, const char * reason)
-{
-}
-
-static inline void trace_usb_uhci_packet_add(uint32_t token, uint32_t addr)
-{
-}
-
-static inline void trace_usb_uhci_packet_link_async(uint32_t token, uint32_t addr)
-{
-}
-
-static inline void trace_usb_uhci_packet_unlink_async(uint32_t token, uint32_t addr)
-{
-}
-
-static inline void trace_usb_uhci_packet_cancel(uint32_t token, uint32_t addr, int done)
-{
-}
-
-static inline void trace_usb_uhci_packet_complete_success(uint32_t token, uint32_t addr)
-{
-}
-
-static inline void trace_usb_uhci_packet_complete_shortxfer(uint32_t token, uint32_t addr)
-{
-}
-
-static inline void trace_usb_uhci_packet_complete_stall(uint32_t token, uint32_t addr)
-{
-}
-
-static inline void trace_usb_uhci_packet_complete_babble(uint32_t token, uint32_t addr)
-{
-}
-
-static inline void trace_usb_uhci_packet_complete_error(uint32_t token, uint32_t addr)
-{
-}
-
-static inline void trace_usb_uhci_packet_del(uint32_t token, uint32_t addr)
-{
-}
-
-static inline void trace_usb_uhci_qh_load(uint32_t qh)
-{
-}
-
-static inline void trace_usb_uhci_td_load(uint32_t qh, uint32_t td, uint32_t ctrl, uint32_t token)
-{
-}
-
-static inline void trace_usb_uhci_td_queue(uint32_t td, uint32_t ctrl, uint32_t token)
-{
-}
-
-static inline void trace_usb_uhci_td_nextqh(uint32_t qh, uint32_t td)
-{
-}
-
-static inline void trace_usb_uhci_td_async(uint32_t qh, uint32_t td)
-{
-}
-
-static inline void trace_usb_uhci_td_complete(uint32_t qh, uint32_t td)
-{
-}
-
-static inline void trace_usb_xhci_reset(void)
-{
-}
-
-static inline void trace_usb_xhci_run(void)
-{
-}
-
-static inline void trace_usb_xhci_stop(void)
-{
-}
-
-static inline void trace_usb_xhci_cap_read(uint32_t off, uint32_t val)
-{
-}
-
-static inline void trace_usb_xhci_oper_read(uint32_t off, uint32_t val)
-{
-}
-
-static inline void trace_usb_xhci_port_read(uint32_t port, uint32_t off, uint32_t val)
-{
-}
-
-static inline void trace_usb_xhci_runtime_read(uint32_t off, uint32_t val)
-{
-}
-
-static inline void trace_usb_xhci_doorbell_read(uint32_t off, uint32_t val)
-{
-}
-
-static inline void trace_usb_xhci_oper_write(uint32_t off, uint32_t val)
-{
-}
-
-static inline void trace_usb_xhci_port_write(uint32_t port, uint32_t off, uint32_t val)
-{
-}
-
-static inline void trace_usb_xhci_runtime_write(uint32_t off, uint32_t val)
-{
-}
-
-static inline void trace_usb_xhci_doorbell_write(uint32_t off, uint32_t val)
-{
-}
-
-static inline void trace_usb_xhci_irq_intx(uint32_t level)
-{
-}
-
-static inline void trace_usb_xhci_irq_msi(uint32_t nr)
-{
-}
-
-static inline void trace_usb_xhci_irq_msix(uint32_t nr)
-{
-}
-
-static inline void trace_usb_xhci_irq_msix_use(uint32_t nr)
-{
-}
-
-static inline void trace_usb_xhci_irq_msix_unuse(uint32_t nr)
-{
-}
-
-static inline void trace_usb_xhci_queue_event(uint32_t vector, uint32_t idx, const char * trb, const char * evt, uint64_t param, uint32_t status, uint32_t control)
-{
-}
-
-static inline void trace_usb_xhci_fetch_trb(uint64_t addr, const char * name, uint64_t param, uint32_t status, uint32_t control)
-{
-}
-
-static inline void trace_usb_xhci_port_reset(uint32_t port)
-{
-}
-
-static inline void trace_usb_xhci_port_link(uint32_t port, uint32_t pls)
-{
-}
-
-static inline void trace_usb_xhci_port_notify(uint32_t port, uint32_t pls)
-{
-}
-
-static inline void trace_usb_xhci_slot_enable(uint32_t slotid)
-{
-}
-
-static inline void trace_usb_xhci_slot_disable(uint32_t slotid)
-{
-}
-
-static inline void trace_usb_xhci_slot_address(uint32_t slotid)
-{
-}
-
-static inline void trace_usb_xhci_slot_configure(uint32_t slotid)
-{
-}
-
-static inline void trace_usb_xhci_slot_evaluate(uint32_t slotid)
-{
-}
-
-static inline void trace_usb_xhci_slot_reset(uint32_t slotid)
-{
-}
-
-static inline void trace_usb_xhci_ep_enable(uint32_t slotid, uint32_t epid)
-{
-}
-
-static inline void trace_usb_xhci_ep_disable(uint32_t slotid, uint32_t epid)
-{
-}
-
-static inline void trace_usb_xhci_ep_set_dequeue(uint32_t slotid, uint32_t epid, uint32_t streamid, uint64_t param)
-{
-}
-
-static inline void trace_usb_xhci_ep_kick(uint32_t slotid, uint32_t epid, uint32_t streamid)
-{
-}
-
-static inline void trace_usb_xhci_ep_stop(uint32_t slotid, uint32_t epid)
-{
-}
-
-static inline void trace_usb_xhci_ep_reset(uint32_t slotid, uint32_t epid)
-{
-}
-
-static inline void trace_usb_xhci_xfer_start(void * xfer, uint32_t slotid, uint32_t epid, uint32_t streamid)
-{
-}
-
-static inline void trace_usb_xhci_xfer_async(void * xfer)
-{
-}
-
-static inline void trace_usb_xhci_xfer_nak(void * xfer)
-{
-}
-
-static inline void trace_usb_xhci_xfer_retry(void * xfer)
-{
-}
-
-static inline void trace_usb_xhci_xfer_success(void * xfer, uint32_t bytes)
-{
-}
-
-static inline void trace_usb_xhci_xfer_error(void * xfer, uint32_t ret)
-{
-}
-
-static inline void trace_usb_xhci_unimplemented(const char * item, int nr)
-{
-}
-
-static inline void trace_usb_desc_device(int addr, int len, int ret)
-{
-}
-
-static inline void trace_usb_desc_device_qualifier(int addr, int len, int ret)
-{
-}
-
-static inline void trace_usb_desc_config(int addr, int index, int len, int ret)
-{
-}
-
-static inline void trace_usb_desc_other_speed_config(int addr, int index, int len, int ret)
-{
-}
-
-static inline void trace_usb_desc_string(int addr, int index, int len, int ret)
-{
-}
-
-static inline void trace_usb_desc_bos(int addr, int len, int ret)
-{
-}
-
-static inline void trace_usb_set_addr(int addr)
-{
-}
-
-static inline void trace_usb_set_config(int addr, int config, int ret)
-{
-}
-
-static inline void trace_usb_set_interface(int addr, int iface, int alt, int ret)
-{
-}
-
-static inline void trace_usb_clear_device_feature(int addr, int feature, int ret)
-{
-}
-
-static inline void trace_usb_set_device_feature(int addr, int feature, int ret)
-{
-}
-
-static inline void trace_usb_hub_reset(int addr)
-{
-}
-
-static inline void trace_usb_hub_control(int addr, int request, int value, int index, int length)
-{
-}
-
-static inline void trace_usb_hub_get_port_status(int addr, int nr, int status, int changed)
-{
-}
-
-static inline void trace_usb_hub_set_port_feature(int addr, int nr, const char * f)
-{
-}
-
-static inline void trace_usb_hub_clear_port_feature(int addr, int nr, const char * f)
-{
-}
-
-static inline void trace_usb_hub_attach(int addr, int nr)
-{
-}
-
-static inline void trace_usb_hub_detach(int addr, int nr)
-{
-}
-
-static inline void trace_usb_uas_reset(int addr)
-{
-}
-
-static inline void trace_usb_uas_command(int addr, uint16_t tag, int lun, uint32_t lun64_1, uint32_t lun64_2)
-{
-}
-
-static inline void trace_usb_uas_response(int addr, uint16_t tag, uint8_t code)
-{
-}
-
-static inline void trace_usb_uas_sense(int addr, uint16_t tag, uint8_t status)
-{
-}
-
-static inline void trace_usb_uas_read_ready(int addr, uint16_t tag)
-{
-}
-
-static inline void trace_usb_uas_write_ready(int addr, uint16_t tag)
-{
-}
-
-static inline void trace_usb_uas_xfer_data(int addr, uint16_t tag, uint32_t copy, uint32_t uoff, uint32_t usize, uint32_t soff, uint32_t ssize)
-{
-}
-
-static inline void trace_usb_uas_scsi_data(int addr, uint16_t tag, uint32_t bytes)
-{
-}
-
-static inline void trace_usb_uas_scsi_complete(int addr, uint16_t tag, uint32_t status, uint32_t resid)
-{
-}
-
-static inline void trace_usb_uas_tmf_abort_task(int addr, uint16_t tag, uint16_t task_tag)
-{
-}
-
-static inline void trace_usb_uas_tmf_logical_unit_reset(int addr, uint16_t tag, int lun)
-{
-}
-
-static inline void trace_usb_uas_tmf_unsupported(int addr, uint16_t tag, uint32_t function)
-{
-}
-
-static inline void trace_usb_host_open_started(int bus, int addr)
-{
-}
-
-static inline void trace_usb_host_open_success(int bus, int addr)
-{
-}
-
-static inline void trace_usb_host_open_failure(int bus, int addr)
-{
-}
-
-static inline void trace_usb_host_disconnect(int bus, int addr)
-{
-}
-
-static inline void trace_usb_host_close(int bus, int addr)
-{
-}
-
-static inline void trace_usb_host_attach_kernel(int bus, int addr, int interface)
-{
-}
-
-static inline void trace_usb_host_detach_kernel(int bus, int addr, int interface)
-{
-}
-
-static inline void trace_usb_host_set_address(int bus, int addr, int config)
-{
-}
-
-static inline void trace_usb_host_set_config(int bus, int addr, int config)
-{
-}
-
-static inline void trace_usb_host_set_interface(int bus, int addr, int interface, int alt)
-{
-}
-
-static inline void trace_usb_host_claim_interfaces(int bus, int addr, int config, int nif)
-{
-}
-
-static inline void trace_usb_host_claim_interface(int bus, int addr, int config, int interface)
-{
-}
-
-static inline void trace_usb_host_release_interfaces(int bus, int addr)
-{
-}
-
-static inline void trace_usb_host_release_interface(int bus, int addr, int interface)
-{
-}
-
-static inline void trace_usb_host_req_control(int bus, int addr, void * p, int req, int value, int index)
-{
-}
-
-static inline void trace_usb_host_req_data(int bus, int addr, void * p, int in, int ep, int size)
-{
-}
-
-static inline void trace_usb_host_req_complete(int bus, int addr, void * p, int status, int length)
-{
-}
-
-static inline void trace_usb_host_req_emulated(int bus, int addr, void * p, int status)
-{
-}
-
-static inline void trace_usb_host_req_canceled(int bus, int addr, void * p)
-{
-}
-
-static inline void trace_usb_host_urb_submit(int bus, int addr, void * aurb, int length, int more)
-{
-}
-
-static inline void trace_usb_host_urb_complete(int bus, int addr, void * aurb, int status, int length, int more)
-{
-}
-
-static inline void trace_usb_host_urb_canceled(int bus, int addr, void * aurb)
-{
-}
-
-static inline void trace_usb_host_ep_set_halt(int bus, int addr, int ep)
-{
-}
-
-static inline void trace_usb_host_ep_clear_halt(int bus, int addr, int ep)
-{
-}
-
-static inline void trace_usb_host_iso_start(int bus, int addr, int ep)
-{
-}
-
-static inline void trace_usb_host_iso_stop(int bus, int addr, int ep)
-{
-}
-
-static inline void trace_usb_host_iso_out_of_bufs(int bus, int addr, int ep)
-{
-}
-
-static inline void trace_usb_host_iso_many_urbs(int bus, int addr, int count)
-{
-}
-
-static inline void trace_usb_host_reset(int bus, int addr)
-{
-}
-
-static inline void trace_usb_host_auto_scan_enabled(void)
-{
-}
-
-static inline void trace_usb_host_auto_scan_disabled(void)
-{
-}
-
-static inline void trace_usb_host_claim_port(int bus, int hub, int port)
-{
-}
-
-static inline void trace_usb_host_parse_device(int bus, int addr, int vendor, int product)
-{
-}
-
-static inline void trace_usb_host_parse_config(int bus, int addr, int value, int active)
-{
-}
-
-static inline void trace_usb_host_parse_interface(int bus, int addr, int num, int alt, int active)
-{
-}
-
-static inline void trace_usb_host_parse_endpoint(int bus, int addr, int ep, const char * dir, const char * type, int active)
-{
-}
-
-static inline void trace_usb_host_parse_unknown(int bus, int addr, int len, int type)
-{
-}
-
-static inline void trace_usb_host_parse_error(int bus, int addr, const char * errmsg)
-{
-}
-
-static inline void trace_scsi_req_alloc(int target, int lun, int tag)
-{
-}
-
-static inline void trace_scsi_req_cancel(int target, int lun, int tag)
-{
-}
-
-static inline void trace_scsi_req_data(int target, int lun, int tag, int len)
-{
-}
-
-static inline void trace_scsi_req_data_canceled(int target, int lun, int tag, int len)
-{
-}
-
-static inline void trace_scsi_req_dequeue(int target, int lun, int tag)
-{
-}
-
-static inline void trace_scsi_req_continue(int target, int lun, int tag)
-{
-}
-
-static inline void trace_scsi_req_continue_canceled(int target, int lun, int tag)
-{
-}
-
-static inline void trace_scsi_req_parsed(int target, int lun, int tag, int cmd, int mode, int xfer)
-{
-}
-
-static inline void trace_scsi_req_parsed_lba(int target, int lun, int tag, int cmd, uint64_t lba)
-{
-}
-
-static inline void trace_scsi_req_parse_bad(int target, int lun, int tag, int cmd)
-{
-}
-
-static inline void trace_scsi_req_build_sense(int target, int lun, int tag, int key, int asc, int ascq)
-{
-}
-
-static inline void trace_scsi_device_set_ua(int target, int lun, int key, int asc, int ascq)
-{
-}
-
-static inline void trace_scsi_report_luns(int target, int lun, int tag)
-{
-}
-
-static inline void trace_scsi_inquiry(int target, int lun, int tag, int cdb1, int cdb2)
-{
-}
-
-static inline void trace_scsi_test_unit_ready(int target, int lun, int tag)
-{
-}
-
-static inline void trace_scsi_request_sense(int target, int lun, int tag)
-{
-}
-
-static inline void trace_vm_state_notify(int running, int reason)
-{
-}
-
-static inline void trace_load_file(const char * name, const char * path)
-{
-}
-
-static inline void trace_runstate_set(int new_state)
-{
-}
-
-static inline void trace_qcow2_writev_start_req(void * co, int64_t sector, int nb_sectors)
-{
-}
-
-static inline void trace_qcow2_writev_done_req(void * co, int ret)
-{
-}
-
-static inline void trace_qcow2_writev_start_part(void * co)
-{
-}
-
-static inline void trace_qcow2_writev_done_part(void * co, int cur_nr_sectors)
-{
-}
-
-static inline void trace_qcow2_writev_data(void * co, uint64_t offset)
-{
-}
-
-static inline void trace_qcow2_alloc_clusters_offset(void * co, uint64_t offset, int n_start, int n_end)
-{
-}
-
-static inline void trace_qcow2_handle_copied(void * co, uint64_t guest_offset, uint64_t host_offset, uint64_t bytes)
-{
-}
-
-static inline void trace_qcow2_handle_alloc(void * co, uint64_t guest_offset, uint64_t host_offset, uint64_t bytes)
-{
-}
-
-static inline void trace_qcow2_do_alloc_clusters_offset(void * co, uint64_t guest_offset, uint64_t host_offset, int nb_clusters)
-{
-}
-
-static inline void trace_qcow2_cluster_alloc_phys(void * co)
-{
-}
-
-static inline void trace_qcow2_cluster_link_l2(void * co, int nb_clusters)
-{
-}
-
-static inline void trace_qcow2_l2_allocate(void * bs, int l1_index)
-{
-}
-
-static inline void trace_qcow2_l2_allocate_get_empty(void * bs, int l1_index)
-{
-}
-
-static inline void trace_qcow2_l2_allocate_write_l2(void * bs, int l1_index)
-{
-}
-
-static inline void trace_qcow2_l2_allocate_write_l1(void * bs, int l1_index)
-{
-}
-
-static inline void trace_qcow2_l2_allocate_done(void * bs, int l1_index, int ret)
-{
-}
-
-static inline void trace_qcow2_cache_get(void * co, int c, uint64_t offset, bool read_from_disk)
-{
-}
-
-static inline void trace_qcow2_cache_get_replace_entry(void * co, int c, int i)
-{
-}
-
-static inline void trace_qcow2_cache_get_read(void * co, int c, int i)
-{
-}
-
-static inline void trace_qcow2_cache_get_done(void * co, int c, int i)
-{
-}
-
-static inline void trace_qcow2_cache_flush(void * co, int c)
-{
-}
-
-static inline void trace_qcow2_cache_entry_flush(void * co, int c, int i)
-{
-}
-
-static inline void trace_qed_alloc_l2_cache_entry(void * l2_cache, void * entry)
-{
-}
-
-static inline void trace_qed_unref_l2_cache_entry(void * entry, int ref)
-{
-}
-
-static inline void trace_qed_find_l2_cache_entry(void * l2_cache, void * entry, uint64_t offset, int ref)
-{
-}
-
-static inline void trace_qed_read_table(void * s, uint64_t offset, void * table)
-{
-}
-
-static inline void trace_qed_read_table_cb(void * s, void * table, int ret)
-{
-}
-
-static inline void trace_qed_write_table(void * s, uint64_t offset, void * table, unsigned int index, unsigned int n)
-{
-}
-
-static inline void trace_qed_write_table_cb(void * s, void * table, int flush, int ret)
-{
-}
-
-static inline void trace_qed_need_check_timer_cb(void * s)
-{
-}
-
-static inline void trace_qed_start_need_check_timer(void * s)
-{
-}
-
-static inline void trace_qed_cancel_need_check_timer(void * s)
-{
-}
-
-static inline void trace_qed_aio_complete(void * s, void * acb, int ret)
-{
-}
-
-static inline void trace_qed_aio_setup(void * s, void * acb, int64_t sector_num, int nb_sectors, void * opaque, int flags)
-{
-}
-
-static inline void trace_qed_aio_next_io(void * s, void * acb, int ret, uint64_t cur_pos)
-{
-}
-
-static inline void trace_qed_aio_read_data(void * s, void * acb, int ret, uint64_t offset, size_t len)
-{
-}
-
-static inline void trace_qed_aio_write_data(void * s, void * acb, int ret, uint64_t offset, size_t len)
-{
-}
-
-static inline void trace_qed_aio_write_prefill(void * s, void * acb, uint64_t start, size_t len, uint64_t offset)
-{
-}
-
-static inline void trace_qed_aio_write_postfill(void * s, void * acb, uint64_t start, size_t len, uint64_t offset)
-{
-}
-
-static inline void trace_qed_aio_write_main(void * s, void * acb, int ret, uint64_t offset, size_t len)
-{
-}
-
-static inline void trace_g364fb_read(uint64_t addr, uint32_t val)
-{
-}
-
-static inline void trace_g364fb_write(uint64_t addr, uint32_t new)
-{
-}
-
-static inline void trace_grlib_gptimer_enable(int id, uint32_t count)
-{
-}
-
-static inline void trace_grlib_gptimer_disabled(int id, uint32_t config)
-{
-}
-
-static inline void trace_grlib_gptimer_restart(int id, uint32_t reload)
-{
-}
-
-static inline void trace_grlib_gptimer_set_scaler(uint32_t scaler, uint32_t freq)
-{
-}
-
-static inline void trace_grlib_gptimer_hit(int id)
-{
-}
-
-static inline void trace_grlib_gptimer_readl(int id, uint64_t addr, uint32_t val)
-{
-}
-
-static inline void trace_grlib_gptimer_writel(int id, uint64_t addr, uint32_t val)
-{
-}
-
-static inline void trace_grlib_irqmp_check_irqs(uint32_t pend, uint32_t force, uint32_t mask, uint32_t lvl1, uint32_t lvl2)
-{
-}
-
-static inline void trace_grlib_irqmp_ack(int intno)
-{
-}
-
-static inline void trace_grlib_irqmp_set_irq(int irq)
-{
-}
-
-static inline void trace_grlib_irqmp_readl_unknown(uint64_t addr)
-{
-}
-
-static inline void trace_grlib_irqmp_writel_unknown(uint64_t addr, uint32_t value)
-{
-}
-
-static inline void trace_grlib_apbuart_event(int event)
-{
-}
-
-static inline void trace_grlib_apbuart_writel_unknown(uint64_t addr, uint32_t value)
-{
-}
-
-static inline void trace_grlib_apbuart_readl_unknown(uint64_t addr)
-{
-}
-
-static inline void trace_leon3_set_irq(int intno)
-{
-}
-
-static inline void trace_leon3_reset_irq(int intno)
-{
-}
-
-static inline void trace_spice_vmc_write(ssize_t out, int len)
-{
-}
-
-static inline void trace_spice_vmc_read(int bytes, int len)
-{
-}
-
-static inline void trace_spice_vmc_register_interface(void * scd)
-{
-}
-
-static inline void trace_spice_vmc_unregister_interface(void * scd)
-{
-}
-
-static inline void trace_spice_vmc_event(int event)
-{
-}
-
-static inline void trace_lm32_pic_raise_irq(void)
-{
-}
-
-static inline void trace_lm32_pic_lower_irq(void)
-{
-}
-
-static inline void trace_lm32_pic_interrupt(int irq, int level)
-{
-}
-
-static inline void trace_lm32_pic_set_im(uint32_t im)
-{
-}
-
-static inline void trace_lm32_pic_set_ip(uint32_t ip)
-{
-}
-
-static inline void trace_lm32_pic_get_im(uint32_t im)
-{
-}
-
-static inline void trace_lm32_pic_get_ip(uint32_t ip)
-{
-}
-
-static inline void trace_lm32_juart_get_jtx(uint32_t value)
-{
-}
-
-static inline void trace_lm32_juart_set_jtx(uint32_t value)
-{
-}
-
-static inline void trace_lm32_juart_get_jrx(uint32_t value)
-{
-}
-
-static inline void trace_lm32_juart_set_jrx(uint32_t value)
-{
-}
-
-static inline void trace_lm32_timer_memory_write(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_lm32_timer_memory_read(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_lm32_timer_hit(void)
-{
-}
-
-static inline void trace_lm32_timer_irq_state(int level)
-{
-}
-
-static inline void trace_lm32_uart_memory_write(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_lm32_uart_memory_read(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_lm32_uart_irq_state(int level)
-{
-}
-
-static inline void trace_lm32_sys_memory_write(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_megasas_init_firmware(uint64_t pa)
-{
-}
-
-static inline void trace_megasas_init_queue(uint64_t queue_pa, int queue_len, uint64_t head, uint64_t tail, uint32_t flags)
-{
-}
-
-static inline void trace_megasas_initq_map_failed(int frame)
-{
-}
-
-static inline void trace_megasas_initq_mismatch(int queue_len, int fw_cmds)
-{
-}
-
-static inline void trace_megasas_qf_found(unsigned int index, uint64_t pa)
-{
-}
-
-static inline void trace_megasas_qf_new(unsigned int index, void * cmd)
-{
-}
-
-static inline void trace_megasas_qf_failed(unsigned long pa)
-{
-}
-
-static inline void trace_megasas_qf_enqueue(unsigned int index, unsigned int count, uint64_t context, unsigned int tail, int busy)
-{
-}
-
-static inline void trace_megasas_qf_update(unsigned int head, unsigned int busy)
-{
-}
-
-static inline void trace_megasas_qf_dequeue(unsigned int index)
-{
-}
-
-static inline void trace_megasas_qf_map_failed(int cmd, unsigned long frame)
-{
-}
-
-static inline void trace_megasas_qf_complete_noirq(uint64_t context)
-{
-}
-
-static inline void trace_megasas_qf_complete(uint64_t context, unsigned int tail, unsigned int offset, int busy, unsigned int doorbell)
-{
-}
-
-static inline void trace_megasas_handle_frame(const char * cmd, uint64_t addr, uint64_t context, uint32_t count)
-{
-}
-
-static inline void trace_megasas_frame_busy(uint64_t addr)
-{
-}
-
-static inline void trace_megasas_unhandled_frame_cmd(int cmd, uint8_t frame_cmd)
-{
-}
-
-static inline void trace_megasas_handle_scsi(const char * frame, int bus, int dev, int lun, void * sdev, unsigned long size)
-{
-}
-
-static inline void trace_megasas_scsi_target_not_present(const char * frame, int bus, int dev, int lun)
-{
-}
-
-static inline void trace_megasas_scsi_invalid_cdb_len(const char * frame, int bus, int dev, int lun, int len)
-{
-}
-
-static inline void trace_megasas_iov_read_overflow(int cmd, int bytes, int len)
-{
-}
-
-static inline void trace_megasas_iov_write_overflow(int cmd, int bytes, int len)
-{
-}
-
-static inline void trace_megasas_iov_read_underflow(int cmd, int bytes, int len)
-{
-}
-
-static inline void trace_megasas_iov_write_underflow(int cmd, int bytes, int len)
-{
-}
-
-static inline void trace_megasas_scsi_req_alloc_failed(const char * frame, int dev, int lun)
-{
-}
-
-static inline void trace_megasas_scsi_read_start(int cmd, int len)
-{
-}
-
-static inline void trace_megasas_scsi_write_start(int cmd, int len)
-{
-}
-
-static inline void trace_megasas_scsi_nodata(int cmd)
-{
-}
-
-static inline void trace_megasas_scsi_complete(int cmd, uint32_t status, int len, int xfer)
-{
-}
-
-static inline void trace_megasas_command_complete(int cmd, uint32_t status, uint32_t resid)
-{
-}
-
-static inline void trace_megasas_handle_io(int cmd, const char * frame, int dev, int lun, unsigned long lba, unsigned long count)
-{
-}
-
-static inline void trace_megasas_io_target_not_present(int cmd, const char * frame, int dev, int lun)
-{
-}
-
-static inline void trace_megasas_io_read_start(int cmd, unsigned long lba, unsigned long count, unsigned long len)
-{
-}
-
-static inline void trace_megasas_io_write_start(int cmd, unsigned long lba, unsigned long count, unsigned long len)
-{
-}
-
-static inline void trace_megasas_io_complete(int cmd, uint32_t len)
-{
-}
-
-static inline void trace_megasas_io_read(int cmd, int bytes, int len, unsigned long offset)
-{
-}
-
-static inline void trace_megasas_io_write(int cmd, int bytes, int len, unsigned long offset)
-{
-}
-
-static inline void trace_megasas_io_continue(int cmd, int bytes)
-{
-}
-
-static inline void trace_megasas_iovec_map_failed(int cmd, int index, unsigned long iov_size)
-{
-}
-
-static inline void trace_megasas_iovec_sgl_overflow(int cmd, int index, int limit)
-{
-}
-
-static inline void trace_megasas_iovec_sgl_underflow(int cmd, int index)
-{
-}
-
-static inline void trace_megasas_iovec_sgl_invalid(int cmd, int index, uint64_t pa, uint32_t len)
-{
-}
-
-static inline void trace_megasas_iovec_overflow(int cmd, int len, int limit)
-{
-}
-
-static inline void trace_megasas_iovec_underflow(int cmd, int len, int limit)
-{
-}
-
-static inline void trace_megasas_handle_dcmd(int cmd, int opcode)
-{
-}
-
-static inline void trace_megasas_finish_dcmd(int cmd, int size)
-{
-}
-
-static inline void trace_megasas_dcmd_req_alloc_failed(int cmd, const char * desc)
-{
-}
-
-static inline void trace_megasas_dcmd_internal_submit(int cmd, const char * desc, int dev)
-{
-}
-
-static inline void trace_megasas_dcmd_internal_finish(int cmd, int opcode, int lun)
-{
-}
-
-static inline void trace_megasas_dcmd_internal_invalid(int cmd, int opcode)
-{
-}
-
-static inline void trace_megasas_dcmd_unhandled(int cmd, int opcode, int len)
-{
-}
-
-static inline void trace_megasas_dcmd_zero_sge(int cmd)
-{
-}
-
-static inline void trace_megasas_dcmd_invalid_sge(int cmd, int count)
-{
-}
-
-static inline void trace_megasas_dcmd_map_failed(int cmd)
-{
-}
-
-static inline void trace_megasas_dcmd_invalid_xfer_len(int cmd, unsigned long size, unsigned long max)
-{
-}
-
-static inline void trace_megasas_dcmd_enter(int cmd, const char * dcmd, int len)
-{
-}
-
-static inline void trace_megasas_dcmd_dummy(int cmd, unsigned long size)
-{
-}
-
-static inline void trace_megasas_dcmd_set_fw_time(int cmd, unsigned long time)
-{
-}
-
-static inline void trace_megasas_dcmd_pd_get_list(int cmd, int num, int max, int offset)
-{
-}
-
-static inline void trace_megasas_dcmd_ld_get_list(int cmd, int num, int max)
-{
-}
-
-static inline void trace_megasas_dcmd_ld_get_info(int cmd, int ld_id)
-{
-}
-
-static inline void trace_megasas_dcmd_pd_get_info(int cmd, int pd_id)
-{
-}
-
-static inline void trace_megasas_dcmd_pd_list_query(int cmd, int flags)
-{
-}
-
-static inline void trace_megasas_dcmd_unsupported(int cmd, unsigned long size)
-{
-}
-
-static inline void trace_megasas_abort_frame(int cmd, int abort_cmd)
-{
-}
-
-static inline void trace_megasas_abort_no_cmd(int cmd, uint64_t context)
-{
-}
-
-static inline void trace_megasas_abort_invalid_context(int cmd, uint64_t context, int abort_cmd)
-{
-}
-
-static inline void trace_megasas_reset(void)
-{
-}
-
-static inline void trace_megasas_init(int sges, int cmds, const char * intr, const char * mode)
-{
-}
-
-static inline void trace_megasas_msix_raise(int vector)
-{
-}
-
-static inline void trace_megasas_irq_lower(void)
-{
-}
-
-static inline void trace_megasas_irq_raise(void)
-{
-}
-
-static inline void trace_megasas_intr_enabled(void)
-{
-}
-
-static inline void trace_megasas_intr_disabled(void)
-{
-}
-
-static inline void trace_megasas_mmio_readl(unsigned long addr, uint32_t val)
-{
-}
-
-static inline void trace_megasas_mmio_invalid_readl(unsigned long addr)
-{
-}
-
-static inline void trace_megasas_mmio_writel(uint32_t addr, uint32_t val)
-{
-}
-
-static inline void trace_megasas_mmio_invalid_writel(uint32_t addr, uint32_t val)
-{
-}
-
-static inline void trace_milkymist_ac97_memory_read(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_milkymist_ac97_memory_write(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_milkymist_ac97_pulse_irq_crrequest(void)
-{
-}
-
-static inline void trace_milkymist_ac97_pulse_irq_crreply(void)
-{
-}
-
-static inline void trace_milkymist_ac97_pulse_irq_dmaw(void)
-{
-}
-
-static inline void trace_milkymist_ac97_pulse_irq_dmar(void)
-{
-}
-
-static inline void trace_milkymist_ac97_in_cb(int avail, uint32_t remaining)
-{
-}
-
-static inline void trace_milkymist_ac97_in_cb_transferred(int transferred)
-{
-}
-
-static inline void trace_milkymist_ac97_out_cb(int free, uint32_t remaining)
-{
-}
-
-static inline void trace_milkymist_ac97_out_cb_transferred(int transferred)
-{
-}
-
-static inline void trace_milkymist_hpdmc_memory_read(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_milkymist_hpdmc_memory_write(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_milkymist_memcard_memory_read(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_milkymist_memcard_memory_write(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_milkymist_minimac2_memory_read(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_milkymist_minimac2_memory_write(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_milkymist_minimac2_mdio_write(uint8_t phy_addr, uint8_t addr, uint16_t value)
-{
-}
-
-static inline void trace_milkymist_minimac2_mdio_read(uint8_t phy_addr, uint8_t addr, uint16_t value)
-{
-}
-
-static inline void trace_milkymist_minimac2_tx_frame(uint32_t length)
-{
-}
-
-static inline void trace_milkymist_minimac2_rx_frame(const void * buf, uint32_t length)
-{
-}
-
-static inline void trace_milkymist_minimac2_drop_rx_frame(const void * buf)
-{
-}
-
-static inline void trace_milkymist_minimac2_rx_transfer(const void * buf, uint32_t length)
-{
-}
-
-static inline void trace_milkymist_minimac2_raise_irq_rx(void)
-{
-}
-
-static inline void trace_milkymist_minimac2_lower_irq_rx(void)
-{
-}
-
-static inline void trace_milkymist_minimac2_pulse_irq_tx(void)
-{
-}
-
-static inline void trace_milkymist_pfpu_memory_read(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_milkymist_pfpu_memory_write(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_milkymist_pfpu_vectout(uint32_t a, uint32_t b, uint32_t dma_ptr)
-{
-}
-
-static inline void trace_milkymist_pfpu_pulse_irq(void)
-{
-}
-
-static inline void trace_milkymist_softusb_memory_read(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_milkymist_softusb_memory_write(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_milkymist_softusb_mevt(uint8_t m)
-{
-}
-
-static inline void trace_milkymist_softusb_kevt(uint8_t m)
-{
-}
-
-static inline void trace_milkymist_softusb_mouse_event(int dx, int dy, int dz, int bs)
-{
-}
-
-static inline void trace_milkymist_softusb_pulse_irq(void)
-{
-}
-
-static inline void trace_milkymist_sysctl_memory_read(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_milkymist_sysctl_memory_write(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_milkymist_sysctl_icap_write(uint32_t value)
-{
-}
-
-static inline void trace_milkymist_sysctl_start_timer0(void)
-{
-}
-
-static inline void trace_milkymist_sysctl_stop_timer0(void)
-{
-}
-
-static inline void trace_milkymist_sysctl_start_timer1(void)
-{
-}
-
-static inline void trace_milkymist_sysctl_stop_timer1(void)
-{
-}
-
-static inline void trace_milkymist_sysctl_pulse_irq_timer0(void)
-{
-}
-
-static inline void trace_milkymist_sysctl_pulse_irq_timer1(void)
-{
-}
-
-static inline void trace_milkymist_tmu2_memory_read(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_milkymist_tmu2_memory_write(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_milkymist_tmu2_start(void)
-{
-}
-
-static inline void trace_milkymist_tmu2_pulse_irq(void)
-{
-}
-
-static inline void trace_milkymist_uart_memory_read(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_milkymist_uart_memory_write(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_milkymist_uart_raise_irq(void)
-{
-}
-
-static inline void trace_milkymist_uart_lower_irq(void)
-{
-}
-
-static inline void trace_milkymist_vgafb_memory_read(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_milkymist_vgafb_memory_write(uint32_t addr, uint32_t value)
-{
-}
-
-static inline void trace_mipsnet_send(uint32_t size)
-{
-}
-
-static inline void trace_mipsnet_receive(uint32_t size)
-{
-}
-
-static inline void trace_mipsnet_read(uint64_t addr, uint32_t val)
-{
-}
-
-static inline void trace_mipsnet_write(uint64_t addr, uint64_t val)
-{
-}
-
-static inline void trace_mipsnet_irq(uint32_t isr, uint32_t intctl)
-{
-}
-
-static inline void trace_pc87312_io_read(uint32_t addr, uint32_t val)
-{
-}
-
-static inline void trace_pc87312_io_write(uint32_t addr, uint32_t val)
-{
-}
-
-static inline void trace_pc87312_info_floppy(uint32_t base)
-{
-}
-
-static inline void trace_pc87312_info_ide(uint32_t base)
-{
-}
-
-static inline void trace_pc87312_info_parallel(uint32_t base, uint32_t irq)
-{
-}
-
-static inline void trace_pc87312_info_serial(int n, uint32_t base, uint32_t irq)
-{
-}
-
-static inline void trace_pvscsi_ring_init_data(uint32_t txr_len_log2, uint32_t rxr_len_log2)
-{
-}
-
-static inline void trace_pvscsi_ring_init_msg(uint32_t len_log2)
-{
-}
-
-static inline void trace_pvscsi_ring_flush_cmp(uint64_t filled_cmp_ptr)
-{
-}
-
-static inline void trace_pvscsi_ring_flush_msg(uint64_t filled_cmp_ptr)
-{
-}
-
-static inline void trace_pvscsi_update_irq_level(bool raise, uint64_t mask, uint64_t status)
-{
-}
-
-static inline void trace_pvscsi_update_irq_msi(void)
-{
-}
-
-static inline void trace_pvscsi_cmp_ring_put(unsigned long addr)
-{
-}
-
-static inline void trace_pvscsi_msg_ring_put(unsigned long addr)
-{
-}
-
-static inline void trace_pvscsi_complete_request(uint64_t context, uint64_t len, uint8_t sense_key)
-{
-}
-
-static inline void trace_pvscsi_get_sg_list(int nsg, size_t size)
-{
-}
-
-static inline void trace_pvscsi_get_next_sg_elem(uint32_t flags)
-{
-}
-
-static inline void trace_pvscsi_command_complete_not_found(uint32_t tag)
-{
-}
-
-static inline void trace_pvscsi_command_complete_data_run(void)
-{
-}
-
-static inline void trace_pvscsi_command_complete_sense_len(int len)
-{
-}
-
-static inline void trace_pvscsi_convert_sglist(uint64_t context, unsigned long addr, uint32_t resid)
-{
-}
-
-static inline void trace_pvscsi_process_req_descr(uint8_t cmd, uint64_t ctx)
-{
-}
-
-static inline void trace_pvscsi_process_req_descr_unknown_device(void)
-{
-}
-
-static inline void trace_pvscsi_process_req_descr_invalid_dir(void)
-{
-}
-
-static inline void trace_pvscsi_process_io(unsigned long addr)
-{
-}
-
-static inline void trace_pvscsi_on_cmd_noimpl(const char* cmd)
-{
-}
-
-static inline void trace_pvscsi_on_cmd_reset_dev(uint32_t tgt, int lun, void* dev)
-{
-}
-
-static inline void trace_pvscsi_on_cmd_arrived(const char* cmd)
-{
-}
-
-static inline void trace_pvscsi_on_cmd_abort(uint64_t ctx, uint32_t tgt)
-{
-}
-
-static inline void trace_pvscsi_on_cmd_unknown(uint64_t cmd_id)
-{
-}
-
-static inline void trace_pvscsi_on_cmd_unknown_data(uint32_t data)
-{
-}
-
-static inline void trace_pvscsi_io_write(const char* cmd, uint64_t val)
-{
-}
-
-static inline void trace_pvscsi_io_write_unknown(unsigned long addr, unsigned sz, uint64_t val)
-{
-}
-
-static inline void trace_pvscsi_io_read(const char* cmd, uint64_t status)
-{
-}
-
-static inline void trace_pvscsi_io_read_unknown(unsigned long addr, unsigned sz)
-{
-}
-
-static inline void trace_pvscsi_init_msi_fail(int res)
-{
-}
-
-static inline void trace_pvscsi_state(const char* state)
-{
-}
-
-static inline void trace_pvscsi_tx_rings_ppn(const char* label, uint64_t ppn)
-{
-}
-
-static inline void trace_pvscsi_tx_rings_num_pages(const char* label, uint32_t num)
-{
-}
-
-static inline void trace_xen_ram_alloc(unsigned long ram_addr, unsigned long size)
-{
-}
-
-static inline void trace_xen_client_set_memory(uint64_t start_addr, unsigned long size, bool log_dirty)
-{
-}
-
-static inline void trace_xen_map_cache(uint64_t phys_addr)
-{
-}
-
-static inline void trace_xen_remap_bucket(uint64_t index)
-{
-}
-
-static inline void trace_xen_map_cache_return(void* ptr)
-{
-}
-
-static inline void trace_xen_map_block(uint64_t phys_addr, uint64_t size)
-{
-}
-
-static inline void trace_xen_unmap_block(void* addr, unsigned long size)
-{
-}
-
-static inline void trace_xen_platform_log(char * s)
-{
-}
-
-static inline void trace_qemu_coroutine_enter(void * from, void * to, void * opaque)
-{
-}
-
-static inline void trace_qemu_coroutine_yield(void * from, void * to)
-{
-}
-
-static inline void trace_qemu_coroutine_terminate(void * co)
-{
-}
-
-static inline void trace_qemu_co_queue_run_restart(void * co)
-{
-}
-
-static inline void trace_qemu_co_queue_next(void * nxt)
-{
-}
-
-static inline void trace_qemu_co_mutex_lock_entry(void * mutex, void * self)
-{
-}
-
-static inline void trace_qemu_co_mutex_lock_return(void * mutex, void * self)
-{
-}
-
-static inline void trace_qemu_co_mutex_unlock_entry(void * mutex, void * self)
-{
-}
-
-static inline void trace_qemu_co_mutex_unlock_return(void * mutex, void * self)
-{
-}
-
-static inline void trace_escc_put_queue(char channel, int b)
-{
-}
-
-static inline void trace_escc_get_queue(char channel, int val)
-{
-}
-
-static inline void trace_escc_update_irq(int irq)
-{
-}
-
-static inline void trace_escc_update_parameters(char channel, int speed, int parity, int data_bits, int stop_bits)
-{
-}
-
-static inline void trace_escc_mem_writeb_ctrl(char channel, uint32_t reg, uint32_t val)
-{
-}
-
-static inline void trace_escc_mem_writeb_data(char channel, uint32_t val)
-{
-}
-
-static inline void trace_escc_mem_readb_ctrl(char channel, uint32_t reg, uint8_t val)
-{
-}
-
-static inline void trace_escc_mem_readb_data(char channel, uint32_t ret)
-{
-}
-
-static inline void trace_escc_serial_receive_byte(char channel, int ch)
-{
-}
-
-static inline void trace_escc_sunkbd_event_in(int ch)
-{
-}
-
-static inline void trace_escc_sunkbd_event_out(int ch)
-{
-}
-
-static inline void trace_escc_kbd_command(int val)
-{
-}
-
-static inline void trace_escc_sunmouse_event(int dx, int dy, int buttons_state)
-{
-}
-
-static inline void trace_iscsi_aio_write16_cb(void * iscsi, int status, void * acb, int canceled)
-{
-}
-
-static inline void trace_iscsi_aio_writev(void * iscsi, int64_t sector_num, int nb_sectors, void * opaque, void * acb)
-{
-}
-
-static inline void trace_iscsi_aio_read16_cb(void * iscsi, int status, void * acb, int canceled)
-{
-}
-
-static inline void trace_iscsi_aio_readv(void * iscsi, int64_t sector_num, int nb_sectors, void * opaque, void * acb)
-{
-}
-
-static inline void trace_esp_error_fifo_overrun(void)
-{
-}
-
-static inline void trace_esp_error_unhandled_command(uint32_t val)
-{
-}
-
-static inline void trace_esp_error_invalid_write(uint32_t val, uint32_t addr)
-{
-}
-
-static inline void trace_esp_raise_irq(void)
-{
-}
-
-static inline void trace_esp_lower_irq(void)
-{
-}
-
-static inline void trace_esp_dma_enable(void)
-{
-}
-
-static inline void trace_esp_dma_disable(void)
-{
-}
-
-static inline void trace_esp_get_cmd(uint32_t dmalen, int target)
-{
-}
-
-static inline void trace_esp_do_busid_cmd(uint8_t busid)
-{
-}
-
-static inline void trace_esp_handle_satn_stop(uint32_t cmdlen)
-{
-}
-
-static inline void trace_esp_write_response(uint32_t status)
-{
-}
-
-static inline void trace_esp_do_dma(uint32_t cmdlen, uint32_t len)
-{
-}
-
-static inline void trace_esp_command_complete(void)
-{
-}
-
-static inline void trace_esp_command_complete_unexpected(void)
-{
-}
-
-static inline void trace_esp_command_complete_fail(void)
-{
-}
-
-static inline void trace_esp_transfer_data(uint32_t dma_left, int32_t ti_size)
-{
-}
-
-static inline void trace_esp_handle_ti(uint32_t minlen)
-{
-}
-
-static inline void trace_esp_handle_ti_cmd(uint32_t cmdlen)
-{
-}
-
-static inline void trace_esp_mem_readb(uint32_t saddr, uint8_t reg)
-{
-}
-
-static inline void trace_esp_mem_writeb(uint32_t saddr, uint8_t reg, uint32_t val)
-{
-}
-
-static inline void trace_esp_mem_writeb_cmd_nop(uint32_t val)
-{
-}
-
-static inline void trace_esp_mem_writeb_cmd_flush(uint32_t val)
-{
-}
-
-static inline void trace_esp_mem_writeb_cmd_reset(uint32_t val)
-{
-}
-
-static inline void trace_esp_mem_writeb_cmd_bus_reset(uint32_t val)
-{
-}
-
-static inline void trace_esp_mem_writeb_cmd_iccs(uint32_t val)
-{
-}
-
-static inline void trace_esp_mem_writeb_cmd_msgacc(uint32_t val)
-{
-}
-
-static inline void trace_esp_mem_writeb_cmd_pad(uint32_t val)
-{
-}
-
-static inline void trace_esp_mem_writeb_cmd_satn(uint32_t val)
-{
-}
-
-static inline void trace_esp_mem_writeb_cmd_rstatn(uint32_t val)
-{
-}
-
-static inline void trace_esp_mem_writeb_cmd_sel(uint32_t val)
-{
-}
-
-static inline void trace_esp_mem_writeb_cmd_selatn(uint32_t val)
-{
-}
-
-static inline void trace_esp_mem_writeb_cmd_selatns(uint32_t val)
-{
-}
-
-static inline void trace_esp_mem_writeb_cmd_ensel(uint32_t val)
-{
-}
-
-static inline void trace_esp_mem_writeb_cmd_dissel(uint32_t val)
-{
-}
-
-static inline void trace_esp_pci_error_invalid_dma_direction(void)
-{
-}
-
-static inline void trace_esp_pci_error_invalid_read(uint32_t reg)
-{
-}
-
-static inline void trace_esp_pci_error_invalid_write(uint32_t reg)
-{
-}
-
-static inline void trace_esp_pci_error_invalid_write_dma(uint32_t val, uint32_t addr)
-{
-}
-
-static inline void trace_esp_pci_dma_read(uint32_t saddr, uint32_t reg)
-{
-}
-
-static inline void trace_esp_pci_dma_write(uint32_t saddr, uint32_t reg, uint32_t val)
-{
-}
-
-static inline void trace_esp_pci_dma_idle(uint32_t val)
-{
-}
-
-static inline void trace_esp_pci_dma_blast(uint32_t val)
-{
-}
-
-static inline void trace_esp_pci_dma_abort(uint32_t val)
-{
-}
-
-static inline void trace_esp_pci_dma_start(uint32_t val)
-{
-}
-
-static inline void trace_esp_pci_sbac_read(uint32_t reg)
-{
-}
-
-static inline void trace_esp_pci_sbac_write(uint32_t reg, uint32_t val)
-{
-}
-
-static inline void trace_handle_qmp_command(void * mon, const char * cmd_name)
-{
-}
-
-static inline void trace_monitor_protocol_emitter(void * mon)
-{
-}
-
-static inline void trace_monitor_protocol_event(uint32_t event, const char * evname, void * data)
-{
-}
-
-static inline void trace_monitor_protocol_event_handler(uint32_t event, void * data, uint64_t last, uint64_t now)
-{
-}
-
-static inline void trace_monitor_protocol_event_emit(uint32_t event, void * data)
-{
-}
-
-static inline void trace_monitor_protocol_event_queue(uint32_t event, void * data, uint64_t rate, uint64_t last, uint64_t now)
-{
-}
-
-static inline void trace_monitor_protocol_event_throttle(uint32_t event, uint64_t rate)
-{
-}
-
-static inline void trace_open_eth_mii_write(unsigned idx, uint16_t v)
-{
-}
-
-static inline void trace_open_eth_mii_read(unsigned idx, uint16_t v)
-{
-}
-
-static inline void trace_open_eth_update_irq(uint32_t v)
-{
-}
-
-static inline void trace_open_eth_receive(unsigned len)
-{
-}
-
-static inline void trace_open_eth_receive_mcast(unsigned idx, uint32_t h0, uint32_t h1)
-{
-}
-
-static inline void trace_open_eth_receive_reject(void)
-{
-}
-
-static inline void trace_open_eth_receive_desc(uint32_t addr, uint32_t len_flags)
-{
-}
-
-static inline void trace_open_eth_start_xmit(uint32_t addr, unsigned len, unsigned tx_len)
-{
-}
-
-static inline void trace_open_eth_reg_read(uint32_t addr, uint32_t v)
-{
-}
-
-static inline void trace_open_eth_reg_write(uint32_t addr, uint32_t v)
-{
-}
-
-static inline void trace_open_eth_desc_read(uint32_t addr, uint32_t v)
-{
-}
-
-static inline void trace_open_eth_desc_write(uint32_t addr, uint32_t v)
-{
-}
-
-static inline void trace_v9fs_rerror(uint16_t tag, uint8_t id, int err)
-{
-}
-
-static inline void trace_v9fs_version(uint16_t tag, uint8_t id, int32_t msize, char* version)
-{
-}
-
-static inline void trace_v9fs_version_return(uint16_t tag, uint8_t id, int32_t msize, char* version)
-{
-}
-
-static inline void trace_v9fs_attach(uint16_t tag, uint8_t id, int32_t fid, int32_t afid, char* uname, char* aname)
-{
-}
-
-static inline void trace_v9fs_attach_return(uint16_t tag, uint8_t id, int8_t type, int32_t version, int64_t path)
-{
-}
-
-static inline void trace_v9fs_stat(uint16_t tag, uint8_t id, int32_t fid)
-{
-}
-
-static inline void trace_v9fs_stat_return(uint16_t tag, uint8_t id, int32_t mode, int32_t atime, int32_t mtime, int64_t length)
-{
-}
-
-static inline void trace_v9fs_getattr(uint16_t tag, uint8_t id, int32_t fid, uint64_t request_mask)
-{
-}
-
-static inline void trace_v9fs_getattr_return(uint16_t tag, uint8_t id, uint64_t result_mask, uint32_t mode, uint32_t uid, uint32_t gid)
-{
-}
-
-static inline void trace_v9fs_walk(uint16_t tag, uint8_t id, int32_t fid, int32_t newfid, uint16_t nwnames)
-{
-}
-
-static inline void trace_v9fs_walk_return(uint16_t tag, uint8_t id, uint16_t nwnames, void* qids)
-{
-}
-
-static inline void trace_v9fs_open(uint16_t tag, uint8_t id, int32_t fid, int32_t mode)
-{
-}
-
-static inline void trace_v9fs_open_return(uint16_t tag, uint8_t id, int8_t type, int32_t version, int64_t path, int iounit)
-{
-}
-
-static inline void trace_v9fs_lcreate(uint16_t tag, uint8_t id, int32_t dfid, int32_t flags, int32_t mode, uint32_t gid)
-{
-}
-
-static inline void trace_v9fs_lcreate_return(uint16_t tag, uint8_t id, int8_t type, int32_t version, int64_t path, int32_t iounit)
-{
-}
-
-static inline void trace_v9fs_fsync(uint16_t tag, uint8_t id, int32_t fid, int datasync)
-{
-}
-
-static inline void trace_v9fs_clunk(uint16_t tag, uint8_t id, int32_t fid)
-{
-}
-
-static inline void trace_v9fs_read(uint16_t tag, uint8_t id, int32_t fid, uint64_t off, uint32_t max_count)
-{
-}
-
-static inline void trace_v9fs_read_return(uint16_t tag, uint8_t id, int32_t count, ssize_t err)
-{
-}
-
-static inline void trace_v9fs_readdir(uint16_t tag, uint8_t id, int32_t fid, uint64_t offset, uint32_t max_count)
-{
-}
-
-static inline void trace_v9fs_readdir_return(uint16_t tag, uint8_t id, uint32_t count, ssize_t retval)
-{
-}
-
-static inline void trace_v9fs_write(uint16_t tag, uint8_t id, int32_t fid, uint64_t off, uint32_t count, int cnt)
-{
-}
-
-static inline void trace_v9fs_write_return(uint16_t tag, uint8_t id, int32_t total, ssize_t err)
-{
-}
-
-static inline void trace_v9fs_create(uint16_t tag, uint8_t id, int32_t fid, char* name, int32_t perm, int8_t mode)
-{
-}
-
-static inline void trace_v9fs_create_return(uint16_t tag, uint8_t id, int8_t type, int32_t version, int64_t path, int iounit)
-{
-}
-
-static inline void trace_v9fs_symlink(uint16_t tag, uint8_t id, int32_t fid, char* name, char* symname, uint32_t gid)
-{
-}
-
-static inline void trace_v9fs_symlink_return(uint16_t tag, uint8_t id, int8_t type, int32_t version, int64_t path)
-{
-}
-
-static inline void trace_v9fs_flush(uint16_t tag, uint8_t id, int16_t flush_tag)
-{
-}
-
-static inline void trace_v9fs_link(uint16_t tag, uint8_t id, int32_t dfid, int32_t oldfid, char* name)
-{
-}
-
-static inline void trace_v9fs_remove(uint16_t tag, uint8_t id, int32_t fid)
-{
-}
-
-static inline void trace_v9fs_wstat(uint16_t tag, uint8_t id, int32_t fid, int32_t mode, int32_t atime, int32_t mtime)
-{
-}
-
-static inline void trace_v9fs_mknod(uint16_t tag, uint8_t id, int32_t fid, int mode, int major, int minor)
-{
-}
-
-static inline void trace_v9fs_mknod_return(uint16_t tag, uint8_t id, int8_t type, int32_t version, int64_t path)
-{
-}
-
-static inline void trace_v9fs_lock(uint16_t tag, uint8_t id, int32_t fid, uint8_t type, uint64_t start, uint64_t length)
-{
-}
-
-static inline void trace_v9fs_lock_return(uint16_t tag, uint8_t id, int8_t status)
-{
-}
-
-static inline void trace_v9fs_getlock(uint16_t tag, uint8_t id, int32_t fid, uint8_t type, uint64_t start, uint64_t length)
-{
-}
-
-static inline void trace_v9fs_getlock_return(uint16_t tag, uint8_t id, uint8_t type, uint64_t start, uint64_t length, uint32_t proc_id)
-{
-}
-
-static inline void trace_v9fs_mkdir(uint16_t tag, uint8_t id, int32_t fid, char* name, int mode, uint32_t gid)
-{
-}
-
-static inline void trace_v9fs_mkdir_return(uint16_t tag, uint8_t id, int8_t type, int32_t version, int64_t path, int err)
-{
-}
-
-static inline void trace_v9fs_xattrwalk(uint16_t tag, uint8_t id, int32_t fid, int32_t newfid, char* name)
-{
-}
-
-static inline void trace_v9fs_xattrwalk_return(uint16_t tag, uint8_t id, int64_t size)
-{
-}
-
-static inline void trace_v9fs_xattrcreate(uint16_t tag, uint8_t id, int32_t fid, char* name, int64_t size, int flags)
-{
-}
-
-static inline void trace_v9fs_readlink(uint16_t tag, uint8_t id, int32_t fid)
-{
-}
-
-static inline void trace_v9fs_readlink_return(uint16_t tag, uint8_t id, char* target)
-{
-}
-
-static inline void trace_mmu_helper_dfault(uint64_t address, uint64_t context, int mmu_idx, uint32_t tl)
-{
-}
-
-static inline void trace_mmu_helper_dprot(uint64_t address, uint64_t context, int mmu_idx, uint32_t tl)
-{
-}
-
-static inline void trace_mmu_helper_dmiss(uint64_t address, uint64_t context)
-{
-}
-
-static inline void trace_mmu_helper_tfault(uint64_t address, uint64_t context)
-{
-}
-
-static inline void trace_mmu_helper_tmiss(uint64_t address, uint64_t context)
-{
-}
-
-static inline void trace_mmu_helper_get_phys_addr_code(uint32_t tl, int mmu_idx, uint64_t prim_context, uint64_t sec_context, uint64_t address)
-{
-}
-
-static inline void trace_mmu_helper_get_phys_addr_data(uint32_t tl, int mmu_idx, uint64_t prim_context, uint64_t sec_context, uint64_t address)
-{
-}
-
-static inline void trace_mmu_helper_mmu_fault(uint64_t address, uint64_t paddr, int mmu_idx, uint32_t tl, uint64_t prim_context, uint64_t sec_context)
-{
-}
-
-static inline void trace_int_helper_set_softint(uint32_t softint)
-{
-}
-
-static inline void trace_int_helper_clear_softint(uint32_t softint)
-{
-}
-
-static inline void trace_int_helper_write_softint(uint32_t softint)
-{
-}
-
-static inline void trace_int_helper_icache_freeze(void)
-{
-}
-
-static inline void trace_int_helper_dcache_freeze(void)
-{
-}
-
-static inline void trace_win_helper_gregset_error(uint32_t pstate)
-{
-}
-
-static inline void trace_win_helper_switch_pstate(uint32_t pstate_regs, uint32_t new_pstate_regs)
-{
-}
-
-static inline void trace_win_helper_no_switch_pstate(uint32_t new_pstate_regs)
-{
-}
-
-static inline void trace_win_helper_wrpil(uint32_t psrpil, uint32_t new_pil)
-{
-}
-
-static inline void trace_win_helper_done(uint32_t tl)
-{
-}
-
-static inline void trace_win_helper_retry(uint32_t tl)
-{
-}
-
-static inline void trace_dma_bdrv_io(void * dbs, void * bs, int64_t sector_num, bool to_dev)
-{
-}
-
-static inline void trace_dma_aio_cancel(void * dbs)
-{
-}
-
-static inline void trace_dma_complete(void * dbs, int ret, void * cb)
-{
-}
-
-static inline void trace_dma_bdrv_cb(void * dbs, int ret)
-{
-}
-
-static inline void trace_dma_map_wait(void * dbs)
-{
-}
-
-static inline void trace_console_gfx_new(void)
-{
-}
-
-static inline void trace_console_txt_new(int w, int h)
-{
-}
-
-static inline void trace_console_select(int nr)
-{
-}
-
-static inline void trace_console_refresh(int interval)
-{
-}
-
-static inline void trace_displaysurface_create(void * display_surface, int w, int h)
-{
-}
-
-static inline void trace_displaysurface_create_from(void * display_surface, int w, int h, int bpp, int swap)
-{
-}
-
-static inline void trace_displaysurface_free(void * display_surface)
-{
-}
-
-static inline void trace_displaychangelistener_register(void * dcl, const char * name)
-{
-}
-
-static inline void trace_displaychangelistener_unregister(void * dcl, const char * name)
-{
-}
-
-static inline void trace_ppm_save(const char * filename, void * display_surface)
-{
-}
-
-static inline void trace_vmware_value_read(uint32_t index, uint32_t value)
-{
-}
-
-static inline void trace_vmware_value_write(uint32_t index, uint32_t value)
-{
-}
-
-static inline void trace_vmware_palette_read(uint32_t index, uint32_t value)
-{
-}
-
-static inline void trace_vmware_palette_write(uint32_t index, uint32_t value)
-{
-}
-
-static inline void trace_vmware_scratch_read(uint32_t index, uint32_t value)
-{
-}
-
-static inline void trace_vmware_scratch_write(uint32_t index, uint32_t value)
-{
-}
-
-static inline void trace_vmware_setmode(uint32_t w, uint32_t h, uint32_t bpp)
-{
-}
-
-static inline void trace_savevm_section_start(void)
-{
-}
-
-static inline void trace_savevm_section_end(unsigned int section_id)
-{
-}
-
-static inline void trace_migration_bitmap_sync_start(void)
-{
-}
-
-static inline void trace_migration_bitmap_sync_end(uint64_t dirty_pages)
-{
-}
-
-static inline void trace_migration_throttle(void)
-{
-}
-
-static inline void trace_qxl_create_guest_primary(int qid, uint32_t width, uint32_t height, uint64_t mem, uint32_t format, uint32_t position)
-{
-}
-
-static inline void trace_qxl_create_guest_primary_rest(int qid, int32_t stride, uint32_t type, uint32_t flags)
-{
-}
-
-static inline void trace_qxl_destroy_primary(int qid)
-{
-}
-
-static inline void trace_qxl_enter_vga_mode(int qid)
-{
-}
-
-static inline void trace_qxl_exit_vga_mode(int qid)
-{
-}
-
-static inline void trace_qxl_hard_reset(int qid, int64_t loadvm)
-{
-}
-
-static inline void trace_qxl_interface_async_complete_io(int qid, uint32_t current_async, void * cookie)
-{
-}
-
-static inline void trace_qxl_interface_attach_worker(int qid)
-{
-}
-
-static inline void trace_qxl_interface_get_init_info(int qid)
-{
-}
-
-static inline void trace_qxl_interface_set_compression_level(int qid, int64_t level)
-{
-}
-
-static inline void trace_qxl_interface_update_area_complete(int qid, uint32_t surface_id, uint32_t dirty_left, uint32_t dirty_right, uint32_t dirty_top, uint32_t dirty_bottom)
-{
-}
-
-static inline void trace_qxl_interface_update_area_complete_rest(int qid, uint32_t num_updated_rects)
-{
-}
-
-static inline void trace_qxl_interface_update_area_complete_overflow(int qid, int max)
-{
-}
-
-static inline void trace_qxl_interface_update_area_complete_schedule_bh(int qid, uint32_t num_dirty)
-{
-}
-
-static inline void trace_qxl_io_destroy_primary_ignored(int qid, const char * mode)
-{
-}
-
-static inline void trace_qxl_io_log(int qid, const uint8_t * log_buf)
-{
-}
-
-static inline void trace_qxl_io_read_unexpected(int qid)
-{
-}
-
-static inline void trace_qxl_io_unexpected_vga_mode(int qid, uint64_t addr, uint64_t val, const char * desc)
-{
-}
-
-static inline void trace_qxl_io_write(int qid, const char * mode, uint64_t addr, uint64_t val, unsigned size, int async)
-{
-}
-
-static inline void trace_qxl_memslot_add_guest(int qid, uint32_t slot_id, uint64_t guest_start, uint64_t guest_end)
-{
-}
-
-static inline void trace_qxl_post_load(int qid, const char * mode)
-{
-}
-
-static inline void trace_qxl_pre_load(int qid)
-{
-}
-
-static inline void trace_qxl_pre_save(int qid)
-{
-}
-
-static inline void trace_qxl_reset_surfaces(int qid)
-{
-}
-
-static inline void trace_qxl_ring_command_check(int qid, const char * mode)
-{
-}
-
-static inline void trace_qxl_ring_command_get(int qid, const char * mode)
-{
-}
-
-static inline void trace_qxl_ring_command_req_notification(int qid)
-{
-}
-
-static inline void trace_qxl_ring_cursor_check(int qid, const char * mode)
-{
-}
-
-static inline void trace_qxl_ring_cursor_get(int qid, const char * mode)
-{
-}
-
-static inline void trace_qxl_ring_cursor_req_notification(int qid)
-{
-}
-
-static inline void trace_qxl_ring_res_push(int qid, const char * mode, uint32_t surface_count, uint32_t free_res, void * last_release, const char * notify)
-{
-}
-
-static inline void trace_qxl_ring_res_push_rest(int qid, uint32_t ring_has, uint32_t ring_size, uint32_t prod, uint32_t cons)
-{
-}
-
-static inline void trace_qxl_ring_res_put(int qid, uint32_t free_res)
-{
-}
-
-static inline void trace_qxl_set_mode(int qid, int modenr, uint32_t x_res, uint32_t y_res, uint32_t bits, uint64_t devmem)
-{
-}
-
-static inline void trace_qxl_soft_reset(int qid)
-{
-}
-
-static inline void trace_qemu_spice_add_memslot(int qid, uint32_t slot_id, unsigned long virt_start, unsigned long virt_end, int async)
-{
-}
-
-static inline void trace_qemu_spice_del_memslot(int qid, uint32_t gid, uint32_t slot_id)
-{
-}
-
-static inline void trace_qemu_spice_create_primary_surface(int qid, uint32_t sid, void * surface, int async)
-{
-}
-
-static inline void trace_qemu_spice_destroy_primary_surface(int qid, uint32_t sid, int async)
-{
-}
-
-static inline void trace_qemu_spice_wakeup(uint32_t qid)
-{
-}
-
-static inline void trace_qemu_spice_start(uint32_t qid)
-{
-}
-
-static inline void trace_qemu_spice_stop(uint32_t qid)
-{
-}
-
-static inline void trace_qemu_spice_create_update(uint32_t left, uint32_t right, uint32_t top, uint32_t bottom)
-{
-}
-
-static inline void trace_qxl_spice_destroy_surfaces_complete(int qid)
-{
-}
-
-static inline void trace_qxl_spice_destroy_surfaces(int qid, int async)
-{
-}
-
-static inline void trace_qxl_spice_destroy_surface_wait_complete(int qid, uint32_t id)
-{
-}
-
-static inline void trace_qxl_spice_destroy_surface_wait(int qid, uint32_t id, int async)
-{
-}
-
-static inline void trace_qxl_spice_flush_surfaces_async(int qid, uint32_t surface_count, uint32_t num_free_res)
-{
-}
-
-static inline void trace_qxl_spice_monitors_config(int qid)
-{
-}
-
-static inline void trace_qxl_spice_loadvm_commands(int qid, void * ext, uint32_t count)
-{
-}
-
-static inline void trace_qxl_spice_oom(int qid)
-{
-}
-
-static inline void trace_qxl_spice_reset_cursor(int qid)
-{
-}
-
-static inline void trace_qxl_spice_reset_image_cache(int qid)
-{
-}
-
-static inline void trace_qxl_spice_reset_memslots(int qid)
-{
-}
-
-static inline void trace_qxl_spice_update_area(int qid, uint32_t surface_id, uint32_t left, uint32_t right, uint32_t top, uint32_t bottom)
-{
-}
-
-static inline void trace_qxl_spice_update_area_rest(int qid, uint32_t num_dirty_rects, uint32_t clear_dirty_region)
-{
-}
-
-static inline void trace_qxl_surfaces_dirty(int qid, int surface, int offset, int size)
-{
-}
-
-static inline void trace_qxl_send_events(int qid, uint32_t events)
-{
-}
-
-static inline void trace_qxl_send_events_vm_stopped(int qid, uint32_t events)
-{
-}
-
-static inline void trace_qxl_set_guest_bug(int qid)
-{
-}
-
-static inline void trace_qxl_interrupt_client_monitors_config(int qid, int num_heads, void * heads)
-{
-}
-
-static inline void trace_qxl_client_monitors_config_unsupported_by_guest(int qid, uint32_t int_mask, void * client_monitors_config)
-{
-}
-
-static inline void trace_qxl_client_monitors_config_unsupported_by_device(int qid, int revision)
-{
-}
-
-static inline void trace_qxl_client_monitors_config_capped(int qid, int requested, int limit)
-{
-}
-
-static inline void trace_qxl_client_monitors_config_crc(int qid, unsigned size, uint32_t crc32)
-{
-}
-
-static inline void trace_qxl_set_client_capabilities_unsupported_by_revision(int qid, int revision)
-{
-}
-
-static inline void trace_qxl_render_blit_guest_primary_initialized(void)
-{
-}
-
-static inline void trace_qxl_render_blit(int32_t stride, int32_t left, int32_t right, int32_t top, int32_t bottom)
-{
-}
-
-static inline void trace_qxl_render_guest_primary_resized(int32_t width, int32_t height, int32_t stride, int32_t bytes_pp, int32_t bits_pp)
-{
-}
-
-static inline void trace_qxl_render_update_area_done(void * cookie)
-{
-}
-
-static inline void trace_spapr_pci_msi(const char * msg, uint32_t n, uint32_t ca)
-{
-}
-
-static inline void trace_spapr_pci_msi_setup(const char * name, unsigned vector, uint64_t addr)
-{
-}
-
-static inline void trace_spapr_pci_rtas_ibm_change_msi(unsigned func, unsigned req)
-{
-}
-
-static inline void trace_spapr_pci_rtas_ibm_query_interrupt_source_number(unsigned ioa, unsigned intr)
-{
-}
-
-static inline void trace_spapr_pci_msi_write(uint64_t addr, uint64_t data, uint32_t dt_irq)
-{
-}
-
-static inline void trace_spapr_pci_lsi_set(const char * busname, int pin, uint32_t irq)
-{
-}
-
-static inline void trace_xics_icp_check_ipi(int server, uint8_t mfrr)
-{
-}
-
-static inline void trace_xics_icp_accept(uint32_t old_xirr, uint32_t new_xirr)
-{
-}
-
-static inline void trace_xics_icp_eoi(int server, uint32_t xirr, uint32_t new_xirr)
-{
-}
-
-static inline void trace_xics_icp_irq(int server, int nr, uint8_t priority)
-{
-}
-
-static inline void trace_xics_icp_raise(uint32_t xirr, uint8_t pending_priority)
-{
-}
-
-static inline void trace_xics_set_irq_msi(int srcno, int nr)
-{
-}
-
-static inline void trace_xics_masked_pending(void)
-{
-}
-
-static inline void trace_xics_set_irq_lsi(int srcno, int nr)
-{
-}
-
-static inline void trace_xics_ics_write_xive(int nr, int srcno, int server, uint8_t priority)
-{
-}
-
-static inline void trace_xics_ics_reject(int nr, int srcno)
-{
-}
-
-static inline void trace_xics_ics_eoi(int nr)
-{
-}
-
-static inline void trace_hbitmap_iter_skip_words(const void * hb, void * hbi, uint64_t pos, unsigned long cur)
-{
-}
-
-static inline void trace_hbitmap_reset(void * hb, uint64_t start, uint64_t count, uint64_t sbit, uint64_t ebit)
-{
-}
-
-static inline void trace_hbitmap_set(void * hb, uint64_t start, uint64_t count, uint64_t sbit, uint64_t ebit)
-{
-}
-
-static inline void trace_ioinst(const char * insn)
-{
-}
-
-static inline void trace_ioinst_sch_id(const char * insn, int cssid, int ssid, int schid)
-{
-}
-
-static inline void trace_ioinst_chp_id(const char * insn, int cssid, int chpid)
-{
-}
-
-static inline void trace_ioinst_chsc_cmd(uint16_t cmd, uint16_t len)
-{
-}
-
-static inline void trace_css_enable_facility(const char * facility)
-{
-}
-
-static inline void trace_css_crw(uint8_t rsc, uint8_t erc, uint16_t rsid, const char * chained)
-{
-}
-
-static inline void trace_css_chpid_add(uint8_t cssid, uint8_t chpid, uint8_t type)
-{
-}
-
-static inline void trace_css_new_image(uint8_t cssid, const char * default_cssid)
-{
-}
-
-static inline void trace_css_assign_subch(const char * do_assign, uint8_t cssid, uint8_t ssid, uint16_t schid, uint16_t devno)
-{
-}
-
-static inline void trace_css_io_interrupt(int cssid, int ssid, int schid, uint32_t intparm, uint8_t isc, const char * conditional)
-{
-}
-
-static inline void trace_virtio_ccw_interpret_ccw(int cssid, int ssid, int schid, int cmd_code)
-{
-}
-
-static inline void trace_virtio_ccw_new_device(int cssid, int ssid, int schid, int devno, const char * devno_mode)
-{
-}
-
-static inline void trace_migrate_set_state(int new_state)
-{
-}
-
-static inline void trace_kvm_ioctl(int type, void * arg)
-{
-}
-
-static inline void trace_kvm_vm_ioctl(int type, void * arg)
-{
-}
-
-static inline void trace_kvm_vcpu_ioctl(int cpu_index, int type, void * arg)
-{
-}
-
-static inline void trace_kvm_run_exit(int cpu_index, uint32_t reason)
-{
-}
-
-static inline void trace_object_dynamic_cast_assert(const char * type, const char * target, const char * file, int line, const char * func)
-{
-}
-
-static inline void trace_object_class_dynamic_cast_assert(const char * type, const char * target, const char * file, int line, const char * func)
-{
-}
-#endif /* TRACE__GENERATED_TRACERS_H */
diff --git a/contrib/qemu/util/aes.c b/contrib/qemu/util/aes.c
deleted file mode 100644
index 91e97fa6e7f..00000000000
--- a/contrib/qemu/util/aes.c
+++ /dev/null
@@ -1,1314 +0,0 @@
-/**
- *
- * aes.c - integrated in QEMU by Fabrice Bellard from the OpenSSL project.
- */
-/*
- * rijndael-alg-fst.c
- *
- * @version 3.0 (December 2000)
- *
- * Optimised ANSI C code for the Rijndael cipher (now AES)
- *
- * @author Vincent Rijmen <vincent.rijmen@esat.kuleuven.ac.be>
- * @author Antoon Bosselaers <antoon.bosselaers@esat.kuleuven.ac.be>
- * @author Paulo Barreto <paulo.barreto@terra.com.br>
- *
- * This code is hereby placed in the public domain.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHORS ''AS IS'' AND ANY EXPRESS
- * OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
- * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
- * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
- * OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
- * EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- */
-#include "qemu-common.h"
-#include "qemu/aes.h"
-
-#ifndef NDEBUG
-#define NDEBUG
-#endif
-
-typedef uint32_t u32;
-typedef uint16_t u16;
-typedef uint8_t u8;
-
-/* This controls loop-unrolling in aes_core.c */
-#undef FULL_UNROLL
-# define GETU32(pt) (((u32)(pt)[0] << 24) ^ ((u32)(pt)[1] << 16) ^ ((u32)(pt)[2] <<  8) ^ ((u32)(pt)[3]))
-# define PUTU32(ct, st) { (ct)[0] = (u8)((st) >> 24); (ct)[1] = (u8)((st) >> 16); (ct)[2] = (u8)((st) >>  8); (ct)[3] = (u8)(st); }
-
-/*
-AES_Te0[x] = S [x].[02, 01, 01, 03];
-AES_Te1[x] = S [x].[03, 02, 01, 01];
-AES_Te2[x] = S [x].[01, 03, 02, 01];
-AES_Te3[x] = S [x].[01, 01, 03, 02];
-AES_Te4[x] = S [x].[01, 01, 01, 01];
-
-AES_Td0[x] = Si[x].[0e, 09, 0d, 0b];
-AES_Td1[x] = Si[x].[0b, 0e, 09, 0d];
-AES_Td2[x] = Si[x].[0d, 0b, 0e, 09];
-AES_Td3[x] = Si[x].[09, 0d, 0b, 0e];
-AES_Td4[x] = Si[x].[01, 01, 01, 01];
-*/
-
-const uint32_t AES_Te0[256] = {
-    0xc66363a5U, 0xf87c7c84U, 0xee777799U, 0xf67b7b8dU,
-    0xfff2f20dU, 0xd66b6bbdU, 0xde6f6fb1U, 0x91c5c554U,
-    0x60303050U, 0x02010103U, 0xce6767a9U, 0x562b2b7dU,
-    0xe7fefe19U, 0xb5d7d762U, 0x4dababe6U, 0xec76769aU,
-    0x8fcaca45U, 0x1f82829dU, 0x89c9c940U, 0xfa7d7d87U,
-    0xeffafa15U, 0xb25959ebU, 0x8e4747c9U, 0xfbf0f00bU,
-    0x41adadecU, 0xb3d4d467U, 0x5fa2a2fdU, 0x45afafeaU,
-    0x239c9cbfU, 0x53a4a4f7U, 0xe4727296U, 0x9bc0c05bU,
-    0x75b7b7c2U, 0xe1fdfd1cU, 0x3d9393aeU, 0x4c26266aU,
-    0x6c36365aU, 0x7e3f3f41U, 0xf5f7f702U, 0x83cccc4fU,
-    0x6834345cU, 0x51a5a5f4U, 0xd1e5e534U, 0xf9f1f108U,
-    0xe2717193U, 0xabd8d873U, 0x62313153U, 0x2a15153fU,
-    0x0804040cU, 0x95c7c752U, 0x46232365U, 0x9dc3c35eU,
-    0x30181828U, 0x379696a1U, 0x0a05050fU, 0x2f9a9ab5U,
-    0x0e070709U, 0x24121236U, 0x1b80809bU, 0xdfe2e23dU,
-    0xcdebeb26U, 0x4e272769U, 0x7fb2b2cdU, 0xea75759fU,
-    0x1209091bU, 0x1d83839eU, 0x582c2c74U, 0x341a1a2eU,
-    0x361b1b2dU, 0xdc6e6eb2U, 0xb45a5aeeU, 0x5ba0a0fbU,
-    0xa45252f6U, 0x763b3b4dU, 0xb7d6d661U, 0x7db3b3ceU,
-    0x5229297bU, 0xdde3e33eU, 0x5e2f2f71U, 0x13848497U,
-    0xa65353f5U, 0xb9d1d168U, 0x00000000U, 0xc1eded2cU,
-    0x40202060U, 0xe3fcfc1fU, 0x79b1b1c8U, 0xb65b5bedU,
-    0xd46a6abeU, 0x8dcbcb46U, 0x67bebed9U, 0x7239394bU,
-    0x944a4adeU, 0x984c4cd4U, 0xb05858e8U, 0x85cfcf4aU,
-    0xbbd0d06bU, 0xc5efef2aU, 0x4faaaae5U, 0xedfbfb16U,
-    0x864343c5U, 0x9a4d4dd7U, 0x66333355U, 0x11858594U,
-    0x8a4545cfU, 0xe9f9f910U, 0x04020206U, 0xfe7f7f81U,
-    0xa05050f0U, 0x783c3c44U, 0x259f9fbaU, 0x4ba8a8e3U,
-    0xa25151f3U, 0x5da3a3feU, 0x804040c0U, 0x058f8f8aU,
-    0x3f9292adU, 0x219d9dbcU, 0x70383848U, 0xf1f5f504U,
-    0x63bcbcdfU, 0x77b6b6c1U, 0xafdada75U, 0x42212163U,
-    0x20101030U, 0xe5ffff1aU, 0xfdf3f30eU, 0xbfd2d26dU,
-    0x81cdcd4cU, 0x180c0c14U, 0x26131335U, 0xc3ecec2fU,
-    0xbe5f5fe1U, 0x359797a2U, 0x884444ccU, 0x2e171739U,
-    0x93c4c457U, 0x55a7a7f2U, 0xfc7e7e82U, 0x7a3d3d47U,
-    0xc86464acU, 0xba5d5de7U, 0x3219192bU, 0xe6737395U,
-    0xc06060a0U, 0x19818198U, 0x9e4f4fd1U, 0xa3dcdc7fU,
-    0x44222266U, 0x542a2a7eU, 0x3b9090abU, 0x0b888883U,
-    0x8c4646caU, 0xc7eeee29U, 0x6bb8b8d3U, 0x2814143cU,
-    0xa7dede79U, 0xbc5e5ee2U, 0x160b0b1dU, 0xaddbdb76U,
-    0xdbe0e03bU, 0x64323256U, 0x743a3a4eU, 0x140a0a1eU,
-    0x924949dbU, 0x0c06060aU, 0x4824246cU, 0xb85c5ce4U,
-    0x9fc2c25dU, 0xbdd3d36eU, 0x43acacefU, 0xc46262a6U,
-    0x399191a8U, 0x319595a4U, 0xd3e4e437U, 0xf279798bU,
-    0xd5e7e732U, 0x8bc8c843U, 0x6e373759U, 0xda6d6db7U,
-    0x018d8d8cU, 0xb1d5d564U, 0x9c4e4ed2U, 0x49a9a9e0U,
-    0xd86c6cb4U, 0xac5656faU, 0xf3f4f407U, 0xcfeaea25U,
-    0xca6565afU, 0xf47a7a8eU, 0x47aeaee9U, 0x10080818U,
-    0x6fbabad5U, 0xf0787888U, 0x4a25256fU, 0x5c2e2e72U,
-    0x381c1c24U, 0x57a6a6f1U, 0x73b4b4c7U, 0x97c6c651U,
-    0xcbe8e823U, 0xa1dddd7cU, 0xe874749cU, 0x3e1f1f21U,
-    0x964b4bddU, 0x61bdbddcU, 0x0d8b8b86U, 0x0f8a8a85U,
-    0xe0707090U, 0x7c3e3e42U, 0x71b5b5c4U, 0xcc6666aaU,
-    0x904848d8U, 0x06030305U, 0xf7f6f601U, 0x1c0e0e12U,
-    0xc26161a3U, 0x6a35355fU, 0xae5757f9U, 0x69b9b9d0U,
-    0x17868691U, 0x99c1c158U, 0x3a1d1d27U, 0x279e9eb9U,
-    0xd9e1e138U, 0xebf8f813U, 0x2b9898b3U, 0x22111133U,
-    0xd26969bbU, 0xa9d9d970U, 0x078e8e89U, 0x339494a7U,
-    0x2d9b9bb6U, 0x3c1e1e22U, 0x15878792U, 0xc9e9e920U,
-    0x87cece49U, 0xaa5555ffU, 0x50282878U, 0xa5dfdf7aU,
-    0x038c8c8fU, 0x59a1a1f8U, 0x09898980U, 0x1a0d0d17U,
-    0x65bfbfdaU, 0xd7e6e631U, 0x844242c6U, 0xd06868b8U,
-    0x824141c3U, 0x299999b0U, 0x5a2d2d77U, 0x1e0f0f11U,
-    0x7bb0b0cbU, 0xa85454fcU, 0x6dbbbbd6U, 0x2c16163aU,
-};
-const uint32_t AES_Te1[256] = {
-    0xa5c66363U, 0x84f87c7cU, 0x99ee7777U, 0x8df67b7bU,
-    0x0dfff2f2U, 0xbdd66b6bU, 0xb1de6f6fU, 0x5491c5c5U,
-    0x50603030U, 0x03020101U, 0xa9ce6767U, 0x7d562b2bU,
-    0x19e7fefeU, 0x62b5d7d7U, 0xe64dababU, 0x9aec7676U,
-    0x458fcacaU, 0x9d1f8282U, 0x4089c9c9U, 0x87fa7d7dU,
-    0x15effafaU, 0xebb25959U, 0xc98e4747U, 0x0bfbf0f0U,
-    0xec41adadU, 0x67b3d4d4U, 0xfd5fa2a2U, 0xea45afafU,
-    0xbf239c9cU, 0xf753a4a4U, 0x96e47272U, 0x5b9bc0c0U,
-    0xc275b7b7U, 0x1ce1fdfdU, 0xae3d9393U, 0x6a4c2626U,
-    0x5a6c3636U, 0x417e3f3fU, 0x02f5f7f7U, 0x4f83ccccU,
-    0x5c683434U, 0xf451a5a5U, 0x34d1e5e5U, 0x08f9f1f1U,
-    0x93e27171U, 0x73abd8d8U, 0x53623131U, 0x3f2a1515U,
-    0x0c080404U, 0x5295c7c7U, 0x65462323U, 0x5e9dc3c3U,
-    0x28301818U, 0xa1379696U, 0x0f0a0505U, 0xb52f9a9aU,
-    0x090e0707U, 0x36241212U, 0x9b1b8080U, 0x3ddfe2e2U,
-    0x26cdebebU, 0x694e2727U, 0xcd7fb2b2U, 0x9fea7575U,
-    0x1b120909U, 0x9e1d8383U, 0x74582c2cU, 0x2e341a1aU,
-    0x2d361b1bU, 0xb2dc6e6eU, 0xeeb45a5aU, 0xfb5ba0a0U,
-    0xf6a45252U, 0x4d763b3bU, 0x61b7d6d6U, 0xce7db3b3U,
-    0x7b522929U, 0x3edde3e3U, 0x715e2f2fU, 0x97138484U,
-    0xf5a65353U, 0x68b9d1d1U, 0x00000000U, 0x2cc1ededU,
-    0x60402020U, 0x1fe3fcfcU, 0xc879b1b1U, 0xedb65b5bU,
-    0xbed46a6aU, 0x468dcbcbU, 0xd967bebeU, 0x4b723939U,
-    0xde944a4aU, 0xd4984c4cU, 0xe8b05858U, 0x4a85cfcfU,
-    0x6bbbd0d0U, 0x2ac5efefU, 0xe54faaaaU, 0x16edfbfbU,
-    0xc5864343U, 0xd79a4d4dU, 0x55663333U, 0x94118585U,
-    0xcf8a4545U, 0x10e9f9f9U, 0x06040202U, 0x81fe7f7fU,
-    0xf0a05050U, 0x44783c3cU, 0xba259f9fU, 0xe34ba8a8U,
-    0xf3a25151U, 0xfe5da3a3U, 0xc0804040U, 0x8a058f8fU,
-    0xad3f9292U, 0xbc219d9dU, 0x48703838U, 0x04f1f5f5U,
-    0xdf63bcbcU, 0xc177b6b6U, 0x75afdadaU, 0x63422121U,
-    0x30201010U, 0x1ae5ffffU, 0x0efdf3f3U, 0x6dbfd2d2U,
-    0x4c81cdcdU, 0x14180c0cU, 0x35261313U, 0x2fc3ececU,
-    0xe1be5f5fU, 0xa2359797U, 0xcc884444U, 0x392e1717U,
-    0x5793c4c4U, 0xf255a7a7U, 0x82fc7e7eU, 0x477a3d3dU,
-    0xacc86464U, 0xe7ba5d5dU, 0x2b321919U, 0x95e67373U,
-    0xa0c06060U, 0x98198181U, 0xd19e4f4fU, 0x7fa3dcdcU,
-    0x66442222U, 0x7e542a2aU, 0xab3b9090U, 0x830b8888U,
-    0xca8c4646U, 0x29c7eeeeU, 0xd36bb8b8U, 0x3c281414U,
-    0x79a7dedeU, 0xe2bc5e5eU, 0x1d160b0bU, 0x76addbdbU,
-    0x3bdbe0e0U, 0x56643232U, 0x4e743a3aU, 0x1e140a0aU,
-    0xdb924949U, 0x0a0c0606U, 0x6c482424U, 0xe4b85c5cU,
-    0x5d9fc2c2U, 0x6ebdd3d3U, 0xef43acacU, 0xa6c46262U,
-    0xa8399191U, 0xa4319595U, 0x37d3e4e4U, 0x8bf27979U,
-    0x32d5e7e7U, 0x438bc8c8U, 0x596e3737U, 0xb7da6d6dU,
-    0x8c018d8dU, 0x64b1d5d5U, 0xd29c4e4eU, 0xe049a9a9U,
-    0xb4d86c6cU, 0xfaac5656U, 0x07f3f4f4U, 0x25cfeaeaU,
-    0xafca6565U, 0x8ef47a7aU, 0xe947aeaeU, 0x18100808U,
-    0xd56fbabaU, 0x88f07878U, 0x6f4a2525U, 0x725c2e2eU,
-    0x24381c1cU, 0xf157a6a6U, 0xc773b4b4U, 0x5197c6c6U,
-    0x23cbe8e8U, 0x7ca1ddddU, 0x9ce87474U, 0x213e1f1fU,
-    0xdd964b4bU, 0xdc61bdbdU, 0x860d8b8bU, 0x850f8a8aU,
-    0x90e07070U, 0x427c3e3eU, 0xc471b5b5U, 0xaacc6666U,
-    0xd8904848U, 0x05060303U, 0x01f7f6f6U, 0x121c0e0eU,
-    0xa3c26161U, 0x5f6a3535U, 0xf9ae5757U, 0xd069b9b9U,
-    0x91178686U, 0x5899c1c1U, 0x273a1d1dU, 0xb9279e9eU,
-    0x38d9e1e1U, 0x13ebf8f8U, 0xb32b9898U, 0x33221111U,
-    0xbbd26969U, 0x70a9d9d9U, 0x89078e8eU, 0xa7339494U,
-    0xb62d9b9bU, 0x223c1e1eU, 0x92158787U, 0x20c9e9e9U,
-    0x4987ceceU, 0xffaa5555U, 0x78502828U, 0x7aa5dfdfU,
-    0x8f038c8cU, 0xf859a1a1U, 0x80098989U, 0x171a0d0dU,
-    0xda65bfbfU, 0x31d7e6e6U, 0xc6844242U, 0xb8d06868U,
-    0xc3824141U, 0xb0299999U, 0x775a2d2dU, 0x111e0f0fU,
-    0xcb7bb0b0U, 0xfca85454U, 0xd66dbbbbU, 0x3a2c1616U,
-};
-const uint32_t AES_Te2[256] = {
-    0x63a5c663U, 0x7c84f87cU, 0x7799ee77U, 0x7b8df67bU,
-    0xf20dfff2U, 0x6bbdd66bU, 0x6fb1de6fU, 0xc55491c5U,
-    0x30506030U, 0x01030201U, 0x67a9ce67U, 0x2b7d562bU,
-    0xfe19e7feU, 0xd762b5d7U, 0xabe64dabU, 0x769aec76U,
-    0xca458fcaU, 0x829d1f82U, 0xc94089c9U, 0x7d87fa7dU,
-    0xfa15effaU, 0x59ebb259U, 0x47c98e47U, 0xf00bfbf0U,
-    0xadec41adU, 0xd467b3d4U, 0xa2fd5fa2U, 0xafea45afU,
-    0x9cbf239cU, 0xa4f753a4U, 0x7296e472U, 0xc05b9bc0U,
-    0xb7c275b7U, 0xfd1ce1fdU, 0x93ae3d93U, 0x266a4c26U,
-    0x365a6c36U, 0x3f417e3fU, 0xf702f5f7U, 0xcc4f83ccU,
-    0x345c6834U, 0xa5f451a5U, 0xe534d1e5U, 0xf108f9f1U,
-    0x7193e271U, 0xd873abd8U, 0x31536231U, 0x153f2a15U,
-    0x040c0804U, 0xc75295c7U, 0x23654623U, 0xc35e9dc3U,
-    0x18283018U, 0x96a13796U, 0x050f0a05U, 0x9ab52f9aU,
-    0x07090e07U, 0x12362412U, 0x809b1b80U, 0xe23ddfe2U,
-    0xeb26cdebU, 0x27694e27U, 0xb2cd7fb2U, 0x759fea75U,
-    0x091b1209U, 0x839e1d83U, 0x2c74582cU, 0x1a2e341aU,
-    0x1b2d361bU, 0x6eb2dc6eU, 0x5aeeb45aU, 0xa0fb5ba0U,
-    0x52f6a452U, 0x3b4d763bU, 0xd661b7d6U, 0xb3ce7db3U,
-    0x297b5229U, 0xe33edde3U, 0x2f715e2fU, 0x84971384U,
-    0x53f5a653U, 0xd168b9d1U, 0x00000000U, 0xed2cc1edU,
-    0x20604020U, 0xfc1fe3fcU, 0xb1c879b1U, 0x5bedb65bU,
-    0x6abed46aU, 0xcb468dcbU, 0xbed967beU, 0x394b7239U,
-    0x4ade944aU, 0x4cd4984cU, 0x58e8b058U, 0xcf4a85cfU,
-    0xd06bbbd0U, 0xef2ac5efU, 0xaae54faaU, 0xfb16edfbU,
-    0x43c58643U, 0x4dd79a4dU, 0x33556633U, 0x85941185U,
-    0x45cf8a45U, 0xf910e9f9U, 0x02060402U, 0x7f81fe7fU,
-    0x50f0a050U, 0x3c44783cU, 0x9fba259fU, 0xa8e34ba8U,
-    0x51f3a251U, 0xa3fe5da3U, 0x40c08040U, 0x8f8a058fU,
-    0x92ad3f92U, 0x9dbc219dU, 0x38487038U, 0xf504f1f5U,
-    0xbcdf63bcU, 0xb6c177b6U, 0xda75afdaU, 0x21634221U,
-    0x10302010U, 0xff1ae5ffU, 0xf30efdf3U, 0xd26dbfd2U,
-    0xcd4c81cdU, 0x0c14180cU, 0x13352613U, 0xec2fc3ecU,
-    0x5fe1be5fU, 0x97a23597U, 0x44cc8844U, 0x17392e17U,
-    0xc45793c4U, 0xa7f255a7U, 0x7e82fc7eU, 0x3d477a3dU,
-    0x64acc864U, 0x5de7ba5dU, 0x192b3219U, 0x7395e673U,
-    0x60a0c060U, 0x81981981U, 0x4fd19e4fU, 0xdc7fa3dcU,
-    0x22664422U, 0x2a7e542aU, 0x90ab3b90U, 0x88830b88U,
-    0x46ca8c46U, 0xee29c7eeU, 0xb8d36bb8U, 0x143c2814U,
-    0xde79a7deU, 0x5ee2bc5eU, 0x0b1d160bU, 0xdb76addbU,
-    0xe03bdbe0U, 0x32566432U, 0x3a4e743aU, 0x0a1e140aU,
-    0x49db9249U, 0x060a0c06U, 0x246c4824U, 0x5ce4b85cU,
-    0xc25d9fc2U, 0xd36ebdd3U, 0xacef43acU, 0x62a6c462U,
-    0x91a83991U, 0x95a43195U, 0xe437d3e4U, 0x798bf279U,
-    0xe732d5e7U, 0xc8438bc8U, 0x37596e37U, 0x6db7da6dU,
-    0x8d8c018dU, 0xd564b1d5U, 0x4ed29c4eU, 0xa9e049a9U,
-    0x6cb4d86cU, 0x56faac56U, 0xf407f3f4U, 0xea25cfeaU,
-    0x65afca65U, 0x7a8ef47aU, 0xaee947aeU, 0x08181008U,
-    0xbad56fbaU, 0x7888f078U, 0x256f4a25U, 0x2e725c2eU,
-    0x1c24381cU, 0xa6f157a6U, 0xb4c773b4U, 0xc65197c6U,
-    0xe823cbe8U, 0xdd7ca1ddU, 0x749ce874U, 0x1f213e1fU,
-    0x4bdd964bU, 0xbddc61bdU, 0x8b860d8bU, 0x8a850f8aU,
-    0x7090e070U, 0x3e427c3eU, 0xb5c471b5U, 0x66aacc66U,
-    0x48d89048U, 0x03050603U, 0xf601f7f6U, 0x0e121c0eU,
-    0x61a3c261U, 0x355f6a35U, 0x57f9ae57U, 0xb9d069b9U,
-    0x86911786U, 0xc15899c1U, 0x1d273a1dU, 0x9eb9279eU,
-    0xe138d9e1U, 0xf813ebf8U, 0x98b32b98U, 0x11332211U,
-    0x69bbd269U, 0xd970a9d9U, 0x8e89078eU, 0x94a73394U,
-    0x9bb62d9bU, 0x1e223c1eU, 0x87921587U, 0xe920c9e9U,
-    0xce4987ceU, 0x55ffaa55U, 0x28785028U, 0xdf7aa5dfU,
-    0x8c8f038cU, 0xa1f859a1U, 0x89800989U, 0x0d171a0dU,
-    0xbfda65bfU, 0xe631d7e6U, 0x42c68442U, 0x68b8d068U,
-    0x41c38241U, 0x99b02999U, 0x2d775a2dU, 0x0f111e0fU,
-    0xb0cb7bb0U, 0x54fca854U, 0xbbd66dbbU, 0x163a2c16U,
-};
-const uint32_t AES_Te3[256] = {
-
-    0x6363a5c6U, 0x7c7c84f8U, 0x777799eeU, 0x7b7b8df6U,
-    0xf2f20dffU, 0x6b6bbdd6U, 0x6f6fb1deU, 0xc5c55491U,
-    0x30305060U, 0x01010302U, 0x6767a9ceU, 0x2b2b7d56U,
-    0xfefe19e7U, 0xd7d762b5U, 0xababe64dU, 0x76769aecU,
-    0xcaca458fU, 0x82829d1fU, 0xc9c94089U, 0x7d7d87faU,
-    0xfafa15efU, 0x5959ebb2U, 0x4747c98eU, 0xf0f00bfbU,
-    0xadadec41U, 0xd4d467b3U, 0xa2a2fd5fU, 0xafafea45U,
-    0x9c9cbf23U, 0xa4a4f753U, 0x727296e4U, 0xc0c05b9bU,
-    0xb7b7c275U, 0xfdfd1ce1U, 0x9393ae3dU, 0x26266a4cU,
-    0x36365a6cU, 0x3f3f417eU, 0xf7f702f5U, 0xcccc4f83U,
-    0x34345c68U, 0xa5a5f451U, 0xe5e534d1U, 0xf1f108f9U,
-    0x717193e2U, 0xd8d873abU, 0x31315362U, 0x15153f2aU,
-    0x04040c08U, 0xc7c75295U, 0x23236546U, 0xc3c35e9dU,
-    0x18182830U, 0x9696a137U, 0x05050f0aU, 0x9a9ab52fU,
-    0x0707090eU, 0x12123624U, 0x80809b1bU, 0xe2e23ddfU,
-    0xebeb26cdU, 0x2727694eU, 0xb2b2cd7fU, 0x75759feaU,
-    0x09091b12U, 0x83839e1dU, 0x2c2c7458U, 0x1a1a2e34U,
-    0x1b1b2d36U, 0x6e6eb2dcU, 0x5a5aeeb4U, 0xa0a0fb5bU,
-    0x5252f6a4U, 0x3b3b4d76U, 0xd6d661b7U, 0xb3b3ce7dU,
-    0x29297b52U, 0xe3e33eddU, 0x2f2f715eU, 0x84849713U,
-    0x5353f5a6U, 0xd1d168b9U, 0x00000000U, 0xeded2cc1U,
-    0x20206040U, 0xfcfc1fe3U, 0xb1b1c879U, 0x5b5bedb6U,
-    0x6a6abed4U, 0xcbcb468dU, 0xbebed967U, 0x39394b72U,
-    0x4a4ade94U, 0x4c4cd498U, 0x5858e8b0U, 0xcfcf4a85U,
-    0xd0d06bbbU, 0xefef2ac5U, 0xaaaae54fU, 0xfbfb16edU,
-    0x4343c586U, 0x4d4dd79aU, 0x33335566U, 0x85859411U,
-    0x4545cf8aU, 0xf9f910e9U, 0x02020604U, 0x7f7f81feU,
-    0x5050f0a0U, 0x3c3c4478U, 0x9f9fba25U, 0xa8a8e34bU,
-    0x5151f3a2U, 0xa3a3fe5dU, 0x4040c080U, 0x8f8f8a05U,
-    0x9292ad3fU, 0x9d9dbc21U, 0x38384870U, 0xf5f504f1U,
-    0xbcbcdf63U, 0xb6b6c177U, 0xdada75afU, 0x21216342U,
-    0x10103020U, 0xffff1ae5U, 0xf3f30efdU, 0xd2d26dbfU,
-    0xcdcd4c81U, 0x0c0c1418U, 0x13133526U, 0xecec2fc3U,
-    0x5f5fe1beU, 0x9797a235U, 0x4444cc88U, 0x1717392eU,
-    0xc4c45793U, 0xa7a7f255U, 0x7e7e82fcU, 0x3d3d477aU,
-    0x6464acc8U, 0x5d5de7baU, 0x19192b32U, 0x737395e6U,
-    0x6060a0c0U, 0x81819819U, 0x4f4fd19eU, 0xdcdc7fa3U,
-    0x22226644U, 0x2a2a7e54U, 0x9090ab3bU, 0x8888830bU,
-    0x4646ca8cU, 0xeeee29c7U, 0xb8b8d36bU, 0x14143c28U,
-    0xdede79a7U, 0x5e5ee2bcU, 0x0b0b1d16U, 0xdbdb76adU,
-    0xe0e03bdbU, 0x32325664U, 0x3a3a4e74U, 0x0a0a1e14U,
-    0x4949db92U, 0x06060a0cU, 0x24246c48U, 0x5c5ce4b8U,
-    0xc2c25d9fU, 0xd3d36ebdU, 0xacacef43U, 0x6262a6c4U,
-    0x9191a839U, 0x9595a431U, 0xe4e437d3U, 0x79798bf2U,
-    0xe7e732d5U, 0xc8c8438bU, 0x3737596eU, 0x6d6db7daU,
-    0x8d8d8c01U, 0xd5d564b1U, 0x4e4ed29cU, 0xa9a9e049U,
-    0x6c6cb4d8U, 0x5656faacU, 0xf4f407f3U, 0xeaea25cfU,
-    0x6565afcaU, 0x7a7a8ef4U, 0xaeaee947U, 0x08081810U,
-    0xbabad56fU, 0x787888f0U, 0x25256f4aU, 0x2e2e725cU,
-    0x1c1c2438U, 0xa6a6f157U, 0xb4b4c773U, 0xc6c65197U,
-    0xe8e823cbU, 0xdddd7ca1U, 0x74749ce8U, 0x1f1f213eU,
-    0x4b4bdd96U, 0xbdbddc61U, 0x8b8b860dU, 0x8a8a850fU,
-    0x707090e0U, 0x3e3e427cU, 0xb5b5c471U, 0x6666aaccU,
-    0x4848d890U, 0x03030506U, 0xf6f601f7U, 0x0e0e121cU,
-    0x6161a3c2U, 0x35355f6aU, 0x5757f9aeU, 0xb9b9d069U,
-    0x86869117U, 0xc1c15899U, 0x1d1d273aU, 0x9e9eb927U,
-    0xe1e138d9U, 0xf8f813ebU, 0x9898b32bU, 0x11113322U,
-    0x6969bbd2U, 0xd9d970a9U, 0x8e8e8907U, 0x9494a733U,
-    0x9b9bb62dU, 0x1e1e223cU, 0x87879215U, 0xe9e920c9U,
-    0xcece4987U, 0x5555ffaaU, 0x28287850U, 0xdfdf7aa5U,
-    0x8c8c8f03U, 0xa1a1f859U, 0x89898009U, 0x0d0d171aU,
-    0xbfbfda65U, 0xe6e631d7U, 0x4242c684U, 0x6868b8d0U,
-    0x4141c382U, 0x9999b029U, 0x2d2d775aU, 0x0f0f111eU,
-    0xb0b0cb7bU, 0x5454fca8U, 0xbbbbd66dU, 0x16163a2cU,
-};
-const uint32_t AES_Te4[256] = {
-    0x63636363U, 0x7c7c7c7cU, 0x77777777U, 0x7b7b7b7bU,
-    0xf2f2f2f2U, 0x6b6b6b6bU, 0x6f6f6f6fU, 0xc5c5c5c5U,
-    0x30303030U, 0x01010101U, 0x67676767U, 0x2b2b2b2bU,
-    0xfefefefeU, 0xd7d7d7d7U, 0xababababU, 0x76767676U,
-    0xcacacacaU, 0x82828282U, 0xc9c9c9c9U, 0x7d7d7d7dU,
-    0xfafafafaU, 0x59595959U, 0x47474747U, 0xf0f0f0f0U,
-    0xadadadadU, 0xd4d4d4d4U, 0xa2a2a2a2U, 0xafafafafU,
-    0x9c9c9c9cU, 0xa4a4a4a4U, 0x72727272U, 0xc0c0c0c0U,
-    0xb7b7b7b7U, 0xfdfdfdfdU, 0x93939393U, 0x26262626U,
-    0x36363636U, 0x3f3f3f3fU, 0xf7f7f7f7U, 0xccccccccU,
-    0x34343434U, 0xa5a5a5a5U, 0xe5e5e5e5U, 0xf1f1f1f1U,
-    0x71717171U, 0xd8d8d8d8U, 0x31313131U, 0x15151515U,
-    0x04040404U, 0xc7c7c7c7U, 0x23232323U, 0xc3c3c3c3U,
-    0x18181818U, 0x96969696U, 0x05050505U, 0x9a9a9a9aU,
-    0x07070707U, 0x12121212U, 0x80808080U, 0xe2e2e2e2U,
-    0xebebebebU, 0x27272727U, 0xb2b2b2b2U, 0x75757575U,
-    0x09090909U, 0x83838383U, 0x2c2c2c2cU, 0x1a1a1a1aU,
-    0x1b1b1b1bU, 0x6e6e6e6eU, 0x5a5a5a5aU, 0xa0a0a0a0U,
-    0x52525252U, 0x3b3b3b3bU, 0xd6d6d6d6U, 0xb3b3b3b3U,
-    0x29292929U, 0xe3e3e3e3U, 0x2f2f2f2fU, 0x84848484U,
-    0x53535353U, 0xd1d1d1d1U, 0x00000000U, 0xededededU,
-    0x20202020U, 0xfcfcfcfcU, 0xb1b1b1b1U, 0x5b5b5b5bU,
-    0x6a6a6a6aU, 0xcbcbcbcbU, 0xbebebebeU, 0x39393939U,
-    0x4a4a4a4aU, 0x4c4c4c4cU, 0x58585858U, 0xcfcfcfcfU,
-    0xd0d0d0d0U, 0xefefefefU, 0xaaaaaaaaU, 0xfbfbfbfbU,
-    0x43434343U, 0x4d4d4d4dU, 0x33333333U, 0x85858585U,
-    0x45454545U, 0xf9f9f9f9U, 0x02020202U, 0x7f7f7f7fU,
-    0x50505050U, 0x3c3c3c3cU, 0x9f9f9f9fU, 0xa8a8a8a8U,
-    0x51515151U, 0xa3a3a3a3U, 0x40404040U, 0x8f8f8f8fU,
-    0x92929292U, 0x9d9d9d9dU, 0x38383838U, 0xf5f5f5f5U,
-    0xbcbcbcbcU, 0xb6b6b6b6U, 0xdadadadaU, 0x21212121U,
-    0x10101010U, 0xffffffffU, 0xf3f3f3f3U, 0xd2d2d2d2U,
-    0xcdcdcdcdU, 0x0c0c0c0cU, 0x13131313U, 0xececececU,
-    0x5f5f5f5fU, 0x97979797U, 0x44444444U, 0x17171717U,
-    0xc4c4c4c4U, 0xa7a7a7a7U, 0x7e7e7e7eU, 0x3d3d3d3dU,
-    0x64646464U, 0x5d5d5d5dU, 0x19191919U, 0x73737373U,
-    0x60606060U, 0x81818181U, 0x4f4f4f4fU, 0xdcdcdcdcU,
-    0x22222222U, 0x2a2a2a2aU, 0x90909090U, 0x88888888U,
-    0x46464646U, 0xeeeeeeeeU, 0xb8b8b8b8U, 0x14141414U,
-    0xdedededeU, 0x5e5e5e5eU, 0x0b0b0b0bU, 0xdbdbdbdbU,
-    0xe0e0e0e0U, 0x32323232U, 0x3a3a3a3aU, 0x0a0a0a0aU,
-    0x49494949U, 0x06060606U, 0x24242424U, 0x5c5c5c5cU,
-    0xc2c2c2c2U, 0xd3d3d3d3U, 0xacacacacU, 0x62626262U,
-    0x91919191U, 0x95959595U, 0xe4e4e4e4U, 0x79797979U,
-    0xe7e7e7e7U, 0xc8c8c8c8U, 0x37373737U, 0x6d6d6d6dU,
-    0x8d8d8d8dU, 0xd5d5d5d5U, 0x4e4e4e4eU, 0xa9a9a9a9U,
-    0x6c6c6c6cU, 0x56565656U, 0xf4f4f4f4U, 0xeaeaeaeaU,
-    0x65656565U, 0x7a7a7a7aU, 0xaeaeaeaeU, 0x08080808U,
-    0xbabababaU, 0x78787878U, 0x25252525U, 0x2e2e2e2eU,
-    0x1c1c1c1cU, 0xa6a6a6a6U, 0xb4b4b4b4U, 0xc6c6c6c6U,
-    0xe8e8e8e8U, 0xddddddddU, 0x74747474U, 0x1f1f1f1fU,
-    0x4b4b4b4bU, 0xbdbdbdbdU, 0x8b8b8b8bU, 0x8a8a8a8aU,
-    0x70707070U, 0x3e3e3e3eU, 0xb5b5b5b5U, 0x66666666U,
-    0x48484848U, 0x03030303U, 0xf6f6f6f6U, 0x0e0e0e0eU,
-    0x61616161U, 0x35353535U, 0x57575757U, 0xb9b9b9b9U,
-    0x86868686U, 0xc1c1c1c1U, 0x1d1d1d1dU, 0x9e9e9e9eU,
-    0xe1e1e1e1U, 0xf8f8f8f8U, 0x98989898U, 0x11111111U,
-    0x69696969U, 0xd9d9d9d9U, 0x8e8e8e8eU, 0x94949494U,
-    0x9b9b9b9bU, 0x1e1e1e1eU, 0x87878787U, 0xe9e9e9e9U,
-    0xcecececeU, 0x55555555U, 0x28282828U, 0xdfdfdfdfU,
-    0x8c8c8c8cU, 0xa1a1a1a1U, 0x89898989U, 0x0d0d0d0dU,
-    0xbfbfbfbfU, 0xe6e6e6e6U, 0x42424242U, 0x68686868U,
-    0x41414141U, 0x99999999U, 0x2d2d2d2dU, 0x0f0f0f0fU,
-    0xb0b0b0b0U, 0x54545454U, 0xbbbbbbbbU, 0x16161616U,
-};
-const uint32_t AES_Td0[256] = {
-    0x51f4a750U, 0x7e416553U, 0x1a17a4c3U, 0x3a275e96U,
-    0x3bab6bcbU, 0x1f9d45f1U, 0xacfa58abU, 0x4be30393U,
-    0x2030fa55U, 0xad766df6U, 0x88cc7691U, 0xf5024c25U,
-    0x4fe5d7fcU, 0xc52acbd7U, 0x26354480U, 0xb562a38fU,
-    0xdeb15a49U, 0x25ba1b67U, 0x45ea0e98U, 0x5dfec0e1U,
-    0xc32f7502U, 0x814cf012U, 0x8d4697a3U, 0x6bd3f9c6U,
-    0x038f5fe7U, 0x15929c95U, 0xbf6d7aebU, 0x955259daU,
-    0xd4be832dU, 0x587421d3U, 0x49e06929U, 0x8ec9c844U,
-    0x75c2896aU, 0xf48e7978U, 0x99583e6bU, 0x27b971ddU,
-    0xbee14fb6U, 0xf088ad17U, 0xc920ac66U, 0x7dce3ab4U,
-    0x63df4a18U, 0xe51a3182U, 0x97513360U, 0x62537f45U,
-    0xb16477e0U, 0xbb6bae84U, 0xfe81a01cU, 0xf9082b94U,
-    0x70486858U, 0x8f45fd19U, 0x94de6c87U, 0x527bf8b7U,
-    0xab73d323U, 0x724b02e2U, 0xe31f8f57U, 0x6655ab2aU,
-    0xb2eb2807U, 0x2fb5c203U, 0x86c57b9aU, 0xd33708a5U,
-    0x302887f2U, 0x23bfa5b2U, 0x02036abaU, 0xed16825cU,
-    0x8acf1c2bU, 0xa779b492U, 0xf307f2f0U, 0x4e69e2a1U,
-    0x65daf4cdU, 0x0605bed5U, 0xd134621fU, 0xc4a6fe8aU,
-    0x342e539dU, 0xa2f355a0U, 0x058ae132U, 0xa4f6eb75U,
-    0x0b83ec39U, 0x4060efaaU, 0x5e719f06U, 0xbd6e1051U,
-    0x3e218af9U, 0x96dd063dU, 0xdd3e05aeU, 0x4de6bd46U,
-    0x91548db5U, 0x71c45d05U, 0x0406d46fU, 0x605015ffU,
-    0x1998fb24U, 0xd6bde997U, 0x894043ccU, 0x67d99e77U,
-    0xb0e842bdU, 0x07898b88U, 0xe7195b38U, 0x79c8eedbU,
-    0xa17c0a47U, 0x7c420fe9U, 0xf8841ec9U, 0x00000000U,
-    0x09808683U, 0x322bed48U, 0x1e1170acU, 0x6c5a724eU,
-    0xfd0efffbU, 0x0f853856U, 0x3daed51eU, 0x362d3927U,
-    0x0a0fd964U, 0x685ca621U, 0x9b5b54d1U, 0x24362e3aU,
-    0x0c0a67b1U, 0x9357e70fU, 0xb4ee96d2U, 0x1b9b919eU,
-    0x80c0c54fU, 0x61dc20a2U, 0x5a774b69U, 0x1c121a16U,
-    0xe293ba0aU, 0xc0a02ae5U, 0x3c22e043U, 0x121b171dU,
-    0x0e090d0bU, 0xf28bc7adU, 0x2db6a8b9U, 0x141ea9c8U,
-    0x57f11985U, 0xaf75074cU, 0xee99ddbbU, 0xa37f60fdU,
-    0xf701269fU, 0x5c72f5bcU, 0x44663bc5U, 0x5bfb7e34U,
-    0x8b432976U, 0xcb23c6dcU, 0xb6edfc68U, 0xb8e4f163U,
-    0xd731dccaU, 0x42638510U, 0x13972240U, 0x84c61120U,
-    0x854a247dU, 0xd2bb3df8U, 0xaef93211U, 0xc729a16dU,
-    0x1d9e2f4bU, 0xdcb230f3U, 0x0d8652ecU, 0x77c1e3d0U,
-    0x2bb3166cU, 0xa970b999U, 0x119448faU, 0x47e96422U,
-    0xa8fc8cc4U, 0xa0f03f1aU, 0x567d2cd8U, 0x223390efU,
-    0x87494ec7U, 0xd938d1c1U, 0x8ccaa2feU, 0x98d40b36U,
-    0xa6f581cfU, 0xa57ade28U, 0xdab78e26U, 0x3fadbfa4U,
-    0x2c3a9de4U, 0x5078920dU, 0x6a5fcc9bU, 0x547e4662U,
-    0xf68d13c2U, 0x90d8b8e8U, 0x2e39f75eU, 0x82c3aff5U,
-    0x9f5d80beU, 0x69d0937cU, 0x6fd52da9U, 0xcf2512b3U,
-    0xc8ac993bU, 0x10187da7U, 0xe89c636eU, 0xdb3bbb7bU,
-    0xcd267809U, 0x6e5918f4U, 0xec9ab701U, 0x834f9aa8U,
-    0xe6956e65U, 0xaaffe67eU, 0x21bccf08U, 0xef15e8e6U,
-    0xbae79bd9U, 0x4a6f36ceU, 0xea9f09d4U, 0x29b07cd6U,
-    0x31a4b2afU, 0x2a3f2331U, 0xc6a59430U, 0x35a266c0U,
-    0x744ebc37U, 0xfc82caa6U, 0xe090d0b0U, 0x33a7d815U,
-    0xf104984aU, 0x41ecdaf7U, 0x7fcd500eU, 0x1791f62fU,
-    0x764dd68dU, 0x43efb04dU, 0xccaa4d54U, 0xe49604dfU,
-    0x9ed1b5e3U, 0x4c6a881bU, 0xc12c1fb8U, 0x4665517fU,
-    0x9d5eea04U, 0x018c355dU, 0xfa877473U, 0xfb0b412eU,
-    0xb3671d5aU, 0x92dbd252U, 0xe9105633U, 0x6dd64713U,
-    0x9ad7618cU, 0x37a10c7aU, 0x59f8148eU, 0xeb133c89U,
-    0xcea927eeU, 0xb761c935U, 0xe11ce5edU, 0x7a47b13cU,
-    0x9cd2df59U, 0x55f2733fU, 0x1814ce79U, 0x73c737bfU,
-    0x53f7cdeaU, 0x5ffdaa5bU, 0xdf3d6f14U, 0x7844db86U,
-    0xcaaff381U, 0xb968c43eU, 0x3824342cU, 0xc2a3405fU,
-    0x161dc372U, 0xbce2250cU, 0x283c498bU, 0xff0d9541U,
-    0x39a80171U, 0x080cb3deU, 0xd8b4e49cU, 0x6456c190U,
-    0x7bcb8461U, 0xd532b670U, 0x486c5c74U, 0xd0b85742U,
-};
-const uint32_t AES_Td1[256] = {
-    0x5051f4a7U, 0x537e4165U, 0xc31a17a4U, 0x963a275eU,
-    0xcb3bab6bU, 0xf11f9d45U, 0xabacfa58U, 0x934be303U,
-    0x552030faU, 0xf6ad766dU, 0x9188cc76U, 0x25f5024cU,
-    0xfc4fe5d7U, 0xd7c52acbU, 0x80263544U, 0x8fb562a3U,
-    0x49deb15aU, 0x6725ba1bU, 0x9845ea0eU, 0xe15dfec0U,
-    0x02c32f75U, 0x12814cf0U, 0xa38d4697U, 0xc66bd3f9U,
-    0xe7038f5fU, 0x9515929cU, 0xebbf6d7aU, 0xda955259U,
-    0x2dd4be83U, 0xd3587421U, 0x2949e069U, 0x448ec9c8U,
-    0x6a75c289U, 0x78f48e79U, 0x6b99583eU, 0xdd27b971U,
-    0xb6bee14fU, 0x17f088adU, 0x66c920acU, 0xb47dce3aU,
-    0x1863df4aU, 0x82e51a31U, 0x60975133U, 0x4562537fU,
-    0xe0b16477U, 0x84bb6baeU, 0x1cfe81a0U, 0x94f9082bU,
-    0x58704868U, 0x198f45fdU, 0x8794de6cU, 0xb7527bf8U,
-    0x23ab73d3U, 0xe2724b02U, 0x57e31f8fU, 0x2a6655abU,
-    0x07b2eb28U, 0x032fb5c2U, 0x9a86c57bU, 0xa5d33708U,
-    0xf2302887U, 0xb223bfa5U, 0xba02036aU, 0x5ced1682U,
-    0x2b8acf1cU, 0x92a779b4U, 0xf0f307f2U, 0xa14e69e2U,
-    0xcd65daf4U, 0xd50605beU, 0x1fd13462U, 0x8ac4a6feU,
-    0x9d342e53U, 0xa0a2f355U, 0x32058ae1U, 0x75a4f6ebU,
-    0x390b83ecU, 0xaa4060efU, 0x065e719fU, 0x51bd6e10U,
-    0xf93e218aU, 0x3d96dd06U, 0xaedd3e05U, 0x464de6bdU,
-    0xb591548dU, 0x0571c45dU, 0x6f0406d4U, 0xff605015U,
-    0x241998fbU, 0x97d6bde9U, 0xcc894043U, 0x7767d99eU,
-    0xbdb0e842U, 0x8807898bU, 0x38e7195bU, 0xdb79c8eeU,
-    0x47a17c0aU, 0xe97c420fU, 0xc9f8841eU, 0x00000000U,
-    0x83098086U, 0x48322bedU, 0xac1e1170U, 0x4e6c5a72U,
-    0xfbfd0effU, 0x560f8538U, 0x1e3daed5U, 0x27362d39U,
-    0x640a0fd9U, 0x21685ca6U, 0xd19b5b54U, 0x3a24362eU,
-    0xb10c0a67U, 0x0f9357e7U, 0xd2b4ee96U, 0x9e1b9b91U,
-    0x4f80c0c5U, 0xa261dc20U, 0x695a774bU, 0x161c121aU,
-    0x0ae293baU, 0xe5c0a02aU, 0x433c22e0U, 0x1d121b17U,
-    0x0b0e090dU, 0xadf28bc7U, 0xb92db6a8U, 0xc8141ea9U,
-    0x8557f119U, 0x4caf7507U, 0xbbee99ddU, 0xfda37f60U,
-    0x9ff70126U, 0xbc5c72f5U, 0xc544663bU, 0x345bfb7eU,
-    0x768b4329U, 0xdccb23c6U, 0x68b6edfcU, 0x63b8e4f1U,
-    0xcad731dcU, 0x10426385U, 0x40139722U, 0x2084c611U,
-    0x7d854a24U, 0xf8d2bb3dU, 0x11aef932U, 0x6dc729a1U,
-    0x4b1d9e2fU, 0xf3dcb230U, 0xec0d8652U, 0xd077c1e3U,
-    0x6c2bb316U, 0x99a970b9U, 0xfa119448U, 0x2247e964U,
-    0xc4a8fc8cU, 0x1aa0f03fU, 0xd8567d2cU, 0xef223390U,
-    0xc787494eU, 0xc1d938d1U, 0xfe8ccaa2U, 0x3698d40bU,
-    0xcfa6f581U, 0x28a57adeU, 0x26dab78eU, 0xa43fadbfU,
-    0xe42c3a9dU, 0x0d507892U, 0x9b6a5fccU, 0x62547e46U,
-    0xc2f68d13U, 0xe890d8b8U, 0x5e2e39f7U, 0xf582c3afU,
-    0xbe9f5d80U, 0x7c69d093U, 0xa96fd52dU, 0xb3cf2512U,
-    0x3bc8ac99U, 0xa710187dU, 0x6ee89c63U, 0x7bdb3bbbU,
-    0x09cd2678U, 0xf46e5918U, 0x01ec9ab7U, 0xa8834f9aU,
-    0x65e6956eU, 0x7eaaffe6U, 0x0821bccfU, 0xe6ef15e8U,
-    0xd9bae79bU, 0xce4a6f36U, 0xd4ea9f09U, 0xd629b07cU,
-    0xaf31a4b2U, 0x312a3f23U, 0x30c6a594U, 0xc035a266U,
-    0x37744ebcU, 0xa6fc82caU, 0xb0e090d0U, 0x1533a7d8U,
-    0x4af10498U, 0xf741ecdaU, 0x0e7fcd50U, 0x2f1791f6U,
-    0x8d764dd6U, 0x4d43efb0U, 0x54ccaa4dU, 0xdfe49604U,
-    0xe39ed1b5U, 0x1b4c6a88U, 0xb8c12c1fU, 0x7f466551U,
-    0x049d5eeaU, 0x5d018c35U, 0x73fa8774U, 0x2efb0b41U,
-    0x5ab3671dU, 0x5292dbd2U, 0x33e91056U, 0x136dd647U,
-    0x8c9ad761U, 0x7a37a10cU, 0x8e59f814U, 0x89eb133cU,
-    0xeecea927U, 0x35b761c9U, 0xede11ce5U, 0x3c7a47b1U,
-    0x599cd2dfU, 0x3f55f273U, 0x791814ceU, 0xbf73c737U,
-    0xea53f7cdU, 0x5b5ffdaaU, 0x14df3d6fU, 0x867844dbU,
-    0x81caaff3U, 0x3eb968c4U, 0x2c382434U, 0x5fc2a340U,
-    0x72161dc3U, 0x0cbce225U, 0x8b283c49U, 0x41ff0d95U,
-    0x7139a801U, 0xde080cb3U, 0x9cd8b4e4U, 0x906456c1U,
-    0x617bcb84U, 0x70d532b6U, 0x74486c5cU, 0x42d0b857U,
-};
-const uint32_t AES_Td2[256] = {
-    0xa75051f4U, 0x65537e41U, 0xa4c31a17U, 0x5e963a27U,
-    0x6bcb3babU, 0x45f11f9dU, 0x58abacfaU, 0x03934be3U,
-    0xfa552030U, 0x6df6ad76U, 0x769188ccU, 0x4c25f502U,
-    0xd7fc4fe5U, 0xcbd7c52aU, 0x44802635U, 0xa38fb562U,
-    0x5a49deb1U, 0x1b6725baU, 0x0e9845eaU, 0xc0e15dfeU,
-    0x7502c32fU, 0xf012814cU, 0x97a38d46U, 0xf9c66bd3U,
-    0x5fe7038fU, 0x9c951592U, 0x7aebbf6dU, 0x59da9552U,
-    0x832dd4beU, 0x21d35874U, 0x692949e0U, 0xc8448ec9U,
-    0x896a75c2U, 0x7978f48eU, 0x3e6b9958U, 0x71dd27b9U,
-    0x4fb6bee1U, 0xad17f088U, 0xac66c920U, 0x3ab47dceU,
-    0x4a1863dfU, 0x3182e51aU, 0x33609751U, 0x7f456253U,
-    0x77e0b164U, 0xae84bb6bU, 0xa01cfe81U, 0x2b94f908U,
-    0x68587048U, 0xfd198f45U, 0x6c8794deU, 0xf8b7527bU,
-    0xd323ab73U, 0x02e2724bU, 0x8f57e31fU, 0xab2a6655U,
-    0x2807b2ebU, 0xc2032fb5U, 0x7b9a86c5U, 0x08a5d337U,
-    0x87f23028U, 0xa5b223bfU, 0x6aba0203U, 0x825ced16U,
-    0x1c2b8acfU, 0xb492a779U, 0xf2f0f307U, 0xe2a14e69U,
-    0xf4cd65daU, 0xbed50605U, 0x621fd134U, 0xfe8ac4a6U,
-    0x539d342eU, 0x55a0a2f3U, 0xe132058aU, 0xeb75a4f6U,
-    0xec390b83U, 0xefaa4060U, 0x9f065e71U, 0x1051bd6eU,
-
-    0x8af93e21U, 0x063d96ddU, 0x05aedd3eU, 0xbd464de6U,
-    0x8db59154U, 0x5d0571c4U, 0xd46f0406U, 0x15ff6050U,
-    0xfb241998U, 0xe997d6bdU, 0x43cc8940U, 0x9e7767d9U,
-    0x42bdb0e8U, 0x8b880789U, 0x5b38e719U, 0xeedb79c8U,
-    0x0a47a17cU, 0x0fe97c42U, 0x1ec9f884U, 0x00000000U,
-    0x86830980U, 0xed48322bU, 0x70ac1e11U, 0x724e6c5aU,
-    0xfffbfd0eU, 0x38560f85U, 0xd51e3daeU, 0x3927362dU,
-    0xd9640a0fU, 0xa621685cU, 0x54d19b5bU, 0x2e3a2436U,
-    0x67b10c0aU, 0xe70f9357U, 0x96d2b4eeU, 0x919e1b9bU,
-    0xc54f80c0U, 0x20a261dcU, 0x4b695a77U, 0x1a161c12U,
-    0xba0ae293U, 0x2ae5c0a0U, 0xe0433c22U, 0x171d121bU,
-    0x0d0b0e09U, 0xc7adf28bU, 0xa8b92db6U, 0xa9c8141eU,
-    0x198557f1U, 0x074caf75U, 0xddbbee99U, 0x60fda37fU,
-    0x269ff701U, 0xf5bc5c72U, 0x3bc54466U, 0x7e345bfbU,
-    0x29768b43U, 0xc6dccb23U, 0xfc68b6edU, 0xf163b8e4U,
-    0xdccad731U, 0x85104263U, 0x22401397U, 0x112084c6U,
-    0x247d854aU, 0x3df8d2bbU, 0x3211aef9U, 0xa16dc729U,
-    0x2f4b1d9eU, 0x30f3dcb2U, 0x52ec0d86U, 0xe3d077c1U,
-    0x166c2bb3U, 0xb999a970U, 0x48fa1194U, 0x642247e9U,
-    0x8cc4a8fcU, 0x3f1aa0f0U, 0x2cd8567dU, 0x90ef2233U,
-    0x4ec78749U, 0xd1c1d938U, 0xa2fe8ccaU, 0x0b3698d4U,
-    0x81cfa6f5U, 0xde28a57aU, 0x8e26dab7U, 0xbfa43fadU,
-    0x9de42c3aU, 0x920d5078U, 0xcc9b6a5fU, 0x4662547eU,
-    0x13c2f68dU, 0xb8e890d8U, 0xf75e2e39U, 0xaff582c3U,
-    0x80be9f5dU, 0x937c69d0U, 0x2da96fd5U, 0x12b3cf25U,
-    0x993bc8acU, 0x7da71018U, 0x636ee89cU, 0xbb7bdb3bU,
-    0x7809cd26U, 0x18f46e59U, 0xb701ec9aU, 0x9aa8834fU,
-    0x6e65e695U, 0xe67eaaffU, 0xcf0821bcU, 0xe8e6ef15U,
-    0x9bd9bae7U, 0x36ce4a6fU, 0x09d4ea9fU, 0x7cd629b0U,
-    0xb2af31a4U, 0x23312a3fU, 0x9430c6a5U, 0x66c035a2U,
-    0xbc37744eU, 0xcaa6fc82U, 0xd0b0e090U, 0xd81533a7U,
-    0x984af104U, 0xdaf741ecU, 0x500e7fcdU, 0xf62f1791U,
-    0xd68d764dU, 0xb04d43efU, 0x4d54ccaaU, 0x04dfe496U,
-    0xb5e39ed1U, 0x881b4c6aU, 0x1fb8c12cU, 0x517f4665U,
-    0xea049d5eU, 0x355d018cU, 0x7473fa87U, 0x412efb0bU,
-    0x1d5ab367U, 0xd25292dbU, 0x5633e910U, 0x47136dd6U,
-    0x618c9ad7U, 0x0c7a37a1U, 0x148e59f8U, 0x3c89eb13U,
-    0x27eecea9U, 0xc935b761U, 0xe5ede11cU, 0xb13c7a47U,
-    0xdf599cd2U, 0x733f55f2U, 0xce791814U, 0x37bf73c7U,
-    0xcdea53f7U, 0xaa5b5ffdU, 0x6f14df3dU, 0xdb867844U,
-    0xf381caafU, 0xc43eb968U, 0x342c3824U, 0x405fc2a3U,
-    0xc372161dU, 0x250cbce2U, 0x498b283cU, 0x9541ff0dU,
-    0x017139a8U, 0xb3de080cU, 0xe49cd8b4U, 0xc1906456U,
-    0x84617bcbU, 0xb670d532U, 0x5c74486cU, 0x5742d0b8U,
-};
-const uint32_t AES_Td3[256] = {
-    0xf4a75051U, 0x4165537eU, 0x17a4c31aU, 0x275e963aU,
-    0xab6bcb3bU, 0x9d45f11fU, 0xfa58abacU, 0xe303934bU,
-    0x30fa5520U, 0x766df6adU, 0xcc769188U, 0x024c25f5U,
-    0xe5d7fc4fU, 0x2acbd7c5U, 0x35448026U, 0x62a38fb5U,
-    0xb15a49deU, 0xba1b6725U, 0xea0e9845U, 0xfec0e15dU,
-    0x2f7502c3U, 0x4cf01281U, 0x4697a38dU, 0xd3f9c66bU,
-    0x8f5fe703U, 0x929c9515U, 0x6d7aebbfU, 0x5259da95U,
-    0xbe832dd4U, 0x7421d358U, 0xe0692949U, 0xc9c8448eU,
-    0xc2896a75U, 0x8e7978f4U, 0x583e6b99U, 0xb971dd27U,
-    0xe14fb6beU, 0x88ad17f0U, 0x20ac66c9U, 0xce3ab47dU,
-    0xdf4a1863U, 0x1a3182e5U, 0x51336097U, 0x537f4562U,
-    0x6477e0b1U, 0x6bae84bbU, 0x81a01cfeU, 0x082b94f9U,
-    0x48685870U, 0x45fd198fU, 0xde6c8794U, 0x7bf8b752U,
-    0x73d323abU, 0x4b02e272U, 0x1f8f57e3U, 0x55ab2a66U,
-    0xeb2807b2U, 0xb5c2032fU, 0xc57b9a86U, 0x3708a5d3U,
-    0x2887f230U, 0xbfa5b223U, 0x036aba02U, 0x16825cedU,
-    0xcf1c2b8aU, 0x79b492a7U, 0x07f2f0f3U, 0x69e2a14eU,
-    0xdaf4cd65U, 0x05bed506U, 0x34621fd1U, 0xa6fe8ac4U,
-    0x2e539d34U, 0xf355a0a2U, 0x8ae13205U, 0xf6eb75a4U,
-    0x83ec390bU, 0x60efaa40U, 0x719f065eU, 0x6e1051bdU,
-    0x218af93eU, 0xdd063d96U, 0x3e05aeddU, 0xe6bd464dU,
-    0x548db591U, 0xc45d0571U, 0x06d46f04U, 0x5015ff60U,
-    0x98fb2419U, 0xbde997d6U, 0x4043cc89U, 0xd99e7767U,
-    0xe842bdb0U, 0x898b8807U, 0x195b38e7U, 0xc8eedb79U,
-    0x7c0a47a1U, 0x420fe97cU, 0x841ec9f8U, 0x00000000U,
-    0x80868309U, 0x2bed4832U, 0x1170ac1eU, 0x5a724e6cU,
-    0x0efffbfdU, 0x8538560fU, 0xaed51e3dU, 0x2d392736U,
-    0x0fd9640aU, 0x5ca62168U, 0x5b54d19bU, 0x362e3a24U,
-    0x0a67b10cU, 0x57e70f93U, 0xee96d2b4U, 0x9b919e1bU,
-    0xc0c54f80U, 0xdc20a261U, 0x774b695aU, 0x121a161cU,
-    0x93ba0ae2U, 0xa02ae5c0U, 0x22e0433cU, 0x1b171d12U,
-    0x090d0b0eU, 0x8bc7adf2U, 0xb6a8b92dU, 0x1ea9c814U,
-    0xf1198557U, 0x75074cafU, 0x99ddbbeeU, 0x7f60fda3U,
-    0x01269ff7U, 0x72f5bc5cU, 0x663bc544U, 0xfb7e345bU,
-    0x4329768bU, 0x23c6dccbU, 0xedfc68b6U, 0xe4f163b8U,
-    0x31dccad7U, 0x63851042U, 0x97224013U, 0xc6112084U,
-    0x4a247d85U, 0xbb3df8d2U, 0xf93211aeU, 0x29a16dc7U,
-    0x9e2f4b1dU, 0xb230f3dcU, 0x8652ec0dU, 0xc1e3d077U,
-    0xb3166c2bU, 0x70b999a9U, 0x9448fa11U, 0xe9642247U,
-    0xfc8cc4a8U, 0xf03f1aa0U, 0x7d2cd856U, 0x3390ef22U,
-    0x494ec787U, 0x38d1c1d9U, 0xcaa2fe8cU, 0xd40b3698U,
-    0xf581cfa6U, 0x7ade28a5U, 0xb78e26daU, 0xadbfa43fU,
-    0x3a9de42cU, 0x78920d50U, 0x5fcc9b6aU, 0x7e466254U,
-    0x8d13c2f6U, 0xd8b8e890U, 0x39f75e2eU, 0xc3aff582U,
-    0x5d80be9fU, 0xd0937c69U, 0xd52da96fU, 0x2512b3cfU,
-    0xac993bc8U, 0x187da710U, 0x9c636ee8U, 0x3bbb7bdbU,
-    0x267809cdU, 0x5918f46eU, 0x9ab701ecU, 0x4f9aa883U,
-    0x956e65e6U, 0xffe67eaaU, 0xbccf0821U, 0x15e8e6efU,
-    0xe79bd9baU, 0x6f36ce4aU, 0x9f09d4eaU, 0xb07cd629U,
-    0xa4b2af31U, 0x3f23312aU, 0xa59430c6U, 0xa266c035U,
-    0x4ebc3774U, 0x82caa6fcU, 0x90d0b0e0U, 0xa7d81533U,
-    0x04984af1U, 0xecdaf741U, 0xcd500e7fU, 0x91f62f17U,
-    0x4dd68d76U, 0xefb04d43U, 0xaa4d54ccU, 0x9604dfe4U,
-    0xd1b5e39eU, 0x6a881b4cU, 0x2c1fb8c1U, 0x65517f46U,
-    0x5eea049dU, 0x8c355d01U, 0x877473faU, 0x0b412efbU,
-    0x671d5ab3U, 0xdbd25292U, 0x105633e9U, 0xd647136dU,
-    0xd7618c9aU, 0xa10c7a37U, 0xf8148e59U, 0x133c89ebU,
-    0xa927eeceU, 0x61c935b7U, 0x1ce5ede1U, 0x47b13c7aU,
-    0xd2df599cU, 0xf2733f55U, 0x14ce7918U, 0xc737bf73U,
-    0xf7cdea53U, 0xfdaa5b5fU, 0x3d6f14dfU, 0x44db8678U,
-    0xaff381caU, 0x68c43eb9U, 0x24342c38U, 0xa3405fc2U,
-    0x1dc37216U, 0xe2250cbcU, 0x3c498b28U, 0x0d9541ffU,
-    0xa8017139U, 0x0cb3de08U, 0xb4e49cd8U, 0x56c19064U,
-    0xcb84617bU, 0x32b670d5U, 0x6c5c7448U, 0xb85742d0U,
-};
-const uint32_t AES_Td4[256] = {
-    0x52525252U, 0x09090909U, 0x6a6a6a6aU, 0xd5d5d5d5U,
-    0x30303030U, 0x36363636U, 0xa5a5a5a5U, 0x38383838U,
-    0xbfbfbfbfU, 0x40404040U, 0xa3a3a3a3U, 0x9e9e9e9eU,
-    0x81818181U, 0xf3f3f3f3U, 0xd7d7d7d7U, 0xfbfbfbfbU,
-    0x7c7c7c7cU, 0xe3e3e3e3U, 0x39393939U, 0x82828282U,
-    0x9b9b9b9bU, 0x2f2f2f2fU, 0xffffffffU, 0x87878787U,
-    0x34343434U, 0x8e8e8e8eU, 0x43434343U, 0x44444444U,
-    0xc4c4c4c4U, 0xdedededeU, 0xe9e9e9e9U, 0xcbcbcbcbU,
-    0x54545454U, 0x7b7b7b7bU, 0x94949494U, 0x32323232U,
-    0xa6a6a6a6U, 0xc2c2c2c2U, 0x23232323U, 0x3d3d3d3dU,
-    0xeeeeeeeeU, 0x4c4c4c4cU, 0x95959595U, 0x0b0b0b0bU,
-    0x42424242U, 0xfafafafaU, 0xc3c3c3c3U, 0x4e4e4e4eU,
-    0x08080808U, 0x2e2e2e2eU, 0xa1a1a1a1U, 0x66666666U,
-    0x28282828U, 0xd9d9d9d9U, 0x24242424U, 0xb2b2b2b2U,
-    0x76767676U, 0x5b5b5b5bU, 0xa2a2a2a2U, 0x49494949U,
-    0x6d6d6d6dU, 0x8b8b8b8bU, 0xd1d1d1d1U, 0x25252525U,
-    0x72727272U, 0xf8f8f8f8U, 0xf6f6f6f6U, 0x64646464U,
-    0x86868686U, 0x68686868U, 0x98989898U, 0x16161616U,
-    0xd4d4d4d4U, 0xa4a4a4a4U, 0x5c5c5c5cU, 0xccccccccU,
-    0x5d5d5d5dU, 0x65656565U, 0xb6b6b6b6U, 0x92929292U,
-    0x6c6c6c6cU, 0x70707070U, 0x48484848U, 0x50505050U,
-    0xfdfdfdfdU, 0xededededU, 0xb9b9b9b9U, 0xdadadadaU,
-    0x5e5e5e5eU, 0x15151515U, 0x46464646U, 0x57575757U,
-    0xa7a7a7a7U, 0x8d8d8d8dU, 0x9d9d9d9dU, 0x84848484U,
-    0x90909090U, 0xd8d8d8d8U, 0xababababU, 0x00000000U,
-    0x8c8c8c8cU, 0xbcbcbcbcU, 0xd3d3d3d3U, 0x0a0a0a0aU,
-    0xf7f7f7f7U, 0xe4e4e4e4U, 0x58585858U, 0x05050505U,
-    0xb8b8b8b8U, 0xb3b3b3b3U, 0x45454545U, 0x06060606U,
-    0xd0d0d0d0U, 0x2c2c2c2cU, 0x1e1e1e1eU, 0x8f8f8f8fU,
-    0xcacacacaU, 0x3f3f3f3fU, 0x0f0f0f0fU, 0x02020202U,
-    0xc1c1c1c1U, 0xafafafafU, 0xbdbdbdbdU, 0x03030303U,
-    0x01010101U, 0x13131313U, 0x8a8a8a8aU, 0x6b6b6b6bU,
-    0x3a3a3a3aU, 0x91919191U, 0x11111111U, 0x41414141U,
-    0x4f4f4f4fU, 0x67676767U, 0xdcdcdcdcU, 0xeaeaeaeaU,
-    0x97979797U, 0xf2f2f2f2U, 0xcfcfcfcfU, 0xcecececeU,
-    0xf0f0f0f0U, 0xb4b4b4b4U, 0xe6e6e6e6U, 0x73737373U,
-    0x96969696U, 0xacacacacU, 0x74747474U, 0x22222222U,
-    0xe7e7e7e7U, 0xadadadadU, 0x35353535U, 0x85858585U,
-    0xe2e2e2e2U, 0xf9f9f9f9U, 0x37373737U, 0xe8e8e8e8U,
-    0x1c1c1c1cU, 0x75757575U, 0xdfdfdfdfU, 0x6e6e6e6eU,
-    0x47474747U, 0xf1f1f1f1U, 0x1a1a1a1aU, 0x71717171U,
-    0x1d1d1d1dU, 0x29292929U, 0xc5c5c5c5U, 0x89898989U,
-    0x6f6f6f6fU, 0xb7b7b7b7U, 0x62626262U, 0x0e0e0e0eU,
-    0xaaaaaaaaU, 0x18181818U, 0xbebebebeU, 0x1b1b1b1bU,
-    0xfcfcfcfcU, 0x56565656U, 0x3e3e3e3eU, 0x4b4b4b4bU,
-    0xc6c6c6c6U, 0xd2d2d2d2U, 0x79797979U, 0x20202020U,
-    0x9a9a9a9aU, 0xdbdbdbdbU, 0xc0c0c0c0U, 0xfefefefeU,
-    0x78787878U, 0xcdcdcdcdU, 0x5a5a5a5aU, 0xf4f4f4f4U,
-    0x1f1f1f1fU, 0xddddddddU, 0xa8a8a8a8U, 0x33333333U,
-    0x88888888U, 0x07070707U, 0xc7c7c7c7U, 0x31313131U,
-    0xb1b1b1b1U, 0x12121212U, 0x10101010U, 0x59595959U,
-    0x27272727U, 0x80808080U, 0xececececU, 0x5f5f5f5fU,
-    0x60606060U, 0x51515151U, 0x7f7f7f7fU, 0xa9a9a9a9U,
-    0x19191919U, 0xb5b5b5b5U, 0x4a4a4a4aU, 0x0d0d0d0dU,
-    0x2d2d2d2dU, 0xe5e5e5e5U, 0x7a7a7a7aU, 0x9f9f9f9fU,
-    0x93939393U, 0xc9c9c9c9U, 0x9c9c9c9cU, 0xefefefefU,
-    0xa0a0a0a0U, 0xe0e0e0e0U, 0x3b3b3b3bU, 0x4d4d4d4dU,
-    0xaeaeaeaeU, 0x2a2a2a2aU, 0xf5f5f5f5U, 0xb0b0b0b0U,
-    0xc8c8c8c8U, 0xebebebebU, 0xbbbbbbbbU, 0x3c3c3c3cU,
-    0x83838383U, 0x53535353U, 0x99999999U, 0x61616161U,
-    0x17171717U, 0x2b2b2b2bU, 0x04040404U, 0x7e7e7e7eU,
-    0xbabababaU, 0x77777777U, 0xd6d6d6d6U, 0x26262626U,
-    0xe1e1e1e1U, 0x69696969U, 0x14141414U, 0x63636363U,
-    0x55555555U, 0x21212121U, 0x0c0c0c0cU, 0x7d7d7d7dU,
-};
-static const u32 rcon[] = {
-	0x01000000, 0x02000000, 0x04000000, 0x08000000,
-	0x10000000, 0x20000000, 0x40000000, 0x80000000,
-	0x1B000000, 0x36000000, /* for 128-bit blocks, Rijndael never uses more than 10 rcon values */
-};
-
-/**
- * Expand the cipher key into the encryption key schedule.
- */
-int AES_set_encrypt_key(const unsigned char *userKey, const int bits,
-			AES_KEY *key) {
-
-	u32 *rk;
-   	int i = 0;
-	u32 temp;
-
-	if (!userKey || !key)
-		return -1;
-	if (bits != 128 && bits != 192 && bits != 256)
-		return -2;
-
-	rk = key->rd_key;
-
-	if (bits==128)
-		key->rounds = 10;
-	else if (bits==192)
-		key->rounds = 12;
-	else
-		key->rounds = 14;
-
-	rk[0] = GETU32(userKey     );
-	rk[1] = GETU32(userKey +  4);
-	rk[2] = GETU32(userKey +  8);
-	rk[3] = GETU32(userKey + 12);
-	if (bits == 128) {
-		while (1) {
-			temp  = rk[3];
-			rk[4] = rk[0] ^
-                                (AES_Te4[(temp >> 16) & 0xff] & 0xff000000) ^
-                                (AES_Te4[(temp >>  8) & 0xff] & 0x00ff0000) ^
-                                (AES_Te4[(temp      ) & 0xff] & 0x0000ff00) ^
-                                (AES_Te4[(temp >> 24)       ] & 0x000000ff) ^
-				rcon[i];
-			rk[5] = rk[1] ^ rk[4];
-			rk[6] = rk[2] ^ rk[5];
-			rk[7] = rk[3] ^ rk[6];
-			if (++i == 10) {
-				return 0;
-			}
-			rk += 4;
-		}
-	}
-	rk[4] = GETU32(userKey + 16);
-	rk[5] = GETU32(userKey + 20);
-	if (bits == 192) {
-		while (1) {
-			temp = rk[ 5];
-			rk[ 6] = rk[ 0] ^
-                                (AES_Te4[(temp >> 16) & 0xff] & 0xff000000) ^
-                                (AES_Te4[(temp >>  8) & 0xff] & 0x00ff0000) ^
-                                (AES_Te4[(temp      ) & 0xff] & 0x0000ff00) ^
-                                (AES_Te4[(temp >> 24)       ] & 0x000000ff) ^
-				rcon[i];
-			rk[ 7] = rk[ 1] ^ rk[ 6];
-			rk[ 8] = rk[ 2] ^ rk[ 7];
-			rk[ 9] = rk[ 3] ^ rk[ 8];
-			if (++i == 8) {
-				return 0;
-			}
-			rk[10] = rk[ 4] ^ rk[ 9];
-			rk[11] = rk[ 5] ^ rk[10];
-			rk += 6;
-		}
-	}
-	rk[6] = GETU32(userKey + 24);
-	rk[7] = GETU32(userKey + 28);
-	if (bits == 256) {
-		while (1) {
-			temp = rk[ 7];
-			rk[ 8] = rk[ 0] ^
-                                (AES_Te4[(temp >> 16) & 0xff] & 0xff000000) ^
-                                (AES_Te4[(temp >>  8) & 0xff] & 0x00ff0000) ^
-                                (AES_Te4[(temp      ) & 0xff] & 0x0000ff00) ^
-                                (AES_Te4[(temp >> 24)       ] & 0x000000ff) ^
-				rcon[i];
-			rk[ 9] = rk[ 1] ^ rk[ 8];
-			rk[10] = rk[ 2] ^ rk[ 9];
-			rk[11] = rk[ 3] ^ rk[10];
-			if (++i == 7) {
-				return 0;
-			}
-			temp = rk[11];
-			rk[12] = rk[ 4] ^
-                                (AES_Te4[(temp >> 24)       ] & 0xff000000) ^
-                                (AES_Te4[(temp >> 16) & 0xff] & 0x00ff0000) ^
-                                (AES_Te4[(temp >>  8) & 0xff] & 0x0000ff00) ^
-                                (AES_Te4[(temp      ) & 0xff] & 0x000000ff);
-			rk[13] = rk[ 5] ^ rk[12];
-			rk[14] = rk[ 6] ^ rk[13];
-			rk[15] = rk[ 7] ^ rk[14];
-
-			rk += 8;
-        	}
-	}
-	return 0;
-}
-
-/**
- * Expand the cipher key into the decryption key schedule.
- */
-int AES_set_decrypt_key(const unsigned char *userKey, const int bits,
-			 AES_KEY *key) {
-
-        u32 *rk;
-	int i, j, status;
-	u32 temp;
-
-	/* first, start with an encryption schedule */
-	status = AES_set_encrypt_key(userKey, bits, key);
-	if (status < 0)
-		return status;
-
-	rk = key->rd_key;
-
-	/* invert the order of the round keys: */
-	for (i = 0, j = 4*(key->rounds); i < j; i += 4, j -= 4) {
-		temp = rk[i    ]; rk[i    ] = rk[j    ]; rk[j    ] = temp;
-		temp = rk[i + 1]; rk[i + 1] = rk[j + 1]; rk[j + 1] = temp;
-		temp = rk[i + 2]; rk[i + 2] = rk[j + 2]; rk[j + 2] = temp;
-		temp = rk[i + 3]; rk[i + 3] = rk[j + 3]; rk[j + 3] = temp;
-	}
-	/* apply the inverse MixColumn transform to all round keys but the first and the last: */
-	for (i = 1; i < (key->rounds); i++) {
-		rk += 4;
-		rk[0] =
-                        AES_Td0[AES_Te4[(rk[0] >> 24)       ] & 0xff] ^
-                        AES_Td1[AES_Te4[(rk[0] >> 16) & 0xff] & 0xff] ^
-                        AES_Td2[AES_Te4[(rk[0] >>  8) & 0xff] & 0xff] ^
-                        AES_Td3[AES_Te4[(rk[0]      ) & 0xff] & 0xff];
-		rk[1] =
-                        AES_Td0[AES_Te4[(rk[1] >> 24)       ] & 0xff] ^
-                        AES_Td1[AES_Te4[(rk[1] >> 16) & 0xff] & 0xff] ^
-                        AES_Td2[AES_Te4[(rk[1] >>  8) & 0xff] & 0xff] ^
-                        AES_Td3[AES_Te4[(rk[1]      ) & 0xff] & 0xff];
-		rk[2] =
-                        AES_Td0[AES_Te4[(rk[2] >> 24)       ] & 0xff] ^
-                        AES_Td1[AES_Te4[(rk[2] >> 16) & 0xff] & 0xff] ^
-                        AES_Td2[AES_Te4[(rk[2] >>  8) & 0xff] & 0xff] ^
-                        AES_Td3[AES_Te4[(rk[2]      ) & 0xff] & 0xff];
-		rk[3] =
-                        AES_Td0[AES_Te4[(rk[3] >> 24)       ] & 0xff] ^
-                        AES_Td1[AES_Te4[(rk[3] >> 16) & 0xff] & 0xff] ^
-                        AES_Td2[AES_Te4[(rk[3] >>  8) & 0xff] & 0xff] ^
-                        AES_Td3[AES_Te4[(rk[3]      ) & 0xff] & 0xff];
-	}
-	return 0;
-}
-
-#ifndef AES_ASM
-/*
- * Encrypt a single block
- * in and out can overlap
- */
-void AES_encrypt(const unsigned char *in, unsigned char *out,
-		 const AES_KEY *key) {
-
-	const u32 *rk;
-	u32 s0, s1, s2, s3, t0, t1, t2, t3;
-#ifndef FULL_UNROLL
-	int r;
-#endif /* ?FULL_UNROLL */
-
-	assert(in && out && key);
-	rk = key->rd_key;
-
-	/*
-	 * map byte array block to cipher state
-	 * and add initial round key:
-	 */
-	s0 = GETU32(in     ) ^ rk[0];
-	s1 = GETU32(in +  4) ^ rk[1];
-	s2 = GETU32(in +  8) ^ rk[2];
-	s3 = GETU32(in + 12) ^ rk[3];
-#ifdef FULL_UNROLL
-	/* round 1: */
-        t0 = AES_Te0[s0 >> 24] ^ AES_Te1[(s1 >> 16) & 0xff] ^ AES_Te2[(s2 >>  8) & 0xff] ^ AES_Te3[s3 & 0xff] ^ rk[ 4];
-        t1 = AES_Te0[s1 >> 24] ^ AES_Te1[(s2 >> 16) & 0xff] ^ AES_Te2[(s3 >>  8) & 0xff] ^ AES_Te3[s0 & 0xff] ^ rk[ 5];
-        t2 = AES_Te0[s2 >> 24] ^ AES_Te1[(s3 >> 16) & 0xff] ^ AES_Te2[(s0 >>  8) & 0xff] ^ AES_Te3[s1 & 0xff] ^ rk[ 6];
-        t3 = AES_Te0[s3 >> 24] ^ AES_Te1[(s0 >> 16) & 0xff] ^ AES_Te2[(s1 >>  8) & 0xff] ^ AES_Te3[s2 & 0xff] ^ rk[ 7];
-   	/* round 2: */
-        s0 = AES_Te0[t0 >> 24] ^ AES_Te1[(t1 >> 16) & 0xff] ^ AES_Te2[(t2 >>  8) & 0xff] ^ AES_Te3[t3 & 0xff] ^ rk[ 8];
-        s1 = AES_Te0[t1 >> 24] ^ AES_Te1[(t2 >> 16) & 0xff] ^ AES_Te2[(t3 >>  8) & 0xff] ^ AES_Te3[t0 & 0xff] ^ rk[ 9];
-        s2 = AES_Te0[t2 >> 24] ^ AES_Te1[(t3 >> 16) & 0xff] ^ AES_Te2[(t0 >>  8) & 0xff] ^ AES_Te3[t1 & 0xff] ^ rk[10];
-        s3 = AES_Te0[t3 >> 24] ^ AES_Te1[(t0 >> 16) & 0xff] ^ AES_Te2[(t1 >>  8) & 0xff] ^ AES_Te3[t2 & 0xff] ^ rk[11];
-	/* round 3: */
-        t0 = AES_Te0[s0 >> 24] ^ AES_Te1[(s1 >> 16) & 0xff] ^ AES_Te2[(s2 >>  8) & 0xff] ^ AES_Te3[s3 & 0xff] ^ rk[12];
-        t1 = AES_Te0[s1 >> 24] ^ AES_Te1[(s2 >> 16) & 0xff] ^ AES_Te2[(s3 >>  8) & 0xff] ^ AES_Te3[s0 & 0xff] ^ rk[13];
-        t2 = AES_Te0[s2 >> 24] ^ AES_Te1[(s3 >> 16) & 0xff] ^ AES_Te2[(s0 >>  8) & 0xff] ^ AES_Te3[s1 & 0xff] ^ rk[14];
-        t3 = AES_Te0[s3 >> 24] ^ AES_Te1[(s0 >> 16) & 0xff] ^ AES_Te2[(s1 >>  8) & 0xff] ^ AES_Te3[s2 & 0xff] ^ rk[15];
-   	/* round 4: */
-        s0 = AES_Te0[t0 >> 24] ^ AES_Te1[(t1 >> 16) & 0xff] ^ AES_Te2[(t2 >>  8) & 0xff] ^ AES_Te3[t3 & 0xff] ^ rk[16];
-        s1 = AES_Te0[t1 >> 24] ^ AES_Te1[(t2 >> 16) & 0xff] ^ AES_Te2[(t3 >>  8) & 0xff] ^ AES_Te3[t0 & 0xff] ^ rk[17];
-        s2 = AES_Te0[t2 >> 24] ^ AES_Te1[(t3 >> 16) & 0xff] ^ AES_Te2[(t0 >>  8) & 0xff] ^ AES_Te3[t1 & 0xff] ^ rk[18];
-        s3 = AES_Te0[t3 >> 24] ^ AES_Te1[(t0 >> 16) & 0xff] ^ AES_Te2[(t1 >>  8) & 0xff] ^ AES_Te3[t2 & 0xff] ^ rk[19];
-	/* round 5: */
-        t0 = AES_Te0[s0 >> 24] ^ AES_Te1[(s1 >> 16) & 0xff] ^ AES_Te2[(s2 >>  8) & 0xff] ^ AES_Te3[s3 & 0xff] ^ rk[20];
-        t1 = AES_Te0[s1 >> 24] ^ AES_Te1[(s2 >> 16) & 0xff] ^ AES_Te2[(s3 >>  8) & 0xff] ^ AES_Te3[s0 & 0xff] ^ rk[21];
-        t2 = AES_Te0[s2 >> 24] ^ AES_Te1[(s3 >> 16) & 0xff] ^ AES_Te2[(s0 >>  8) & 0xff] ^ AES_Te3[s1 & 0xff] ^ rk[22];
-        t3 = AES_Te0[s3 >> 24] ^ AES_Te1[(s0 >> 16) & 0xff] ^ AES_Te2[(s1 >>  8) & 0xff] ^ AES_Te3[s2 & 0xff] ^ rk[23];
-   	/* round 6: */
-        s0 = AES_Te0[t0 >> 24] ^ AES_Te1[(t1 >> 16) & 0xff] ^ AES_Te2[(t2 >>  8) & 0xff] ^ AES_Te3[t3 & 0xff] ^ rk[24];
-        s1 = AES_Te0[t1 >> 24] ^ AES_Te1[(t2 >> 16) & 0xff] ^ AES_Te2[(t3 >>  8) & 0xff] ^ AES_Te3[t0 & 0xff] ^ rk[25];
-        s2 = AES_Te0[t2 >> 24] ^ AES_Te1[(t3 >> 16) & 0xff] ^ AES_Te2[(t0 >>  8) & 0xff] ^ AES_Te3[t1 & 0xff] ^ rk[26];
-        s3 = AES_Te0[t3 >> 24] ^ AES_Te1[(t0 >> 16) & 0xff] ^ AES_Te2[(t1 >>  8) & 0xff] ^ AES_Te3[t2 & 0xff] ^ rk[27];
-	/* round 7: */
-        t0 = AES_Te0[s0 >> 24] ^ AES_Te1[(s1 >> 16) & 0xff] ^ AES_Te2[(s2 >>  8) & 0xff] ^ AES_Te3[s3 & 0xff] ^ rk[28];
-        t1 = AES_Te0[s1 >> 24] ^ AES_Te1[(s2 >> 16) & 0xff] ^ AES_Te2[(s3 >>  8) & 0xff] ^ AES_Te3[s0 & 0xff] ^ rk[29];
-        t2 = AES_Te0[s2 >> 24] ^ AES_Te1[(s3 >> 16) & 0xff] ^ AES_Te2[(s0 >>  8) & 0xff] ^ AES_Te3[s1 & 0xff] ^ rk[30];
-        t3 = AES_Te0[s3 >> 24] ^ AES_Te1[(s0 >> 16) & 0xff] ^ AES_Te2[(s1 >>  8) & 0xff] ^ AES_Te3[s2 & 0xff] ^ rk[31];
-   	/* round 8: */
-        s0 = AES_Te0[t0 >> 24] ^ AES_Te1[(t1 >> 16) & 0xff] ^ AES_Te2[(t2 >>  8) & 0xff] ^ AES_Te3[t3 & 0xff] ^ rk[32];
-        s1 = AES_Te0[t1 >> 24] ^ AES_Te1[(t2 >> 16) & 0xff] ^ AES_Te2[(t3 >>  8) & 0xff] ^ AES_Te3[t0 & 0xff] ^ rk[33];
-        s2 = AES_Te0[t2 >> 24] ^ AES_Te1[(t3 >> 16) & 0xff] ^ AES_Te2[(t0 >>  8) & 0xff] ^ AES_Te3[t1 & 0xff] ^ rk[34];
-        s3 = AES_Te0[t3 >> 24] ^ AES_Te1[(t0 >> 16) & 0xff] ^ AES_Te2[(t1 >>  8) & 0xff] ^ AES_Te3[t2 & 0xff] ^ rk[35];
-	/* round 9: */
-        t0 = AES_Te0[s0 >> 24] ^ AES_Te1[(s1 >> 16) & 0xff] ^ AES_Te2[(s2 >>  8) & 0xff] ^ AES_Te3[s3 & 0xff] ^ rk[36];
-        t1 = AES_Te0[s1 >> 24] ^ AES_Te1[(s2 >> 16) & 0xff] ^ AES_Te2[(s3 >>  8) & 0xff] ^ AES_Te3[s0 & 0xff] ^ rk[37];
-        t2 = AES_Te0[s2 >> 24] ^ AES_Te1[(s3 >> 16) & 0xff] ^ AES_Te2[(s0 >>  8) & 0xff] ^ AES_Te3[s1 & 0xff] ^ rk[38];
-        t3 = AES_Te0[s3 >> 24] ^ AES_Te1[(s0 >> 16) & 0xff] ^ AES_Te2[(s1 >>  8) & 0xff] ^ AES_Te3[s2 & 0xff] ^ rk[39];
-    if (key->rounds > 10) {
-        /* round 10: */
-        s0 = AES_Te0[t0 >> 24] ^ AES_Te1[(t1 >> 16) & 0xff] ^ AES_Te2[(t2 >>  8) & 0xff] ^ AES_Te3[t3 & 0xff] ^ rk[40];
-        s1 = AES_Te0[t1 >> 24] ^ AES_Te1[(t2 >> 16) & 0xff] ^ AES_Te2[(t3 >>  8) & 0xff] ^ AES_Te3[t0 & 0xff] ^ rk[41];
-        s2 = AES_Te0[t2 >> 24] ^ AES_Te1[(t3 >> 16) & 0xff] ^ AES_Te2[(t0 >>  8) & 0xff] ^ AES_Te3[t1 & 0xff] ^ rk[42];
-        s3 = AES_Te0[t3 >> 24] ^ AES_Te1[(t0 >> 16) & 0xff] ^ AES_Te2[(t1 >>  8) & 0xff] ^ AES_Te3[t2 & 0xff] ^ rk[43];
-        /* round 11: */
-        t0 = AES_Te0[s0 >> 24] ^ AES_Te1[(s1 >> 16) & 0xff] ^ AES_Te2[(s2 >>  8) & 0xff] ^ AES_Te3[s3 & 0xff] ^ rk[44];
-        t1 = AES_Te0[s1 >> 24] ^ AES_Te1[(s2 >> 16) & 0xff] ^ AES_Te2[(s3 >>  8) & 0xff] ^ AES_Te3[s0 & 0xff] ^ rk[45];
-        t2 = AES_Te0[s2 >> 24] ^ AES_Te1[(s3 >> 16) & 0xff] ^ AES_Te2[(s0 >>  8) & 0xff] ^ AES_Te3[s1 & 0xff] ^ rk[46];
-        t3 = AES_Te0[s3 >> 24] ^ AES_Te1[(s0 >> 16) & 0xff] ^ AES_Te2[(s1 >>  8) & 0xff] ^ AES_Te3[s2 & 0xff] ^ rk[47];
-        if (key->rounds > 12) {
-            /* round 12: */
-            s0 = AES_Te0[t0 >> 24] ^ AES_Te1[(t1 >> 16) & 0xff] ^ AES_Te2[(t2 >>  8) & 0xff] ^ AES_Te3[t3 & 0xff] ^ rk[48];
-            s1 = AES_Te0[t1 >> 24] ^ AES_Te1[(t2 >> 16) & 0xff] ^ AES_Te2[(t3 >>  8) & 0xff] ^ AES_Te3[t0 & 0xff] ^ rk[49];
-            s2 = AES_Te0[t2 >> 24] ^ AES_Te1[(t3 >> 16) & 0xff] ^ AES_Te2[(t0 >>  8) & 0xff] ^ AES_Te3[t1 & 0xff] ^ rk[50];
-            s3 = AES_Te0[t3 >> 24] ^ AES_Te1[(t0 >> 16) & 0xff] ^ AES_Te2[(t1 >>  8) & 0xff] ^ AES_Te3[t2 & 0xff] ^ rk[51];
-            /* round 13: */
-            t0 = AES_Te0[s0 >> 24] ^ AES_Te1[(s1 >> 16) & 0xff] ^ AES_Te2[(s2 >>  8) & 0xff] ^ AES_Te3[s3 & 0xff] ^ rk[52];
-            t1 = AES_Te0[s1 >> 24] ^ AES_Te1[(s2 >> 16) & 0xff] ^ AES_Te2[(s3 >>  8) & 0xff] ^ AES_Te3[s0 & 0xff] ^ rk[53];
-            t2 = AES_Te0[s2 >> 24] ^ AES_Te1[(s3 >> 16) & 0xff] ^ AES_Te2[(s0 >>  8) & 0xff] ^ AES_Te3[s1 & 0xff] ^ rk[54];
-            t3 = AES_Te0[s3 >> 24] ^ AES_Te1[(s0 >> 16) & 0xff] ^ AES_Te2[(s1 >>  8) & 0xff] ^ AES_Te3[s2 & 0xff] ^ rk[55];
-        }
-    }
-    rk += key->rounds << 2;
-#else  /* !FULL_UNROLL */
-    /*
-     * Nr - 1 full rounds:
-     */
-    r = key->rounds >> 1;
-    for (;;) {
-        t0 =
-            AES_Te0[(s0 >> 24)       ] ^
-            AES_Te1[(s1 >> 16) & 0xff] ^
-            AES_Te2[(s2 >>  8) & 0xff] ^
-            AES_Te3[(s3      ) & 0xff] ^
-            rk[4];
-        t1 =
-            AES_Te0[(s1 >> 24)       ] ^
-            AES_Te1[(s2 >> 16) & 0xff] ^
-            AES_Te2[(s3 >>  8) & 0xff] ^
-            AES_Te3[(s0      ) & 0xff] ^
-            rk[5];
-        t2 =
-            AES_Te0[(s2 >> 24)       ] ^
-            AES_Te1[(s3 >> 16) & 0xff] ^
-            AES_Te2[(s0 >>  8) & 0xff] ^
-            AES_Te3[(s1      ) & 0xff] ^
-            rk[6];
-        t3 =
-            AES_Te0[(s3 >> 24)       ] ^
-            AES_Te1[(s0 >> 16) & 0xff] ^
-            AES_Te2[(s1 >>  8) & 0xff] ^
-            AES_Te3[(s2      ) & 0xff] ^
-            rk[7];
-
-        rk += 8;
-        if (--r == 0) {
-            break;
-        }
-
-        s0 =
-            AES_Te0[(t0 >> 24)       ] ^
-            AES_Te1[(t1 >> 16) & 0xff] ^
-            AES_Te2[(t2 >>  8) & 0xff] ^
-            AES_Te3[(t3      ) & 0xff] ^
-            rk[0];
-        s1 =
-            AES_Te0[(t1 >> 24)       ] ^
-            AES_Te1[(t2 >> 16) & 0xff] ^
-            AES_Te2[(t3 >>  8) & 0xff] ^
-            AES_Te3[(t0      ) & 0xff] ^
-            rk[1];
-        s2 =
-            AES_Te0[(t2 >> 24)       ] ^
-            AES_Te1[(t3 >> 16) & 0xff] ^
-            AES_Te2[(t0 >>  8) & 0xff] ^
-            AES_Te3[(t1      ) & 0xff] ^
-            rk[2];
-        s3 =
-            AES_Te0[(t3 >> 24)       ] ^
-            AES_Te1[(t0 >> 16) & 0xff] ^
-            AES_Te2[(t1 >>  8) & 0xff] ^
-            AES_Te3[(t2      ) & 0xff] ^
-            rk[3];
-    }
-#endif /* ?FULL_UNROLL */
-    /*
-	 * apply last round and
-	 * map cipher state to byte array block:
-	 */
-	s0 =
-                (AES_Te4[(t0 >> 24)       ] & 0xff000000) ^
-                (AES_Te4[(t1 >> 16) & 0xff] & 0x00ff0000) ^
-                (AES_Te4[(t2 >>  8) & 0xff] & 0x0000ff00) ^
-                (AES_Te4[(t3      ) & 0xff] & 0x000000ff) ^
-		rk[0];
-	PUTU32(out     , s0);
-	s1 =
-                (AES_Te4[(t1 >> 24)       ] & 0xff000000) ^
-                (AES_Te4[(t2 >> 16) & 0xff] & 0x00ff0000) ^
-                (AES_Te4[(t3 >>  8) & 0xff] & 0x0000ff00) ^
-                (AES_Te4[(t0      ) & 0xff] & 0x000000ff) ^
-		rk[1];
-	PUTU32(out +  4, s1);
-	s2 =
-                (AES_Te4[(t2 >> 24)       ] & 0xff000000) ^
-                (AES_Te4[(t3 >> 16) & 0xff] & 0x00ff0000) ^
-                (AES_Te4[(t0 >>  8) & 0xff] & 0x0000ff00) ^
-                (AES_Te4[(t1      ) & 0xff] & 0x000000ff) ^
-		rk[2];
-	PUTU32(out +  8, s2);
-	s3 =
-                (AES_Te4[(t3 >> 24)       ] & 0xff000000) ^
-                (AES_Te4[(t0 >> 16) & 0xff] & 0x00ff0000) ^
-                (AES_Te4[(t1 >>  8) & 0xff] & 0x0000ff00) ^
-                (AES_Te4[(t2      ) & 0xff] & 0x000000ff) ^
-		rk[3];
-	PUTU32(out + 12, s3);
-}
-
-/*
- * Decrypt a single block
- * in and out can overlap
- */
-void AES_decrypt(const unsigned char *in, unsigned char *out,
-		 const AES_KEY *key) {
-
-	const u32 *rk;
-	u32 s0, s1, s2, s3, t0, t1, t2, t3;
-#ifndef FULL_UNROLL
-	int r;
-#endif /* ?FULL_UNROLL */
-
-	assert(in && out && key);
-	rk = key->rd_key;
-
-	/*
-	 * map byte array block to cipher state
-	 * and add initial round key:
-	 */
-    s0 = GETU32(in     ) ^ rk[0];
-    s1 = GETU32(in +  4) ^ rk[1];
-    s2 = GETU32(in +  8) ^ rk[2];
-    s3 = GETU32(in + 12) ^ rk[3];
-#ifdef FULL_UNROLL
-    /* round 1: */
-    t0 = AES_Td0[s0 >> 24] ^ AES_Td1[(s3 >> 16) & 0xff] ^ AES_Td2[(s2 >>  8) & 0xff] ^ AES_Td3[s1 & 0xff] ^ rk[ 4];
-    t1 = AES_Td0[s1 >> 24] ^ AES_Td1[(s0 >> 16) & 0xff] ^ AES_Td2[(s3 >>  8) & 0xff] ^ AES_Td3[s2 & 0xff] ^ rk[ 5];
-    t2 = AES_Td0[s2 >> 24] ^ AES_Td1[(s1 >> 16) & 0xff] ^ AES_Td2[(s0 >>  8) & 0xff] ^ AES_Td3[s3 & 0xff] ^ rk[ 6];
-    t3 = AES_Td0[s3 >> 24] ^ AES_Td1[(s2 >> 16) & 0xff] ^ AES_Td2[(s1 >>  8) & 0xff] ^ AES_Td3[s0 & 0xff] ^ rk[ 7];
-    /* round 2: */
-    s0 = AES_Td0[t0 >> 24] ^ AES_Td1[(t3 >> 16) & 0xff] ^ AES_Td2[(t2 >>  8) & 0xff] ^ AES_Td3[t1 & 0xff] ^ rk[ 8];
-    s1 = AES_Td0[t1 >> 24] ^ AES_Td1[(t0 >> 16) & 0xff] ^ AES_Td2[(t3 >>  8) & 0xff] ^ AES_Td3[t2 & 0xff] ^ rk[ 9];
-    s2 = AES_Td0[t2 >> 24] ^ AES_Td1[(t1 >> 16) & 0xff] ^ AES_Td2[(t0 >>  8) & 0xff] ^ AES_Td3[t3 & 0xff] ^ rk[10];
-    s3 = AES_Td0[t3 >> 24] ^ AES_Td1[(t2 >> 16) & 0xff] ^ AES_Td2[(t1 >>  8) & 0xff] ^ AES_Td3[t0 & 0xff] ^ rk[11];
-    /* round 3: */
-    t0 = AES_Td0[s0 >> 24] ^ AES_Td1[(s3 >> 16) & 0xff] ^ AES_Td2[(s2 >>  8) & 0xff] ^ AES_Td3[s1 & 0xff] ^ rk[12];
-    t1 = AES_Td0[s1 >> 24] ^ AES_Td1[(s0 >> 16) & 0xff] ^ AES_Td2[(s3 >>  8) & 0xff] ^ AES_Td3[s2 & 0xff] ^ rk[13];
-    t2 = AES_Td0[s2 >> 24] ^ AES_Td1[(s1 >> 16) & 0xff] ^ AES_Td2[(s0 >>  8) & 0xff] ^ AES_Td3[s3 & 0xff] ^ rk[14];
-    t3 = AES_Td0[s3 >> 24] ^ AES_Td1[(s2 >> 16) & 0xff] ^ AES_Td2[(s1 >>  8) & 0xff] ^ AES_Td3[s0 & 0xff] ^ rk[15];
-    /* round 4: */
-    s0 = AES_Td0[t0 >> 24] ^ AES_Td1[(t3 >> 16) & 0xff] ^ AES_Td2[(t2 >>  8) & 0xff] ^ AES_Td3[t1 & 0xff] ^ rk[16];
-    s1 = AES_Td0[t1 >> 24] ^ AES_Td1[(t0 >> 16) & 0xff] ^ AES_Td2[(t3 >>  8) & 0xff] ^ AES_Td3[t2 & 0xff] ^ rk[17];
-    s2 = AES_Td0[t2 >> 24] ^ AES_Td1[(t1 >> 16) & 0xff] ^ AES_Td2[(t0 >>  8) & 0xff] ^ AES_Td3[t3 & 0xff] ^ rk[18];
-    s3 = AES_Td0[t3 >> 24] ^ AES_Td1[(t2 >> 16) & 0xff] ^ AES_Td2[(t1 >>  8) & 0xff] ^ AES_Td3[t0 & 0xff] ^ rk[19];
-    /* round 5: */
-    t0 = AES_Td0[s0 >> 24] ^ AES_Td1[(s3 >> 16) & 0xff] ^ AES_Td2[(s2 >>  8) & 0xff] ^ AES_Td3[s1 & 0xff] ^ rk[20];
-    t1 = AES_Td0[s1 >> 24] ^ AES_Td1[(s0 >> 16) & 0xff] ^ AES_Td2[(s3 >>  8) & 0xff] ^ AES_Td3[s2 & 0xff] ^ rk[21];
-    t2 = AES_Td0[s2 >> 24] ^ AES_Td1[(s1 >> 16) & 0xff] ^ AES_Td2[(s0 >>  8) & 0xff] ^ AES_Td3[s3 & 0xff] ^ rk[22];
-    t3 = AES_Td0[s3 >> 24] ^ AES_Td1[(s2 >> 16) & 0xff] ^ AES_Td2[(s1 >>  8) & 0xff] ^ AES_Td3[s0 & 0xff] ^ rk[23];
-    /* round 6: */
-    s0 = AES_Td0[t0 >> 24] ^ AES_Td1[(t3 >> 16) & 0xff] ^ AES_Td2[(t2 >>  8) & 0xff] ^ AES_Td3[t1 & 0xff] ^ rk[24];
-    s1 = AES_Td0[t1 >> 24] ^ AES_Td1[(t0 >> 16) & 0xff] ^ AES_Td2[(t3 >>  8) & 0xff] ^ AES_Td3[t2 & 0xff] ^ rk[25];
-    s2 = AES_Td0[t2 >> 24] ^ AES_Td1[(t1 >> 16) & 0xff] ^ AES_Td2[(t0 >>  8) & 0xff] ^ AES_Td3[t3 & 0xff] ^ rk[26];
-    s3 = AES_Td0[t3 >> 24] ^ AES_Td1[(t2 >> 16) & 0xff] ^ AES_Td2[(t1 >>  8) & 0xff] ^ AES_Td3[t0 & 0xff] ^ rk[27];
-    /* round 7: */
-    t0 = AES_Td0[s0 >> 24] ^ AES_Td1[(s3 >> 16) & 0xff] ^ AES_Td2[(s2 >>  8) & 0xff] ^ AES_Td3[s1 & 0xff] ^ rk[28];
-    t1 = AES_Td0[s1 >> 24] ^ AES_Td1[(s0 >> 16) & 0xff] ^ AES_Td2[(s3 >>  8) & 0xff] ^ AES_Td3[s2 & 0xff] ^ rk[29];
-    t2 = AES_Td0[s2 >> 24] ^ AES_Td1[(s1 >> 16) & 0xff] ^ AES_Td2[(s0 >>  8) & 0xff] ^ AES_Td3[s3 & 0xff] ^ rk[30];
-    t3 = AES_Td0[s3 >> 24] ^ AES_Td1[(s2 >> 16) & 0xff] ^ AES_Td2[(s1 >>  8) & 0xff] ^ AES_Td3[s0 & 0xff] ^ rk[31];
-    /* round 8: */
-    s0 = AES_Td0[t0 >> 24] ^ AES_Td1[(t3 >> 16) & 0xff] ^ AES_Td2[(t2 >>  8) & 0xff] ^ AES_Td3[t1 & 0xff] ^ rk[32];
-    s1 = AES_Td0[t1 >> 24] ^ AES_Td1[(t0 >> 16) & 0xff] ^ AES_Td2[(t3 >>  8) & 0xff] ^ AES_Td3[t2 & 0xff] ^ rk[33];
-    s2 = AES_Td0[t2 >> 24] ^ AES_Td1[(t1 >> 16) & 0xff] ^ AES_Td2[(t0 >>  8) & 0xff] ^ AES_Td3[t3 & 0xff] ^ rk[34];
-    s3 = AES_Td0[t3 >> 24] ^ AES_Td1[(t2 >> 16) & 0xff] ^ AES_Td2[(t1 >>  8) & 0xff] ^ AES_Td3[t0 & 0xff] ^ rk[35];
-    /* round 9: */
-    t0 = AES_Td0[s0 >> 24] ^ AES_Td1[(s3 >> 16) & 0xff] ^ AES_Td2[(s2 >>  8) & 0xff] ^ AES_Td3[s1 & 0xff] ^ rk[36];
-    t1 = AES_Td0[s1 >> 24] ^ AES_Td1[(s0 >> 16) & 0xff] ^ AES_Td2[(s3 >>  8) & 0xff] ^ AES_Td3[s2 & 0xff] ^ rk[37];
-    t2 = AES_Td0[s2 >> 24] ^ AES_Td1[(s1 >> 16) & 0xff] ^ AES_Td2[(s0 >>  8) & 0xff] ^ AES_Td3[s3 & 0xff] ^ rk[38];
-    t3 = AES_Td0[s3 >> 24] ^ AES_Td1[(s2 >> 16) & 0xff] ^ AES_Td2[(s1 >>  8) & 0xff] ^ AES_Td3[s0 & 0xff] ^ rk[39];
-    if (key->rounds > 10) {
-        /* round 10: */
-        s0 = AES_Td0[t0 >> 24] ^ AES_Td1[(t3 >> 16) & 0xff] ^ AES_Td2[(t2 >>  8) & 0xff] ^ AES_Td3[t1 & 0xff] ^ rk[40];
-        s1 = AES_Td0[t1 >> 24] ^ AES_Td1[(t0 >> 16) & 0xff] ^ AES_Td2[(t3 >>  8) & 0xff] ^ AES_Td3[t2 & 0xff] ^ rk[41];
-        s2 = AES_Td0[t2 >> 24] ^ AES_Td1[(t1 >> 16) & 0xff] ^ AES_Td2[(t0 >>  8) & 0xff] ^ AES_Td3[t3 & 0xff] ^ rk[42];
-        s3 = AES_Td0[t3 >> 24] ^ AES_Td1[(t2 >> 16) & 0xff] ^ AES_Td2[(t1 >>  8) & 0xff] ^ AES_Td3[t0 & 0xff] ^ rk[43];
-        /* round 11: */
-        t0 = AES_Td0[s0 >> 24] ^ AES_Td1[(s3 >> 16) & 0xff] ^ AES_Td2[(s2 >>  8) & 0xff] ^ AES_Td3[s1 & 0xff] ^ rk[44];
-        t1 = AES_Td0[s1 >> 24] ^ AES_Td1[(s0 >> 16) & 0xff] ^ AES_Td2[(s3 >>  8) & 0xff] ^ AES_Td3[s2 & 0xff] ^ rk[45];
-        t2 = AES_Td0[s2 >> 24] ^ AES_Td1[(s1 >> 16) & 0xff] ^ AES_Td2[(s0 >>  8) & 0xff] ^ AES_Td3[s3 & 0xff] ^ rk[46];
-        t3 = AES_Td0[s3 >> 24] ^ AES_Td1[(s2 >> 16) & 0xff] ^ AES_Td2[(s1 >>  8) & 0xff] ^ AES_Td3[s0 & 0xff] ^ rk[47];
-        if (key->rounds > 12) {
-            /* round 12: */
-            s0 = AES_Td0[t0 >> 24] ^ AES_Td1[(t3 >> 16) & 0xff] ^ AES_Td2[(t2 >>  8) & 0xff] ^ AES_Td3[t1 & 0xff] ^ rk[48];
-            s1 = AES_Td0[t1 >> 24] ^ AES_Td1[(t0 >> 16) & 0xff] ^ AES_Td2[(t3 >>  8) & 0xff] ^ AES_Td3[t2 & 0xff] ^ rk[49];
-            s2 = AES_Td0[t2 >> 24] ^ AES_Td1[(t1 >> 16) & 0xff] ^ AES_Td2[(t0 >>  8) & 0xff] ^ AES_Td3[t3 & 0xff] ^ rk[50];
-            s3 = AES_Td0[t3 >> 24] ^ AES_Td1[(t2 >> 16) & 0xff] ^ AES_Td2[(t1 >>  8) & 0xff] ^ AES_Td3[t0 & 0xff] ^ rk[51];
-            /* round 13: */
-            t0 = AES_Td0[s0 >> 24] ^ AES_Td1[(s3 >> 16) & 0xff] ^ AES_Td2[(s2 >>  8) & 0xff] ^ AES_Td3[s1 & 0xff] ^ rk[52];
-            t1 = AES_Td0[s1 >> 24] ^ AES_Td1[(s0 >> 16) & 0xff] ^ AES_Td2[(s3 >>  8) & 0xff] ^ AES_Td3[s2 & 0xff] ^ rk[53];
-            t2 = AES_Td0[s2 >> 24] ^ AES_Td1[(s1 >> 16) & 0xff] ^ AES_Td2[(s0 >>  8) & 0xff] ^ AES_Td3[s3 & 0xff] ^ rk[54];
-            t3 = AES_Td0[s3 >> 24] ^ AES_Td1[(s2 >> 16) & 0xff] ^ AES_Td2[(s1 >>  8) & 0xff] ^ AES_Td3[s0 & 0xff] ^ rk[55];
-        }
-    }
-	rk += key->rounds << 2;
-#else  /* !FULL_UNROLL */
-    /*
-     * Nr - 1 full rounds:
-     */
-    r = key->rounds >> 1;
-    for (;;) {
-        t0 =
-            AES_Td0[(s0 >> 24)       ] ^
-            AES_Td1[(s3 >> 16) & 0xff] ^
-            AES_Td2[(s2 >>  8) & 0xff] ^
-            AES_Td3[(s1      ) & 0xff] ^
-            rk[4];
-        t1 =
-            AES_Td0[(s1 >> 24)       ] ^
-            AES_Td1[(s0 >> 16) & 0xff] ^
-            AES_Td2[(s3 >>  8) & 0xff] ^
-            AES_Td3[(s2      ) & 0xff] ^
-            rk[5];
-        t2 =
-            AES_Td0[(s2 >> 24)       ] ^
-            AES_Td1[(s1 >> 16) & 0xff] ^
-            AES_Td2[(s0 >>  8) & 0xff] ^
-            AES_Td3[(s3      ) & 0xff] ^
-            rk[6];
-        t3 =
-            AES_Td0[(s3 >> 24)       ] ^
-            AES_Td1[(s2 >> 16) & 0xff] ^
-            AES_Td2[(s1 >>  8) & 0xff] ^
-            AES_Td3[(s0      ) & 0xff] ^
-            rk[7];
-
-        rk += 8;
-        if (--r == 0) {
-            break;
-        }
-
-        s0 =
-            AES_Td0[(t0 >> 24)       ] ^
-            AES_Td1[(t3 >> 16) & 0xff] ^
-            AES_Td2[(t2 >>  8) & 0xff] ^
-            AES_Td3[(t1      ) & 0xff] ^
-            rk[0];
-        s1 =
-            AES_Td0[(t1 >> 24)       ] ^
-            AES_Td1[(t0 >> 16) & 0xff] ^
-            AES_Td2[(t3 >>  8) & 0xff] ^
-            AES_Td3[(t2      ) & 0xff] ^
-            rk[1];
-        s2 =
-            AES_Td0[(t2 >> 24)       ] ^
-            AES_Td1[(t1 >> 16) & 0xff] ^
-            AES_Td2[(t0 >>  8) & 0xff] ^
-            AES_Td3[(t3      ) & 0xff] ^
-            rk[2];
-        s3 =
-            AES_Td0[(t3 >> 24)       ] ^
-            AES_Td1[(t2 >> 16) & 0xff] ^
-            AES_Td2[(t1 >>  8) & 0xff] ^
-            AES_Td3[(t0      ) & 0xff] ^
-            rk[3];
-    }
-#endif /* ?FULL_UNROLL */
-    /*
-	 * apply last round and
-	 * map cipher state to byte array block:
-	 */
-   	s0 =
-                (AES_Td4[(t0 >> 24)       ] & 0xff000000) ^
-                (AES_Td4[(t3 >> 16) & 0xff] & 0x00ff0000) ^
-                (AES_Td4[(t2 >>  8) & 0xff] & 0x0000ff00) ^
-                (AES_Td4[(t1      ) & 0xff] & 0x000000ff) ^
-   		rk[0];
-	PUTU32(out     , s0);
-   	s1 =
-                (AES_Td4[(t1 >> 24)       ] & 0xff000000) ^
-                (AES_Td4[(t0 >> 16) & 0xff] & 0x00ff0000) ^
-                (AES_Td4[(t3 >>  8) & 0xff] & 0x0000ff00) ^
-                (AES_Td4[(t2      ) & 0xff] & 0x000000ff) ^
-   		rk[1];
-	PUTU32(out +  4, s1);
-   	s2 =
-                (AES_Td4[(t2 >> 24)       ] & 0xff000000) ^
-                (AES_Td4[(t1 >> 16) & 0xff] & 0x00ff0000) ^
-                (AES_Td4[(t0 >>  8) & 0xff] & 0x0000ff00) ^
-                (AES_Td4[(t3      ) & 0xff] & 0x000000ff) ^
-   		rk[2];
-	PUTU32(out +  8, s2);
-   	s3 =
-                (AES_Td4[(t3 >> 24)       ] & 0xff000000) ^
-                (AES_Td4[(t2 >> 16) & 0xff] & 0x00ff0000) ^
-                (AES_Td4[(t1 >>  8) & 0xff] & 0x0000ff00) ^
-                (AES_Td4[(t0      ) & 0xff] & 0x000000ff) ^
-   		rk[3];
-	PUTU32(out + 12, s3);
-}
-
-#endif /* AES_ASM */
-
-void AES_cbc_encrypt(const unsigned char *in, unsigned char *out,
-		     const unsigned long length, const AES_KEY *key,
-		     unsigned char *ivec, const int enc)
-{
-
-	unsigned long n;
-	unsigned long len = length;
-	unsigned char tmp[AES_BLOCK_SIZE];
-
-	assert(in && out && key && ivec);
-
-	if (enc) {
-		while (len >= AES_BLOCK_SIZE) {
-			for(n=0; n < AES_BLOCK_SIZE; ++n)
-				tmp[n] = in[n] ^ ivec[n];
-			AES_encrypt(tmp, out, key);
-			memcpy(ivec, out, AES_BLOCK_SIZE);
-			len -= AES_BLOCK_SIZE;
-			in += AES_BLOCK_SIZE;
-			out += AES_BLOCK_SIZE;
-		}
-		if (len) {
-			for(n=0; n < len; ++n)
-				tmp[n] = in[n] ^ ivec[n];
-			for(n=len; n < AES_BLOCK_SIZE; ++n)
-				tmp[n] = ivec[n];
-			AES_encrypt(tmp, tmp, key);
-			memcpy(out, tmp, AES_BLOCK_SIZE);
-			memcpy(ivec, tmp, AES_BLOCK_SIZE);
-		}
-	} else {
-		while (len >= AES_BLOCK_SIZE) {
-			memcpy(tmp, in, AES_BLOCK_SIZE);
-			AES_decrypt(in, out, key);
-			for(n=0; n < AES_BLOCK_SIZE; ++n)
-				out[n] ^= ivec[n];
-			memcpy(ivec, tmp, AES_BLOCK_SIZE);
-			len -= AES_BLOCK_SIZE;
-			in += AES_BLOCK_SIZE;
-			out += AES_BLOCK_SIZE;
-		}
-		if (len) {
-			memcpy(tmp, in, AES_BLOCK_SIZE);
-			AES_decrypt(tmp, tmp, key);
-			for(n=0; n < len; ++n)
-				out[n] = tmp[n] ^ ivec[n];
-			memcpy(ivec, tmp, AES_BLOCK_SIZE);
-		}
-	}
-}
diff --git a/contrib/qemu/util/bitmap.c b/contrib/qemu/util/bitmap.c
deleted file mode 100644
index 687841dcec0..00000000000
--- a/contrib/qemu/util/bitmap.c
+++ /dev/null
@@ -1,256 +0,0 @@
-/*
- * Bitmap Module
- *
- * Stolen from linux/src/lib/bitmap.c
- *
- * Copyright (C) 2010 Corentin Chary
- *
- * This source code is licensed under the GNU General Public License,
- * Version 2.
- */
-
-#include "qemu/bitops.h"
-#include "qemu/bitmap.h"
-
-/*
- * bitmaps provide an array of bits, implemented using an an
- * array of unsigned longs.  The number of valid bits in a
- * given bitmap does _not_ need to be an exact multiple of
- * BITS_PER_LONG.
- *
- * The possible unused bits in the last, partially used word
- * of a bitmap are 'don't care'.  The implementation makes
- * no particular effort to keep them zero.  It ensures that
- * their value will not affect the results of any operation.
- * The bitmap operations that return Boolean (bitmap_empty,
- * for example) or scalar (bitmap_weight, for example) results
- * carefully filter out these unused bits from impacting their
- * results.
- *
- * These operations actually hold to a slightly stronger rule:
- * if you don't input any bitmaps to these ops that have some
- * unused bits set, then they won't output any set unused bits
- * in output bitmaps.
- *
- * The byte ordering of bitmaps is more natural on little
- * endian architectures.
- */
-
-int slow_bitmap_empty(const unsigned long *bitmap, int bits)
-{
-    int k, lim = bits/BITS_PER_LONG;
-
-    for (k = 0; k < lim; ++k) {
-        if (bitmap[k]) {
-            return 0;
-        }
-    }
-    if (bits % BITS_PER_LONG) {
-        if (bitmap[k] & BITMAP_LAST_WORD_MASK(bits)) {
-            return 0;
-        }
-    }
-
-    return 1;
-}
-
-int slow_bitmap_full(const unsigned long *bitmap, int bits)
-{
-    int k, lim = bits/BITS_PER_LONG;
-
-    for (k = 0; k < lim; ++k) {
-        if (~bitmap[k]) {
-            return 0;
-        }
-    }
-
-    if (bits % BITS_PER_LONG) {
-        if (~bitmap[k] & BITMAP_LAST_WORD_MASK(bits)) {
-            return 0;
-        }
-    }
-
-    return 1;
-}
-
-int slow_bitmap_equal(const unsigned long *bitmap1,
-                      const unsigned long *bitmap2, int bits)
-{
-    int k, lim = bits/BITS_PER_LONG;
-
-    for (k = 0; k < lim; ++k) {
-        if (bitmap1[k] != bitmap2[k]) {
-            return 0;
-        }
-    }
-
-    if (bits % BITS_PER_LONG) {
-        if ((bitmap1[k] ^ bitmap2[k]) & BITMAP_LAST_WORD_MASK(bits)) {
-            return 0;
-        }
-    }
-
-    return 1;
-}
-
-void slow_bitmap_complement(unsigned long *dst, const unsigned long *src,
-                            int bits)
-{
-    int k, lim = bits/BITS_PER_LONG;
-
-    for (k = 0; k < lim; ++k) {
-        dst[k] = ~src[k];
-    }
-
-    if (bits % BITS_PER_LONG) {
-        dst[k] = ~src[k] & BITMAP_LAST_WORD_MASK(bits);
-    }
-}
-
-int slow_bitmap_and(unsigned long *dst, const unsigned long *bitmap1,
-                    const unsigned long *bitmap2, int bits)
-{
-    int k;
-    int nr = BITS_TO_LONGS(bits);
-    unsigned long result = 0;
-
-    for (k = 0; k < nr; k++) {
-        result |= (dst[k] = bitmap1[k] & bitmap2[k]);
-    }
-    return result != 0;
-}
-
-void slow_bitmap_or(unsigned long *dst, const unsigned long *bitmap1,
-                    const unsigned long *bitmap2, int bits)
-{
-    int k;
-    int nr = BITS_TO_LONGS(bits);
-
-    for (k = 0; k < nr; k++) {
-        dst[k] = bitmap1[k] | bitmap2[k];
-    }
-}
-
-void slow_bitmap_xor(unsigned long *dst, const unsigned long *bitmap1,
-                     const unsigned long *bitmap2, int bits)
-{
-    int k;
-    int nr = BITS_TO_LONGS(bits);
-
-    for (k = 0; k < nr; k++) {
-        dst[k] = bitmap1[k] ^ bitmap2[k];
-    }
-}
-
-int slow_bitmap_andnot(unsigned long *dst, const unsigned long *bitmap1,
-                       const unsigned long *bitmap2, int bits)
-{
-    int k;
-    int nr = BITS_TO_LONGS(bits);
-    unsigned long result = 0;
-
-    for (k = 0; k < nr; k++) {
-        result |= (dst[k] = bitmap1[k] & ~bitmap2[k]);
-    }
-    return result != 0;
-}
-
-#define BITMAP_FIRST_WORD_MASK(start) (~0UL << ((start) % BITS_PER_LONG))
-
-void bitmap_set(unsigned long *map, int start, int nr)
-{
-    unsigned long *p = map + BIT_WORD(start);
-    const int size = start + nr;
-    int bits_to_set = BITS_PER_LONG - (start % BITS_PER_LONG);
-    unsigned long mask_to_set = BITMAP_FIRST_WORD_MASK(start);
-
-    while (nr - bits_to_set >= 0) {
-        *p |= mask_to_set;
-        nr -= bits_to_set;
-        bits_to_set = BITS_PER_LONG;
-        mask_to_set = ~0UL;
-        p++;
-    }
-    if (nr) {
-        mask_to_set &= BITMAP_LAST_WORD_MASK(size);
-        *p |= mask_to_set;
-    }
-}
-
-void bitmap_clear(unsigned long *map, int start, int nr)
-{
-    unsigned long *p = map + BIT_WORD(start);
-    const int size = start + nr;
-    int bits_to_clear = BITS_PER_LONG - (start % BITS_PER_LONG);
-    unsigned long mask_to_clear = BITMAP_FIRST_WORD_MASK(start);
-
-    while (nr - bits_to_clear >= 0) {
-        *p &= ~mask_to_clear;
-        nr -= bits_to_clear;
-        bits_to_clear = BITS_PER_LONG;
-        mask_to_clear = ~0UL;
-        p++;
-    }
-    if (nr) {
-        mask_to_clear &= BITMAP_LAST_WORD_MASK(size);
-        *p &= ~mask_to_clear;
-    }
-}
-
-#define ALIGN_MASK(x,mask)      (((x)+(mask))&~(mask))
-
-/**
- * bitmap_find_next_zero_area - find a contiguous aligned zero area
- * @map: The address to base the search on
- * @size: The bitmap size in bits
- * @start: The bitnumber to start searching at
- * @nr: The number of zeroed bits we're looking for
- * @align_mask: Alignment mask for zero area
- *
- * The @align_mask should be one less than a power of 2; the effect is that
- * the bit offset of all zero areas this function finds is multiples of that
- * power of 2. A @align_mask of 0 means no alignment is required.
- */
-unsigned long bitmap_find_next_zero_area(unsigned long *map,
-					 unsigned long size,
-					 unsigned long start,
-					 unsigned int nr,
-					 unsigned long align_mask)
-{
-    unsigned long index, end, i;
-again:
-    index = find_next_zero_bit(map, size, start);
-
-    /* Align allocation */
-    index = ALIGN_MASK(index, align_mask);
-
-    end = index + nr;
-    if (end > size) {
-        return end;
-    }
-    i = find_next_bit(map, end, index);
-    if (i < end) {
-        start = i + 1;
-        goto again;
-    }
-    return index;
-}
-
-int slow_bitmap_intersects(const unsigned long *bitmap1,
-                           const unsigned long *bitmap2, int bits)
-{
-    int k, lim = bits/BITS_PER_LONG;
-
-    for (k = 0; k < lim; ++k) {
-        if (bitmap1[k] & bitmap2[k]) {
-            return 1;
-        }
-    }
-
-    if (bits % BITS_PER_LONG) {
-        if ((bitmap1[k] & bitmap2[k]) & BITMAP_LAST_WORD_MASK(bits)) {
-            return 1;
-        }
-    }
-    return 0;
-}
diff --git a/contrib/qemu/util/bitops.c b/contrib/qemu/util/bitops.c
deleted file mode 100644
index 227c38b883d..00000000000
--- a/contrib/qemu/util/bitops.c
+++ /dev/null
@@ -1,158 +0,0 @@
-/*
- * Copyright (C) 2004 Red Hat, Inc. All Rights Reserved.
- * Written by David Howells (dhowells@redhat.com)
- * Copyright (C) 2008 IBM Corporation
- * Written by Rusty Russell <rusty@rustcorp.com.au>
- * (Inspired by David Howell's find_next_bit implementation)
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * as published by the Free Software Foundation; either version
- * 2 of the License, or (at your option) any later version.
- */
-
-#include "qemu/bitops.h"
-
-#define BITOP_WORD(nr)		((nr) / BITS_PER_LONG)
-
-/*
- * Find the next set bit in a memory region.
- */
-unsigned long find_next_bit(const unsigned long *addr, unsigned long size,
-			    unsigned long offset)
-{
-    const unsigned long *p = addr + BITOP_WORD(offset);
-    unsigned long result = offset & ~(BITS_PER_LONG-1);
-    unsigned long tmp;
-
-    if (offset >= size) {
-        return size;
-    }
-    size -= result;
-    offset %= BITS_PER_LONG;
-    if (offset) {
-        tmp = *(p++);
-        tmp &= (~0UL << offset);
-        if (size < BITS_PER_LONG) {
-            goto found_first;
-        }
-        if (tmp) {
-            goto found_middle;
-        }
-        size -= BITS_PER_LONG;
-        result += BITS_PER_LONG;
-    }
-    while (size >= 4*BITS_PER_LONG) {
-        unsigned long d1, d2, d3;
-        tmp = *p;
-        d1 = *(p+1);
-        d2 = *(p+2);
-        d3 = *(p+3);
-        if (tmp) {
-            goto found_middle;
-        }
-        if (d1 | d2 | d3) {
-            break;
-        }
-        p += 4;
-        result += 4*BITS_PER_LONG;
-        size -= 4*BITS_PER_LONG;
-    }
-    while (size >= BITS_PER_LONG) {
-        if ((tmp = *(p++))) {
-            goto found_middle;
-        }
-        result += BITS_PER_LONG;
-        size -= BITS_PER_LONG;
-    }
-    if (!size) {
-        return result;
-    }
-    tmp = *p;
-
-found_first:
-    tmp &= (~0UL >> (BITS_PER_LONG - size));
-    if (tmp == 0UL) {		/* Are any bits set? */
-        return result + size;	/* Nope. */
-    }
-found_middle:
-    return result + ctzl(tmp);
-}
-
-/*
- * This implementation of find_{first,next}_zero_bit was stolen from
- * Linus' asm-alpha/bitops.h.
- */
-unsigned long find_next_zero_bit(const unsigned long *addr, unsigned long size,
-				 unsigned long offset)
-{
-    const unsigned long *p = addr + BITOP_WORD(offset);
-    unsigned long result = offset & ~(BITS_PER_LONG-1);
-    unsigned long tmp;
-
-    if (offset >= size) {
-        return size;
-    }
-    size -= result;
-    offset %= BITS_PER_LONG;
-    if (offset) {
-        tmp = *(p++);
-        tmp |= ~0UL >> (BITS_PER_LONG - offset);
-        if (size < BITS_PER_LONG) {
-            goto found_first;
-        }
-        if (~tmp) {
-            goto found_middle;
-        }
-        size -= BITS_PER_LONG;
-        result += BITS_PER_LONG;
-    }
-    while (size & ~(BITS_PER_LONG-1)) {
-        if (~(tmp = *(p++))) {
-            goto found_middle;
-        }
-        result += BITS_PER_LONG;
-        size -= BITS_PER_LONG;
-    }
-    if (!size) {
-        return result;
-    }
-    tmp = *p;
-
-found_first:
-    tmp |= ~0UL << size;
-    if (tmp == ~0UL) {	/* Are any bits zero? */
-        return result + size;	/* Nope. */
-    }
-found_middle:
-    return result + ctzl(~tmp);
-}
-
-unsigned long find_last_bit(const unsigned long *addr, unsigned long size)
-{
-    unsigned long words;
-    unsigned long tmp;
-
-    /* Start at final word. */
-    words = size / BITS_PER_LONG;
-
-    /* Partial final word? */
-    if (size & (BITS_PER_LONG-1)) {
-        tmp = (addr[words] & (~0UL >> (BITS_PER_LONG
-                                       - (size & (BITS_PER_LONG-1)))));
-        if (tmp) {
-            goto found;
-        }
-    }
-
-    while (words) {
-        tmp = addr[--words];
-        if (tmp) {
-        found:
-            return words * BITS_PER_LONG + BITS_PER_LONG - 1 - clzl(tmp);
-        }
-    }
-
-    /* Not found */
-    return size;
-}
diff --git a/contrib/qemu/util/cutils.c b/contrib/qemu/util/cutils.c
deleted file mode 100644
index 6caa4b8ddc4..00000000000
--- a/contrib/qemu/util/cutils.c
+++ /dev/null
@@ -1,532 +0,0 @@
-/*
- * Simple C functions to supplement the C library
- *
- * Copyright (c) 2006 Fabrice Bellard
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-#include "qemu-common.h"
-#include "qemu/host-utils.h"
-#include <math.h>
-
-#include "qemu/sockets.h"
-#include "qemu/iov.h"
-
-void strpadcpy(char *buf, int buf_size, const char *str, char pad)
-{
-    int len = qemu_strnlen(str, buf_size);
-    memcpy(buf, str, len);
-    memset(buf + len, pad, buf_size - len);
-}
-
-void pstrcpy(char *buf, int buf_size, const char *str)
-{
-    int c;
-    char *q = buf;
-
-    if (buf_size <= 0)
-        return;
-
-    for(;;) {
-        c = *str++;
-        if (c == 0 || q >= buf + buf_size - 1)
-            break;
-        *q++ = c;
-    }
-    *q = '\0';
-}
-
-/* strcat and truncate. */
-char *pstrcat(char *buf, int buf_size, const char *s)
-{
-    int len;
-    len = strlen(buf);
-    if (len < buf_size)
-        pstrcpy(buf + len, buf_size - len, s);
-    return buf;
-}
-
-int strstart(const char *str, const char *val, const char **ptr)
-{
-    const char *p, *q;
-    p = str;
-    q = val;
-    while (*q != '\0') {
-        if (*p != *q)
-            return 0;
-        p++;
-        q++;
-    }
-    if (ptr)
-        *ptr = p;
-    return 1;
-}
-
-int stristart(const char *str, const char *val, const char **ptr)
-{
-    const char *p, *q;
-    p = str;
-    q = val;
-    while (*q != '\0') {
-        if (qemu_toupper(*p) != qemu_toupper(*q))
-            return 0;
-        p++;
-        q++;
-    }
-    if (ptr)
-        *ptr = p;
-    return 1;
-}
-
-/* XXX: use host strnlen if available ? */
-int qemu_strnlen(const char *s, int max_len)
-{
-    int i;
-
-    for(i = 0; i < max_len; i++) {
-        if (s[i] == '\0') {
-            break;
-        }
-    }
-    return i;
-}
-
-char *qemu_strsep(char **input, const char *delim)
-{
-    char *result = *input;
-    if (result != NULL) {
-        char *p;
-
-        for (p = result; *p != '\0'; p++) {
-            if (strchr(delim, *p)) {
-                break;
-            }
-        }
-        if (*p == '\0') {
-            *input = NULL;
-        } else {
-            *p = '\0';
-            *input = p + 1;
-        }
-    }
-    return result;
-}
-
-time_t mktimegm(struct tm *tm)
-{
-    time_t t;
-    int y = tm->tm_year + 1900, m = tm->tm_mon + 1, d = tm->tm_mday;
-    if (m < 3) {
-        m += 12;
-        y--;
-    }
-    t = 86400ULL * (d + (153 * m - 457) / 5 + 365 * y + y / 4 - y / 100 + 
-                 y / 400 - 719469);
-    t += 3600 * tm->tm_hour + 60 * tm->tm_min + tm->tm_sec;
-    return t;
-}
-
-int qemu_fls(int i)
-{
-    return 32 - clz32(i);
-}
-
-/*
- * Make sure data goes on disk, but if possible do not bother to
- * write out the inode just for timestamp updates.
- *
- * Unfortunately even in 2009 many operating systems do not support
- * fdatasync and have to fall back to fsync.
- */
-int qemu_fdatasync(int fd)
-{
-#ifdef HAVE_FDATASYNC
-    return fdatasync(fd);
-#else
-    return fsync(fd);
-#endif
-}
-
-/*
- * Searches for an area with non-zero content in a buffer
- *
- * Attention! The len must be a multiple of
- * BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR * sizeof(VECTYPE)
- * and addr must be a multiple of sizeof(VECTYPE) due to
- * restriction of optimizations in this function.
- *
- * can_use_buffer_find_nonzero_offset() can be used to check
- * these requirements.
- *
- * The return value is the offset of the non-zero area rounded
- * down to a multiple of sizeof(VECTYPE) for the first
- * BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR chunks and down to
- * BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR * sizeof(VECTYPE)
- * afterwards.
- *
- * If the buffer is all zero the return value is equal to len.
- */
-
-size_t buffer_find_nonzero_offset(const void *buf, size_t len)
-{
-    const VECTYPE *p = buf;
-    const VECTYPE zero = (VECTYPE){0};
-    size_t i;
-
-    assert(can_use_buffer_find_nonzero_offset(buf, len));
-
-    if (!len) {
-        return 0;
-    }
-
-    for (i = 0; i < BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR; i++) {
-        if (!ALL_EQ(p[i], zero)) {
-            return i * sizeof(VECTYPE);
-        }
-    }
-
-    for (i = BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR;
-         i < len / sizeof(VECTYPE);
-         i += BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR) {
-        VECTYPE tmp0 = p[i + 0] | p[i + 1];
-        VECTYPE tmp1 = p[i + 2] | p[i + 3];
-        VECTYPE tmp2 = p[i + 4] | p[i + 5];
-        VECTYPE tmp3 = p[i + 6] | p[i + 7];
-        VECTYPE tmp01 = tmp0 | tmp1;
-        VECTYPE tmp23 = tmp2 | tmp3;
-        if (!ALL_EQ(tmp01 | tmp23, zero)) {
-            break;
-        }
-    }
-
-    return i * sizeof(VECTYPE);
-}
-
-/*
- * Checks if a buffer is all zeroes
- *
- * Attention! The len must be a multiple of 4 * sizeof(long) due to
- * restriction of optimizations in this function.
- */
-bool buffer_is_zero(const void *buf, size_t len)
-{
-    /*
-     * Use long as the biggest available internal data type that fits into the
-     * CPU register and unroll the loop to smooth out the effect of memory
-     * latency.
-     */
-
-    size_t i;
-    long d0, d1, d2, d3;
-    const long * const data = buf;
-
-    /* use vector optimized zero check if possible */
-    if (can_use_buffer_find_nonzero_offset(buf, len)) {
-        return buffer_find_nonzero_offset(buf, len) == len;
-    }
-
-    assert(len % (4 * sizeof(long)) == 0);
-    len /= sizeof(long);
-
-    for (i = 0; i < len; i += 4) {
-        d0 = data[i + 0];
-        d1 = data[i + 1];
-        d2 = data[i + 2];
-        d3 = data[i + 3];
-
-        if (d0 || d1 || d2 || d3) {
-            return false;
-        }
-    }
-
-    return true;
-}
-
-#ifndef _WIN32
-/* Sets a specific flag */
-int fcntl_setfl(int fd, int flag)
-{
-    int flags;
-
-    flags = fcntl(fd, F_GETFL);
-    if (flags == -1)
-        return -errno;
-
-    if (fcntl(fd, F_SETFL, flags | flag) == -1)
-        return -errno;
-
-    return 0;
-}
-#endif
-
-static int64_t suffix_mul(char suffix, int64_t unit)
-{
-    switch (qemu_toupper(suffix)) {
-    case STRTOSZ_DEFSUFFIX_B:
-        return 1;
-    case STRTOSZ_DEFSUFFIX_KB:
-        return unit;
-    case STRTOSZ_DEFSUFFIX_MB:
-        return unit * unit;
-    case STRTOSZ_DEFSUFFIX_GB:
-        return unit * unit * unit;
-    case STRTOSZ_DEFSUFFIX_TB:
-        return unit * unit * unit * unit;
-    case STRTOSZ_DEFSUFFIX_PB:
-        return unit * unit * unit * unit * unit;
-    case STRTOSZ_DEFSUFFIX_EB:
-        return unit * unit * unit * unit * unit * unit;
-    }
-    return -1;
-}
-
-/*
- * Convert string to bytes, allowing either B/b for bytes, K/k for KB,
- * M/m for MB, G/g for GB or T/t for TB. End pointer will be returned
- * in *end, if not NULL. Return -ERANGE on overflow, Return -EINVAL on
- * other error.
- */
-int64_t strtosz_suffix_unit(const char *nptr, char **end,
-                            const char default_suffix, int64_t unit)
-{
-    int64_t retval = -EINVAL;
-    char *endptr;
-    unsigned char c;
-    int mul_required = 0;
-    double val, mul, integral, fraction;
-
-    errno = 0;
-    val = strtod(nptr, &endptr);
-    if (isnan(val) || endptr == nptr || errno != 0) {
-        goto fail;
-    }
-    fraction = modf(val, &integral);
-    if (fraction != 0) {
-        mul_required = 1;
-    }
-    c = *endptr;
-    mul = suffix_mul(c, unit);
-    if (mul >= 0) {
-        endptr++;
-    } else {
-        mul = suffix_mul(default_suffix, unit);
-        assert(mul >= 0);
-    }
-    if (mul == 1 && mul_required) {
-        goto fail;
-    }
-    if ((val * mul >= INT64_MAX) || val < 0) {
-        retval = -ERANGE;
-        goto fail;
-    }
-    retval = val * mul;
-
-fail:
-    if (end) {
-        *end = endptr;
-    }
-
-    return retval;
-}
-
-int64_t strtosz_suffix(const char *nptr, char **end, const char default_suffix)
-{
-    return strtosz_suffix_unit(nptr, end, default_suffix, 1024);
-}
-
-int64_t strtosz(const char *nptr, char **end)
-{
-    return strtosz_suffix(nptr, end, STRTOSZ_DEFSUFFIX_MB);
-}
-
-/**
- * parse_uint:
- *
- * @s: String to parse
- * @value: Destination for parsed integer value
- * @endptr: Destination for pointer to first character not consumed
- * @base: integer base, between 2 and 36 inclusive, or 0
- *
- * Parse unsigned integer
- *
- * Parsed syntax is like strtoull()'s: arbitrary whitespace, a single optional
- * '+' or '-', an optional "0x" if @base is 0 or 16, one or more digits.
- *
- * If @s is null, or @base is invalid, or @s doesn't start with an
- * integer in the syntax above, set *@value to 0, *@endptr to @s, and
- * return -EINVAL.
- *
- * Set *@endptr to point right beyond the parsed integer (even if the integer
- * overflows or is negative, all digits will be parsed and *@endptr will
- * point right beyond them).
- *
- * If the integer is negative, set *@value to 0, and return -ERANGE.
- *
- * If the integer overflows unsigned long long, set *@value to
- * ULLONG_MAX, and return -ERANGE.
- *
- * Else, set *@value to the parsed integer, and return 0.
- */
-int parse_uint(const char *s, unsigned long long *value, char **endptr,
-               int base)
-{
-    int r = 0;
-    char *endp = (char *)s;
-    unsigned long long val = 0;
-
-    if (!s) {
-        r = -EINVAL;
-        goto out;
-    }
-
-    errno = 0;
-    val = strtoull(s, &endp, base);
-    if (errno) {
-        r = -errno;
-        goto out;
-    }
-
-    if (endp == s) {
-        r = -EINVAL;
-        goto out;
-    }
-
-    /* make sure we reject negative numbers: */
-    while (isspace((unsigned char)*s)) {
-        s++;
-    }
-    if (*s == '-') {
-        val = 0;
-        r = -ERANGE;
-        goto out;
-    }
-
-out:
-    *value = val;
-    *endptr = endp;
-    return r;
-}
-
-/**
- * parse_uint_full:
- *
- * @s: String to parse
- * @value: Destination for parsed integer value
- * @base: integer base, between 2 and 36 inclusive, or 0
- *
- * Parse unsigned integer from entire string
- *
- * Have the same behavior of parse_uint(), but with an additional check
- * for additional data after the parsed number. If extra characters are present
- * after the parsed number, the function will return -EINVAL, and *@v will
- * be set to 0.
- */
-int parse_uint_full(const char *s, unsigned long long *value, int base)
-{
-    char *endp;
-    int r;
-
-    r = parse_uint(s, value, &endp, base);
-    if (r < 0) {
-        return r;
-    }
-    if (*endp) {
-        *value = 0;
-        return -EINVAL;
-    }
-
-    return 0;
-}
-
-int qemu_parse_fd(const char *param)
-{
-    int fd;
-    char *endptr = NULL;
-
-    fd = strtol(param, &endptr, 10);
-    if (*endptr || (fd == 0 && param == endptr)) {
-        return -1;
-    }
-    return fd;
-}
-
-/* round down to the nearest power of 2*/
-int64_t pow2floor(int64_t value)
-{
-    if (!is_power_of_2(value)) {
-        value = 0x8000000000000000ULL >> clz64(value);
-    }
-    return value;
-}
-
-/*
- * Implementation of  ULEB128 (http://en.wikipedia.org/wiki/LEB128)
- * Input is limited to 14-bit numbers
- */
-int uleb128_encode_small(uint8_t *out, uint32_t n)
-{
-    g_assert(n <= 0x3fff);
-    if (n < 0x80) {
-        *out++ = n;
-        return 1;
-    } else {
-        *out++ = (n & 0x7f) | 0x80;
-        *out++ = n >> 7;
-        return 2;
-    }
-}
-
-int uleb128_decode_small(const uint8_t *in, uint32_t *n)
-{
-    if (!(*in & 0x80)) {
-        *n = *in++;
-        return 1;
-    } else {
-        *n = *in++ & 0x7f;
-        /* we exceed 14 bit number */
-        if (*in & 0x80) {
-            return -1;
-        }
-        *n |= *in++ << 7;
-        return 2;
-    }
-}
-
-/*
- * helper to parse debug environment variables
- */
-int parse_debug_env(const char *name, int max, int initial)
-{
-    char *debug_env = getenv(name);
-    char *inv = NULL;
-    int debug;
-
-    if (!debug_env) {
-        return initial;
-    }
-    debug = strtol(debug_env, &inv, 10);
-    if (inv == debug_env) {
-        return initial;
-    }
-    if (debug < 0 || debug > max) {
-        fprintf(stderr, "warning: %s not in [0, %d]", name, max);
-        return initial;
-    }
-    return debug;
-}
diff --git a/contrib/qemu/util/error.c b/contrib/qemu/util/error.c
deleted file mode 100644
index 53b04354aef..00000000000
--- a/contrib/qemu/util/error.c
+++ /dev/null
@@ -1,120 +0,0 @@
-/*
- * QEMU Error Objects
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.  See
- * the COPYING.LIB file in the top-level directory.
- */
-
-#include "qemu-common.h"
-#include "qapi/error.h"
-#include "qapi/qmp/qjson.h"
-#include "qapi/qmp/qdict.h"
-#include "qapi-types.h"
-#include "qapi/qmp/qerror.h"
-
-struct Error
-{
-    char *msg;
-    ErrorClass err_class;
-};
-
-void error_set(Error **errp, ErrorClass err_class, const char *fmt, ...)
-{
-    Error *err;
-    va_list ap;
-
-    if (errp == NULL) {
-        return;
-    }
-    assert(*errp == NULL);
-
-    err = g_malloc0(sizeof(*err));
-
-    va_start(ap, fmt);
-    err->msg = g_strdup_vprintf(fmt, ap);
-    va_end(ap);
-    err->err_class = err_class;
-
-    *errp = err;
-}
-
-void error_set_errno(Error **errp, int os_errno, ErrorClass err_class,
-                     const char *fmt, ...)
-{
-    Error *err;
-    char *msg1;
-    va_list ap;
-
-    if (errp == NULL) {
-        return;
-    }
-    assert(*errp == NULL);
-
-    err = g_malloc0(sizeof(*err));
-
-    va_start(ap, fmt);
-    msg1 = g_strdup_vprintf(fmt, ap);
-    if (os_errno != 0) {
-        err->msg = g_strdup_printf("%s: %s", msg1, strerror(os_errno));
-        g_free(msg1);
-    } else {
-        err->msg = msg1;
-    }
-    va_end(ap);
-    err->err_class = err_class;
-
-    *errp = err;
-}
-
-void error_setg_file_open(Error **errp, int os_errno, const char *filename)
-{
-    error_setg_errno(errp, os_errno, "Could not open '%s'", filename);
-}
-
-Error *error_copy(const Error *err)
-{
-    Error *err_new;
-
-    err_new = g_malloc0(sizeof(*err));
-    err_new->msg = g_strdup(err->msg);
-    err_new->err_class = err->err_class;
-
-    return err_new;
-}
-
-bool error_is_set(Error **errp)
-{
-    return (errp && *errp);
-}
-
-ErrorClass error_get_class(const Error *err)
-{
-    return err->err_class;
-}
-
-const char *error_get_pretty(Error *err)
-{
-    return err->msg;
-}
-
-void error_free(Error *err)
-{
-    if (err) {
-        g_free(err->msg);
-        g_free(err);
-    }
-}
-
-void error_propagate(Error **dst_err, Error *local_err)
-{
-    if (dst_err && !*dst_err) {
-        *dst_err = local_err;
-    } else if (local_err) {
-        error_free(local_err);
-    }
-}
diff --git a/contrib/qemu/util/hbitmap.c b/contrib/qemu/util/hbitmap.c
deleted file mode 100644
index e063e681f52..00000000000
--- a/contrib/qemu/util/hbitmap.c
+++ /dev/null
@@ -1,404 +0,0 @@
-/*
- * Hierarchical Bitmap Data Type
- *
- * Copyright Red Hat, Inc., 2012
- *
- * Author: Paolo Bonzini <pbonzini@redhat.com>
- *
- * This work is licensed under the terms of the GNU GPL, version 2 or
- * later.  See the COPYING file in the top-level directory.
- */
-
-#include <string.h>
-#include <glib.h>
-#include <assert.h>
-#include "qemu/osdep.h"
-#include "qemu/hbitmap.h"
-#include "qemu/host-utils.h"
-#include "trace.h"
-
-/* HBitmaps provides an array of bits.  The bits are stored as usual in an
- * array of unsigned longs, but HBitmap is also optimized to provide fast
- * iteration over set bits; going from one bit to the next is O(logB n)
- * worst case, with B = sizeof(long) * CHAR_BIT: the result is low enough
- * that the number of levels is in fact fixed.
- *
- * In order to do this, it stacks multiple bitmaps with progressively coarser
- * granularity; in all levels except the last, bit N is set iff the N-th
- * unsigned long is nonzero in the immediately next level.  When iteration
- * completes on the last level it can examine the 2nd-last level to quickly
- * skip entire words, and even do so recursively to skip blocks of 64 words or
- * powers thereof (32 on 32-bit machines).
- *
- * Given an index in the bitmap, it can be split in group of bits like
- * this (for the 64-bit case):
- *
- *   bits 0-57 => word in the last bitmap     | bits 58-63 => bit in the word
- *   bits 0-51 => word in the 2nd-last bitmap | bits 52-57 => bit in the word
- *   bits 0-45 => word in the 3rd-last bitmap | bits 46-51 => bit in the word
- *
- * So it is easy to move up simply by shifting the index right by
- * log2(BITS_PER_LONG) bits.  To move down, you shift the index left
- * similarly, and add the word index within the group.  Iteration uses
- * ffs (find first set bit) to find the next word to examine; this
- * operation can be done in constant time in most current architectures.
- *
- * Setting or clearing a range of m bits on all levels, the work to perform
- * is O(m + m/W + m/W^2 + ...), which is O(m) like on a regular bitmap.
- *
- * When iterating on a bitmap, each bit (on any level) is only visited
- * once.  Hence, The total cost of visiting a bitmap with m bits in it is
- * the number of bits that are set in all bitmaps.  Unless the bitmap is
- * extremely sparse, this is also O(m + m/W + m/W^2 + ...), so the amortized
- * cost of advancing from one bit to the next is usually constant (worst case
- * O(logB n) as in the non-amortized complexity).
- */
-
-struct HBitmap {
-    /* Number of total bits in the bottom level.  */
-    uint64_t size;
-
-    /* Number of set bits in the bottom level.  */
-    uint64_t count;
-
-    /* A scaling factor.  Given a granularity of G, each bit in the bitmap will
-     * will actually represent a group of 2^G elements.  Each operation on a
-     * range of bits first rounds the bits to determine which group they land
-     * in, and then affect the entire page; iteration will only visit the first
-     * bit of each group.  Here is an example of operations in a size-16,
-     * granularity-1 HBitmap:
-     *
-     *    initial state            00000000
-     *    set(start=0, count=9)    11111000 (iter: 0, 2, 4, 6, 8)
-     *    reset(start=1, count=3)  00111000 (iter: 4, 6, 8)
-     *    set(start=9, count=2)    00111100 (iter: 4, 6, 8, 10)
-     *    reset(start=5, count=5)  00000000
-     *
-     * From an implementation point of view, when setting or resetting bits,
-     * the bitmap will scale bit numbers right by this amount of bits.  When
-     * iterating, the bitmap will scale bit numbers left by this amount of
-     * bits.
-     */
-    int granularity;
-
-    /* A number of progressively less coarse bitmaps (i.e. level 0 is the
-     * coarsest).  Each bit in level N represents a word in level N+1 that
-     * has a set bit, except the last level where each bit represents the
-     * actual bitmap.
-     *
-     * Note that all bitmaps have the same number of levels.  Even a 1-bit
-     * bitmap will still allocate HBITMAP_LEVELS arrays.
-     */
-    unsigned long *levels[HBITMAP_LEVELS];
-};
-
-#ifndef __NetBSD__ /* we have it in <strings.h> */
-static inline int popcountl(unsigned long l)
-{
-    return BITS_PER_LONG == 32 ? ctpop32(l) : ctpop64(l);
-}
-#endif
-
-/* Advance hbi to the next nonzero word and return it.  hbi->pos
- * is updated.  Returns zero if we reach the end of the bitmap.
- */
-unsigned long hbitmap_iter_skip_words(HBitmapIter *hbi)
-{
-    size_t pos = hbi->pos;
-    const HBitmap *hb = hbi->hb;
-    unsigned i = HBITMAP_LEVELS - 1;
-
-    unsigned long cur;
-    do {
-        cur = hbi->cur[--i];
-        pos >>= BITS_PER_LEVEL;
-    } while (cur == 0);
-
-    /* Check for end of iteration.  We always use fewer than BITS_PER_LONG
-     * bits in the level 0 bitmap; thus we can repurpose the most significant
-     * bit as a sentinel.  The sentinel is set in hbitmap_alloc and ensures
-     * that the above loop ends even without an explicit check on i.
-     */
-
-    if (i == 0 && cur == (1UL << (BITS_PER_LONG - 1))) {
-        return 0;
-    }
-    for (; i < HBITMAP_LEVELS - 1; i++) {
-        /* Shift back pos to the left, matching the right shifts above.
-         * The index of this word's least significant set bit provides
-         * the low-order bits.
-         */
-        assert(cur);
-        pos = (pos << BITS_PER_LEVEL) + ctzl(cur);
-        hbi->cur[i] = cur & (cur - 1);
-
-        /* Set up next level for iteration.  */
-        cur = hb->levels[i + 1][pos];
-    }
-
-    hbi->pos = pos;
-    trace_hbitmap_iter_skip_words(hbi->hb, hbi, pos, cur);
-
-    assert(cur);
-    return cur;
-}
-
-void hbitmap_iter_init(HBitmapIter *hbi, const HBitmap *hb, uint64_t first)
-{
-    unsigned i, bit;
-    uint64_t pos;
-
-    hbi->hb = hb;
-    pos = first >> hb->granularity;
-    assert(pos < hb->size);
-    hbi->pos = pos >> BITS_PER_LEVEL;
-    hbi->granularity = hb->granularity;
-
-    for (i = HBITMAP_LEVELS; i-- > 0; ) {
-        bit = pos & (BITS_PER_LONG - 1);
-        pos >>= BITS_PER_LEVEL;
-
-        /* Drop bits representing items before first.  */
-        hbi->cur[i] = hb->levels[i][pos] & ~((1UL << bit) - 1);
-
-        /* We have already added level i+1, so the lowest set bit has
-         * been processed.  Clear it.
-         */
-        if (i != HBITMAP_LEVELS - 1) {
-            hbi->cur[i] &= ~(1UL << bit);
-        }
-    }
-}
-
-bool hbitmap_empty(const HBitmap *hb)
-{
-    return hb->count == 0;
-}
-
-int hbitmap_granularity(const HBitmap *hb)
-{
-    return hb->granularity;
-}
-
-uint64_t hbitmap_count(const HBitmap *hb)
-{
-    return hb->count << hb->granularity;
-}
-
-/* Count the number of set bits between start and end, not accounting for
- * the granularity.  Also an example of how to use hbitmap_iter_next_word.
- */
-static uint64_t hb_count_between(HBitmap *hb, uint64_t start, uint64_t last)
-{
-    HBitmapIter hbi;
-    uint64_t count = 0;
-    uint64_t end = last + 1;
-    unsigned long cur;
-    size_t pos;
-
-    hbitmap_iter_init(&hbi, hb, start << hb->granularity);
-    for (;;) {
-        pos = hbitmap_iter_next_word(&hbi, &cur);
-        if (pos >= (end >> BITS_PER_LEVEL)) {
-            break;
-        }
-        count += popcountl(cur);
-    }
-
-    if (pos == (end >> BITS_PER_LEVEL)) {
-        /* Drop bits representing the END-th and subsequent items.  */
-        int bit = end & (BITS_PER_LONG - 1);
-        cur &= (1UL << bit) - 1;
-        count += popcountl(cur);
-    }
-
-    return count;
-}
-
-/* Setting starts at the last layer and propagates up if an element
- * changes from zero to non-zero.
- */
-static inline bool hb_set_elem(unsigned long *elem, uint64_t start, uint64_t last)
-{
-    unsigned long mask;
-    bool changed;
-
-    assert((last >> BITS_PER_LEVEL) == (start >> BITS_PER_LEVEL));
-    assert(start <= last);
-
-    mask = 2UL << (last & (BITS_PER_LONG - 1));
-    mask -= 1UL << (start & (BITS_PER_LONG - 1));
-    changed = (*elem == 0);
-    *elem |= mask;
-    return changed;
-}
-
-/* The recursive workhorse (the depth is limited to HBITMAP_LEVELS)... */
-static void hb_set_between(HBitmap *hb, int level, uint64_t start, uint64_t last)
-{
-    size_t pos = start >> BITS_PER_LEVEL;
-    size_t lastpos = last >> BITS_PER_LEVEL;
-    bool changed = false;
-    size_t i;
-
-    i = pos;
-    if (i < lastpos) {
-        uint64_t next = (start | (BITS_PER_LONG - 1)) + 1;
-        changed |= hb_set_elem(&hb->levels[level][i], start, next - 1);
-        for (;;) {
-            start = next;
-            next += BITS_PER_LONG;
-            if (++i == lastpos) {
-                break;
-            }
-            changed |= (hb->levels[level][i] == 0);
-            hb->levels[level][i] = ~0UL;
-        }
-    }
-    changed |= hb_set_elem(&hb->levels[level][i], start, last);
-
-    /* If there was any change in this layer, we may have to update
-     * the one above.
-     */
-    if (level > 0 && changed) {
-        hb_set_between(hb, level - 1, pos, lastpos);
-    }
-}
-
-void hbitmap_set(HBitmap *hb, uint64_t start, uint64_t count)
-{
-    /* Compute range in the last layer.  */
-    uint64_t last = start + count - 1;
-
-    trace_hbitmap_set(hb, start, count,
-                      start >> hb->granularity, last >> hb->granularity);
-
-    start >>= hb->granularity;
-    last >>= hb->granularity;
-    count = last - start + 1;
-
-    hb->count += count - hb_count_between(hb, start, last);
-    hb_set_between(hb, HBITMAP_LEVELS - 1, start, last);
-}
-
-/* Resetting works the other way round: propagate up if the new
- * value is zero.
- */
-static inline bool hb_reset_elem(unsigned long *elem, uint64_t start, uint64_t last)
-{
-    unsigned long mask;
-    bool blanked;
-
-    assert((last >> BITS_PER_LEVEL) == (start >> BITS_PER_LEVEL));
-    assert(start <= last);
-
-    mask = 2UL << (last & (BITS_PER_LONG - 1));
-    mask -= 1UL << (start & (BITS_PER_LONG - 1));
-    blanked = *elem != 0 && ((*elem & ~mask) == 0);
-    *elem &= ~mask;
-    return blanked;
-}
-
-/* The recursive workhorse (the depth is limited to HBITMAP_LEVELS)... */
-static void hb_reset_between(HBitmap *hb, int level, uint64_t start, uint64_t last)
-{
-    size_t pos = start >> BITS_PER_LEVEL;
-    size_t lastpos = last >> BITS_PER_LEVEL;
-    bool changed = false;
-    size_t i;
-
-    i = pos;
-    if (i < lastpos) {
-        uint64_t next = (start | (BITS_PER_LONG - 1)) + 1;
-
-        /* Here we need a more complex test than when setting bits.  Even if
-         * something was changed, we must not blank bits in the upper level
-         * unless the lower-level word became entirely zero.  So, remove pos
-         * from the upper-level range if bits remain set.
-         */
-        if (hb_reset_elem(&hb->levels[level][i], start, next - 1)) {
-            changed = true;
-        } else {
-            pos++;
-        }
-
-        for (;;) {
-            start = next;
-            next += BITS_PER_LONG;
-            if (++i == lastpos) {
-                break;
-            }
-            changed |= (hb->levels[level][i] != 0);
-            hb->levels[level][i] = 0UL;
-        }
-    }
-
-    /* Same as above, this time for lastpos.  */
-    if (hb_reset_elem(&hb->levels[level][i], start, last)) {
-        changed = true;
-    } else {
-        lastpos--;
-    }
-
-    if (level > 0 && changed) {
-        hb_reset_between(hb, level - 1, pos, lastpos);
-    }
-}
-
-void hbitmap_reset(HBitmap *hb, uint64_t start, uint64_t count)
-{
-    /* Compute range in the last layer.  */
-    uint64_t last = start + count - 1;
-
-    trace_hbitmap_reset(hb, start, count,
-                        start >> hb->granularity, last >> hb->granularity);
-
-    start >>= hb->granularity;
-    last >>= hb->granularity;
-
-    hb->count -= hb_count_between(hb, start, last);
-    hb_reset_between(hb, HBITMAP_LEVELS - 1, start, last);
-}
-
-bool hbitmap_get(const HBitmap *hb, uint64_t item)
-{
-    /* Compute position and bit in the last layer.  */
-    uint64_t pos = item >> hb->granularity;
-    unsigned long bit = 1UL << (pos & (BITS_PER_LONG - 1));
-
-    return (hb->levels[HBITMAP_LEVELS - 1][pos >> BITS_PER_LEVEL] & bit) != 0;
-}
-
-void hbitmap_free(HBitmap *hb)
-{
-    unsigned i;
-    for (i = HBITMAP_LEVELS; i-- > 0; ) {
-        g_free(hb->levels[i]);
-    }
-    g_free(hb);
-}
-
-HBitmap *hbitmap_alloc(uint64_t size, int granularity)
-{
-    HBitmap *hb = g_malloc0(sizeof (struct HBitmap));
-    unsigned i;
-
-    assert(granularity >= 0 && granularity < 64);
-    size = (size + (1ULL << granularity) - 1) >> granularity;
-    assert(size <= ((uint64_t)1 << HBITMAP_LOG_MAX_SIZE));
-
-    hb->size = size;
-    hb->granularity = granularity;
-    for (i = HBITMAP_LEVELS; i-- > 0; ) {
-        size = MAX((size + BITS_PER_LONG - 1) >> BITS_PER_LEVEL, 1);
-        hb->levels[i] = g_malloc0(size * sizeof(unsigned long));
-    }
-
-    /* We necessarily have free bits in level 0 due to the definition
-     * of HBITMAP_LEVELS, so use one for a sentinel.  This speeds up
-     * hbitmap_iter_skip_words.
-     */
-    assert(size == 1);
-    hb->levels[0][0] |= 1UL << (BITS_PER_LONG - 1);
-    return hb;
-}
diff --git a/contrib/qemu/util/hexdump.c b/contrib/qemu/util/hexdump.c
deleted file mode 100644
index 969b3406c07..00000000000
--- a/contrib/qemu/util/hexdump.c
+++ /dev/null
@@ -1,37 +0,0 @@
-/*
- * Helper to hexdump a buffer
- *
- * Copyright (c) 2013 Red Hat, Inc.
- * Copyright (c) 2013 Gerd Hoffmann <kraxel@redhat.com>
- * Copyright (c) 2013 Peter Crosthwaite <peter.crosthwaite@xilinx.com>
- * Copyright (c) 2013 Xilinx, Inc
- *
- * This work is licensed under the terms of the GNU GPL, version 2.  See
- * the COPYING file in the top-level directory.
- *
- * Contributions after 2012-01-13 are licensed under the terms of the
- * GNU GPL, version 2 or (at your option) any later version.
- */
-
-#include "qemu-common.h"
-
-void qemu_hexdump(const char *buf, FILE *fp, const char *prefix, size_t size)
-{
-    unsigned int b;
-
-    for (b = 0; b < size; b++) {
-        if ((b % 16) == 0) {
-            fprintf(fp, "%s: %04x:", prefix, b);
-        }
-        if ((b % 4) == 0) {
-            fprintf(fp, " ");
-        }
-        fprintf(fp, " %02x", (unsigned char)buf[b]);
-        if ((b % 16) == 15) {
-            fprintf(fp, "\n");
-        }
-    }
-    if ((b % 16) != 0) {
-        fprintf(fp, "\n");
-    }
-}
diff --git a/contrib/qemu/util/iov.c b/contrib/qemu/util/iov.c
deleted file mode 100644
index cc6e837c836..00000000000
--- a/contrib/qemu/util/iov.c
+++ /dev/null
@@ -1,426 +0,0 @@
-/*
- * Helpers for getting linearized buffers from iov / filling buffers into iovs
- *
- * Copyright IBM, Corp. 2007, 2008
- * Copyright (C) 2010 Red Hat, Inc.
- *
- * Author(s):
- *  Anthony Liguori <aliguori@us.ibm.com>
- *  Amit Shah <amit.shah@redhat.com>
- *  Michael Tokarev <mjt@tls.msk.ru>
- *
- * This work is licensed under the terms of the GNU GPL, version 2.  See
- * the COPYING file in the top-level directory.
- *
- * Contributions after 2012-01-13 are licensed under the terms of the
- * GNU GPL, version 2 or (at your option) any later version.
- */
-
-#include "qemu/iov.h"
-
-#ifdef _WIN32
-# include <windows.h>
-# include <winsock2.h>
-#else
-# include <sys/types.h>
-# include <sys/socket.h>
-#endif
-
-size_t iov_from_buf(const struct iovec *iov, unsigned int iov_cnt,
-                    size_t offset, const void *buf, size_t bytes)
-{
-    size_t done;
-    unsigned int i;
-    for (i = 0, done = 0; (offset || done < bytes) && i < iov_cnt; i++) {
-        if (offset < iov[i].iov_len) {
-            size_t len = MIN(iov[i].iov_len - offset, bytes - done);
-            memcpy(iov[i].iov_base + offset, buf + done, len);
-            done += len;
-            offset = 0;
-        } else {
-            offset -= iov[i].iov_len;
-        }
-    }
-    assert(offset == 0);
-    return done;
-}
-
-size_t iov_to_buf(const struct iovec *iov, const unsigned int iov_cnt,
-                  size_t offset, void *buf, size_t bytes)
-{
-    size_t done;
-    unsigned int i;
-    for (i = 0, done = 0; (offset || done < bytes) && i < iov_cnt; i++) {
-        if (offset < iov[i].iov_len) {
-            size_t len = MIN(iov[i].iov_len - offset, bytes - done);
-            memcpy(buf + done, iov[i].iov_base + offset, len);
-            done += len;
-            offset = 0;
-        } else {
-            offset -= iov[i].iov_len;
-        }
-    }
-    assert(offset == 0);
-    return done;
-}
-
-size_t iov_memset(const struct iovec *iov, const unsigned int iov_cnt,
-                  size_t offset, int fillc, size_t bytes)
-{
-    size_t done;
-    unsigned int i;
-    for (i = 0, done = 0; (offset || done < bytes) && i < iov_cnt; i++) {
-        if (offset < iov[i].iov_len) {
-            size_t len = MIN(iov[i].iov_len - offset, bytes - done);
-            memset(iov[i].iov_base + offset, fillc, len);
-            done += len;
-            offset = 0;
-        } else {
-            offset -= iov[i].iov_len;
-        }
-    }
-    assert(offset == 0);
-    return done;
-}
-
-size_t iov_size(const struct iovec *iov, const unsigned int iov_cnt)
-{
-    size_t len;
-    unsigned int i;
-
-    len = 0;
-    for (i = 0; i < iov_cnt; i++) {
-        len += iov[i].iov_len;
-    }
-    return len;
-}
-
-/* helper function for iov_send_recv() */
-static ssize_t
-do_send_recv(int sockfd, struct iovec *iov, unsigned iov_cnt, bool do_send)
-{
-#ifdef CONFIG_POSIX
-    ssize_t ret;
-    struct msghdr msg;
-    memset(&msg, 0, sizeof(msg));
-    msg.msg_iov = iov;
-    msg.msg_iovlen = iov_cnt;
-    do {
-        ret = do_send
-            ? sendmsg(sockfd, &msg, 0)
-            : recvmsg(sockfd, &msg, 0);
-    } while (ret < 0 && errno == EINTR);
-    return ret;
-#else
-    /* else send piece-by-piece */
-    /*XXX Note: windows has WSASend() and WSARecv() */
-    unsigned i = 0;
-    ssize_t ret = 0;
-    while (i < iov_cnt) {
-        ssize_t r = do_send
-            ? send(sockfd, iov[i].iov_base, iov[i].iov_len, 0)
-            : recv(sockfd, iov[i].iov_base, iov[i].iov_len, 0);
-        if (r > 0) {
-            ret += r;
-        } else if (!r) {
-            break;
-        } else if (errno == EINTR) {
-            continue;
-        } else {
-            /* else it is some "other" error,
-             * only return if there was no data processed. */
-            if (ret == 0) {
-                ret = -1;
-            }
-            break;
-        }
-        i++;
-    }
-    return ret;
-#endif
-}
-
-ssize_t iov_send_recv(int sockfd, struct iovec *iov, unsigned iov_cnt,
-                      size_t offset, size_t bytes,
-                      bool do_send)
-{
-    ssize_t total = 0;
-    ssize_t ret;
-    size_t orig_len, tail;
-    unsigned niov;
-
-    while (bytes > 0) {
-        /* Find the start position, skipping `offset' bytes:
-         * first, skip all full-sized vector elements, */
-        for (niov = 0; niov < iov_cnt && offset >= iov[niov].iov_len; ++niov) {
-            offset -= iov[niov].iov_len;
-        }
-
-        /* niov == iov_cnt would only be valid if bytes == 0, which
-         * we already ruled out in the loop condition.  */
-        assert(niov < iov_cnt);
-        iov += niov;
-        iov_cnt -= niov;
-
-        if (offset) {
-            /* second, skip `offset' bytes from the (now) first element,
-             * undo it on exit */
-            iov[0].iov_base += offset;
-            iov[0].iov_len -= offset;
-        }
-        /* Find the end position skipping `bytes' bytes: */
-        /* first, skip all full-sized elements */
-        tail = bytes;
-        for (niov = 0; niov < iov_cnt && iov[niov].iov_len <= tail; ++niov) {
-            tail -= iov[niov].iov_len;
-        }
-        if (tail) {
-            /* second, fixup the last element, and remember the original
-             * length */
-            assert(niov < iov_cnt);
-            assert(iov[niov].iov_len > tail);
-            orig_len = iov[niov].iov_len;
-            iov[niov++].iov_len = tail;
-        }
-
-        ret = do_send_recv(sockfd, iov, niov, do_send);
-
-        /* Undo the changes above before checking for errors */
-        if (tail) {
-            iov[niov-1].iov_len = orig_len;
-        }
-        if (offset) {
-            iov[0].iov_base -= offset;
-            iov[0].iov_len += offset;
-        }
-
-        if (ret < 0) {
-            assert(errno != EINTR);
-            if (errno == EAGAIN && total > 0) {
-                return total;
-            }
-            return -1;
-        }
-
-        /* Prepare for the next iteration */
-        offset += ret;
-        total += ret;
-        bytes -= ret;
-    }
-
-    return total;
-}
-
-
-void iov_hexdump(const struct iovec *iov, const unsigned int iov_cnt,
-                 FILE *fp, const char *prefix, size_t limit)
-{
-    int v;
-    size_t size = 0;
-    char *buf;
-
-    for (v = 0; v < iov_cnt; v++) {
-        size += iov[v].iov_len;
-    }
-    size = size > limit ? limit : size;
-    buf = g_malloc(size);
-    iov_to_buf(iov, iov_cnt, 0, buf, size);
-    qemu_hexdump(buf, fp, prefix, size);
-    g_free(buf);
-}
-
-unsigned iov_copy(struct iovec *dst_iov, unsigned int dst_iov_cnt,
-                 const struct iovec *iov, unsigned int iov_cnt,
-                 size_t offset, size_t bytes)
-{
-    size_t len;
-    unsigned int i, j;
-    for (i = 0, j = 0; i < iov_cnt && j < dst_iov_cnt && bytes; i++) {
-        if (offset >= iov[i].iov_len) {
-            offset -= iov[i].iov_len;
-            continue;
-        }
-        len = MIN(bytes, iov[i].iov_len - offset);
-
-        dst_iov[j].iov_base = iov[i].iov_base + offset;
-        dst_iov[j].iov_len = len;
-        j++;
-        bytes -= len;
-        offset = 0;
-    }
-    assert(offset == 0);
-    return j;
-}
-
-/* io vectors */
-
-void qemu_iovec_init(QEMUIOVector *qiov, int alloc_hint)
-{
-    qiov->iov = g_malloc(alloc_hint * sizeof(struct iovec));
-    qiov->niov = 0;
-    qiov->nalloc = alloc_hint;
-    qiov->size = 0;
-}
-
-void qemu_iovec_init_external(QEMUIOVector *qiov, struct iovec *iov, int niov)
-{
-    int i;
-
-    qiov->iov = iov;
-    qiov->niov = niov;
-    qiov->nalloc = -1;
-    qiov->size = 0;
-    for (i = 0; i < niov; i++)
-        qiov->size += iov[i].iov_len;
-}
-
-void qemu_iovec_add(QEMUIOVector *qiov, void *base, size_t len)
-{
-    assert(qiov->nalloc != -1);
-
-    if (qiov->niov == qiov->nalloc) {
-        qiov->nalloc = 2 * qiov->nalloc + 1;
-        qiov->iov = g_realloc(qiov->iov, qiov->nalloc * sizeof(struct iovec));
-    }
-    qiov->iov[qiov->niov].iov_base = base;
-    qiov->iov[qiov->niov].iov_len = len;
-    qiov->size += len;
-    ++qiov->niov;
-}
-
-/*
- * Concatenates (partial) iovecs from src_iov to the end of dst.
- * It starts copying after skipping `soffset' bytes at the
- * beginning of src and adds individual vectors from src to
- * dst copies up to `sbytes' bytes total, or up to the end
- * of src_iov if it comes first.  This way, it is okay to specify
- * very large value for `sbytes' to indicate "up to the end
- * of src".
- * Only vector pointers are processed, not the actual data buffers.
- */
-void qemu_iovec_concat_iov(QEMUIOVector *dst,
-                           struct iovec *src_iov, unsigned int src_cnt,
-                           size_t soffset, size_t sbytes)
-{
-    int i;
-    size_t done;
-
-    if (!sbytes) {
-        return;
-    }
-    assert(dst->nalloc != -1);
-    for (i = 0, done = 0; done < sbytes && i < src_cnt; i++) {
-        if (soffset < src_iov[i].iov_len) {
-            size_t len = MIN(src_iov[i].iov_len - soffset, sbytes - done);
-            qemu_iovec_add(dst, src_iov[i].iov_base + soffset, len);
-            done += len;
-            soffset = 0;
-        } else {
-            soffset -= src_iov[i].iov_len;
-        }
-    }
-    assert(soffset == 0); /* offset beyond end of src */
-}
-
-/*
- * Concatenates (partial) iovecs from src to the end of dst.
- * It starts copying after skipping `soffset' bytes at the
- * beginning of src and adds individual vectors from src to
- * dst copies up to `sbytes' bytes total, or up to the end
- * of src if it comes first.  This way, it is okay to specify
- * very large value for `sbytes' to indicate "up to the end
- * of src".
- * Only vector pointers are processed, not the actual data buffers.
- */
-void qemu_iovec_concat(QEMUIOVector *dst,
-                       QEMUIOVector *src, size_t soffset, size_t sbytes)
-{
-    qemu_iovec_concat_iov(dst, src->iov, src->niov, soffset, sbytes);
-}
-
-void qemu_iovec_destroy(QEMUIOVector *qiov)
-{
-    assert(qiov->nalloc != -1);
-
-    qemu_iovec_reset(qiov);
-    g_free(qiov->iov);
-    qiov->nalloc = 0;
-    qiov->iov = NULL;
-}
-
-void qemu_iovec_reset(QEMUIOVector *qiov)
-{
-    assert(qiov->nalloc != -1);
-
-    qiov->niov = 0;
-    qiov->size = 0;
-}
-
-size_t qemu_iovec_to_buf(QEMUIOVector *qiov, size_t offset,
-                         void *buf, size_t bytes)
-{
-    return iov_to_buf(qiov->iov, qiov->niov, offset, buf, bytes);
-}
-
-size_t qemu_iovec_from_buf(QEMUIOVector *qiov, size_t offset,
-                           const void *buf, size_t bytes)
-{
-    return iov_from_buf(qiov->iov, qiov->niov, offset, buf, bytes);
-}
-
-size_t qemu_iovec_memset(QEMUIOVector *qiov, size_t offset,
-                         int fillc, size_t bytes)
-{
-    return iov_memset(qiov->iov, qiov->niov, offset, fillc, bytes);
-}
-
-size_t iov_discard_front(struct iovec **iov, unsigned int *iov_cnt,
-                         size_t bytes)
-{
-    size_t total = 0;
-    struct iovec *cur;
-
-    for (cur = *iov; *iov_cnt > 0; cur++) {
-        if (cur->iov_len > bytes) {
-            cur->iov_base += bytes;
-            cur->iov_len -= bytes;
-            total += bytes;
-            break;
-        }
-
-        bytes -= cur->iov_len;
-        total += cur->iov_len;
-        *iov_cnt -= 1;
-    }
-
-    *iov = cur;
-    return total;
-}
-
-size_t iov_discard_back(struct iovec *iov, unsigned int *iov_cnt,
-                        size_t bytes)
-{
-    size_t total = 0;
-    struct iovec *cur;
-
-    if (*iov_cnt == 0) {
-        return 0;
-    }
-
-    cur = iov + (*iov_cnt - 1);
-
-    while (*iov_cnt > 0) {
-        if (cur->iov_len > bytes) {
-            cur->iov_len -= bytes;
-            total += bytes;
-            break;
-        }
-
-        bytes -= cur->iov_len;
-        total += cur->iov_len;
-        cur--;
-        *iov_cnt -= 1;
-    }
-
-    return total;
-}
diff --git a/contrib/qemu/util/module.c b/contrib/qemu/util/module.c
deleted file mode 100644
index 7acc33d076a..00000000000
--- a/contrib/qemu/util/module.c
+++ /dev/null
@@ -1,81 +0,0 @@
-/*
- * QEMU Module Infrastructure
- *
- * Copyright IBM, Corp. 2009
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU GPL, version 2.  See
- * the COPYING file in the top-level directory.
- *
- * Contributions after 2012-01-13 are licensed under the terms of the
- * GNU GPL, version 2 or (at your option) any later version.
- */
-
-#include "qemu-common.h"
-#include "qemu/queue.h"
-#include "qemu/module.h"
-
-typedef struct ModuleEntry
-{
-    void (*init)(void);
-    QTAILQ_ENTRY(ModuleEntry) node;
-} ModuleEntry;
-
-typedef QTAILQ_HEAD(, ModuleEntry) ModuleTypeList;
-
-static ModuleTypeList init_type_list[MODULE_INIT_MAX];
-
-static void init_types(void)
-{
-    static int inited;
-    int i;
-
-    if (inited) {
-        return;
-    }
-
-    for (i = 0; i < MODULE_INIT_MAX; i++) {
-        QTAILQ_INIT(&init_type_list[i]);
-    }
-
-    inited = 1;
-}
-
-
-static ModuleTypeList *find_type(module_init_type type)
-{
-    ModuleTypeList *l;
-
-    init_types();
-
-    l = &init_type_list[type];
-
-    return l;
-}
-
-void register_module_init(void (*fn)(void), module_init_type type)
-{
-    ModuleEntry *e;
-    ModuleTypeList *l;
-
-    e = g_malloc0(sizeof(*e));
-    e->init = fn;
-
-    l = find_type(type);
-
-    QTAILQ_INSERT_TAIL(l, e, node);
-}
-
-void module_call_init(module_init_type type)
-{
-    ModuleTypeList *l;
-    ModuleEntry *e;
-
-    l = find_type(type);
-
-    QTAILQ_FOREACH(e, l, node) {
-        e->init();
-    }
-}
diff --git a/contrib/qemu/util/oslib-posix.c b/contrib/qemu/util/oslib-posix.c
deleted file mode 100644
index bac4c1a158e..00000000000
--- a/contrib/qemu/util/oslib-posix.c
+++ /dev/null
@@ -1,248 +0,0 @@
-/*
- * os-posix-lib.c
- *
- * Copyright (c) 2003-2008 Fabrice Bellard
- * Copyright (c) 2010 Red Hat, Inc.
- *
- * QEMU library functions on POSIX which are shared between QEMU and
- * the QEMU tools.
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-
-/* The following block of code temporarily renames the daemon() function so the
-   compiler does not see the warning associated with it in stdlib.h on OSX */
-#ifdef __APPLE__
-#define daemon qemu_fake_daemon_function
-#include <stdlib.h>
-#undef daemon
-extern int daemon(int, int);
-#endif
-
-#if defined(__linux__) && (defined(__x86_64__) || defined(__arm__))
-   /* Use 2 MiB alignment so transparent hugepages can be used by KVM.
-      Valgrind does not support alignments larger than 1 MiB,
-      therefore we need special code which handles running on Valgrind. */
-#  define QEMU_VMALLOC_ALIGN (512 * 4096)
-#elif defined(__linux__) && defined(__s390x__)
-   /* Use 1 MiB (segment size) alignment so gmap can be used by KVM. */
-#  define QEMU_VMALLOC_ALIGN (256 * 4096)
-#else
-#  define QEMU_VMALLOC_ALIGN getpagesize()
-#endif
-
-#include <glib/gprintf.h>
-
-#include "config-host.h"
-#include "qemu-common.h"
-#include "sysemu/sysemu.h"
-#include "trace.h"
-#include "qemu/sockets.h"
-#include <sys/mman.h>
-
-#ifdef CONFIG_LINUX
-#include <sys/syscall.h>
-#endif
-
-int qemu_get_thread_id(void)
-{
-#if defined(__linux__)
-    return syscall(SYS_gettid);
-#else
-    return getpid();
-#endif
-}
-
-int qemu_daemon(int nochdir, int noclose)
-{
-    return daemon(nochdir, noclose);
-}
-
-void *qemu_oom_check(void *ptr)
-{
-    if (ptr == NULL) {
-        fprintf(stderr, "Failed to allocate memory: %s\n", strerror(errno));
-        abort();
-    }
-    return ptr;
-}
-
-void *qemu_memalign(size_t alignment, size_t size)
-{
-    void *ptr;
-#if defined(_POSIX_C_SOURCE) && !defined(__sun__) || defined(__APPLE__)
-    int ret;
-    ret = posix_memalign(&ptr, alignment, size);
-    if (ret != 0) {
-        fprintf(stderr, "Failed to allocate %zu B: %s\n",
-                size, strerror(ret));
-        abort();
-    }
-#elif defined(GF_BSD_HOST_OS)
-    ptr = qemu_oom_check(valloc(size));
-#else
-    ptr = qemu_oom_check(memalign(alignment, size));
-#endif
-    trace_qemu_memalign(alignment, size, ptr);
-    return ptr;
-}
-
-/* alloc shared memory pages */
-void *qemu_anon_ram_alloc(size_t size)
-{
-    size_t align = QEMU_VMALLOC_ALIGN;
-    size_t total = size + align - getpagesize();
-    void *ptr = mmap(0, total, PROT_READ | PROT_WRITE,
-                     MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
-    size_t offset = QEMU_ALIGN_UP((uintptr_t)ptr, align) - (uintptr_t)ptr;
-
-    if (ptr == MAP_FAILED) {
-        fprintf(stderr, "Failed to allocate %zu B: %s\n",
-                size, strerror(errno));
-        abort();
-    }
-
-    ptr += offset;
-    total -= offset;
-
-    if (offset > 0) {
-        munmap(ptr - offset, offset);
-    }
-    if (total > size) {
-        munmap(ptr + size, total - size);
-    }
-
-    trace_qemu_anon_ram_alloc(size, ptr);
-    return ptr;
-}
-
-void qemu_vfree(void *ptr)
-{
-    trace_qemu_vfree(ptr);
-    free(ptr);
-}
-
-void qemu_anon_ram_free(void *ptr, size_t size)
-{
-    trace_qemu_anon_ram_free(ptr, size);
-    if (ptr) {
-        munmap(ptr, size);
-    }
-}
-
-void qemu_set_block(int fd)
-{
-    int f;
-    f = fcntl(fd, F_GETFL);
-    fcntl(fd, F_SETFL, f & ~O_NONBLOCK);
-}
-
-void qemu_set_nonblock(int fd)
-{
-    int f;
-    f = fcntl(fd, F_GETFL);
-    fcntl(fd, F_SETFL, f | O_NONBLOCK);
-}
-
-void qemu_set_cloexec(int fd)
-{
-    int f;
-    f = fcntl(fd, F_GETFD);
-    fcntl(fd, F_SETFD, f | FD_CLOEXEC);
-}
-
-/*
- * Creates a pipe with FD_CLOEXEC set on both file descriptors
- */
-int qemu_pipe(int pipefd[2])
-{
-    int ret;
-
-#ifdef CONFIG_PIPE2
-#ifdef pipe2
-    ret = pipe2(pipefd, O_CLOEXEC);
-    if (ret != -1 || errno != ENOSYS) {
-        return ret;
-    }
-#endif
-#endif
-    ret = pipe(pipefd);
-    if (ret == 0) {
-        qemu_set_cloexec(pipefd[0]);
-        qemu_set_cloexec(pipefd[1]);
-    }
-
-    return ret;
-}
-
-int qemu_utimens(const char *path, const struct timespec *times)
-{
-    struct timeval tv[2], tv_now;
-    struct stat st;
-    int i;
-#if defined(CONFIG_UTIMENSAT)
-#ifdef utimensat
-    int ret;
-
-    ret = utimensat(AT_FDCWD, path, times, AT_SYMLINK_NOFOLLOW);
-    if (ret != -1 || errno != ENOSYS) {
-        return ret;
-    }
-#endif
-#endif
-    /* Fallback: use utimes() instead of utimensat() */
-
-    /* happy if special cases */
-    if (times[0].tv_nsec == UTIME_OMIT && times[1].tv_nsec == UTIME_OMIT) {
-        return 0;
-    }
-    if (times[0].tv_nsec == UTIME_NOW && times[1].tv_nsec == UTIME_NOW) {
-        return utimes(path, NULL);
-    }
-
-    /* prepare for hard cases */
-    if (times[0].tv_nsec == UTIME_NOW || times[1].tv_nsec == UTIME_NOW) {
-        gettimeofday(&tv_now, NULL);
-    }
-    if (times[0].tv_nsec == UTIME_OMIT || times[1].tv_nsec == UTIME_OMIT) {
-        stat(path, &st);
-    }
-
-    for (i = 0; i < 2; i++) {
-        if (times[i].tv_nsec == UTIME_NOW) {
-            tv[i].tv_sec = tv_now.tv_sec;
-            tv[i].tv_usec = tv_now.tv_usec;
-        } else if (times[i].tv_nsec == UTIME_OMIT) {
-            tv[i].tv_sec = (i == 0) ? st.st_atime : st.st_mtime;
-            tv[i].tv_usec = 0;
-        } else {
-            tv[i].tv_sec = times[i].tv_sec;
-            tv[i].tv_usec = times[i].tv_nsec / 1000;
-        }
-    }
-
-    return utimes(path, &tv[0]);
-}
-
-char *
-qemu_get_local_state_pathname(const char *relative_pathname)
-{
-    return g_strdup_printf("%s/%s", CONFIG_QEMU_LOCALSTATEDIR,
-                           relative_pathname);
-}
diff --git a/contrib/qemu/util/qemu-error.c b/contrib/qemu/util/qemu-error.c
deleted file mode 100644
index fec02c60754..00000000000
--- a/contrib/qemu/util/qemu-error.c
+++ /dev/null
@@ -1,225 +0,0 @@
-/*
- * Error reporting
- *
- * Copyright (C) 2010 Red Hat Inc.
- *
- * Authors:
- *  Markus Armbruster <armbru@redhat.com>,
- *
- * This work is licensed under the terms of the GNU GPL, version 2 or later.
- * See the COPYING file in the top-level directory.
- */
-
-#include <stdio.h>
-#include "monitor/monitor.h"
-
-/*
- * Print to current monitor if we have one, else to stderr.
- * TODO should return int, so callers can calculate width, but that
- * requires surgery to monitor_vprintf().  Left for another day.
- */
-void error_vprintf(const char *fmt, va_list ap)
-{
-    if (cur_mon) {
-        monitor_vprintf(cur_mon, fmt, ap);
-    } else {
-        vfprintf(stderr, fmt, ap);
-    }
-}
-
-/*
- * Print to current monitor if we have one, else to stderr.
- * TODO just like error_vprintf()
- */
-void error_printf(const char *fmt, ...)
-{
-    va_list ap;
-
-    va_start(ap, fmt);
-    error_vprintf(fmt, ap);
-    va_end(ap);
-}
-
-void error_printf_unless_qmp(const char *fmt, ...)
-{
-    va_list ap;
-
-    if (!monitor_cur_is_qmp()) {
-        va_start(ap, fmt);
-        error_vprintf(fmt, ap);
-        va_end(ap);
-    }
-}
-
-static Location std_loc = {
-    .kind = LOC_NONE
-};
-static Location *cur_loc = &std_loc;
-
-/*
- * Push location saved in LOC onto the location stack, return it.
- * The top of that stack is the current location.
- * Needs a matching loc_pop().
- */
-Location *loc_push_restore(Location *loc)
-{
-    assert(!loc->prev);
-    loc->prev = cur_loc;
-    cur_loc = loc;
-    return loc;
-}
-
-/*
- * Initialize *LOC to "nowhere", push it onto the location stack.
- * The top of that stack is the current location.
- * Needs a matching loc_pop().
- * Return LOC.
- */
-Location *loc_push_none(Location *loc)
-{
-    loc->kind = LOC_NONE;
-    loc->prev = NULL;
-    return loc_push_restore(loc);
-}
-
-/*
- * Pop the location stack.
- * LOC must be the current location, i.e. the top of the stack.
- */
-Location *loc_pop(Location *loc)
-{
-    assert(cur_loc == loc && loc->prev);
-    cur_loc = loc->prev;
-    loc->prev = NULL;
-    return loc;
-}
-
-/*
- * Save the current location in LOC, return LOC.
- */
-Location *loc_save(Location *loc)
-{
-    *loc = *cur_loc;
-    loc->prev = NULL;
-    return loc;
-}
-
-/*
- * Change the current location to the one saved in LOC.
- */
-void loc_restore(Location *loc)
-{
-    Location *prev = cur_loc->prev;
-    assert(!loc->prev);
-    *cur_loc = *loc;
-    cur_loc->prev = prev;
-}
-
-/*
- * Change the current location to "nowhere in particular".
- */
-void loc_set_none(void)
-{
-    cur_loc->kind = LOC_NONE;
-}
-
-/*
- * Change the current location to argument ARGV[IDX..IDX+CNT-1].
- */
-void loc_set_cmdline(char **argv, int idx, int cnt)
-{
-    cur_loc->kind = LOC_CMDLINE;
-    cur_loc->num = cnt;
-    cur_loc->ptr = argv + idx;
-}
-
-/*
- * Change the current location to file FNAME, line LNO.
- */
-void loc_set_file(const char *fname, int lno)
-{
-    assert (fname || cur_loc->kind == LOC_FILE);
-    cur_loc->kind = LOC_FILE;
-    cur_loc->num = lno;
-    if (fname) {
-        cur_loc->ptr = fname;
-    }
-}
-
-static const char *progname;
-
-/*
- * Set the program name for error_print_loc().
- */
-void error_set_progname(const char *argv0)
-{
-    const char *p = strrchr(argv0, '/');
-    progname = p ? p + 1 : argv0;
-}
-
-const char *error_get_progname(void)
-{
-    return progname;
-}
-
-/*
- * Print current location to current monitor if we have one, else to stderr.
- */
-void error_print_loc(void)
-{
-    const char *sep = "";
-    int i;
-    const char *const *argp;
-
-    if (!cur_mon && progname) {
-        fprintf(stderr, "%s:", progname);
-        sep = " ";
-    }
-    switch (cur_loc->kind) {
-    case LOC_CMDLINE:
-        argp = cur_loc->ptr;
-        for (i = 0; i < cur_loc->num; i++) {
-            error_printf("%s%s", sep, argp[i]);
-            sep = " ";
-        }
-        error_printf(": ");
-        break;
-    case LOC_FILE:
-        error_printf("%s:", (const char *)cur_loc->ptr);
-        if (cur_loc->num) {
-            error_printf("%d:", cur_loc->num);
-        }
-        error_printf(" ");
-        break;
-    default:
-        error_printf("%s", sep);
-    }
-}
-
-bool enable_timestamp_msg;
-/*
- * Print an error message to current monitor if we have one, else to stderr.
- * Format arguments like sprintf().  The result should not contain
- * newlines.
- * Prepend the current location and append a newline.
- * It's wrong to call this in a QMP monitor.  Use qerror_report() there.
- */
-void error_report(const char *fmt, ...)
-{
-    va_list ap;
-    GTimeVal tv;
-    gchar *timestr;
-
-    if (enable_timestamp_msg) {
-        g_get_current_time(&tv);
-        timestr = g_time_val_to_iso8601(&tv);
-        error_printf("%s ", timestr);
-        g_free(timestr);
-    }
-
-    error_print_loc();
-    va_start(ap, fmt);
-    error_vprintf(fmt, ap);
-    va_end(ap);
-    error_printf("\n");
-}
diff --git a/contrib/qemu/util/qemu-option.c b/contrib/qemu/util/qemu-option.c
deleted file mode 100644
index e0ef426daa0..00000000000
--- a/contrib/qemu/util/qemu-option.c
+++ /dev/null
@@ -1,1126 +0,0 @@
-/*
- * Commandline option parsing functions
- *
- * Copyright (c) 2003-2008 Fabrice Bellard
- * Copyright (c) 2009 Kevin Wolf <kwolf@redhat.com>
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-
-#include <stdio.h>
-#include <string.h>
-
-#include "qemu-common.h"
-#include "qemu/error-report.h"
-#include "qapi/qmp/types.h"
-#include "qapi/error.h"
-#include "qapi/qmp/qerror.h"
-#include "qemu/option_int.h"
-
-/*
- * Extracts the name of an option from the parameter string (p points at the
- * first byte of the option name)
- *
- * The option name is delimited by delim (usually , or =) or the string end
- * and is copied into buf. If the option name is longer than buf_size, it is
- * truncated. buf is always zero terminated.
- *
- * The return value is the position of the delimiter/zero byte after the option
- * name in p.
- */
-const char *get_opt_name(char *buf, int buf_size, const char *p, char delim)
-{
-    char *q;
-
-    q = buf;
-    while (*p != '\0' && *p != delim) {
-        if (q && (q - buf) < buf_size - 1)
-            *q++ = *p;
-        p++;
-    }
-    if (q)
-        *q = '\0';
-
-    return p;
-}
-
-/*
- * Extracts the value of an option from the parameter string p (p points at the
- * first byte of the option value)
- *
- * This function is comparable to get_opt_name with the difference that the
- * delimiter is fixed to be comma which starts a new option. To specify an
- * option value that contains commas, double each comma.
- */
-const char *get_opt_value(char *buf, int buf_size, const char *p)
-{
-    char *q;
-
-    q = buf;
-    while (*p != '\0') {
-        if (*p == ',') {
-            if (*(p + 1) != ',')
-                break;
-            p++;
-        }
-        if (q && (q - buf) < buf_size - 1)
-            *q++ = *p;
-        p++;
-    }
-    if (q)
-        *q = '\0';
-
-    return p;
-}
-
-int get_next_param_value(char *buf, int buf_size,
-                         const char *tag, const char **pstr)
-{
-    const char *p;
-    char option[128];
-
-    p = *pstr;
-    for(;;) {
-        p = get_opt_name(option, sizeof(option), p, '=');
-        if (*p != '=')
-            break;
-        p++;
-        if (!strcmp(tag, option)) {
-            *pstr = get_opt_value(buf, buf_size, p);
-            if (**pstr == ',') {
-                (*pstr)++;
-            }
-            return strlen(buf);
-        } else {
-            p = get_opt_value(NULL, 0, p);
-        }
-        if (*p != ',')
-            break;
-        p++;
-    }
-    return 0;
-}
-
-int get_param_value(char *buf, int buf_size,
-                    const char *tag, const char *str)
-{
-    return get_next_param_value(buf, buf_size, tag, &str);
-}
-
-/*
- * Searches an option list for an option with the given name
- */
-QEMUOptionParameter *get_option_parameter(QEMUOptionParameter *list,
-    const char *name)
-{
-    while (list && list->name) {
-        if (!strcmp(list->name, name)) {
-            return list;
-        }
-        list++;
-    }
-
-    return NULL;
-}
-
-static void parse_option_bool(const char *name, const char *value, bool *ret,
-                              Error **errp)
-{
-    if (value != NULL) {
-        if (!strcmp(value, "on")) {
-            *ret = 1;
-        } else if (!strcmp(value, "off")) {
-            *ret = 0;
-        } else {
-            error_set(errp,QERR_INVALID_PARAMETER_VALUE, name, "'on' or 'off'");
-        }
-    } else {
-        *ret = 1;
-    }
-}
-
-static void parse_option_number(const char *name, const char *value,
-                                uint64_t *ret, Error **errp)
-{
-    char *postfix;
-    uint64_t number;
-
-    if (value != NULL) {
-        number = strtoull(value, &postfix, 0);
-        if (*postfix != '\0') {
-            error_set(errp, QERR_INVALID_PARAMETER_VALUE, name, "a number");
-            return;
-        }
-        *ret = number;
-    } else {
-        error_set(errp, QERR_INVALID_PARAMETER_VALUE, name, "a number");
-    }
-}
-
-static void parse_option_size(const char *name, const char *value,
-                              uint64_t *ret, Error **errp)
-{
-    char *postfix;
-    double sizef;
-
-    if (value != NULL) {
-        sizef = strtod(value, &postfix);
-        switch (*postfix) {
-        case 'T':
-            sizef *= 1024;
-            /* fall through */
-        case 'G':
-            sizef *= 1024;
-            /* fall through */
-        case 'M':
-            sizef *= 1024;
-            /* fall through */
-        case 'K':
-        case 'k':
-            sizef *= 1024;
-            /* fall through */
-        case 'b':
-        case '\0':
-            *ret = (uint64_t) sizef;
-            break;
-        default:
-            error_set(errp, QERR_INVALID_PARAMETER_VALUE, name, "a size");
-#if 0 /* conversion from qerror_report() to error_set() broke this: */
-            error_printf_unless_qmp("You may use k, M, G or T suffixes for "
-                    "kilobytes, megabytes, gigabytes and terabytes.\n");
-#endif
-            return;
-        }
-    } else {
-        error_set(errp, QERR_INVALID_PARAMETER_VALUE, name, "a size");
-    }
-}
-
-/*
- * Sets the value of a parameter in a given option list. The parsing of the
- * value depends on the type of option:
- *
- * OPT_FLAG (uses value.n):
- *      If no value is given, the flag is set to 1.
- *      Otherwise the value must be "on" (set to 1) or "off" (set to 0)
- *
- * OPT_STRING (uses value.s):
- *      value is strdup()ed and assigned as option value
- *
- * OPT_SIZE (uses value.n):
- *      The value is converted to an integer. Suffixes for kilobytes etc. are
- *      allowed (powers of 1024).
- *
- * Returns 0 on succes, -1 in error cases
- */
-int set_option_parameter(QEMUOptionParameter *list, const char *name,
-    const char *value)
-{
-    bool flag;
-    Error *local_err = NULL;
-
-    // Find a matching parameter
-    list = get_option_parameter(list, name);
-    if (list == NULL) {
-        fprintf(stderr, "Unknown option '%s'\n", name);
-        return -1;
-    }
-
-    // Process parameter
-    switch (list->type) {
-    case OPT_FLAG:
-        parse_option_bool(name, value, &flag, &local_err);
-        if (!error_is_set(&local_err)) {
-            list->value.n = flag;
-        }
-        break;
-
-    case OPT_STRING:
-        if (value != NULL) {
-            list->value.s = g_strdup(value);
-        } else {
-            fprintf(stderr, "Option '%s' needs a parameter\n", name);
-            return -1;
-        }
-        break;
-
-    case OPT_SIZE:
-        parse_option_size(name, value, &list->value.n, &local_err);
-        break;
-
-    default:
-        fprintf(stderr, "Bug: Option '%s' has an unknown type\n", name);
-        return -1;
-    }
-
-    if (error_is_set(&local_err)) {
-        qerror_report_err(local_err);
-        error_free(local_err);
-        return -1;
-    }
-
-    return 0;
-}
-
-/*
- * Sets the given parameter to an integer instead of a string.
- * This function cannot be used to set string options.
- *
- * Returns 0 on success, -1 in error cases
- */
-int set_option_parameter_int(QEMUOptionParameter *list, const char *name,
-    uint64_t value)
-{
-    // Find a matching parameter
-    list = get_option_parameter(list, name);
-    if (list == NULL) {
-        fprintf(stderr, "Unknown option '%s'\n", name);
-        return -1;
-    }
-
-    // Process parameter
-    switch (list->type) {
-    case OPT_FLAG:
-    case OPT_NUMBER:
-    case OPT_SIZE:
-        list->value.n = value;
-        break;
-
-    default:
-        return -1;
-    }
-
-    return 0;
-}
-
-/*
- * Frees a option list. If it contains strings, the strings are freed as well.
- */
-void free_option_parameters(QEMUOptionParameter *list)
-{
-    QEMUOptionParameter *cur = list;
-
-    while (cur && cur->name) {
-        if (cur->type == OPT_STRING) {
-            g_free(cur->value.s);
-        }
-        cur++;
-    }
-
-    g_free(list);
-}
-
-/*
- * Count valid options in list
- */
-static size_t count_option_parameters(QEMUOptionParameter *list)
-{
-    size_t num_options = 0;
-
-    while (list && list->name) {
-        num_options++;
-        list++;
-    }
-
-    return num_options;
-}
-
-/*
- * Append an option list (list) to an option list (dest).
- *
- * If dest is NULL, a new copy of list is created.
- *
- * Returns a pointer to the first element of dest (or the newly allocated copy)
- */
-QEMUOptionParameter *append_option_parameters(QEMUOptionParameter *dest,
-    QEMUOptionParameter *list)
-{
-    size_t num_options, num_dest_options;
-
-    num_options = count_option_parameters(dest);
-    num_dest_options = num_options;
-
-    num_options += count_option_parameters(list);
-
-    dest = g_realloc(dest, (num_options + 1) * sizeof(QEMUOptionParameter));
-    dest[num_dest_options].name = NULL;
-
-    while (list && list->name) {
-        if (get_option_parameter(dest, list->name) == NULL) {
-            dest[num_dest_options++] = *list;
-            dest[num_dest_options].name = NULL;
-        }
-        list++;
-    }
-
-    return dest;
-}
-
-/*
- * Parses a parameter string (param) into an option list (dest).
- *
- * list is the template option list. If dest is NULL, a new copy of list is
- * created. If list is NULL, this function fails.
- *
- * A parameter string consists of one or more parameters, separated by commas.
- * Each parameter consists of its name and possibly of a value. In the latter
- * case, the value is delimited by an = character. To specify a value which
- * contains commas, double each comma so it won't be recognized as the end of
- * the parameter.
- *
- * For more details of the parsing see above.
- *
- * Returns a pointer to the first element of dest (or the newly allocated copy)
- * or NULL in error cases
- */
-QEMUOptionParameter *parse_option_parameters(const char *param,
-    QEMUOptionParameter *list, QEMUOptionParameter *dest)
-{
-    QEMUOptionParameter *allocated = NULL;
-    char name[256];
-    char value[256];
-    char *param_delim, *value_delim;
-    char next_delim;
-
-    if (list == NULL) {
-        return NULL;
-    }
-
-    if (dest == NULL) {
-        dest = allocated = append_option_parameters(NULL, list);
-    }
-
-    while (*param) {
-
-        // Find parameter name and value in the string
-        param_delim = strchr(param, ',');
-        value_delim = strchr(param, '=');
-
-        if (value_delim && (value_delim < param_delim || !param_delim)) {
-            next_delim = '=';
-        } else {
-            next_delim = ',';
-            value_delim = NULL;
-        }
-
-        param = get_opt_name(name, sizeof(name), param, next_delim);
-        if (value_delim) {
-            param = get_opt_value(value, sizeof(value), param + 1);
-        }
-        if (*param != '\0') {
-            param++;
-        }
-
-        // Set the parameter
-        if (set_option_parameter(dest, name, value_delim ? value : NULL)) {
-            goto fail;
-        }
-    }
-
-    return dest;
-
-fail:
-    // Only free the list if it was newly allocated
-    free_option_parameters(allocated);
-    return NULL;
-}
-
-/*
- * Prints all options of a list that have a value to stdout
- */
-void print_option_parameters(QEMUOptionParameter *list)
-{
-    while (list && list->name) {
-        switch (list->type) {
-            case OPT_STRING:
-                 if (list->value.s != NULL) {
-                     printf("%s='%s' ", list->name, list->value.s);
-                 }
-                break;
-            case OPT_FLAG:
-                printf("%s=%s ", list->name, list->value.n ? "on" : "off");
-                break;
-            case OPT_SIZE:
-            case OPT_NUMBER:
-                printf("%s=%" PRId64 " ", list->name, list->value.n);
-                break;
-            default:
-                printf("%s=(unknown type) ", list->name);
-                break;
-        }
-        list++;
-    }
-}
-
-/*
- * Prints an overview of all available options
- */
-void print_option_help(QEMUOptionParameter *list)
-{
-    printf("Supported options:\n");
-    while (list && list->name) {
-        printf("%-16s %s\n", list->name,
-            list->help ? list->help : "No description available");
-        list++;
-    }
-}
-
-/* ------------------------------------------------------------------ */
-
-static QemuOpt *qemu_opt_find(QemuOpts *opts, const char *name)
-{
-    QemuOpt *opt;
-
-    QTAILQ_FOREACH_REVERSE(opt, &opts->head, QemuOptHead, next) {
-        if (strcmp(opt->name, name) != 0)
-            continue;
-        return opt;
-    }
-    return NULL;
-}
-
-const char *qemu_opt_get(QemuOpts *opts, const char *name)
-{
-    QemuOpt *opt = qemu_opt_find(opts, name);
-    return opt ? opt->str : NULL;
-}
-
-bool qemu_opt_has_help_opt(QemuOpts *opts)
-{
-    QemuOpt *opt;
-
-    QTAILQ_FOREACH_REVERSE(opt, &opts->head, QemuOptHead, next) {
-        if (is_help_option(opt->name)) {
-            return true;
-        }
-    }
-    return false;
-}
-
-bool qemu_opt_get_bool(QemuOpts *opts, const char *name, bool defval)
-{
-    QemuOpt *opt = qemu_opt_find(opts, name);
-
-    if (opt == NULL)
-        return defval;
-    assert(opt->desc && opt->desc->type == QEMU_OPT_BOOL);
-    return opt->value.boolean;
-}
-
-uint64_t qemu_opt_get_number(QemuOpts *opts, const char *name, uint64_t defval)
-{
-    QemuOpt *opt = qemu_opt_find(opts, name);
-
-    if (opt == NULL)
-        return defval;
-    assert(opt->desc && opt->desc->type == QEMU_OPT_NUMBER);
-    return opt->value.uint;
-}
-
-uint64_t qemu_opt_get_size(QemuOpts *opts, const char *name, uint64_t defval)
-{
-    QemuOpt *opt = qemu_opt_find(opts, name);
-
-    if (opt == NULL)
-        return defval;
-    assert(opt->desc && opt->desc->type == QEMU_OPT_SIZE);
-    return opt->value.uint;
-}
-
-static void qemu_opt_parse(QemuOpt *opt, Error **errp)
-{
-    if (opt->desc == NULL)
-        return;
-
-    switch (opt->desc->type) {
-    case QEMU_OPT_STRING:
-        /* nothing */
-        return;
-    case QEMU_OPT_BOOL:
-        parse_option_bool(opt->name, opt->str, &opt->value.boolean, errp);
-        break;
-    case QEMU_OPT_NUMBER:
-        parse_option_number(opt->name, opt->str, &opt->value.uint, errp);
-        break;
-    case QEMU_OPT_SIZE:
-        parse_option_size(opt->name, opt->str, &opt->value.uint, errp);
-        break;
-    default:
-        abort();
-    }
-}
-
-static void qemu_opt_del(QemuOpt *opt)
-{
-    QTAILQ_REMOVE(&opt->opts->head, opt, next);
-    g_free((/* !const */ char*)opt->name);
-    g_free((/* !const */ char*)opt->str);
-    g_free(opt);
-}
-
-static bool opts_accepts_any(const QemuOpts *opts)
-{
-    return opts->list->desc[0].name == NULL;
-}
-
-static const QemuOptDesc *find_desc_by_name(const QemuOptDesc *desc,
-                                            const char *name)
-{
-    int i;
-
-    for (i = 0; desc[i].name != NULL; i++) {
-        if (strcmp(desc[i].name, name) == 0) {
-            return &desc[i];
-        }
-    }
-
-    return NULL;
-}
-
-static void opt_set(QemuOpts *opts, const char *name, const char *value,
-                    bool prepend, Error **errp)
-{
-    QemuOpt *opt;
-    const QemuOptDesc *desc;
-    Error *local_err = NULL;
-
-    desc = find_desc_by_name(opts->list->desc, name);
-    if (!desc && !opts_accepts_any(opts)) {
-        error_set(errp, QERR_INVALID_PARAMETER, name);
-        return;
-    }
-
-    opt = g_malloc0(sizeof(*opt));
-    opt->name = g_strdup(name);
-    opt->opts = opts;
-    if (prepend) {
-        QTAILQ_INSERT_HEAD(&opts->head, opt, next);
-    } else {
-        QTAILQ_INSERT_TAIL(&opts->head, opt, next);
-    }
-    opt->desc = desc;
-    opt->str = g_strdup(value);
-    qemu_opt_parse(opt, &local_err);
-    if (error_is_set(&local_err)) {
-        error_propagate(errp, local_err);
-        qemu_opt_del(opt);
-    }
-}
-
-int qemu_opt_set(QemuOpts *opts, const char *name, const char *value)
-{
-    Error *local_err = NULL;
-
-    opt_set(opts, name, value, false, &local_err);
-    if (error_is_set(&local_err)) {
-        qerror_report_err(local_err);
-        error_free(local_err);
-        return -1;
-    }
-
-    return 0;
-}
-
-void qemu_opt_set_err(QemuOpts *opts, const char *name, const char *value,
-                      Error **errp)
-{
-    opt_set(opts, name, value, false, errp);
-}
-
-int qemu_opt_set_bool(QemuOpts *opts, const char *name, bool val)
-{
-    QemuOpt *opt;
-    const QemuOptDesc *desc = opts->list->desc;
-
-    opt = g_malloc0(sizeof(*opt));
-    opt->desc = find_desc_by_name(desc, name);
-    if (!opt->desc && !opts_accepts_any(opts)) {
-        qerror_report(QERR_INVALID_PARAMETER, name);
-        g_free(opt);
-        return -1;
-    }
-
-    opt->name = g_strdup(name);
-    opt->opts = opts;
-    opt->value.boolean = !!val;
-    opt->str = g_strdup(val ? "on" : "off");
-    QTAILQ_INSERT_TAIL(&opts->head, opt, next);
-
-    return 0;
-}
-
-int qemu_opt_set_number(QemuOpts *opts, const char *name, int64_t val)
-{
-    QemuOpt *opt;
-    const QemuOptDesc *desc = opts->list->desc;
-
-    opt = g_malloc0(sizeof(*opt));
-    opt->desc = find_desc_by_name(desc, name);
-    if (!opt->desc && !opts_accepts_any(opts)) {
-        qerror_report(QERR_INVALID_PARAMETER, name);
-        g_free(opt);
-        return -1;
-    }
-
-    opt->name = g_strdup(name);
-    opt->opts = opts;
-    opt->value.uint = val;
-    opt->str = g_strdup_printf("%" PRId64, val);
-    QTAILQ_INSERT_TAIL(&opts->head, opt, next);
-
-    return 0;
-}
-
-int qemu_opt_foreach(QemuOpts *opts, qemu_opt_loopfunc func, void *opaque,
-                     int abort_on_failure)
-{
-    QemuOpt *opt;
-    int rc = 0;
-
-    QTAILQ_FOREACH(opt, &opts->head, next) {
-        rc = func(opt->name, opt->str, opaque);
-        if (abort_on_failure  &&  rc != 0)
-            break;
-    }
-    return rc;
-}
-
-QemuOpts *qemu_opts_find(QemuOptsList *list, const char *id)
-{
-    QemuOpts *opts;
-
-    QTAILQ_FOREACH(opts, &list->head, next) {
-        if (!opts->id && !id) {
-            return opts;
-        }
-        if (opts->id && id && !strcmp(opts->id, id)) {
-            return opts;
-        }
-    }
-    return NULL;
-}
-
-static int id_wellformed(const char *id)
-{
-    int i;
-
-    if (!qemu_isalpha(id[0])) {
-        return 0;
-    }
-    for (i = 1; id[i]; i++) {
-        if (!qemu_isalnum(id[i]) && !strchr("-._", id[i])) {
-            return 0;
-        }
-    }
-    return 1;
-}
-
-QemuOpts *qemu_opts_create(QemuOptsList *list, const char *id,
-                           int fail_if_exists, Error **errp)
-{
-    QemuOpts *opts = NULL;
-
-    if (id) {
-        if (!id_wellformed(id)) {
-            error_set(errp,QERR_INVALID_PARAMETER_VALUE, "id", "an identifier");
-#if 0 /* conversion from qerror_report() to error_set() broke this: */
-            error_printf_unless_qmp("Identifiers consist of letters, digits, '-', '.', '_', starting with a letter.\n");
-#endif
-            return NULL;
-        }
-        opts = qemu_opts_find(list, id);
-        if (opts != NULL) {
-            if (fail_if_exists && !list->merge_lists) {
-                error_set(errp, QERR_DUPLICATE_ID, id, list->name);
-                return NULL;
-            } else {
-                return opts;
-            }
-        }
-    } else if (list->merge_lists) {
-        opts = qemu_opts_find(list, NULL);
-        if (opts) {
-            return opts;
-        }
-    }
-    opts = g_malloc0(sizeof(*opts));
-    opts->id = g_strdup(id);
-    opts->list = list;
-    loc_save(&opts->loc);
-    QTAILQ_INIT(&opts->head);
-    QTAILQ_INSERT_TAIL(&list->head, opts, next);
-    return opts;
-}
-
-QemuOpts *qemu_opts_create_nofail(QemuOptsList *list)
-{
-    QemuOpts *opts;
-    Error *errp = NULL;
-    opts = qemu_opts_create(list, NULL, 0, &errp);
-    assert_no_error(errp);
-    return opts;
-}
-
-void qemu_opts_reset(QemuOptsList *list)
-{
-    QemuOpts *opts, *next_opts;
-
-    QTAILQ_FOREACH_SAFE(opts, &list->head, next, next_opts) {
-        qemu_opts_del(opts);
-    }
-}
-
-void qemu_opts_loc_restore(QemuOpts *opts)
-{
-    loc_restore(&opts->loc);
-}
-
-int qemu_opts_set(QemuOptsList *list, const char *id,
-                  const char *name, const char *value)
-{
-    QemuOpts *opts;
-    Error *local_err = NULL;
-
-    opts = qemu_opts_create(list, id, 1, &local_err);
-    if (error_is_set(&local_err)) {
-        qerror_report_err(local_err);
-        error_free(local_err);
-        return -1;
-    }
-    return qemu_opt_set(opts, name, value);
-}
-
-const char *qemu_opts_id(QemuOpts *opts)
-{
-    return opts->id;
-}
-
-void qemu_opts_del(QemuOpts *opts)
-{
-    QemuOpt *opt;
-
-    for (;;) {
-        opt = QTAILQ_FIRST(&opts->head);
-        if (opt == NULL)
-            break;
-        qemu_opt_del(opt);
-    }
-    QTAILQ_REMOVE(&opts->list->head, opts, next);
-    g_free(opts->id);
-    g_free(opts);
-}
-
-int qemu_opts_print(QemuOpts *opts, void *dummy)
-{
-    QemuOpt *opt;
-
-    fprintf(stderr, "%s: %s:", opts->list->name,
-            opts->id ? opts->id : "<noid>");
-    QTAILQ_FOREACH(opt, &opts->head, next) {
-        fprintf(stderr, " %s=\"%s\"", opt->name, opt->str);
-    }
-    fprintf(stderr, "\n");
-    return 0;
-}
-
-static int opts_do_parse(QemuOpts *opts, const char *params,
-                         const char *firstname, bool prepend)
-{
-    char option[128], value[1024];
-    const char *p,*pe,*pc;
-    Error *local_err = NULL;
-
-    for (p = params; *p != '\0'; p++) {
-        pe = strchr(p, '=');
-        pc = strchr(p, ',');
-        if (!pe || (pc && pc < pe)) {
-            /* found "foo,more" */
-            if (p == params && firstname) {
-                /* implicitly named first option */
-                pstrcpy(option, sizeof(option), firstname);
-                p = get_opt_value(value, sizeof(value), p);
-            } else {
-                /* option without value, probably a flag */
-                p = get_opt_name(option, sizeof(option), p, ',');
-                if (strncmp(option, "no", 2) == 0) {
-                    memmove(option, option+2, strlen(option+2)+1);
-                    pstrcpy(value, sizeof(value), "off");
-                } else {
-                    pstrcpy(value, sizeof(value), "on");
-                }
-            }
-        } else {
-            /* found "foo=bar,more" */
-            p = get_opt_name(option, sizeof(option), p, '=');
-            if (*p != '=') {
-                break;
-            }
-            p++;
-            p = get_opt_value(value, sizeof(value), p);
-        }
-        if (strcmp(option, "id") != 0) {
-            /* store and parse */
-            opt_set(opts, option, value, prepend, &local_err);
-            if (error_is_set(&local_err)) {
-                qerror_report_err(local_err);
-                error_free(local_err);
-                return -1;
-            }
-        }
-        if (*p != ',') {
-            break;
-        }
-    }
-    return 0;
-}
-
-int qemu_opts_do_parse(QemuOpts *opts, const char *params, const char *firstname)
-{
-    return opts_do_parse(opts, params, firstname, false);
-}
-
-static QemuOpts *opts_parse(QemuOptsList *list, const char *params,
-                            int permit_abbrev, bool defaults)
-{
-    const char *firstname;
-    char value[1024], *id = NULL;
-    const char *p;
-    QemuOpts *opts;
-    Error *local_err = NULL;
-
-    assert(!permit_abbrev || list->implied_opt_name);
-    firstname = permit_abbrev ? list->implied_opt_name : NULL;
-
-    if (strncmp(params, "id=", 3) == 0) {
-        get_opt_value(value, sizeof(value), params+3);
-        id = value;
-    } else if ((p = strstr(params, ",id=")) != NULL) {
-        get_opt_value(value, sizeof(value), p+4);
-        id = value;
-    }
-    opts = qemu_opts_create(list, id, !defaults, &local_err);
-    if (opts == NULL) {
-        if (error_is_set(&local_err)) {
-            qerror_report_err(local_err);
-            error_free(local_err);
-        }
-        return NULL;
-    }
-
-    if (opts_do_parse(opts, params, firstname, defaults) != 0) {
-        qemu_opts_del(opts);
-        return NULL;
-    }
-
-    return opts;
-}
-
-QemuOpts *qemu_opts_parse(QemuOptsList *list, const char *params,
-                          int permit_abbrev)
-{
-    return opts_parse(list, params, permit_abbrev, false);
-}
-
-void qemu_opts_set_defaults(QemuOptsList *list, const char *params,
-                            int permit_abbrev)
-{
-    QemuOpts *opts;
-
-    opts = opts_parse(list, params, permit_abbrev, true);
-    assert(opts);
-}
-
-typedef struct OptsFromQDictState {
-    QemuOpts *opts;
-    Error **errp;
-} OptsFromQDictState;
-
-static void qemu_opts_from_qdict_1(const char *key, QObject *obj, void *opaque)
-{
-    OptsFromQDictState *state = opaque;
-    char buf[32];
-    const char *value;
-    int n;
-
-    if (!strcmp(key, "id") || error_is_set(state->errp)) {
-        return;
-    }
-
-    switch (qobject_type(obj)) {
-    case QTYPE_QSTRING:
-        value = qstring_get_str(qobject_to_qstring(obj));
-        break;
-    case QTYPE_QINT:
-        n = snprintf(buf, sizeof(buf), "%" PRId64,
-                     qint_get_int(qobject_to_qint(obj)));
-        assert(n < sizeof(buf));
-        value = buf;
-        break;
-    case QTYPE_QFLOAT:
-        n = snprintf(buf, sizeof(buf), "%.17g",
-                     qfloat_get_double(qobject_to_qfloat(obj)));
-        assert(n < sizeof(buf));
-        value = buf;
-        break;
-    case QTYPE_QBOOL:
-        pstrcpy(buf, sizeof(buf),
-                qbool_get_int(qobject_to_qbool(obj)) ? "on" : "off");
-        value = buf;
-        break;
-    default:
-        return;
-    }
-
-    qemu_opt_set_err(state->opts, key, value, state->errp);
-}
-
-/*
- * Create QemuOpts from a QDict.
- * Use value of key "id" as ID if it exists and is a QString.
- * Only QStrings, QInts, QFloats and QBools are copied.  Entries with
- * other types are silently ignored.
- */
-QemuOpts *qemu_opts_from_qdict(QemuOptsList *list, const QDict *qdict,
-                               Error **errp)
-{
-    OptsFromQDictState state;
-    Error *local_err = NULL;
-    QemuOpts *opts;
-
-    opts = qemu_opts_create(list, qdict_get_try_str(qdict, "id"), 1,
-                            &local_err);
-    if (error_is_set(&local_err)) {
-        error_propagate(errp, local_err);
-        return NULL;
-    }
-
-    assert(opts != NULL);
-
-    state.errp = &local_err;
-    state.opts = opts;
-    qdict_iter(qdict, qemu_opts_from_qdict_1, &state);
-    if (error_is_set(&local_err)) {
-        error_propagate(errp, local_err);
-        qemu_opts_del(opts);
-        return NULL;
-    }
-
-    return opts;
-}
-
-/*
- * Adds all QDict entries to the QemuOpts that can be added and removes them
- * from the QDict. When this function returns, the QDict contains only those
- * entries that couldn't be added to the QemuOpts.
- */
-void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp)
-{
-    const QDictEntry *entry, *next;
-
-    entry = qdict_first(qdict);
-
-    while (entry != NULL) {
-        Error *local_err = NULL;
-        OptsFromQDictState state = {
-            .errp = &local_err,
-            .opts = opts,
-        };
-
-        next = qdict_next(qdict, entry);
-
-        if (find_desc_by_name(opts->list->desc, entry->key)) {
-            qemu_opts_from_qdict_1(entry->key, entry->value, &state);
-            if (error_is_set(&local_err)) {
-                error_propagate(errp, local_err);
-                return;
-            } else {
-                qdict_del(qdict, entry->key);
-            }
-        }
-
-        entry = next;
-    }
-}
-
-/*
- * Convert from QemuOpts to QDict.
- * The QDict values are of type QString.
- * TODO We'll want to use types appropriate for opt->desc->type, but
- * this is enough for now.
- */
-QDict *qemu_opts_to_qdict(QemuOpts *opts, QDict *qdict)
-{
-    QemuOpt *opt;
-    QObject *val;
-
-    if (!qdict) {
-        qdict = qdict_new();
-    }
-    if (opts->id) {
-        qdict_put(qdict, "id", qstring_from_str(opts->id));
-    }
-    QTAILQ_FOREACH(opt, &opts->head, next) {
-        val = QOBJECT(qstring_from_str(opt->str));
-        qdict_put_obj(qdict, opt->name, val);
-    }
-    return qdict;
-}
-
-/* Validate parsed opts against descriptions where no
- * descriptions were provided in the QemuOptsList.
- */
-void qemu_opts_validate(QemuOpts *opts, const QemuOptDesc *desc, Error **errp)
-{
-    QemuOpt *opt;
-    Error *local_err = NULL;
-
-    assert(opts_accepts_any(opts));
-
-    QTAILQ_FOREACH(opt, &opts->head, next) {
-        opt->desc = find_desc_by_name(desc, opt->name);
-        if (!opt->desc) {
-            error_set(errp, QERR_INVALID_PARAMETER, opt->name);
-            return;
-        }
-
-        qemu_opt_parse(opt, &local_err);
-        if (error_is_set(&local_err)) {
-            error_propagate(errp, local_err);
-            return;
-        }
-    }
-}
-
-int qemu_opts_foreach(QemuOptsList *list, qemu_opts_loopfunc func, void *opaque,
-                      int abort_on_failure)
-{
-    Location loc;
-    QemuOpts *opts;
-    int rc = 0;
-
-    loc_push_none(&loc);
-    QTAILQ_FOREACH(opts, &list->head, next) {
-        loc_restore(&opts->loc);
-        rc |= func(opts, opaque);
-        if (abort_on_failure  &&  rc != 0)
-            break;
-    }
-    loc_pop(&loc);
-    return rc;
-}
diff --git a/contrib/qemu/util/qemu-thread-posix.c b/contrib/qemu/util/qemu-thread-posix.c
deleted file mode 100644
index 4489abf1d85..00000000000
--- a/contrib/qemu/util/qemu-thread-posix.c
+++ /dev/null
@@ -1,327 +0,0 @@
-/*
- * Wrappers around mutex/cond/thread functions
- *
- * Copyright Red Hat, Inc. 2009
- *
- * Author:
- *  Marcelo Tosatti <mtosatti@redhat.com>
- *
- * This work is licensed under the terms of the GNU GPL, version 2 or later.
- * See the COPYING file in the top-level directory.
- *
- */
-#include <stdlib.h>
-#include <stdio.h>
-#include <errno.h>
-#include <time.h>
-#include <signal.h>
-#include <stdint.h>
-#include <string.h>
-#include <limits.h>
-#include <unistd.h>
-#include <sys/time.h>
-#include "qemu/thread.h"
-
-static void error_exit(int err, const char *msg)
-{
-    fprintf(stderr, "qemu: %s: %s\n", msg, strerror(err));
-    abort();
-}
-
-void qemu_mutex_init(QemuMutex *mutex)
-{
-    int err;
-    pthread_mutexattr_t mutexattr;
-
-    pthread_mutexattr_init(&mutexattr);
-    pthread_mutexattr_settype(&mutexattr, PTHREAD_MUTEX_ERRORCHECK);
-    err = pthread_mutex_init(&mutex->lock, &mutexattr);
-    pthread_mutexattr_destroy(&mutexattr);
-    if (err)
-        error_exit(err, __func__);
-}
-
-void qemu_mutex_destroy(QemuMutex *mutex)
-{
-    int err;
-
-    err = pthread_mutex_destroy(&mutex->lock);
-    if (err)
-        error_exit(err, __func__);
-}
-
-void qemu_mutex_lock(QemuMutex *mutex)
-{
-    int err;
-
-    err = pthread_mutex_lock(&mutex->lock);
-    if (err)
-        error_exit(err, __func__);
-}
-
-int qemu_mutex_trylock(QemuMutex *mutex)
-{
-    return pthread_mutex_trylock(&mutex->lock);
-}
-
-void qemu_mutex_unlock(QemuMutex *mutex)
-{
-    int err;
-
-    err = pthread_mutex_unlock(&mutex->lock);
-    if (err)
-        error_exit(err, __func__);
-}
-
-void qemu_cond_init(QemuCond *cond)
-{
-    int err;
-
-    err = pthread_cond_init(&cond->cond, NULL);
-    if (err)
-        error_exit(err, __func__);
-}
-
-void qemu_cond_destroy(QemuCond *cond)
-{
-    int err;
-
-    err = pthread_cond_destroy(&cond->cond);
-    if (err)
-        error_exit(err, __func__);
-}
-
-void qemu_cond_signal(QemuCond *cond)
-{
-    int err;
-
-    err = pthread_cond_signal(&cond->cond);
-    if (err)
-        error_exit(err, __func__);
-}
-
-void qemu_cond_broadcast(QemuCond *cond)
-{
-    int err;
-
-    err = pthread_cond_broadcast(&cond->cond);
-    if (err)
-        error_exit(err, __func__);
-}
-
-void qemu_cond_wait(QemuCond *cond, QemuMutex *mutex)
-{
-    int err;
-
-    err = pthread_cond_wait(&cond->cond, &mutex->lock);
-    if (err)
-        error_exit(err, __func__);
-}
-
-void qemu_sem_init(QemuSemaphore *sem, int init)
-{
-    int rc;
-
-#if defined(__APPLE__) || defined(__NetBSD__)
-    rc = pthread_mutex_init(&sem->lock, NULL);
-    if (rc != 0) {
-        error_exit(rc, __func__);
-    }
-    rc = pthread_cond_init(&sem->cond, NULL);
-    if (rc != 0) {
-        error_exit(rc, __func__);
-    }
-    if (init < 0) {
-        error_exit(EINVAL, __func__);
-    }
-    sem->count = init;
-#else
-    rc = sem_init(&sem->sem, 0, init);
-    if (rc < 0) {
-        error_exit(errno, __func__);
-    }
-#endif
-}
-
-void qemu_sem_destroy(QemuSemaphore *sem)
-{
-    int rc;
-
-#if defined(__APPLE__) || defined(__NetBSD__)
-    rc = pthread_cond_destroy(&sem->cond);
-    if (rc < 0) {
-        error_exit(rc, __func__);
-    }
-    rc = pthread_mutex_destroy(&sem->lock);
-    if (rc < 0) {
-        error_exit(rc, __func__);
-    }
-#else
-    rc = sem_destroy(&sem->sem);
-    if (rc < 0) {
-        error_exit(errno, __func__);
-    }
-#endif
-}
-
-void qemu_sem_post(QemuSemaphore *sem)
-{
-    int rc;
-
-#if defined(__APPLE__) || defined(__NetBSD__)
-    pthread_mutex_lock(&sem->lock);
-    if (sem->count == INT_MAX) {
-        rc = EINVAL;
-    } else if (sem->count++ < 0) {
-        rc = pthread_cond_signal(&sem->cond);
-    } else {
-        rc = 0;
-    }
-    pthread_mutex_unlock(&sem->lock);
-    if (rc != 0) {
-        error_exit(rc, __func__);
-    }
-#else
-    rc = sem_post(&sem->sem);
-    if (rc < 0) {
-        error_exit(errno, __func__);
-    }
-#endif
-}
-
-static void compute_abs_deadline(struct timespec *ts, int ms)
-{
-    struct timeval tv;
-    gettimeofday(&tv, NULL);
-    ts->tv_nsec = tv.tv_usec * 1000 + (ms % 1000) * 1000000;
-    ts->tv_sec = tv.tv_sec + ms / 1000;
-    if (ts->tv_nsec >= 1000000000) {
-        ts->tv_sec++;
-        ts->tv_nsec -= 1000000000;
-    }
-}
-
-int qemu_sem_timedwait(QemuSemaphore *sem, int ms)
-{
-    int rc;
-    struct timespec ts;
-
-#if defined(__APPLE__) || defined(__NetBSD__)
-    compute_abs_deadline(&ts, ms);
-    pthread_mutex_lock(&sem->lock);
-    --sem->count;
-    while (sem->count < 0) {
-        rc = pthread_cond_timedwait(&sem->cond, &sem->lock, &ts);
-        if (rc == ETIMEDOUT) {
-            ++sem->count;
-            break;
-        }
-        if (rc != 0) {
-            error_exit(rc, __func__);
-        }
-    }
-    pthread_mutex_unlock(&sem->lock);
-    return (rc == ETIMEDOUT ? -1 : 0);
-#else
-    if (ms <= 0) {
-        /* This is cheaper than sem_timedwait.  */
-        do {
-            rc = sem_trywait(&sem->sem);
-        } while (rc == -1 && errno == EINTR);
-        if (rc == -1 && errno == EAGAIN) {
-            return -1;
-        }
-    } else {
-        compute_abs_deadline(&ts, ms);
-        do {
-            rc = sem_timedwait(&sem->sem, &ts);
-        } while (rc == -1 && errno == EINTR);
-        if (rc == -1 && errno == ETIMEDOUT) {
-            return -1;
-        }
-    }
-    if (rc < 0) {
-        error_exit(errno, __func__);
-    }
-    return 0;
-#endif
-}
-
-void qemu_sem_wait(QemuSemaphore *sem)
-{
-#if defined(__APPLE__) || defined(__NetBSD__)
-    pthread_mutex_lock(&sem->lock);
-    --sem->count;
-    while (sem->count < 0) {
-        pthread_cond_wait(&sem->cond, &sem->lock);
-    }
-    pthread_mutex_unlock(&sem->lock);
-#else
-    int rc;
-
-    do {
-        rc = sem_wait(&sem->sem);
-    } while (rc == -1 && errno == EINTR);
-    if (rc < 0) {
-        error_exit(errno, __func__);
-    }
-#endif
-}
-
-void qemu_thread_create(QemuThread *thread,
-                       void *(*start_routine)(void*),
-                       void *arg, int mode)
-{
-    sigset_t set, oldset;
-    int err;
-    pthread_attr_t attr;
-
-    err = pthread_attr_init(&attr);
-    if (err) {
-        error_exit(err, __func__);
-    }
-    if (mode == QEMU_THREAD_DETACHED) {
-        err = pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);
-        if (err) {
-            error_exit(err, __func__);
-        }
-    }
-
-    /* Leave signal handling to the iothread.  */
-    sigfillset(&set);
-    pthread_sigmask(SIG_SETMASK, &set, &oldset);
-    err = pthread_create(&thread->thread, &attr, start_routine, arg);
-    if (err)
-        error_exit(err, __func__);
-
-    pthread_sigmask(SIG_SETMASK, &oldset, NULL);
-
-    pthread_attr_destroy(&attr);
-}
-
-void qemu_thread_get_self(QemuThread *thread)
-{
-    thread->thread = pthread_self();
-}
-
-bool qemu_thread_is_self(QemuThread *thread)
-{
-   return pthread_equal(pthread_self(), thread->thread);
-}
-
-void qemu_thread_exit(void *retval)
-{
-    pthread_exit(retval);
-}
-
-void *qemu_thread_join(QemuThread *thread)
-{
-    int err;
-    void *ret;
-
-    err = pthread_join(thread->thread, &ret);
-    if (err) {
-        error_exit(err, __func__);
-    }
-    return ret;
-}
diff --git a/contrib/qemu/util/unicode.c b/contrib/qemu/util/unicode.c
deleted file mode 100644
index d1c86588505..00000000000
--- a/contrib/qemu/util/unicode.c
+++ /dev/null
@@ -1,100 +0,0 @@
-/*
- * Dealing with Unicode
- *
- * Copyright (C) 2013 Red Hat, Inc.
- *
- * Authors:
- *  Markus Armbruster <armbru@redhat.com>
- *
- * This work is licensed under the terms of the GNU GPL, version 2 or
- * later.  See the COPYING file in the top-level directory.
- */
-
-#include "qemu-common.h"
-
-/**
- * mod_utf8_codepoint:
- * @s: string encoded in modified UTF-8
- * @n: maximum number of bytes to read from @s, if less than 6
- * @end: set to end of sequence on return
- *
- * Convert the modified UTF-8 sequence at the start of @s.  Modified
- * UTF-8 is exactly like UTF-8, except U+0000 is encoded as
- * "\xC0\x80".
- *
- * If @n is zero or @s points to a zero byte, the sequence is invalid,
- * and @end is set to @s.
- *
- * If @s points to an impossible byte (0xFE or 0xFF) or a continuation
- * byte, the sequence is invalid, and @end is set to @s + 1
- *
- * Else, the first byte determines how many continuation bytes are
- * expected.  If there are fewer, the sequence is invalid, and @end is
- * set to @s + 1 + actual number of continuation bytes.  Else, the
- * sequence is well-formed, and @end is set to @s + 1 + expected
- * number of continuation bytes.
- *
- * A well-formed sequence is valid unless it encodes a codepoint
- * outside the Unicode range U+0000..U+10FFFF, one of Unicode's 66
- * noncharacters, a surrogate codepoint, or is overlong.  Except the
- * overlong sequence "\xC0\x80" is valid.
- *
- * Conversion succeeds if and only if the sequence is valid.
- *
- * Returns: the Unicode codepoint on success, -1 on failure.
- */
-int mod_utf8_codepoint(const char *s, size_t n, char **end)
-{
-    static int min_cp[5] = { 0x80, 0x800, 0x10000, 0x200000, 0x4000000 };
-    const unsigned char *p;
-    unsigned byte, mask, len, i;
-    int cp;
-
-    if (n == 0 || *s == 0) {
-        /* empty sequence */
-        *end = (char *)s;
-        return -1;
-    }
-
-    p = (const unsigned char *)s;
-    byte = *p++;
-    if (byte < 0x80) {
-        cp = byte;              /* one byte sequence */
-    } else if (byte >= 0xFE) {
-        cp = -1;                /* impossible bytes 0xFE, 0xFF */
-    } else if ((byte & 0x40) == 0) {
-        cp = -1;                /* unexpected continuation byte */
-    } else {
-        /* multi-byte sequence */
-        len = 0;
-        for (mask = 0x80; byte & mask; mask >>= 1) {
-            len++;
-        }
-        assert(len > 1 && len < 7);
-        cp = byte & (mask - 1);
-        for (i = 1; i < len; i++) {
-            byte = i < n ? *p : 0;
-            if ((byte & 0xC0) != 0x80) {
-                cp = -1;        /* continuation byte missing */
-                goto out;
-            }
-            p++;
-            cp <<= 6;
-            cp |= byte & 0x3F;
-        }
-        if (cp > 0x10FFFF) {
-            cp = -1;            /* beyond Unicode range */
-        } else if ((cp >= 0xFDD0 && cp <= 0xFDEF)
-                   || (cp & 0xFFFE) == 0xFFFE) {
-            cp = -1;            /* noncharacter */
-        } else if (cp >= 0xD800 && cp <= 0xDFFF) {
-            cp = -1;            /* surrogate code point */
-        } else if (cp < min_cp[len - 2] && !(cp == 0 && len == 2)) {
-            cp = -1;            /* overlong, not \xC0\x80 */
-        }
-    }
-
-out:
-    *end = (char *)p;
-    return cp;
-}
diff --git a/contrib/stdlib/gf_mkostemp.c b/contrib/stdlib/gf_mkostemp.c
deleted file mode 100644
index 931249a4520..00000000000
--- a/contrib/stdlib/gf_mkostemp.c
+++ /dev/null
@@ -1,107 +0,0 @@
-/* Borrowed from glibc-2.16/sysdeps/posix/tempname.c */
-
-/* Copyright (C) 1991-2001, 2006, 2007, 2009 Free Software Foundation, Inc.
-   This file is part of the GNU C Library.
-
-   The GNU C Library is free software; you can redistribute it and/or
-   modify it under the terms of the GNU Lesser General Public
-   License as published by the Free Software Foundation; either
-   version 2.1 of the License, or (at your option) any later version.
-
-   The GNU C Library is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-   Lesser General Public License for more details.
-
-   You should have received a copy of the GNU Lesser General Public
-   License along with the GNU C Library; if not, see
-   <http://www.gnu.org/licenses/>.  */
-
-#include <stdlib.h>
-#include <string.h>
-#include <sys/stat.h>
-#include <unistd.h>
-#include <errno.h>
-#include <sys/time.h>
-#include <fcntl.h>
-#include <sys/types.h>
-#include <time.h>
-#include <inttypes.h>
-
-static const char letters[] =
-"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
-
-/* Generate a temporary file name based on TMPL.  TMPL must match the
-   rules for mk[s]temp (i.e. end in "XXXXXX", possibly with a suffix).
-*/
-
-#if !defined(TMP_MAX)
-#define TMP_MAX 238328
-#endif
-
-int
-gf_mkostemp (char *tmpl, int suffixlen, int flags)
-{
-        int len;
-        char *XXXXXX;
-        static uint64_t value;
-        uint64_t random_time_bits;
-        unsigned int count;
-        int fd = -1;
-
-  /* A lower bound on the number of temporary files to attempt to
-     generate.  The maximum total number of temporary file names that
-     can exist for a given template is 62**6.  It should never be
-     necessary to try all these combinations.  Instead if a reasonable
-     number of names is tried (we define reasonable as 62**3) fail to
-     give the system administrator the chance to remove the problems.  */
-
-        unsigned int attempts = TMP_MAX; /* TMP_MAX == 62³ */
-
-        len = strlen (tmpl);
-        if (len < 6 + suffixlen || memcmp (&tmpl[len - 6 - suffixlen],
-                                           "XXXXXX", 6))
-                return -1;
-
-  /* This is where the Xs start.  */
-        XXXXXX = &tmpl[len - 6 - suffixlen];
-
-  /* Get some more or less random data.  */
-# if HAVE_GETTIMEOFDAY
-        struct timeval tv;
-        gettimeofday (&tv, NULL);
-        random_time_bits = ((uint64_t) tv.tv_usec << 16) ^ tv.tv_sec;
-# else
-        random_time_bits = time (NULL);
-# endif
-
-        value += random_time_bits ^ getpid ();
-
-        for (count = 0; count < attempts; value += 7777, ++count) {
-                uint64_t v = value;
-
-                /* Fill in the random bits.  */
-                XXXXXX[0] = letters[v % 62];
-                v /= 62;
-                XXXXXX[1] = letters[v % 62];
-                v /= 62;
-                XXXXXX[2] = letters[v % 62];
-                v /= 62;
-                XXXXXX[3] = letters[v % 62];
-                v /= 62;
-                XXXXXX[4] = letters[v % 62];
-                v /= 62;
-                XXXXXX[5] = letters[v % 62];
-
-                fd = open (tmpl, (flags & ~O_ACCMODE)
-                           | O_RDWR | O_CREAT | O_EXCL, S_IRUSR | S_IWUSR);
-
-                if (fd >= 0)
-                        return fd;
-                else if (errno != EEXIST)
-                        return -1;
-        }
-
-        /* We got out of the loop because we ran out of combinations to try.  */
-        return -1;
-}
diff --git a/contrib/sunrpc/xdr_sizeof.c b/contrib/sunrpc/xdr_sizeof.c
deleted file mode 100644
index ca1f7bf0a5e..00000000000
--- a/contrib/sunrpc/xdr_sizeof.c
+++ /dev/null
@@ -1,204 +0,0 @@
-/*
- * Copyright (c) 1999 Apple Computer, Inc. All rights reserved.
- *
- * @APPLE_LICENSE_HEADER_START@
- *
- * Portions Copyright (c) 1999 Apple Computer, Inc.  All Rights
- * Reserved.  This file contains Original Code and/or Modifications of
- * Original Code as defined in and that are subject to the Apple Public
- * Source License Version 1.1 (the "License").  You may not use this file
- * except in compliance with the License.  Please obtain a copy of the
- * License at http://www.apple.com/publicsource and read it before using
- * this file.
- *
- * The Original Code and all software distributed under the License are
- * distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, EITHER
- * EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
- * INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE OR NON- INFRINGEMENT.  Please see the
- * License for the specific language governing rights and limitations
- * under the License.
- *
- * @APPLE_LICENSE_HEADER_END@
- */
-
-/*
- * Sun RPC is a product of Sun Microsystems, Inc. and is provided for
- * unrestricted use provided that this legend is included on all tape
- * media and as a part of the software program in whole or part.  Users
- * may copy or modify Sun RPC without charge, but are not authorized
- * to license or distribute it to anyone else except as part of a product or
- * program developed by the user.
- *
- * SUN RPC IS PROVIDED AS IS WITH NO WARRANTIES OF ANY KIND INCLUDING THE
- * WARRANTIES OF DESIGN, MERCHANTIBILITY AND FITNESS FOR A PARTICULAR
- * PURPOSE, OR ARISING FROM A COURSE OF DEALING, USAGE OR TRADE PRACTICE.
- *
- * Sun RPC is provided with no support and without any obligation on the
- * part of Sun Microsystems, Inc. to assist in its use, correction,
- * modification or enhancement.
- *
- * SUN MICROSYSTEMS, INC. SHALL HAVE NO LIABILITY WITH RESPECT TO THE
- * INFRINGEMENT OF COPYRIGHTS, TRADE SECRETS OR ANY PATENTS BY SUN RPC
- * OR ANY PART THEREOF.
- *
- * In no event will Sun Microsystems, Inc. be liable for any lost revenue
- * or profits or other special, indirect and consequential damages, even if
- * Sun has been advised of the possibility of such damages.
- *
- * Sun Microsystems, Inc.
- * 2550 Garcia Avenue
- * Mountain View, California  94043
- */
-
-/*
- * xdr_sizeof.c
- *
- * Copyright 1990 Sun Microsystems, Inc.
- *
- * General purpose routine to see how much space something will use
- * when serialized using XDR.
- */
-
-#ifdef GF_DARWIN_HOST_OS
-
-#include <rpc/types.h>
-#include <rpc/xdr.h>
-#include <sys/types.h>
-#include <sys/cdefs.h>
-
-#include <stdlib.h>
-
-/* ARGSUSED */
-#ifdef GF_DARWIN_HOST_OS
-static bool_t
-x_putlong (XDR *xdrs, const int *longp)
-{
-        xdrs->x_handy += BYTES_PER_XDR_UNIT;
-        return TRUE;
-}
-
-#else
-static bool_t
-x_putlong (XDR *xdrs, const long *longp)
-{
-        xdrs->x_handy += BYTES_PER_XDR_UNIT;
-        return TRUE;
-}
-#endif
-
-/* ARGSUSED */
-static bool_t
-x_putbytes (XDR *xdrs, const char *bp, u_int len)
-{
-        xdrs->x_handy += len;
-        return TRUE;
-}
-
-#ifdef GF_DARWIN_HOST_OS
-static u_int
-x_getpostn (XDR *xdrs)
-{
-        return xdrs->x_handy;
-}
-#else
-static u_int
-x_getpostn (const XDR *xdrs)
-{
-        return xdrs->x_handy;
-}
-#endif
-
-/* ARGSUSED */
-static bool_t
-x_setpostn (XDR *xdrs, u_int len)
-{
-        /* This is not allowed */
-        return FALSE;
-}
-
-static int32_t *
-x_inline (XDR *xdrs, u_int len)
-{
-        if (len == 0)
-                return NULL;
-        if (xdrs->x_op != XDR_ENCODE)
-                return NULL;
-        if (len < (u_int) (long int) xdrs->x_base)
-        {
-                /* x_private was already allocated */
-                xdrs->x_handy += len;
-                return (int32_t *) xdrs->x_private;
-        }
-        else
-        {
-                /* Free the earlier space and allocate new area */
-                free (xdrs->x_private);
-                if ((xdrs->x_private = (caddr_t) malloc (len)) == NULL)
-                {
-                        xdrs->x_base = 0;
-                        return NULL;
-                }
-                xdrs->x_base = (void *) (long) len;
-                xdrs->x_handy += len;
-                return (int32_t *) xdrs->x_private;
-        }
-}
-
-static int
-harmless (void)
-{
-        /* Always return FALSE/NULL, as the case may be */
-        return 0;
-}
-
-static void
-x_destroy (XDR *xdrs)
-{
-        xdrs->x_handy = 0;
-        xdrs->x_base = 0;
-        if (xdrs->x_private)
-        {
-                free (xdrs->x_private);
-                xdrs->x_private = NULL;
-        }
-        return;
-}
-
-unsigned long
-xdr_sizeof (xdrproc_t func, void *data)
-{
-        XDR x;
-        struct xdr_ops ops;
-        bool_t stat;
-
-#ifdef GF_DARWIN_HOST_OS
-        typedef bool_t (*dummyfunc1) (XDR *, int *);
-#else
-        typedef bool_t (*dummyfunc1) (XDR *, long *);
-#endif
-        typedef bool_t (*dummyfunc2) (XDR *, caddr_t, u_int);
-
-        ops.x_putlong = x_putlong;
-        ops.x_putbytes = x_putbytes;
-        ops.x_inline = x_inline;
-        ops.x_getpostn = x_getpostn;
-        ops.x_setpostn = x_setpostn;
-        ops.x_destroy = x_destroy;
-
-        /* the other harmless ones */
-        ops.x_getlong = (dummyfunc1) harmless;
-        ops.x_getbytes = (dummyfunc2) harmless;
-
-        x.x_op = XDR_ENCODE;
-        x.x_ops = &ops;
-        x.x_handy = 0;
-        x.x_private = (caddr_t) NULL;
-        x.x_base = (caddr_t) 0;
-
-        stat = func (&x, data, 0);
-        if (x.x_private)
-                free (x.x_private);
-        return (stat == TRUE ? (unsigned) x.x_handy : 0);
-}
-#endif /* GF_DARWIN_HOST_OS */
diff --git a/contrib/timer-wheel/find_last_bit.c b/contrib/timer-wheel/find_last_bit.c
new file mode 100644
index 00000000000..192fee802a8
--- /dev/null
+++ b/contrib/timer-wheel/find_last_bit.c
@@ -0,0 +1,61 @@
+/* bit search implementation
+ *
+ * Copyright (C) 2004 Red Hat, Inc. All Rights Reserved.
+ * Written by David Howells (dhowells@redhat.com)
+ *
+ * Copyright (C) 2008 IBM Corporation
+ * 'find_last_bit' is written by Rusty Russell <rusty@rustcorp.com.au>
+ * (Inspired by David Howell's find_next_bit implementation)
+ *
+ * Rewritten by Yury Norov <yury.norov@gmail.com> to decrease
+ * size and improve performance, 2015.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+/**
+ * @find_last_bit
+ * optimized implementation of find last bit in
+ */
+
+#ifndef BITS_PER_LONG
+#ifdef __LP64__
+#define BITS_PER_LONG 64
+#else
+#define BITS_PER_LONG 32
+#endif
+#endif
+
+unsigned long gw_tw_fls (unsigned long word)
+{
+        int num = BITS_PER_LONG;
+
+#if BITS_PER_LONG == 64
+        if (!(word & (~0ul << 32))) {
+                num -= 32;
+                word <<= 32;
+        }
+#endif
+        if (!(word & (~0ul << (BITS_PER_LONG-16)))) {
+                num -= 16;
+                word <<= 16;
+        }
+        if (!(word & (~0ul << (BITS_PER_LONG-8)))) {
+                num -= 8;
+                word <<= 8;
+        }
+        if (!(word & (~0ul << (BITS_PER_LONG-4)))) {
+                num -= 4;
+                word <<= 4;
+        }
+        if (!(word & (~0ul << (BITS_PER_LONG-2)))) {
+                num -= 2;
+                word <<= 2;
+        }
+        if (!(word & (~0ul << (BITS_PER_LONG-1))))
+                num -= 1;
+        return num;
+}
diff --git a/contrib/timer-wheel/timer-wheel.c b/contrib/timer-wheel/timer-wheel.c
new file mode 100644
index 00000000000..58e0607bf0c
--- /dev/null
+++ b/contrib/timer-wheel/timer-wheel.c
@@ -0,0 +1,374 @@
+/*
+ *  linux/kernel/timer.c
+ *
+ *  Kernel internal timers
+ *
+ *  Copyright (C) 1991, 1992  Linus Torvalds
+ *
+ */
+/*
+  This program is free software; you can redistribute it and/or modify
+  it under the terms of the GNU General Public License as published by
+  the Free Software Foundation; either version 2 of the License, or
+  (at your option) any later version.
+
+  This program is distributed in the hope that it will be useful,
+  but WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+  GNU General Public License for more details.
+
+  You should have received a copy of the GNU General Public License along
+  with this program; if not, write to the Free Software Foundation, Inc.,
+  51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+*/
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include <unistd.h>
+#include <sys/select.h>
+
+#include "timer-wheel.h"
+
+static inline void
+__gf_tw_add_timer (struct tvec_base *base, struct gf_tw_timer_list *timer)
+{
+        int i;
+        unsigned long idx;
+        unsigned long expires;
+        struct list_head *vec;
+
+        expires = timer->expires;
+
+        idx = expires - base->timer_sec;
+
+        if (idx < TVR_SIZE) {
+                i = expires & TVR_MASK;
+                vec = base->tv1.vec + i;
+        } else if (idx < 1 << (TVR_BITS + TVN_BITS)) {
+                i = (expires >> TVR_BITS) & TVN_MASK;
+                vec = base->tv2.vec + i;
+        } else if (idx < 1 << (TVR_BITS + 2*TVN_BITS)) {
+                i = (expires >> (TVR_BITS + TVN_BITS)) & TVN_MASK;
+                vec = base->tv3.vec + i;
+        } else if (idx < 1 << (TVR_BITS + 3*TVN_BITS)) {
+                i = (expires >> (TVR_BITS + 2*TVN_BITS)) & TVN_MASK;
+                vec = base->tv4.vec + i;
+        } else if (idx < 0) {
+                vec = base->tv1.vec + (base->timer_sec & TVR_MASK);
+        } else {
+                i = (expires >> (TVR_BITS + 3*TVN_BITS)) & TVN_MASK;
+                vec = base->tv5.vec + i;
+        }
+
+        list_add_tail (&timer->entry, vec);
+}
+
+unsigned long gf_tw_find_last_bit(const unsigned long *, unsigned long);
+
+#if defined(__GNUC__) || defined(__clang__)
+static inline unsigned long gf_tw_fls (unsigned long word)
+{
+        return BITS_PER_LONG - __builtin_clzl(word);
+}
+#else
+extern unsigned long gf_tw_fls (unsigned long);
+#endif
+
+static inline unsigned long
+apply_slack(struct tvec_base *base, struct gf_tw_timer_list *timer)
+{
+        long delta;
+        unsigned long mask, expires, expires_limit;
+
+        expires = timer->expires;
+
+        delta = expires - base->timer_sec;
+        if (delta < 256)
+                return expires;
+
+        expires_limit = expires + delta / 256;
+        mask = expires ^ expires_limit;
+        if (mask == 0)
+                return expires;
+
+        int bit = gf_tw_fls (mask);
+        mask = (1UL << bit) - 1;
+
+        expires_limit = expires_limit & ~(mask);
+        return expires_limit;
+}
+
+static inline void
+__gf_tw_detach_timer (struct gf_tw_timer_list *timer)
+{
+        struct list_head *entry = &timer->entry;
+
+        list_del (entry);
+        entry->next = NULL;
+}
+
+static inline int
+cascade (struct tvec_base *base, struct tvec *tv, int index)
+{
+        struct gf_tw_timer_list *timer, *tmp;
+        struct list_head tv_list;
+
+        list_replace_init (tv->vec + index, &tv_list);
+
+        list_for_each_entry_safe (timer, tmp, &tv_list, entry) {
+                __gf_tw_add_timer (base, timer);
+        }
+
+        return index;
+}
+
+#define INDEX(N)  ((base->timer_sec >> (TVR_BITS + N * TVN_BITS)) & TVN_MASK)
+
+/**
+ * run expired timers
+ */
+static inline void
+run_timers (struct tvec_base *base)
+{
+        unsigned long index, call_time;
+        struct gf_tw_timer_list *timer;
+
+        struct list_head work_list;
+        struct list_head *head = &work_list;
+
+        pthread_spin_lock (&base->lock);
+        {
+                index  = base->timer_sec & TVR_MASK;
+
+                if (!index &&
+                    (!cascade (base, &base->tv2, INDEX(0))) &&
+                    (!cascade (base, &base->tv3, INDEX(1))) &&
+                    (!cascade (base, &base->tv4, INDEX(2))))
+                        cascade (base, &base->tv5, INDEX(3));
+
+                call_time = base->timer_sec++;
+                list_replace_init (base->tv1.vec + index, head);
+                while (!list_empty(head)) {
+                        void (*fn)(struct gf_tw_timer_list *, void *, unsigned long);
+                        void *data;
+
+                        timer = list_first_entry (head, struct gf_tw_timer_list, entry);
+                        fn = timer->function;
+                        data = timer->data;
+
+                        __gf_tw_detach_timer (timer);
+                        pthread_spin_unlock(&base->lock);
+                        {
+                            /* It is required to run the actual function outside
+                               of the locked zone, so we don't bother about
+                               locked operations inside that function */
+                            fn(timer, data, call_time);
+                        }
+                        pthread_spin_lock(&base->lock);
+                }
+        }
+        pthread_spin_unlock (&base->lock);
+
+}
+
+void *runner (void *arg)
+{
+        struct timeval tv = {0,};
+        struct tvec_base *base = arg;
+
+        while (1) {
+                run_timers (base);
+
+                tv.tv_sec  = 1;
+                tv.tv_usec = 0;
+                (void) select (0, NULL, NULL, NULL, &tv);
+        }
+
+        return NULL;
+
+}
+
+static inline int timer_pending (struct gf_tw_timer_list *timer)
+{
+        struct list_head *entry = &timer->entry;
+
+        return (entry->next != NULL);
+}
+
+static inline int __detach_if_pending (struct gf_tw_timer_list *timer)
+{
+        if (!timer_pending (timer))
+                return 0;
+
+        __gf_tw_detach_timer (timer);
+        return 1;
+}
+
+static inline int __mod_timer (struct tvec_base *base,
+                               struct gf_tw_timer_list *timer, int pending_only)
+{
+        int ret = 0;
+
+        ret = __detach_if_pending (timer);
+        if (!ret && pending_only)
+                goto done;
+
+        ret = 1;
+        __gf_tw_add_timer (base, timer);
+
+ done:
+        return ret;
+}
+
+/* interface */
+
+/**
+ * Add a timer in the timer wheel
+ */
+void gf_tw_add_timer (struct tvec_base *base, struct gf_tw_timer_list *timer)
+{
+        pthread_spin_lock (&base->lock);
+        {
+                timer->expires += base->timer_sec;
+                timer->expires = apply_slack (base, timer);
+                __gf_tw_add_timer (base, timer);
+        }
+        pthread_spin_unlock (&base->lock);
+}
+
+/**
+ * Remove a timer from the timer wheel
+ */
+int gf_tw_del_timer (struct tvec_base *base, struct gf_tw_timer_list *timer)
+{
+        int ret = 0;
+
+        pthread_spin_lock (&base->lock);
+        {
+                if (timer_pending (timer)) {
+                        ret = 1;
+                        __gf_tw_detach_timer (timer);
+                }
+        }
+        pthread_spin_unlock (&base->lock);
+
+        return ret;
+}
+
+int gf_tw_mod_timer_pending (struct tvec_base *base,
+                             struct gf_tw_timer_list *timer,
+                             unsigned long expires)
+{
+        int ret = 1;
+
+        pthread_spin_lock (&base->lock);
+        {
+                timer->expires = expires + base->timer_sec;
+                timer->expires = apply_slack (base, timer);
+
+                ret = __mod_timer (base, timer, 1);
+        }
+        pthread_spin_unlock (&base->lock);
+
+        return ret;
+}
+
+int gf_tw_mod_timer (struct tvec_base *base,
+                     struct gf_tw_timer_list *timer, unsigned long expires)
+{
+        int ret = 1;
+
+        pthread_spin_lock (&base->lock);
+        {
+                /* fast path optimization */
+                if (timer_pending (timer) && timer->expires == expires)
+                        goto unblock;
+
+                timer->expires = expires + base->timer_sec;
+                timer->expires = apply_slack (base, timer);
+
+                ret = __mod_timer (base, timer, 0);
+        }
+ unblock:
+        pthread_spin_unlock (&base->lock);
+
+        return ret;
+}
+
+int gf_tw_cleanup_timers (struct tvec_base *base)
+{
+        int ret = 0;
+        void *res = NULL;
+
+        /* terminate runner */
+        ret = pthread_cancel (base->runner);
+        if (ret != 0)
+                goto error_return;
+        ret = pthread_join (base->runner, &res);
+        if (ret != 0)
+                goto error_return;
+        if (res != PTHREAD_CANCELED)
+                goto error_return;
+
+        /* destroy lock */
+        ret = pthread_spin_destroy (&base->lock);
+        if (ret != 0)
+                goto error_return;
+
+        /* deallocated timer base */
+        free (base);
+        return 0;
+
+ error_return:
+        return -1;
+}
+
+/**
+ * Initialize various timer wheel lists and spawn a thread that
+ * invokes run_timers()
+ */
+struct tvec_base *gf_tw_init_timers ()
+{
+        int               j    = 0;
+        int               ret  = 0;
+        struct timeval    tv   = {0,};
+        struct tvec_base *base = NULL;
+
+        base = malloc (sizeof (*base));
+        if (!base)
+                goto error_return;
+
+        ret = pthread_spin_init (&base->lock, 0);
+        if (ret != 0)
+                goto error_dealloc;
+
+        for (j = 0; j < TVN_SIZE; j++) {
+                INIT_LIST_HEAD (base->tv5.vec + j);
+                INIT_LIST_HEAD (base->tv4.vec + j);
+                INIT_LIST_HEAD (base->tv3.vec + j);
+                INIT_LIST_HEAD (base->tv2.vec + j);
+        }
+
+        for (j = 0; j < TVR_SIZE; j++) {
+                INIT_LIST_HEAD (base->tv1.vec + j);
+        }
+
+        ret = gettimeofday (&tv, 0);
+        if (ret < 0)
+                goto destroy_lock;
+        base->timer_sec = tv.tv_sec;
+
+        ret = pthread_create (&base->runner, NULL, runner, base);
+        if (ret != 0)
+                goto destroy_lock;
+        return base;
+
+ destroy_lock:
+        (void) pthread_spin_destroy (&base->lock);
+ error_dealloc:
+        free (base);
+ error_return:
+        return NULL;
+}
diff --git a/contrib/timer-wheel/timer-wheel.h b/contrib/timer-wheel/timer-wheel.h
new file mode 100644
index 00000000000..5637735ec22
--- /dev/null
+++ b/contrib/timer-wheel/timer-wheel.h
@@ -0,0 +1,77 @@
+/*
+  This program is free software; you can redistribute it and/or modify
+  it under the terms of the GNU General Public License as published by
+  the Free Software Foundation; either version 2 of the License, or
+  (at your option) any later version.
+
+  This program is distributed in the hope that it will be useful,
+  but WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+  GNU General Public License for more details.
+
+  You should have received a copy of the GNU General Public License along
+  with this program; if not, write to the Free Software Foundation, Inc.,
+  51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+*/
+
+#ifndef __TIMER_WHEEL_H
+#define __TIMER_WHEEL_H
+
+#include "glusterfs/locking.h"
+
+#include "glusterfs/list.h"
+
+#define TVR_BITS  8
+#define TVN_BITS  6
+#define TVR_SIZE  (1 << TVR_BITS)
+#define TVN_SIZE  (1 << TVN_BITS)
+#define TVR_MASK  (TVR_SIZE - 1)
+#define TVN_MASK  (TVN_SIZE - 1)
+
+#define BITS_PER_LONG  64
+
+struct tvec {
+        struct list_head vec[TVN_SIZE];
+};
+
+struct tvec_root {
+        struct list_head vec[TVR_SIZE];
+};
+
+struct tvec_base {
+        pthread_spinlock_t lock;      /* base lock */
+
+        pthread_t runner;             /* run_timer() */
+
+        unsigned long timer_sec;      /* time counter */
+
+        struct tvec_root tv1;
+        struct tvec tv2;
+        struct tvec tv3;
+        struct tvec tv4;
+        struct tvec tv5;
+};
+
+struct gf_tw_timer_list {
+        void *data;
+        unsigned long expires;
+
+        /** callback routine */
+        void (*function)(struct gf_tw_timer_list *, void *, unsigned long);
+
+        struct list_head entry;
+};
+
+/** The API! */
+struct tvec_base *gf_tw_init_timers ();
+int gf_tw_cleanup_timers (struct tvec_base *);
+void gf_tw_add_timer (struct tvec_base *, struct gf_tw_timer_list *);
+int gf_tw_del_timer (struct tvec_base *, struct gf_tw_timer_list *);
+
+int gf_tw_mod_timer_pending (struct tvec_base *,
+                             struct gf_tw_timer_list *, unsigned long);
+
+int gf_tw_mod_timer (struct tvec_base *,
+                     struct gf_tw_timer_list *, unsigned long);
+
+#endif
diff --git a/contrib/umountd/Makefile.am b/contrib/umountd/Makefile.am
index c03b0cbcae6..b39e000f5aa 100644
--- a/contrib/umountd/Makefile.am
+++ b/contrib/umountd/Makefile.am
@@ -4,7 +4,8 @@ umountd_CFLAGS = $(GF_CFLAGS) -DDATADIR=\"$(localstatedir)\"
 umountd_LDADD = $(top_builddir)/libglusterfs/src/libglusterfs.la ${GF_LDADD}
 umountd_LDFLAGS = $(GF_LDFLAGS)
 
-AM_CPPFLAGS = $(GF_CPPFLAGS) -I$(top_srcdir)/libglusterfs/src 
+AM_CPPFLAGS = $(GF_CPPFLAGS) -I$(top_srcdir)/libglusterfs/src \
+	-I$(top_srcdir)/rpc/xdr/src -I$(top_builddir)/rpc/xdr/src
 AM_CFLAGS = -Wall $(GF_CFLAGS)
 
 CLEANFILES =
diff --git a/contrib/umountd/umountd.c b/contrib/umountd/umountd.c
index 0d2c6f20b60..3f933ecb554 100644
--- a/contrib/umountd/umountd.c
+++ b/contrib/umountd/umountd.c
@@ -23,11 +23,11 @@
 #include <sys/stat.h>
 #include <sys/mount.h>
 
-#include "glusterfs.h"
-#include "globals.h"
-#include "logging.h"
-#include "syscall.h"
-#include "mem-types.h"
+#include "glusterfs/glusterfs.h"
+#include "glusterfs/globals.h"
+#include "glusterfs/logging.h"
+#include "glusterfs/syscall.h"
+#include "glusterfs/mem-types.h"
 
 static void
 usage (void)
@@ -56,7 +56,8 @@ sanity_check (char *path, dev_t *devp)
                         break;
                 default:
                         gf_log ("umountd", GF_LOG_ERROR,
-                                "Cannot access %s\n", path, strerror (errno));
+                                "Cannot access %s: %s\n",
+                                path, strerror (errno));
                         goto out;
                 }
         }
@@ -65,12 +66,13 @@ sanity_check (char *path, dev_t *devp)
         if (*devp == -1 && ret == 0)
                 *devp = st.st_dev;
 
-        strncpy (pathtmp, path, PATH_MAX);
+        snprintf (pathtmp, PATH_MAX, "%s", path);
         parent = dirname (pathtmp);
 
         if (stat (parent, &parent_st) != 0) {
                 gf_log ("umountd", GF_LOG_ERROR,
-                        "Cannot access %s\n", parent, strerror (errno));
+                        "Cannot access %s: %s\n",
+                        parent, strerror (errno));
                 goto out;
         }
 
diff --git a/contrib/userspace-rcu/rculist-extra.h b/contrib/userspace-rcu/rculist-extra.h
new file mode 100644
index 00000000000..274cf9f9d7a
--- /dev/null
+++ b/contrib/userspace-rcu/rculist-extra.h
@@ -0,0 +1,42 @@
+/*
+ * Copyright (C) 2002 Free Software Foundation, Inc.
+ * (originally part of the GNU C Library)
+ * Contributed by Ulrich Drepper <drepper@redhat.com>, 2002.
+ *
+ * Copyright (C) 2009 Pierre-Marc Fournier
+ * Conversion to RCU list.
+ * Copyright (C) 2010 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#ifndef URCU_RCULIST_EXTRA_H
+#define URCU_RCULIST_EXTRA_H
+/* Copying this definition from liburcu-0.8 as liburcu-0.7 does not have this
+ * particular list api
+ */
+/* Add new element at the tail of the list. */
+
+static inline
+void cds_list_add_tail_rcu(struct cds_list_head *newp,
+                struct cds_list_head *head)
+{
+        newp->next = head;
+        newp->prev = head->prev;
+        rcu_assign_pointer(head->prev->next, newp);
+        head->prev = newp;
+}
+
+#endif
diff --git a/contrib/userspace-rcu/static-wfcqueue.h b/contrib/userspace-rcu/static-wfcqueue.h
new file mode 100644
index 00000000000..37d14ad674b
--- /dev/null
+++ b/contrib/userspace-rcu/static-wfcqueue.h
@@ -0,0 +1,685 @@
+#ifndef _URCU_WFCQUEUE_STATIC_H
+#define _URCU_WFCQUEUE_STATIC_H
+
+/*
+ * urcu/static/wfcqueue.h
+ *
+ * Userspace RCU library - Concurrent Queue with Wait-Free Enqueue/Blocking Dequeue
+ *
+ * TO BE INCLUDED ONLY IN LGPL-COMPATIBLE CODE. See urcu/wfcqueue.h for
+ * linking dynamically with the userspace rcu library.
+ *
+ * Copyright 2010-2012 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+ * Copyright 2011-2012 - Lai Jiangshan <laijs@cn.fujitsu.com>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+/* Copied from userspace-rcu 0.10 because version 0.7 doesn't contain it. */
+
+#include <pthread.h>
+#include <assert.h>
+#include <poll.h>
+#include <stdbool.h>
+#include <urcu/compiler.h>
+#include <urcu/uatomic.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/*
+ * Concurrent queue with wait-free enqueue/blocking dequeue.
+ *
+ * This queue has been designed and implemented collaboratively by
+ * Mathieu Desnoyers and Lai Jiangshan. Inspired from
+ * half-wait-free/half-blocking queue implementation done by Paul E.
+ * McKenney.
+ *
+ * Mutual exclusion of cds_wfcq_* / __cds_wfcq_* API
+ *
+ * Synchronization table:
+ *
+ * External synchronization techniques described in the API below is
+ * required between pairs marked with "X". No external synchronization
+ * required between pairs marked with "-".
+ *
+ * Legend:
+ * [1] cds_wfcq_enqueue
+ * [2] __cds_wfcq_splice (destination queue)
+ * [3] __cds_wfcq_dequeue
+ * [4] __cds_wfcq_splice (source queue)
+ * [5] __cds_wfcq_first
+ * [6] __cds_wfcq_next
+ *
+ *     [1] [2] [3] [4] [5] [6]
+ * [1]  -   -   -   -   -   -
+ * [2]  -   -   -   -   -   -
+ * [3]  -   -   X   X   X   X
+ * [4]  -   -   X   -   X   X
+ * [5]  -   -   X   X   -   -
+ * [6]  -   -   X   X   -   -
+ *
+ * Mutual exclusion can be ensured by holding cds_wfcq_dequeue_lock().
+ *
+ * For convenience, cds_wfcq_dequeue_blocking() and
+ * cds_wfcq_splice_blocking() hold the dequeue lock.
+ *
+ * Besides locking, mutual exclusion of dequeue, splice and iteration
+ * can be ensured by performing all of those operations from a single
+ * thread, without requiring any lock.
+ */
+
+#define WFCQ_ADAPT_ATTEMPTS		10	/* Retry if being set */
+#define WFCQ_WAIT			10	/* Wait 10 ms if being set */
+
+/*
+ * cds_wfcq_node_init: initialize wait-free queue node.
+ */
+static inline void _cds_wfcq_node_init(struct cds_wfcq_node *node)
+{
+	node->next = NULL;
+}
+
+/*
+ * cds_wfcq_init: initialize wait-free queue (with lock). Pair with
+ * cds_wfcq_destroy().
+ */
+static inline void _cds_wfcq_init(struct cds_wfcq_head *head,
+		struct cds_wfcq_tail *tail)
+{
+	int ret;
+
+	/* Set queue head and tail */
+	_cds_wfcq_node_init(&head->node);
+	tail->p = &head->node;
+	ret = pthread_mutex_init(&head->lock, NULL);
+	assert(!ret);
+}
+
+/*
+ * cds_wfcq_destroy: destroy wait-free queue (with lock). Pair with
+ * cds_wfcq_init().
+ */
+static inline void _cds_wfcq_destroy(struct cds_wfcq_head *head,
+		struct cds_wfcq_tail *tail)
+{
+	int ret = pthread_mutex_destroy(&head->lock);
+	assert(!ret);
+}
+
+/*
+ * __cds_wfcq_init: initialize wait-free queue (without lock). Don't
+ * pair with any destroy function.
+ */
+static inline void ___cds_wfcq_init(struct __cds_wfcq_head *head,
+		struct cds_wfcq_tail *tail)
+{
+	/* Set queue head and tail */
+	_cds_wfcq_node_init(&head->node);
+	tail->p = &head->node;
+}
+
+/*
+ * cds_wfcq_empty: return whether wait-free queue is empty.
+ *
+ * No memory barrier is issued. No mutual exclusion is required.
+ *
+ * We perform the test on head->node.next to check if the queue is
+ * possibly empty, but we confirm this by checking if the tail pointer
+ * points to the head node because the tail pointer is the linearisation
+ * point of the enqueuers. Just checking the head next pointer could
+ * make a queue appear empty if an enqueuer is preempted for a long time
+ * between xchg() and setting the previous node's next pointer.
+ */
+static inline bool _cds_wfcq_empty(cds_wfcq_head_ptr_t u_head,
+		struct cds_wfcq_tail *tail)
+{
+	struct __cds_wfcq_head *head = u_head._h;
+	/*
+	 * Queue is empty if no node is pointed by head->node.next nor
+	 * tail->p. Even though the tail->p check is sufficient to find
+	 * out of the queue is empty, we first check head->node.next as a
+	 * common case to ensure that dequeuers do not frequently access
+	 * enqueuer's tail->p cache line.
+	 */
+	return CMM_LOAD_SHARED(head->node.next) == NULL
+		&& CMM_LOAD_SHARED(tail->p) == &head->node;
+}
+
+static inline void _cds_wfcq_dequeue_lock(struct cds_wfcq_head *head,
+		struct cds_wfcq_tail *tail)
+{
+	int ret;
+
+	ret = pthread_mutex_lock(&head->lock);
+	assert(!ret);
+}
+
+static inline void _cds_wfcq_dequeue_unlock(struct cds_wfcq_head *head,
+		struct cds_wfcq_tail *tail)
+{
+	int ret;
+
+	ret = pthread_mutex_unlock(&head->lock);
+	assert(!ret);
+}
+
+static inline bool ___cds_wfcq_append(cds_wfcq_head_ptr_t u_head,
+		struct cds_wfcq_tail *tail,
+		struct cds_wfcq_node *new_head,
+		struct cds_wfcq_node *new_tail)
+{
+	struct __cds_wfcq_head *head = u_head._h;
+	struct cds_wfcq_node *old_tail;
+
+	/*
+	 * Implicit memory barrier before uatomic_xchg() orders earlier
+	 * stores to data structure containing node and setting
+	 * node->next to NULL before publication.
+	 */
+	old_tail = uatomic_xchg(&tail->p, new_tail);
+
+	/*
+	 * Implicit memory barrier after uatomic_xchg() orders store to
+	 * q->tail before store to old_tail->next.
+	 *
+	 * At this point, dequeuers see a NULL tail->p->next, which
+	 * indicates that the queue is being appended to. The following
+	 * store will append "node" to the queue from a dequeuer
+	 * perspective.
+	 */
+	CMM_STORE_SHARED(old_tail->next, new_head);
+	/*
+	 * Return false if queue was empty prior to adding the node,
+	 * else return true.
+	 */
+	return old_tail != &head->node;
+}
+
+/*
+ * cds_wfcq_enqueue: enqueue a node into a wait-free queue.
+ *
+ * Issues a full memory barrier before enqueue. No mutual exclusion is
+ * required.
+ *
+ * Returns false if the queue was empty prior to adding the node.
+ * Returns true otherwise.
+ */
+static inline bool _cds_wfcq_enqueue(cds_wfcq_head_ptr_t head,
+		struct cds_wfcq_tail *tail,
+		struct cds_wfcq_node *new_tail)
+{
+	return ___cds_wfcq_append(head, tail, new_tail, new_tail);
+}
+
+/*
+ * CDS_WFCQ_WAIT_SLEEP:
+ *
+ * By default, this sleeps for the given @msec milliseconds.
+ * This is a macro which LGPL users may #define themselves before
+ * including wfcqueue.h to override the default behavior (e.g.
+ * to log a warning or perform other background work).
+ */
+#ifndef CDS_WFCQ_WAIT_SLEEP
+#define CDS_WFCQ_WAIT_SLEEP(msec) ___cds_wfcq_wait_sleep(msec)
+#endif
+
+static inline void ___cds_wfcq_wait_sleep(int msec)
+{
+	(void) poll(NULL, 0, msec);
+}
+
+/*
+ * ___cds_wfcq_busy_wait: adaptative busy-wait.
+ *
+ * Returns 1 if nonblocking and needs to block, 0 otherwise.
+ */
+static inline bool
+___cds_wfcq_busy_wait(int *attempt, int blocking)
+{
+	if (!blocking)
+		return 1;
+	if (++(*attempt) >= WFCQ_ADAPT_ATTEMPTS) {
+		CDS_WFCQ_WAIT_SLEEP(WFCQ_WAIT);		/* Wait for 10ms */
+		*attempt = 0;
+	} else {
+		caa_cpu_relax();
+	}
+	return 0;
+}
+
+/*
+ * Waiting for enqueuer to complete enqueue and return the next node.
+ */
+static inline struct cds_wfcq_node *
+___cds_wfcq_node_sync_next(struct cds_wfcq_node *node, int blocking)
+{
+	struct cds_wfcq_node *next;
+	int attempt = 0;
+
+	/*
+	 * Adaptative busy-looping waiting for enqueuer to complete enqueue.
+	 */
+	while ((next = CMM_LOAD_SHARED(node->next)) == NULL) {
+		if (___cds_wfcq_busy_wait(&attempt, blocking))
+			return CDS_WFCQ_WOULDBLOCK;
+	}
+
+	return next;
+}
+
+static inline struct cds_wfcq_node *
+___cds_wfcq_first(cds_wfcq_head_ptr_t u_head,
+		struct cds_wfcq_tail *tail,
+		int blocking)
+{
+	struct __cds_wfcq_head *head = u_head._h;
+	struct cds_wfcq_node *node;
+
+	if (_cds_wfcq_empty(__cds_wfcq_head_cast(head), tail))
+		return NULL;
+	node = ___cds_wfcq_node_sync_next(&head->node, blocking);
+	/* Load head->node.next before loading node's content */
+	cmm_smp_read_barrier_depends();
+	return node;
+}
+
+/*
+ * __cds_wfcq_first_blocking: get first node of a queue, without dequeuing.
+ *
+ * Content written into the node before enqueue is guaranteed to be
+ * consistent, but no other memory ordering is ensured.
+ * Dequeue/splice/iteration mutual exclusion should be ensured by the
+ * caller.
+ *
+ * Used by for-like iteration macros in urcu/wfqueue.h:
+ * __cds_wfcq_for_each_blocking()
+ * __cds_wfcq_for_each_blocking_safe()
+ *
+ * Returns NULL if queue is empty, first node otherwise.
+ */
+static inline struct cds_wfcq_node *
+___cds_wfcq_first_blocking(cds_wfcq_head_ptr_t head,
+		struct cds_wfcq_tail *tail)
+{
+	return ___cds_wfcq_first(head, tail, 1);
+}
+
+
+/*
+ * __cds_wfcq_first_nonblocking: get first node of a queue, without dequeuing.
+ *
+ * Same as __cds_wfcq_first_blocking, but returns CDS_WFCQ_WOULDBLOCK if
+ * it needs to block.
+ */
+static inline struct cds_wfcq_node *
+___cds_wfcq_first_nonblocking(cds_wfcq_head_ptr_t head,
+		struct cds_wfcq_tail *tail)
+{
+	return ___cds_wfcq_first(head, tail, 0);
+}
+
+static inline struct cds_wfcq_node *
+___cds_wfcq_next(cds_wfcq_head_ptr_t head,
+		struct cds_wfcq_tail *tail,
+		struct cds_wfcq_node *node,
+		int blocking)
+{
+	struct cds_wfcq_node *next;
+
+	/*
+	 * Even though the following tail->p check is sufficient to find
+	 * out if we reached the end of the queue, we first check
+	 * node->next as a common case to ensure that iteration on nodes
+	 * do not frequently access enqueuer's tail->p cache line.
+	 */
+	if ((next = CMM_LOAD_SHARED(node->next)) == NULL) {
+		/* Load node->next before tail->p */
+		cmm_smp_rmb();
+		if (CMM_LOAD_SHARED(tail->p) == node)
+			return NULL;
+		next = ___cds_wfcq_node_sync_next(node, blocking);
+	}
+	/* Load node->next before loading next's content */
+	cmm_smp_read_barrier_depends();
+	return next;
+}
+
+/*
+ * __cds_wfcq_next_blocking: get next node of a queue, without dequeuing.
+ *
+ * Content written into the node before enqueue is guaranteed to be
+ * consistent, but no other memory ordering is ensured.
+ * Dequeue/splice/iteration mutual exclusion should be ensured by the
+ * caller.
+ *
+ * Used by for-like iteration macros in urcu/wfqueue.h:
+ * __cds_wfcq_for_each_blocking()
+ * __cds_wfcq_for_each_blocking_safe()
+ *
+ * Returns NULL if reached end of queue, non-NULL next queue node
+ * otherwise.
+ */
+static inline struct cds_wfcq_node *
+___cds_wfcq_next_blocking(cds_wfcq_head_ptr_t head,
+		struct cds_wfcq_tail *tail,
+		struct cds_wfcq_node *node)
+{
+	return ___cds_wfcq_next(head, tail, node, 1);
+}
+
+/*
+ * __cds_wfcq_next_blocking: get next node of a queue, without dequeuing.
+ *
+ * Same as __cds_wfcq_next_blocking, but returns CDS_WFCQ_WOULDBLOCK if
+ * it needs to block.
+ */
+static inline struct cds_wfcq_node *
+___cds_wfcq_next_nonblocking(cds_wfcq_head_ptr_t head,
+		struct cds_wfcq_tail *tail,
+		struct cds_wfcq_node *node)
+{
+	return ___cds_wfcq_next(head, tail, node, 0);
+}
+
+static inline struct cds_wfcq_node *
+___cds_wfcq_dequeue_with_state(cds_wfcq_head_ptr_t u_head,
+		struct cds_wfcq_tail *tail,
+		int *state,
+		int blocking)
+{
+	struct __cds_wfcq_head *head = u_head._h;
+	struct cds_wfcq_node *node, *next;
+
+	if (state)
+		*state = 0;
+
+	if (_cds_wfcq_empty(__cds_wfcq_head_cast(head), tail)) {
+		return NULL;
+	}
+
+	node = ___cds_wfcq_node_sync_next(&head->node, blocking);
+	if (!blocking && node == CDS_WFCQ_WOULDBLOCK) {
+		return CDS_WFCQ_WOULDBLOCK;
+	}
+
+	if ((next = CMM_LOAD_SHARED(node->next)) == NULL) {
+		/*
+		 * @node is probably the only node in the queue.
+		 * Try to move the tail to &q->head.
+		 * q->head.next is set to NULL here, and stays
+		 * NULL if the cmpxchg succeeds. Should the
+		 * cmpxchg fail due to a concurrent enqueue, the
+		 * q->head.next will be set to the next node.
+		 * The implicit memory barrier before
+		 * uatomic_cmpxchg() orders load node->next
+		 * before loading q->tail.
+		 * The implicit memory barrier before uatomic_cmpxchg
+		 * orders load q->head.next before loading node's
+		 * content.
+		 */
+		_cds_wfcq_node_init(&head->node);
+		if (uatomic_cmpxchg(&tail->p, node, &head->node) == node) {
+			if (state)
+				*state |= CDS_WFCQ_STATE_LAST;
+			return node;
+		}
+		next = ___cds_wfcq_node_sync_next(node, blocking);
+		/*
+		 * In nonblocking mode, if we would need to block to
+		 * get node's next, set the head next node pointer
+		 * (currently NULL) back to its original value.
+		 */
+		if (!blocking && next == CDS_WFCQ_WOULDBLOCK) {
+			head->node.next = node;
+			return CDS_WFCQ_WOULDBLOCK;
+		}
+	}
+
+	/*
+	 * Move queue head forward.
+	 */
+	head->node.next = next;
+
+	/* Load q->head.next before loading node's content */
+	cmm_smp_read_barrier_depends();
+	return node;
+}
+
+/*
+ * __cds_wfcq_dequeue_with_state_blocking: dequeue node from queue, with state.
+ *
+ * Content written into the node before enqueue is guaranteed to be
+ * consistent, but no other memory ordering is ensured.
+ * It is valid to reuse and free a dequeued node immediately.
+ * Dequeue/splice/iteration mutual exclusion should be ensured by the
+ * caller.
+ */
+static inline struct cds_wfcq_node *
+___cds_wfcq_dequeue_with_state_blocking(cds_wfcq_head_ptr_t head,
+		struct cds_wfcq_tail *tail, int *state)
+{
+	return ___cds_wfcq_dequeue_with_state(head, tail, state, 1);
+}
+
+/*
+ * ___cds_wfcq_dequeue_blocking: dequeue node from queue.
+ *
+ * Same as __cds_wfcq_dequeue_with_state_blocking, but without saving
+ * state.
+ */
+static inline struct cds_wfcq_node *
+___cds_wfcq_dequeue_blocking(cds_wfcq_head_ptr_t head,
+		struct cds_wfcq_tail *tail)
+{
+	return ___cds_wfcq_dequeue_with_state_blocking(head, tail, NULL);
+}
+
+/*
+ * __cds_wfcq_dequeue_with_state_nonblocking: dequeue node, with state.
+ *
+ * Same as __cds_wfcq_dequeue_blocking, but returns CDS_WFCQ_WOULDBLOCK
+ * if it needs to block.
+ */
+static inline struct cds_wfcq_node *
+___cds_wfcq_dequeue_with_state_nonblocking(cds_wfcq_head_ptr_t head,
+		struct cds_wfcq_tail *tail, int *state)
+{
+	return ___cds_wfcq_dequeue_with_state(head, tail, state, 0);
+}
+
+/*
+ * ___cds_wfcq_dequeue_nonblocking: dequeue node from queue.
+ *
+ * Same as __cds_wfcq_dequeue_with_state_nonblocking, but without saving
+ * state.
+ */
+static inline struct cds_wfcq_node *
+___cds_wfcq_dequeue_nonblocking(cds_wfcq_head_ptr_t head,
+		struct cds_wfcq_tail *tail)
+{
+	return ___cds_wfcq_dequeue_with_state_nonblocking(head, tail, NULL);
+}
+
+/*
+ * __cds_wfcq_splice: enqueue all src_q nodes at the end of dest_q.
+ *
+ * Dequeue all nodes from src_q.
+ * dest_q must be already initialized.
+ * Mutual exclusion for src_q should be ensured by the caller as
+ * specified in the "Synchronisation table".
+ * Returns enum cds_wfcq_ret which indicates the state of the src or
+ * dest queue.
+ */
+static inline enum cds_wfcq_ret
+___cds_wfcq_splice(
+		cds_wfcq_head_ptr_t u_dest_q_head,
+		struct cds_wfcq_tail *dest_q_tail,
+		cds_wfcq_head_ptr_t u_src_q_head,
+		struct cds_wfcq_tail *src_q_tail,
+		int blocking)
+{
+	struct __cds_wfcq_head *dest_q_head = u_dest_q_head._h;
+	struct __cds_wfcq_head *src_q_head = u_src_q_head._h;
+	struct cds_wfcq_node *head, *tail;
+	int attempt = 0;
+
+	/*
+	 * Initial emptiness check to speed up cases where queue is
+	 * empty: only require loads to check if queue is empty.
+	 */
+	if (_cds_wfcq_empty(__cds_wfcq_head_cast(src_q_head), src_q_tail))
+		return CDS_WFCQ_RET_SRC_EMPTY;
+
+	for (;;) {
+		/*
+		 * Open-coded _cds_wfcq_empty() by testing result of
+		 * uatomic_xchg, as well as tail pointer vs head node
+		 * address.
+		 */
+		head = uatomic_xchg(&src_q_head->node.next, NULL);
+		if (head)
+			break;	/* non-empty */
+		if (CMM_LOAD_SHARED(src_q_tail->p) == &src_q_head->node)
+			return CDS_WFCQ_RET_SRC_EMPTY;
+		if (___cds_wfcq_busy_wait(&attempt, blocking))
+			return CDS_WFCQ_RET_WOULDBLOCK;
+	}
+
+	/*
+	 * Memory barrier implied before uatomic_xchg() orders store to
+	 * src_q->head before store to src_q->tail. This is required by
+	 * concurrent enqueue on src_q, which exchanges the tail before
+	 * updating the previous tail's next pointer.
+	 */
+	tail = uatomic_xchg(&src_q_tail->p, &src_q_head->node);
+
+	/*
+	 * Append the spliced content of src_q into dest_q. Does not
+	 * require mutual exclusion on dest_q (wait-free).
+	 */
+	if (___cds_wfcq_append(__cds_wfcq_head_cast(dest_q_head), dest_q_tail,
+			head, tail))
+		return CDS_WFCQ_RET_DEST_NON_EMPTY;
+	else
+		return CDS_WFCQ_RET_DEST_EMPTY;
+}
+
+/*
+ * __cds_wfcq_splice_blocking: enqueue all src_q nodes at the end of dest_q.
+ *
+ * Dequeue all nodes from src_q.
+ * dest_q must be already initialized.
+ * Mutual exclusion for src_q should be ensured by the caller as
+ * specified in the "Synchronisation table".
+ * Returns enum cds_wfcq_ret which indicates the state of the src or
+ * dest queue. Never returns CDS_WFCQ_RET_WOULDBLOCK.
+ */
+static inline enum cds_wfcq_ret
+___cds_wfcq_splice_blocking(
+		cds_wfcq_head_ptr_t dest_q_head,
+		struct cds_wfcq_tail *dest_q_tail,
+		cds_wfcq_head_ptr_t src_q_head,
+		struct cds_wfcq_tail *src_q_tail)
+{
+	return ___cds_wfcq_splice(dest_q_head, dest_q_tail,
+		src_q_head, src_q_tail, 1);
+}
+
+/*
+ * __cds_wfcq_splice_nonblocking: enqueue all src_q nodes at the end of dest_q.
+ *
+ * Same as __cds_wfcq_splice_blocking, but returns
+ * CDS_WFCQ_RET_WOULDBLOCK if it needs to block.
+ */
+static inline enum cds_wfcq_ret
+___cds_wfcq_splice_nonblocking(
+		cds_wfcq_head_ptr_t dest_q_head,
+		struct cds_wfcq_tail *dest_q_tail,
+		cds_wfcq_head_ptr_t src_q_head,
+		struct cds_wfcq_tail *src_q_tail)
+{
+	return ___cds_wfcq_splice(dest_q_head, dest_q_tail,
+		src_q_head, src_q_tail, 0);
+}
+
+/*
+ * cds_wfcq_dequeue_with_state_blocking: dequeue a node from a wait-free queue.
+ *
+ * Content written into the node before enqueue is guaranteed to be
+ * consistent, but no other memory ordering is ensured.
+ * Mutual exclusion with cds_wfcq_splice_blocking and dequeue lock is
+ * ensured.
+ * It is valid to reuse and free a dequeued node immediately.
+ */
+static inline struct cds_wfcq_node *
+_cds_wfcq_dequeue_with_state_blocking(struct cds_wfcq_head *head,
+		struct cds_wfcq_tail *tail, int *state)
+{
+	struct cds_wfcq_node *retval;
+
+	_cds_wfcq_dequeue_lock(head, tail);
+	retval = ___cds_wfcq_dequeue_with_state_blocking(cds_wfcq_head_cast(head),
+			tail, state);
+	_cds_wfcq_dequeue_unlock(head, tail);
+	return retval;
+}
+
+/*
+ * cds_wfcq_dequeue_blocking: dequeue node from queue.
+ *
+ * Same as cds_wfcq_dequeue_blocking, but without saving state.
+ */
+static inline struct cds_wfcq_node *
+_cds_wfcq_dequeue_blocking(struct cds_wfcq_head *head,
+		struct cds_wfcq_tail *tail)
+{
+	return _cds_wfcq_dequeue_with_state_blocking(head, tail, NULL);
+}
+
+/*
+ * cds_wfcq_splice_blocking: enqueue all src_q nodes at the end of dest_q.
+ *
+ * Dequeue all nodes from src_q.
+ * dest_q must be already initialized.
+ * Content written into the node before enqueue is guaranteed to be
+ * consistent, but no other memory ordering is ensured.
+ * Mutual exclusion with cds_wfcq_dequeue_blocking and dequeue lock is
+ * ensured.
+ * Returns enum cds_wfcq_ret which indicates the state of the src or
+ * dest queue. Never returns CDS_WFCQ_RET_WOULDBLOCK.
+ */
+static inline enum cds_wfcq_ret
+_cds_wfcq_splice_blocking(
+		struct cds_wfcq_head *dest_q_head,
+		struct cds_wfcq_tail *dest_q_tail,
+		struct cds_wfcq_head *src_q_head,
+		struct cds_wfcq_tail *src_q_tail)
+{
+	enum cds_wfcq_ret ret;
+
+	_cds_wfcq_dequeue_lock(src_q_head, src_q_tail);
+	ret = ___cds_wfcq_splice_blocking(cds_wfcq_head_cast(dest_q_head), dest_q_tail,
+			cds_wfcq_head_cast(src_q_head), src_q_tail);
+	_cds_wfcq_dequeue_unlock(src_q_head, src_q_tail);
+	return ret;
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _URCU_WFCQUEUE_STATIC_H */
diff --git a/contrib/userspace-rcu/static-wfstack.h b/contrib/userspace-rcu/static-wfstack.h
new file mode 100644
index 00000000000..29b81c3aac3
--- /dev/null
+++ b/contrib/userspace-rcu/static-wfstack.h
@@ -0,0 +1,455 @@
+#ifndef _URCU_STATIC_WFSTACK_H
+#define _URCU_STATIC_WFSTACK_H
+
+/*
+ * urcu/static/wfstack.h
+ *
+ * Userspace RCU library - Stack with with wait-free push, blocking traversal.
+ *
+ * TO BE INCLUDED ONLY IN LGPL-COMPATIBLE CODE. See urcu/wfstack.h for
+ * linking dynamically with the userspace rcu library.
+ *
+ * Copyright 2010-2012 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+/* Adapted from userspace-rcu 0.10 because version 0.7 doesn't support a stack
+ * without mutex. */
+
+#include <pthread.h>
+#include <assert.h>
+#include <poll.h>
+#include <stdbool.h>
+#include <urcu/compiler.h>
+#include <urcu/uatomic.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#define CDS_WFS_END			((void *) 0x1UL)
+#define CDS_WFS_ADAPT_ATTEMPTS		10	/* Retry if being set */
+#define CDS_WFS_WAIT			10	/* Wait 10 ms if being set */
+
+/*
+ * Stack with wait-free push, blocking traversal.
+ *
+ * Stack implementing push, pop, pop_all operations, as well as iterator
+ * on the stack head returned by pop_all.
+ *
+ * Wait-free operations: cds_wfs_push, __cds_wfs_pop_all, cds_wfs_empty,
+ *                       cds_wfs_first.
+ * Blocking operations: cds_wfs_pop, cds_wfs_pop_all, cds_wfs_next,
+ *                      iteration on stack head returned by pop_all.
+ *
+ * Synchronization table:
+ *
+ * External synchronization techniques described in the API below is
+ * required between pairs marked with "X". No external synchronization
+ * required between pairs marked with "-".
+ *
+ *                      cds_wfs_push  __cds_wfs_pop  __cds_wfs_pop_all
+ * cds_wfs_push               -              -                  -
+ * __cds_wfs_pop              -              X                  X
+ * __cds_wfs_pop_all          -              X                  -
+ *
+ * cds_wfs_pop and cds_wfs_pop_all use an internal mutex to provide
+ * synchronization.
+ */
+
+/*
+ * cds_wfs_node_init: initialize wait-free stack node.
+ */
+static inline
+void _cds_wfs_node_init(struct cds_wfs_node *node)
+{
+	node->next = NULL;
+}
+
+/*
+ * __cds_wfs_init: initialize wait-free stack. Don't pair with
+ * any destroy function.
+ */
+static inline void ___cds_wfs_init(struct __cds_wfs_stack *s)
+{
+	s->head = CDS_WFS_END;
+}
+
+/*
+ * cds_wfs_init: initialize wait-free stack. Pair with
+ * cds_wfs_destroy().
+ */
+static inline
+void _cds_wfs_init(struct cds_wfs_stack *s)
+{
+	int ret;
+
+	s->head = CDS_WFS_END;
+	ret = pthread_mutex_init(&s->lock, NULL);
+	assert(!ret);
+}
+
+/*
+ * cds_wfs_destroy: destroy wait-free stack. Pair with
+ * cds_wfs_init().
+ */
+static inline
+void _cds_wfs_destroy(struct cds_wfs_stack *s)
+{
+	int ret = pthread_mutex_destroy(&s->lock);
+	assert(!ret);
+}
+
+static inline bool ___cds_wfs_end(void *node)
+{
+	return node == CDS_WFS_END;
+}
+
+/*
+ * cds_wfs_empty: return whether wait-free stack is empty.
+ *
+ * No memory barrier is issued. No mutual exclusion is required.
+ */
+static inline bool _cds_wfs_empty(cds_wfs_stack_ptr_t u_stack)
+{
+	struct __cds_wfs_stack *s = u_stack._s;
+
+	return ___cds_wfs_end(CMM_LOAD_SHARED(s->head));
+}
+
+/*
+ * cds_wfs_push: push a node into the stack.
+ *
+ * Issues a full memory barrier before push. No mutual exclusion is
+ * required.
+ *
+ * Returns 0 if the stack was empty prior to adding the node.
+ * Returns non-zero otherwise.
+ */
+static inline
+int _cds_wfs_push(cds_wfs_stack_ptr_t u_stack, struct cds_wfs_node *node)
+{
+	struct __cds_wfs_stack *s = u_stack._s;
+	struct cds_wfs_head *old_head, *new_head;
+
+	assert(node->next == NULL);
+	new_head = caa_container_of(node, struct cds_wfs_head, node);
+	/*
+	 * uatomic_xchg() implicit memory barrier orders earlier stores
+	 * to node (setting it to NULL) before publication.
+	 */
+	old_head = uatomic_xchg(&s->head, new_head);
+	/*
+	 * At this point, dequeuers see a NULL node->next, they should
+	 * busy-wait until node->next is set to old_head.
+	 */
+	CMM_STORE_SHARED(node->next, &old_head->node);
+	return !___cds_wfs_end(old_head);
+}
+
+/*
+ * Waiting for push to complete enqueue and return the next node.
+ */
+static inline struct cds_wfs_node *
+___cds_wfs_node_sync_next(struct cds_wfs_node *node, int blocking)
+{
+	struct cds_wfs_node *next;
+	int attempt = 0;
+
+	/*
+	 * Adaptative busy-looping waiting for push to complete.
+	 */
+	while ((next = CMM_LOAD_SHARED(node->next)) == NULL) {
+		if (!blocking)
+			return CDS_WFS_WOULDBLOCK;
+		if (++attempt >= CDS_WFS_ADAPT_ATTEMPTS) {
+			(void) poll(NULL, 0, CDS_WFS_WAIT);	/* Wait for 10ms */
+			attempt = 0;
+		} else {
+			caa_cpu_relax();
+		}
+	}
+
+	return next;
+}
+
+static inline
+struct cds_wfs_node *
+___cds_wfs_pop(cds_wfs_stack_ptr_t u_stack, int *state, int blocking)
+{
+	struct cds_wfs_head *head, *new_head;
+	struct cds_wfs_node *next;
+	struct __cds_wfs_stack *s = u_stack._s;
+
+	if (state)
+		*state = 0;
+	for (;;) {
+		head = CMM_LOAD_SHARED(s->head);
+		if (___cds_wfs_end(head)) {
+			return NULL;
+		}
+		next = ___cds_wfs_node_sync_next(&head->node, blocking);
+		if (!blocking && next == CDS_WFS_WOULDBLOCK) {
+			return CDS_WFS_WOULDBLOCK;
+		}
+		new_head = caa_container_of(next, struct cds_wfs_head, node);
+		if (uatomic_cmpxchg(&s->head, head, new_head) == head) {
+			if (state && ___cds_wfs_end(new_head))
+				*state |= CDS_WFS_STATE_LAST;
+			return &head->node;
+		}
+		if (!blocking) {
+			return CDS_WFS_WOULDBLOCK;
+		}
+		/* busy-loop if head changed under us */
+	}
+}
+
+/*
+ * __cds_wfs_pop_with_state_blocking: pop a node from the stack, with state.
+ *
+ * Returns NULL if stack is empty.
+ *
+ * __cds_wfs_pop_blocking needs to be synchronized using one of the
+ * following techniques:
+ *
+ * 1) Calling __cds_wfs_pop_blocking under rcu read lock critical
+ *    section. The caller must wait for a grace period to pass before
+ *    freeing the returned node or modifying the cds_wfs_node structure.
+ * 2) Using mutual exclusion (e.g. mutexes) to protect
+ *     __cds_wfs_pop_blocking and __cds_wfs_pop_all callers.
+ * 3) Ensuring that only ONE thread can call __cds_wfs_pop_blocking()
+ *    and __cds_wfs_pop_all(). (multi-provider/single-consumer scheme).
+ *
+ * "state" saves state flags atomically sampled with pop operation.
+ */
+static inline
+struct cds_wfs_node *
+___cds_wfs_pop_with_state_blocking(cds_wfs_stack_ptr_t u_stack, int *state)
+{
+	return ___cds_wfs_pop(u_stack, state, 1);
+}
+
+static inline
+struct cds_wfs_node *
+___cds_wfs_pop_blocking(cds_wfs_stack_ptr_t u_stack)
+{
+	return ___cds_wfs_pop_with_state_blocking(u_stack, NULL);
+}
+
+/*
+ * __cds_wfs_pop_with_state_nonblocking: pop a node from the stack.
+ *
+ * Same as __cds_wfs_pop_with_state_blocking, but returns
+ * CDS_WFS_WOULDBLOCK if it needs to block.
+ *
+ * "state" saves state flags atomically sampled with pop operation.
+ */
+static inline
+struct cds_wfs_node *
+___cds_wfs_pop_with_state_nonblocking(cds_wfs_stack_ptr_t u_stack, int *state)
+{
+	return ___cds_wfs_pop(u_stack, state, 0);
+}
+
+/*
+ * __cds_wfs_pop_nonblocking: pop a node from the stack.
+ *
+ * Same as __cds_wfs_pop_blocking, but returns CDS_WFS_WOULDBLOCK if
+ * it needs to block.
+ */
+static inline
+struct cds_wfs_node *
+___cds_wfs_pop_nonblocking(cds_wfs_stack_ptr_t u_stack)
+{
+	return ___cds_wfs_pop_with_state_nonblocking(u_stack, NULL);
+}
+
+/*
+ * __cds_wfs_pop_all: pop all nodes from a stack.
+ *
+ * __cds_wfs_pop_all does not require any synchronization with other
+ * push, nor with other __cds_wfs_pop_all, but requires synchronization
+ * matching the technique used to synchronize __cds_wfs_pop_blocking:
+ *
+ * 1) If __cds_wfs_pop_blocking is called under rcu read lock critical
+ *    section, both __cds_wfs_pop_blocking and cds_wfs_pop_all callers
+ *    must wait for a grace period to pass before freeing the returned
+ *    node or modifying the cds_wfs_node structure. However, no RCU
+ *    read-side critical section is needed around __cds_wfs_pop_all.
+ * 2) Using mutual exclusion (e.g. mutexes) to protect
+ *     __cds_wfs_pop_blocking and __cds_wfs_pop_all callers.
+ * 3) Ensuring that only ONE thread can call __cds_wfs_pop_blocking()
+ *    and __cds_wfs_pop_all(). (multi-provider/single-consumer scheme).
+ */
+static inline
+struct cds_wfs_head *
+___cds_wfs_pop_all(cds_wfs_stack_ptr_t u_stack)
+{
+	struct __cds_wfs_stack *s = u_stack._s;
+	struct cds_wfs_head *head;
+
+	/*
+	 * Implicit memory barrier after uatomic_xchg() matches implicit
+	 * memory barrier before uatomic_xchg() in cds_wfs_push. It
+	 * ensures that all nodes of the returned list are consistent.
+	 * There is no need to issue memory barriers when iterating on
+	 * the returned list, because the full memory barrier issued
+	 * prior to each uatomic_cmpxchg, which each write to head, are
+	 * taking care to order writes to each node prior to the full
+	 * memory barrier after this uatomic_xchg().
+	 */
+	head = uatomic_xchg(&s->head, CDS_WFS_END);
+	if (___cds_wfs_end(head))
+		return NULL;
+	return head;
+}
+
+/*
+ * cds_wfs_pop_lock: lock stack pop-protection mutex.
+ */
+static inline void _cds_wfs_pop_lock(struct cds_wfs_stack *s)
+{
+	int ret;
+
+	ret = pthread_mutex_lock(&s->lock);
+	assert(!ret);
+}
+
+/*
+ * cds_wfs_pop_unlock: unlock stack pop-protection mutex.
+ */
+static inline void _cds_wfs_pop_unlock(struct cds_wfs_stack *s)
+{
+	int ret;
+
+	ret = pthread_mutex_unlock(&s->lock);
+	assert(!ret);
+}
+
+/*
+ * Call __cds_wfs_pop_with_state_blocking with an internal pop mutex held.
+ */
+static inline
+struct cds_wfs_node *
+_cds_wfs_pop_with_state_blocking(struct cds_wfs_stack *s, int *state)
+{
+	struct cds_wfs_node *retnode;
+
+	_cds_wfs_pop_lock(s);
+	retnode = ___cds_wfs_pop_with_state_blocking(s, state);
+	_cds_wfs_pop_unlock(s);
+	return retnode;
+}
+
+/*
+ * Call _cds_wfs_pop_with_state_blocking without saving any state.
+ */
+static inline
+struct cds_wfs_node *
+_cds_wfs_pop_blocking(struct cds_wfs_stack *s)
+{
+	return _cds_wfs_pop_with_state_blocking(s, NULL);
+}
+
+/*
+ * Call __cds_wfs_pop_all with an internal pop mutex held.
+ */
+static inline
+struct cds_wfs_head *
+_cds_wfs_pop_all_blocking(struct cds_wfs_stack *s)
+{
+	struct cds_wfs_head *rethead;
+
+	_cds_wfs_pop_lock(s);
+	rethead = ___cds_wfs_pop_all(s);
+	_cds_wfs_pop_unlock(s);
+	return rethead;
+}
+
+/*
+ * cds_wfs_first: get first node of a popped stack.
+ *
+ * Content written into the node before enqueue is guaranteed to be
+ * consistent, but no other memory ordering is ensured.
+ *
+ * Used by for-like iteration macros in urcu/wfstack.h:
+ * cds_wfs_for_each_blocking()
+ * cds_wfs_for_each_blocking_safe()
+ *
+ * Returns NULL if popped stack is empty, top stack node otherwise.
+ */
+static inline struct cds_wfs_node *
+_cds_wfs_first(struct cds_wfs_head *head)
+{
+	if (___cds_wfs_end(head))
+		return NULL;
+	return &head->node;
+}
+
+static inline struct cds_wfs_node *
+___cds_wfs_next(struct cds_wfs_node *node, int blocking)
+{
+	struct cds_wfs_node *next;
+
+	next = ___cds_wfs_node_sync_next(node, blocking);
+	/*
+	 * CDS_WFS_WOULDBLOCK != CSD_WFS_END, so we can check for end
+	 * even if ___cds_wfs_node_sync_next returns CDS_WFS_WOULDBLOCK,
+	 * and still return CDS_WFS_WOULDBLOCK.
+	 */
+	if (___cds_wfs_end(next))
+		return NULL;
+	return next;
+}
+
+/*
+ * cds_wfs_next_blocking: get next node of a popped stack.
+ *
+ * Content written into the node before enqueue is guaranteed to be
+ * consistent, but no other memory ordering is ensured.
+ *
+ * Used by for-like iteration macros in urcu/wfstack.h:
+ * cds_wfs_for_each_blocking()
+ * cds_wfs_for_each_blocking_safe()
+ *
+ * Returns NULL if reached end of popped stack, non-NULL next stack
+ * node otherwise.
+ */
+static inline struct cds_wfs_node *
+_cds_wfs_next_blocking(struct cds_wfs_node *node)
+{
+	return ___cds_wfs_next(node, 1);
+}
+
+
+/*
+ * cds_wfs_next_nonblocking: get next node of a popped stack.
+ *
+ * Same as cds_wfs_next_blocking, but returns CDS_WFS_WOULDBLOCK if it
+ * needs to block.
+ */
+static inline struct cds_wfs_node *
+_cds_wfs_next_nonblocking(struct cds_wfs_node *node)
+{
+	return ___cds_wfs_next(node, 0);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _URCU_STATIC_WFSTACK_H */
diff --git a/contrib/userspace-rcu/wfcqueue.h b/contrib/userspace-rcu/wfcqueue.h
new file mode 100644
index 00000000000..0292585ac79
--- /dev/null
+++ b/contrib/userspace-rcu/wfcqueue.h
@@ -0,0 +1,216 @@
+#ifndef _URCU_WFCQUEUE_H
+#define _URCU_WFCQUEUE_H
+
+/*
+ * urcu/wfcqueue.h
+ *
+ * Userspace RCU library - Concurrent Queue with Wait-Free Enqueue/Blocking Dequeue
+ *
+ * Copyright 2010-2012 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+ * Copyright 2011-2012 - Lai Jiangshan <laijs@cn.fujitsu.com>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+/* Adapted from userspace-rcu 0.10 because version 0.7 doesn't contain it.
+ * The non-LGPL section has been removed. */
+
+#include <pthread.h>
+#include <assert.h>
+#include <stdbool.h>
+#include <urcu/compiler.h>
+#include <urcu/arch.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/*
+ * Concurrent queue with wait-free enqueue/blocking dequeue.
+ *
+ * This queue has been designed and implemented collaboratively by
+ * Mathieu Desnoyers and Lai Jiangshan. Inspired from
+ * half-wait-free/half-blocking queue implementation done by Paul E.
+ * McKenney.
+ */
+
+#define CDS_WFCQ_WOULDBLOCK	((struct cds_wfcq_node *) -1UL)
+
+enum cds_wfcq_ret {
+	CDS_WFCQ_RET_WOULDBLOCK =	-1,
+	CDS_WFCQ_RET_DEST_EMPTY =	0,
+	CDS_WFCQ_RET_DEST_NON_EMPTY =	1,
+	CDS_WFCQ_RET_SRC_EMPTY = 	2,
+};
+
+enum cds_wfcq_state {
+	CDS_WFCQ_STATE_LAST =		(1U << 0),
+};
+
+struct cds_wfcq_node {
+	struct cds_wfcq_node *next;
+};
+
+/*
+ * Do not put head and tail on the same cache-line if concurrent
+ * enqueue/dequeue are expected from many CPUs. This eliminates
+ * false-sharing between enqueue and dequeue.
+ */
+struct __cds_wfcq_head {
+	struct cds_wfcq_node node;
+};
+
+struct cds_wfcq_head {
+	struct cds_wfcq_node node;
+	pthread_mutex_t lock;
+};
+
+#ifndef __cplusplus
+/*
+ * The transparent union allows calling functions that work on both
+ * struct cds_wfcq_head and struct __cds_wfcq_head on any of those two
+ * types.
+ */
+typedef union {
+	struct __cds_wfcq_head *_h;
+	struct cds_wfcq_head *h;
+} __attribute__((__transparent_union__)) cds_wfcq_head_ptr_t;
+
+/*
+ * This static inline is only present for compatibility with C++. It is
+ * effect-less in C.
+ */
+static inline struct __cds_wfcq_head *__cds_wfcq_head_cast(struct __cds_wfcq_head *head)
+{
+	return head;
+}
+
+/*
+ * This static inline is only present for compatibility with C++. It is
+ * effect-less in C.
+ */
+static inline struct cds_wfcq_head *cds_wfcq_head_cast(struct cds_wfcq_head *head)
+{
+	return head;
+}
+#else /* #ifndef __cplusplus */
+
+/* C++ ignores transparent union. */
+typedef union {
+	struct __cds_wfcq_head *_h;
+	struct cds_wfcq_head *h;
+} cds_wfcq_head_ptr_t;
+
+/* C++ ignores transparent union. Requires an explicit conversion. */
+static inline cds_wfcq_head_ptr_t __cds_wfcq_head_cast(struct __cds_wfcq_head *head)
+{
+	cds_wfcq_head_ptr_t ret = { ._h = head };
+	return ret;
+}
+/* C++ ignores transparent union. Requires an explicit conversion. */
+static inline cds_wfcq_head_ptr_t cds_wfcq_head_cast(struct cds_wfcq_head *head)
+{
+	cds_wfcq_head_ptr_t ret = { .h = head };
+	return ret;
+}
+#endif /* #else #ifndef __cplusplus */
+
+struct cds_wfcq_tail {
+	struct cds_wfcq_node *p;
+};
+
+#include "static-wfcqueue.h"
+
+#define cds_wfcq_node_init		_cds_wfcq_node_init
+#define cds_wfcq_init			_cds_wfcq_init
+#define __cds_wfcq_init			___cds_wfcq_init
+#define cds_wfcq_destroy		_cds_wfcq_destroy
+#define cds_wfcq_empty			_cds_wfcq_empty
+#define cds_wfcq_enqueue		_cds_wfcq_enqueue
+
+/* Dequeue locking */
+#define cds_wfcq_dequeue_lock		_cds_wfcq_dequeue_lock
+#define cds_wfcq_dequeue_unlock		_cds_wfcq_dequeue_unlock
+
+/* Locking performed within cds_wfcq calls. */
+#define cds_wfcq_dequeue_blocking	_cds_wfcq_dequeue_blocking
+#define cds_wfcq_dequeue_with_state_blocking	\
+					_cds_wfcq_dequeue_with_state_blocking
+#define cds_wfcq_splice_blocking	_cds_wfcq_splice_blocking
+#define cds_wfcq_first_blocking		_cds_wfcq_first_blocking
+#define cds_wfcq_next_blocking		_cds_wfcq_next_blocking
+
+/* Locking ensured by caller by holding cds_wfcq_dequeue_lock() */
+#define __cds_wfcq_dequeue_blocking	___cds_wfcq_dequeue_blocking
+#define __cds_wfcq_dequeue_with_state_blocking	\
+					___cds_wfcq_dequeue_with_state_blocking
+#define __cds_wfcq_splice_blocking	___cds_wfcq_splice_blocking
+#define __cds_wfcq_first_blocking	___cds_wfcq_first_blocking
+#define __cds_wfcq_next_blocking	___cds_wfcq_next_blocking
+
+/*
+ * Locking ensured by caller by holding cds_wfcq_dequeue_lock().
+ * Non-blocking: deque, first, next return CDS_WFCQ_WOULDBLOCK if they
+ * need to block. splice returns nonzero if it needs to block.
+ */
+#define __cds_wfcq_dequeue_nonblocking	___cds_wfcq_dequeue_nonblocking
+#define __cds_wfcq_dequeue_with_state_nonblocking	\
+				___cds_wfcq_dequeue_with_state_nonblocking
+#define __cds_wfcq_splice_nonblocking	___cds_wfcq_splice_nonblocking
+#define __cds_wfcq_first_nonblocking	___cds_wfcq_first_nonblocking
+#define __cds_wfcq_next_nonblocking	___cds_wfcq_next_nonblocking
+
+/*
+ * __cds_wfcq_for_each_blocking: Iterate over all nodes in a queue,
+ * without dequeuing them.
+ * @head: head of the queue (struct cds_wfcq_head or __cds_wfcq_head pointer).
+ * @tail: tail of the queue (struct cds_wfcq_tail pointer).
+ * @node: iterator on the queue (struct cds_wfcq_node pointer).
+ *
+ * Content written into each node before enqueue is guaranteed to be
+ * consistent, but no other memory ordering is ensured.
+ * Dequeue/splice/iteration mutual exclusion should be ensured by the
+ * caller.
+ */
+#define __cds_wfcq_for_each_blocking(head, tail, node)		\
+	for (node = __cds_wfcq_first_blocking(head, tail);	\
+		node != NULL;					\
+		node = __cds_wfcq_next_blocking(head, tail, node))
+
+/*
+ * __cds_wfcq_for_each_blocking_safe: Iterate over all nodes in a queue,
+ * without dequeuing them. Safe against deletion.
+ * @head: head of the queue (struct cds_wfcq_head or __cds_wfcq_head pointer).
+ * @tail: tail of the queue (struct cds_wfcq_tail pointer).
+ * @node: iterator on the queue (struct cds_wfcq_node pointer).
+ * @n: struct cds_wfcq_node pointer holding the next pointer (used
+ *     internally).
+ *
+ * Content written into each node before enqueue is guaranteed to be
+ * consistent, but no other memory ordering is ensured.
+ * Dequeue/splice/iteration mutual exclusion should be ensured by the
+ * caller.
+ */
+#define __cds_wfcq_for_each_blocking_safe(head, tail, node, n)		       \
+	for (node = __cds_wfcq_first_blocking(head, tail),		       \
+			n = (node ? __cds_wfcq_next_blocking(head, tail, node) : NULL); \
+		node != NULL;						       \
+		node = n, n = (node ? __cds_wfcq_next_blocking(head, tail, node) : NULL))
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _URCU_WFCQUEUE_H */
diff --git a/contrib/userspace-rcu/wfstack.h b/contrib/userspace-rcu/wfstack.h
new file mode 100644
index 00000000000..738fd1cfd33
--- /dev/null
+++ b/contrib/userspace-rcu/wfstack.h
@@ -0,0 +1,178 @@
+#ifndef _URCU_WFSTACK_H
+#define _URCU_WFSTACK_H
+
+/*
+ * urcu/wfstack.h
+ *
+ * Userspace RCU library - Stack with wait-free push, blocking traversal.
+ *
+ * Copyright 2010-2012 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+/* Adapted from userspace-rcu 0.10 because version 0.7 doesn't support a stack
+ * without mutex. The non-LGPL section has been removed. */
+
+#include <pthread.h>
+#include <assert.h>
+#include <stdbool.h>
+#include <urcu/compiler.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/*
+ * Stack with wait-free push, blocking traversal.
+ *
+ * Stack implementing push, pop, pop_all operations, as well as iterator
+ * on the stack head returned by pop_all.
+ *
+ * Wait-free operations: cds_wfs_push, __cds_wfs_pop_all, cds_wfs_empty,
+ *                       cds_wfs_first.
+ * Blocking operations: cds_wfs_pop, cds_wfs_pop_all, cds_wfs_next,
+ *                      iteration on stack head returned by pop_all.
+ *
+ * Synchronization table:
+ *
+ * External synchronization techniques described in the API below is
+ * required between pairs marked with "X". No external synchronization
+ * required between pairs marked with "-".
+ *
+ *                      cds_wfs_push  __cds_wfs_pop  __cds_wfs_pop_all
+ * cds_wfs_push               -              -                  -
+ * __cds_wfs_pop              -              X                  X
+ * __cds_wfs_pop_all          -              X                  -
+ *
+ * cds_wfs_pop and cds_wfs_pop_all use an internal mutex to provide
+ * synchronization.
+ */
+
+#define CDS_WFS_WOULDBLOCK	((void *) -1UL)
+
+enum cds_wfs_state {
+	CDS_WFS_STATE_LAST =		(1U << 0),
+};
+
+/*
+ * struct cds_wfs_node is returned by __cds_wfs_pop, and also used as
+ * iterator on stack. It is not safe to dereference the node next
+ * pointer when returned by __cds_wfs_pop_blocking.
+ */
+struct cds_wfs_node {
+	struct cds_wfs_node *next;
+};
+
+/*
+ * struct cds_wfs_head is returned by __cds_wfs_pop_all, and can be used
+ * to begin iteration on the stack. "node" needs to be the first field of
+ * cds_wfs_head, so the end-of-stack pointer value can be used for both
+ * types.
+ */
+struct cds_wfs_head {
+	struct cds_wfs_node node;
+};
+
+struct __cds_wfs_stack {
+	struct cds_wfs_head *head;
+};
+
+struct cds_wfs_stack {
+	struct cds_wfs_head *head;
+	pthread_mutex_t lock;
+};
+
+/*
+ * The transparent union allows calling functions that work on both
+ * struct cds_wfs_stack and struct __cds_wfs_stack on any of those two
+ * types.
+ */
+typedef union {
+	struct __cds_wfs_stack *_s;
+	struct cds_wfs_stack *s;
+} __attribute__((__transparent_union__)) cds_wfs_stack_ptr_t;
+
+#include "static-wfstack.h"
+
+#define cds_wfs_node_init		_cds_wfs_node_init
+#define cds_wfs_init			_cds_wfs_init
+#define cds_wfs_destroy			_cds_wfs_destroy
+#define __cds_wfs_init			___cds_wfs_init
+#define cds_wfs_empty			_cds_wfs_empty
+#define cds_wfs_push			_cds_wfs_push
+
+/* Locking performed internally */
+#define cds_wfs_pop_blocking		_cds_wfs_pop_blocking
+#define cds_wfs_pop_with_state_blocking	_cds_wfs_pop_with_state_blocking
+#define cds_wfs_pop_all_blocking	_cds_wfs_pop_all_blocking
+
+/*
+ * For iteration on cds_wfs_head returned by __cds_wfs_pop_all or
+ * cds_wfs_pop_all_blocking.
+ */
+#define cds_wfs_first			_cds_wfs_first
+#define cds_wfs_next_blocking		_cds_wfs_next_blocking
+#define cds_wfs_next_nonblocking	_cds_wfs_next_nonblocking
+
+/* Pop locking with internal mutex */
+#define cds_wfs_pop_lock		_cds_wfs_pop_lock
+#define cds_wfs_pop_unlock		_cds_wfs_pop_unlock
+
+/* Synchronization ensured by the caller. See synchronization table. */
+#define __cds_wfs_pop_blocking		___cds_wfs_pop_blocking
+#define __cds_wfs_pop_with_state_blocking	\
+					___cds_wfs_pop_with_state_blocking
+#define __cds_wfs_pop_nonblocking	___cds_wfs_pop_nonblocking
+#define __cds_wfs_pop_with_state_nonblocking	\
+					___cds_wfs_pop_with_state_nonblocking
+#define __cds_wfs_pop_all		___cds_wfs_pop_all
+
+#ifdef __cplusplus
+}
+#endif
+
+/*
+ * cds_wfs_for_each_blocking: Iterate over all nodes returned by
+ * __cds_wfs_pop_all().
+ * @head: head of the queue (struct cds_wfs_head pointer).
+ * @node: iterator (struct cds_wfs_node pointer).
+ *
+ * Content written into each node before enqueue is guaranteed to be
+ * consistent, but no other memory ordering is ensured.
+ */
+#define cds_wfs_for_each_blocking(head, node)			\
+	for (node = cds_wfs_first(head);			\
+		node != NULL;					\
+		node = cds_wfs_next_blocking(node))
+
+/*
+ * cds_wfs_for_each_blocking_safe: Iterate over all nodes returned by
+ * __cds_wfs_pop_all(). Safe against deletion.
+ * @head: head of the queue (struct cds_wfs_head pointer).
+ * @node: iterator (struct cds_wfs_node pointer).
+ * @n: struct cds_wfs_node pointer holding the next pointer (used
+ *     internally).
+ *
+ * Content written into each node before enqueue is guaranteed to be
+ * consistent, but no other memory ordering is ensured.
+ */
+#define cds_wfs_for_each_blocking_safe(head, node, n)			   \
+	for (node = cds_wfs_first(head),				   \
+			n = (node ? cds_wfs_next_blocking(node) : NULL);   \
+		node != NULL;						   \
+		node = n, n = (node ? cds_wfs_next_blocking(node) : NULL))
+
+#endif /* _URCU_WFSTACK_H */
diff --git a/contrib/uuid/clear.c b/contrib/uuid/clear.c
deleted file mode 100644
index 2d91fee9399..00000000000
--- a/contrib/uuid/clear.c
+++ /dev/null
@@ -1,43 +0,0 @@
-/*
- * clear.c -- Clear a UUID
- *
- * Copyright (C) 1996, 1997 Theodore Ts'o.
- *
- * %Begin-Header%
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, and the entire permission notice in its entirety,
- *    including the disclaimer of warranties.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- * 3. The name of the author may not be used to endorse or promote
- *    products derived from this software without specific prior
- *    written permission.
- *
- * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
- * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
- * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
- * WHICH ARE HEREBY DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
- * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
- * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
- * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
- * DAMAGE.
- * %End-Header%
- */
-
-#include "string.h"
-
-#include "uuidP.h"
-
-void uuid_clear(uuid_t uu)
-{
-	memset(uu, 0, 16);
-}
-
diff --git a/contrib/uuid/compare.c b/contrib/uuid/compare.c
deleted file mode 100644
index f28a72678cf..00000000000
--- a/contrib/uuid/compare.c
+++ /dev/null
@@ -1,55 +0,0 @@
-/*
- * compare.c --- compare whether or not two UUID's are the same
- *
- * Returns 0 if the two UUID's are different, and 1 if they are the same.
- *
- * Copyright (C) 1996, 1997 Theodore Ts'o.
- *
- * %Begin-Header%
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, and the entire permission notice in its entirety,
- *    including the disclaimer of warranties.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- * 3. The name of the author may not be used to endorse or promote
- *    products derived from this software without specific prior
- *    written permission.
- *
- * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
- * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
- * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
- * WHICH ARE HEREBY DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
- * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
- * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
- * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
- * DAMAGE.
- * %End-Header%
- */
-
-#include "uuidP.h"
-#include <string.h>
-
-#define UUCMP(u1,u2) if (u1 != u2) return((u1 < u2) ? -1 : 1);
-
-int uuid_compare(const uuid_t uu1, const uuid_t uu2)
-{
-	struct uuid	uuid1, uuid2;
-
-	uuid_unpack(uu1, &uuid1);
-	uuid_unpack(uu2, &uuid2);
-
-	UUCMP(uuid1.time_low, uuid2.time_low);
-	UUCMP(uuid1.time_mid, uuid2.time_mid);
-	UUCMP(uuid1.time_hi_and_version, uuid2.time_hi_and_version);
-	UUCMP(uuid1.clock_seq, uuid2.clock_seq);
-	return memcmp(uuid1.node, uuid2.node, 6);
-}
-
diff --git a/contrib/uuid/copy.c b/contrib/uuid/copy.c
deleted file mode 100644
index ead33aa26e8..00000000000
--- a/contrib/uuid/copy.c
+++ /dev/null
@@ -1,45 +0,0 @@
-/*
- * copy.c --- copy UUIDs
- *
- * Copyright (C) 1996, 1997 Theodore Ts'o.
- *
- * %Begin-Header%
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, and the entire permission notice in its entirety,
- *    including the disclaimer of warranties.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- * 3. The name of the author may not be used to endorse or promote
- *    products derived from this software without specific prior
- *    written permission.
- *
- * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
- * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
- * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
- * WHICH ARE HEREBY DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
- * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
- * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
- * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
- * DAMAGE.
- * %End-Header%
- */
-
-#include "uuidP.h"
-
-void uuid_copy(uuid_t dst, const uuid_t src)
-{
-	unsigned char		*cp1;
-	const unsigned char	*cp2;
-	int			i;
-
-	for (i=0, cp1 = dst, cp2 = src; i < 16; i++)
-		*cp1++ = *cp2++;
-}
diff --git a/contrib/uuid/gen_uuid.c b/contrib/uuid/gen_uuid.c
deleted file mode 100644
index 4da0dc69b84..00000000000
--- a/contrib/uuid/gen_uuid.c
+++ /dev/null
@@ -1,686 +0,0 @@
-/*
- * gen_uuid.c --- generate a DCE-compatible uuid
- *
- * Copyright (C) 1996, 1997, 1998, 1999 Theodore Ts'o.
- *
- * %Begin-Header%
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, and the entire permission notice in its entirety,
- *    including the disclaimer of warranties.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- * 3. The name of the author may not be used to endorse or promote
- *    products derived from this software without specific prior
- *    written permission.
- *
- * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
- * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
- * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
- * WHICH ARE HEREBY DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
- * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
- * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
- * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
- * DAMAGE.
- * %End-Header%
- */
-
-/*
- * Force inclusion of SVID stuff since we need it if we're compiling in
- * gcc-wall wall mode
- */
-#define _SVID_SOURCE
-
-#include "config.h"
-#ifdef _WIN32
-#define _WIN32_WINNT 0x0500
-#include <windows.h>
-#define UUID MYUUID
-#endif
-#include <stdio.h>
-#ifdef HAVE_UNISTD_H
-#include <unistd.h>
-#endif
-#ifdef HAVE_STDLIB_H
-#include <stdlib.h>
-#endif
-#include <string.h>
-#include <fcntl.h>
-#include <errno.h>
-#include <sys/types.h>
-#ifdef HAVE_SYS_TIME_H
-#include <sys/time.h>
-#endif
-#include <sys/wait.h>
-#include <sys/stat.h>
-#ifdef HAVE_SYS_FILE_H
-#include <sys/file.h>
-#endif
-#ifdef HAVE_SYS_IOCTL_H
-#include <sys/ioctl.h>
-#endif
-#ifdef HAVE_SYS_SOCKET_H
-#include <sys/socket.h>
-#endif
-#ifdef HAVE_SYS_UN_H
-#include <sys/un.h>
-#endif
-#ifdef HAVE_SYS_SOCKIO_H
-#include <sys/sockio.h>
-#endif
-#ifdef HAVE_NET_IF_H
-#include <net/if.h>
-#endif
-#ifdef HAVE_NETINET_IN_H
-#include <netinet/in.h>
-#endif
-#ifdef HAVE_NET_IF_DL_H
-#include <net/if_dl.h>
-#endif
-#if defined(__linux__) && defined(HAVE_SYS_SYSCALL_H)
-#include <sys/syscall.h>
-#endif
-#ifdef HAVE_SYS_RESOURCE_H
-#include <sys/resource.h>
-#endif
-#include <limits.h>
-
-#include "uuidP.h"
-#include "uuidd.h"
-
-#ifdef HAVE_SRANDOM
-#define srand(x) 	srandom(x)
-#define rand() 		random()
-#endif
-
-#ifdef TLS
-#define THREAD_LOCAL static TLS
-#else
-#define THREAD_LOCAL static
-#endif
-
-#if defined(__linux__) && defined(__NR_gettid) && defined(HAVE_JRAND48)
-#define DO_JRAND_MIX
-THREAD_LOCAL unsigned short jrand_seed[3];
-#endif
-
-#ifndef OPEN_MAX
-#define OPEN_MAX 1024
-#endif
-
-#ifdef _WIN32
-static void gettimeofday (struct timeval *tv, void *dummy)
-{
-	FILETIME	ftime;
-	uint64_t	n;
-
-	GetSystemTimeAsFileTime (&ftime);
-	n = (((uint64_t) ftime.dwHighDateTime << 32)
-	     + (uint64_t) ftime.dwLowDateTime);
-	if (n) {
-		n /= 10;
-		n -= ((369 * 365 + 89) * (uint64_t) 86400) * 1000000;
-	}
-
-	tv->tv_sec = n / 1000000;
-	tv->tv_usec = n % 1000000;
-}
-
-static int getuid (void)
-{
-	return 1;
-}
-#endif
-
-static int get_random_fd(void)
-{
-	struct timeval	tv;
-	static int	fd = -2;
-	int		i;
-
-	if (fd == -2) {
-		gettimeofday(&tv, 0);
-#ifndef _WIN32
-		fd = open("/dev/urandom", O_RDONLY);
-		if (fd == -1)
-			fd = open("/dev/random", O_RDONLY | O_NONBLOCK);
-		if (fd >= 0) {
-			i = fcntl(fd, F_GETFD);
-			if (i >= 0)
-				fcntl(fd, F_SETFD, i | FD_CLOEXEC);
-		}
-#endif
-		srand((getpid() << 16) ^ getuid() ^ tv.tv_sec ^ tv.tv_usec);
-#ifdef DO_JRAND_MIX
-		jrand_seed[0] = getpid() ^ (tv.tv_sec & 0xFFFF);
-		jrand_seed[1] = getppid() ^ (tv.tv_usec & 0xFFFF);
-		jrand_seed[2] = (tv.tv_sec ^ tv.tv_usec) >> 16;
-#endif
-	}
-	/* Crank the random number generator a few times */
-	gettimeofday(&tv, 0);
-	for (i = (tv.tv_sec ^ tv.tv_usec) & 0x1F; i > 0; i--)
-		rand();
-	return fd;
-}
-
-
-/*
- * Generate a series of random bytes.  Use /dev/urandom if possible,
- * and if not, use srandom/random.
- */
-static void get_random_bytes(void *buf, int nbytes)
-{
-	int i, n = nbytes, fd = get_random_fd();
-	int lose_counter = 0;
-	unsigned char *cp = (unsigned char *) buf;
-#ifdef DO_JRAND_MIX
-	unsigned short tmp_seed[3];
-#endif
-	if (fd >= 0) {
-		while (n > 0) {
-			i = read(fd, cp, n);
-			if (i <= 0) {
-				if (lose_counter++ > 16)
-					break;
-				continue;
-			}
-			n -= i;
-			cp += i;
-			lose_counter = 0;
-		}
-	}
-
-	/*
-	 * We do this all the time, but this is the only source of
-	 * randomness if /dev/random/urandom is out to lunch.
-	 */
-	for (cp = buf, i = 0; i < nbytes; i++)
-		*cp++ ^= (rand() >> 7) & 0xFF;
-#ifdef DO_JRAND_MIX
-	memcpy(tmp_seed, jrand_seed, sizeof(tmp_seed));
-	jrand_seed[2] = jrand_seed[2] ^ syscall(__NR_gettid);
-	for (cp = buf, i = 0; i < nbytes; i++)
-		*cp++ ^= (jrand48(tmp_seed) >> 7) & 0xFF;
-	memcpy(jrand_seed, tmp_seed,
-	       sizeof(jrand_seed)-sizeof(unsigned short));
-#endif
-
-	return;
-}
-
-/*
- * Get the ethernet hardware address, if we can find it...
- *
- * XXX for a windows version, probably should use GetAdaptersInfo:
- * http://www.codeguru.com/cpp/i-n/network/networkinformation/article.php/c5451
- * commenting out get_node_id just to get gen_uuid to compile under windows
- * is not the right way to go!
- */
-static int get_node_id(unsigned char *node_id)
-{
-#ifdef HAVE_NET_IF_H
-	int 		sd;
-	struct ifreq 	ifr, *ifrp;
-	struct ifconf 	ifc;
-	char buf[1024];
-	int		n, i;
-	unsigned char 	*a;
-#ifdef HAVE_NET_IF_DL_H
-	struct sockaddr_dl *sdlp;
-#endif
-
-/*
- * BSD 4.4 defines the size of an ifreq to be
- * max(sizeof(ifreq), sizeof(ifreq.ifr_name)+ifreq.ifr_addr.sa_len
- * However, under earlier systems, sa_len isn't present, so the size is
- * just sizeof(struct ifreq)
- */
-#ifdef HAVE_SA_LEN
-#ifndef max
-#define max(a,b) ((a) > (b) ? (a) : (b))
-#endif
-#define ifreq_size(i) max(sizeof(struct ifreq),\
-     sizeof((i).ifr_name)+(i).ifr_addr.sa_len)
-#else
-#define ifreq_size(i) sizeof(struct ifreq)
-#endif /* HAVE_SA_LEN*/
-
-	sd = socket(AF_INET, SOCK_DGRAM, IPPROTO_IP);
-	if (sd < 0) {
-		return -1;
-	}
-	memset(buf, 0, sizeof(buf));
-	ifc.ifc_len = sizeof(buf);
-	ifc.ifc_buf = buf;
-	if (ioctl (sd, SIOCGIFCONF, (char *)&ifc) < 0) {
-		close(sd);
-		return -1;
-	}
-	n = ifc.ifc_len;
-	for (i = 0; i < n; i+= ifreq_size(*ifrp) ) {
-		ifrp = (struct ifreq *)((char *) ifc.ifc_buf+i);
-		strncpy(ifr.ifr_name, ifrp->ifr_name, IFNAMSIZ);
-#ifdef SIOCGIFHWADDR
-		if (ioctl(sd, SIOCGIFHWADDR, &ifr) < 0)
-			continue;
-		a = (unsigned char *) &ifr.ifr_hwaddr.sa_data;
-#else
-#ifdef SIOCGENADDR
-		if (ioctl(sd, SIOCGENADDR, &ifr) < 0)
-			continue;
-		a = (unsigned char *) ifr.ifr_enaddr;
-#else
-#ifdef HAVE_NET_IF_DL_H
-		sdlp = (struct sockaddr_dl *) &ifrp->ifr_addr;
-		if ((sdlp->sdl_family != AF_LINK) || (sdlp->sdl_alen != 6))
-			continue;
-		a = (unsigned char *) &sdlp->sdl_data[sdlp->sdl_nlen];
-#else
-		/*
-		 * XXX we don't have a way of getting the hardware
-		 * address
-		 */
-		close(sd);
-		return 0;
-#endif /* HAVE_NET_IF_DL_H */
-#endif /* SIOCGENADDR */
-#endif /* SIOCGIFHWADDR */
-		if (!a[0] && !a[1] && !a[2] && !a[3] && !a[4] && !a[5])
-			continue;
-		if (node_id) {
-			memcpy(node_id, a, 6);
-			close(sd);
-			return 1;
-		}
-	}
-	close(sd);
-#endif
-	return 0;
-}
-
-/* Assume that the gettimeofday() has microsecond granularity */
-#define MAX_ADJUSTMENT 10
-
-static int get_clock(uint32_t *clock_high, uint32_t *clock_low,
-		     uint16_t *ret_clock_seq, int *num)
-{
-	THREAD_LOCAL int		adjustment = 0;
-	THREAD_LOCAL struct timeval	last = {0, 0};
-	THREAD_LOCAL int		state_fd = -2;
-	THREAD_LOCAL FILE		*state_f;
-	THREAD_LOCAL uint16_t		clock_seq;
-	struct timeval 			tv;
-	struct flock			fl;
-	uint64_t			clock_reg;
-	mode_t				save_umask;
-	int				len;
-
-	if (state_fd == -2) {
-		save_umask = umask(0);
-		state_fd = open("/var/lib/libuuid/clock.txt",
-				O_RDWR|O_CREAT, 0660);
-		(void) umask(save_umask);
-		state_f = fdopen(state_fd, "r+");
-		if (!state_f) {
-			close(state_fd);
-			state_fd = -1;
-		}
-	}
-	fl.l_type = F_WRLCK;
-	fl.l_whence = SEEK_SET;
-	fl.l_start = 0;
-	fl.l_len = 0;
-	fl.l_pid = 0;
-	if (state_fd >= 0) {
-		rewind(state_f);
-		while (fcntl(state_fd, F_SETLKW, &fl) < 0) {
-			if ((errno == EAGAIN) || (errno == EINTR))
-				continue;
-			fclose(state_f);
-			close(state_fd);
-			state_fd = -1;
-			break;
-		}
-	}
-	if (state_fd >= 0) {
-		unsigned int cl;
-		unsigned long tv1, tv2;
-		int a;
-
-		if (fscanf(state_f, "clock: %04x tv: %lu %lu adj: %d\n",
-			   &cl, &tv1, &tv2, &a) == 4) {
-			clock_seq = cl & 0x3FFF;
-			last.tv_sec = tv1;
-			last.tv_usec = tv2;
-			adjustment = a;
-		}
-	}
-
-	if ((last.tv_sec == 0) && (last.tv_usec == 0)) {
-		get_random_bytes(&clock_seq, sizeof(clock_seq));
-		clock_seq &= 0x3FFF;
-		gettimeofday(&last, 0);
-		last.tv_sec--;
-	}
-
-try_again:
-	gettimeofday(&tv, 0);
-	if ((tv.tv_sec < last.tv_sec) ||
-	    ((tv.tv_sec == last.tv_sec) &&
-	     (tv.tv_usec < last.tv_usec))) {
-		clock_seq = (clock_seq+1) & 0x3FFF;
-		adjustment = 0;
-		last = tv;
-	} else if ((tv.tv_sec == last.tv_sec) &&
-	    (tv.tv_usec == last.tv_usec)) {
-		if (adjustment >= MAX_ADJUSTMENT)
-			goto try_again;
-		adjustment++;
-	} else {
-		adjustment = 0;
-		last = tv;
-	}
-
-	clock_reg = tv.tv_usec*10 + adjustment;
-	clock_reg += ((uint64_t) tv.tv_sec)*10000000;
-	clock_reg += (((uint64_t) 0x01B21DD2) << 32) + 0x13814000;
-
-	if (num && (*num > 1)) {
-		adjustment += *num - 1;
-		last.tv_usec += adjustment / 10;
-		adjustment = adjustment % 10;
-		last.tv_sec += last.tv_usec / 1000000;
-		last.tv_usec = last.tv_usec % 1000000;
-	}
-
-	if (state_fd > 0) {
-		rewind(state_f);
-		len = fprintf(state_f, 
-			      "clock: %04x tv: %016lu %08lu adj: %08d\n",
-			      clock_seq, last.tv_sec, last.tv_usec, adjustment);
-		fflush(state_f);
-		if (ftruncate(state_fd, len) < 0) {
-			fprintf(state_f, "                   \n");
-			fflush(state_f);
-		}
-		rewind(state_f);
-		fl.l_type = F_UNLCK;
-		fcntl(state_fd, F_SETLK, &fl);
-	}
-
-	*clock_high = clock_reg >> 32;
-	*clock_low = clock_reg;
-	*ret_clock_seq = clock_seq;
-	return 0;
-}
-
-#if defined(USE_UUIDD) && defined(HAVE_SYS_UN_H)
-static ssize_t read_all(int fd, char *buf, size_t count)
-{
-	ssize_t ret;
-	ssize_t c = 0;
-	int tries = 0;
-
-	memset(buf, 0, count);
-	while (count > 0) {
-		ret = read(fd, buf, count);
-		if (ret <= 0) {
-			if ((errno == EAGAIN || errno == EINTR || ret == 0) &&
-			    (tries++ < 5))
-				continue;
-			return c ? c : -1;
-		}
-		if (ret > 0)
-			tries = 0;
-		count -= ret;
-		buf += ret;
-		c += ret;
-	}
-	return c;
-}
-#endif
-
-/*
- * Close all file descriptors
- */
-#if defined(USE_UUIDD) && defined(HAVE_SYS_UN_H)
-static void close_all_fds(void)
-{
-#ifdef F_CLOSEM
-	(void)fcntl(0, F_CLOSEM);
-	(void)open("/dev/null", O_RDWR); /* stdin */
-	(void)open("/dev/null", O_RDWR); /* stdout */
-	(void)open("/dev/null", O_RDWR); /* stderr */
-#else /* F_CLOSEM */
-	int i, max;
-
-#if defined(HAVE_SYSCONF) && defined(_SC_OPEN_MAX)
-	max = sysconf(_SC_OPEN_MAX);
-#elif defined(HAVE_GETDTABLESIZE)
-	max = getdtablesize();
-#elif defined(HAVE_GETRLIMIT) && defined(RLIMIT_NOFILE)
-	struct rlimit rl;
-
-	getrlimit(RLIMIT_NOFILE, &rl);
-	max = rl.rlim_cur;
-#else
-	max = OPEN_MAX;
-#endif
-
-	for (i=0; i < max; i++) {
-		close(i);
-		if (i <= 2)
-			open("/dev/null", O_RDWR);
-	}
-#endif /* F_CLOSEM */
-}
-#endif
-
-
-/*
- * Try using the uuidd daemon to generate the UUID
- *
- * Returns 0 on success, non-zero on failure.
- */
-static int get_uuid_via_daemon(int op, uuid_t out, int *num)
-{
-#if defined(USE_UUIDD) && defined(HAVE_SYS_UN_H)
-	char op_buf[64];
-	int op_len;
-	int s;
-	ssize_t ret;
-	int32_t reply_len = 0, expected = 16;
-	struct sockaddr_un srv_addr;
-	struct stat st;
-	pid_t pid;
-	static const char *uuidd_path = UUIDD_PATH;
-	static int access_ret = -2;
-	static int start_attempts = 0;
-
-	if ((s = socket(AF_UNIX, SOCK_STREAM, 0)) < 0)
-		return -1;
-
-	srv_addr.sun_family = AF_UNIX;
-	strcpy(srv_addr.sun_path, UUIDD_SOCKET_PATH);
-
-	if (connect(s, (const struct sockaddr *) &srv_addr,
-		    sizeof(struct sockaddr_un)) < 0) {
-		if (access_ret == -2)
-			access_ret = access(uuidd_path, X_OK);
-		if (access_ret == 0)
-			access_ret = stat(uuidd_path, &st);
-		if (access_ret == 0 && (st.st_mode & (S_ISUID | S_ISGID)) == 0)
-			access_ret = access(UUIDD_DIR, W_OK);
-		if (access_ret == 0 && start_attempts++ < 5) {
-			if ((pid = fork()) == 0) {
-				close_all_fds();
-				execl(uuidd_path, "uuidd", "-qT", "300",
-				      (char *) NULL);
-				exit(1);
-			}
-			(void) waitpid(pid, 0, 0);
-			if (connect(s, (const struct sockaddr *) &srv_addr,
-				    sizeof(struct sockaddr_un)) < 0)
-				goto fail;
-		} else
-			goto fail;
-	}
-	op_buf[0] = op;
-	op_len = 1;
-	if (op == UUIDD_OP_BULK_TIME_UUID) {
-		memcpy(op_buf+1, num, sizeof(*num));
-		op_len += sizeof(*num);
-		expected += sizeof(*num);
-	}
-
-	ret = write(s, op_buf, op_len);
-	if (ret < 1)
-		goto fail;
-
-	ret = read_all(s, (char *) &reply_len, sizeof(reply_len));
-	if (ret < 0)
-		goto fail;
-
-	if (reply_len != expected)
-		goto fail;
-
-	ret = read_all(s, op_buf, reply_len);
-
-	if (op == UUIDD_OP_BULK_TIME_UUID)
-		memcpy(op_buf+16, num, sizeof(int));
-
-	memcpy(out, op_buf, 16);
-
-	close(s);
-	return ((ret == expected) ? 0 : -1);
-
-fail:
-	close(s);
-#endif
-	return -1;
-}
-
-void uuid__generate_time(uuid_t out, int *num)
-{
-	static unsigned char node_id[6];
-	static int has_init = 0;
-	struct uuid uu;
-	uint32_t	clock_mid;
-
-	if (!has_init) {
-		if (get_node_id(node_id) <= 0) {
-			get_random_bytes(node_id, 6);
-			/*
-			 * Set multicast bit, to prevent conflicts
-			 * with IEEE 802 addresses obtained from
-			 * network cards
-			 */
-			node_id[0] |= 0x01;
-		}
-		has_init = 1;
-	}
-	get_clock(&clock_mid, &uu.time_low, &uu.clock_seq, num);
-	uu.clock_seq |= 0x8000;
-	uu.time_mid = (uint16_t) clock_mid;
-	uu.time_hi_and_version = ((clock_mid >> 16) & 0x0FFF) | 0x1000;
-	memcpy(uu.node, node_id, 6);
-	uuid_pack(&uu, out);
-}
-
-void uuid_generate_time(uuid_t out)
-{
-#ifdef TLS
-	THREAD_LOCAL int		num = 0;
-	THREAD_LOCAL struct uuid	uu;
-	THREAD_LOCAL time_t		last_time = 0;
-	time_t				now;
-
-	if (num > 0) {
-		now = time(0);
-		if (now > last_time+1)
-			num = 0;
-	}
-	if (num <= 0) {
-		num = 1000;
-		if (get_uuid_via_daemon(UUIDD_OP_BULK_TIME_UUID,
-					out, &num) == 0) {
-			last_time = time(0);
-			uuid_unpack(out, &uu);
-			num--;
-			return;
-		}
-		num = 0;
-	}
-	if (num > 0) {
-		uu.time_low++;
-		if (uu.time_low == 0) {
-			uu.time_mid++;
-			if (uu.time_mid == 0)
-				uu.time_hi_and_version++;
-		}
-		num--;
-		uuid_pack(&uu, out);
-		return;
-	}
-#else
-	if (get_uuid_via_daemon(UUIDD_OP_TIME_UUID, out, 0) == 0)
-		return;
-#endif
-
-	uuid__generate_time(out, 0);
-}
-
-
-void uuid__generate_random(uuid_t out, int *num)
-{
-	uuid_t	buf;
-	struct uuid uu;
-	int i, n;
-
-	if (!num || !*num)
-		n = 1;
-	else
-		n = *num;
-
-	for (i = 0; i < n; i++) {
-		get_random_bytes(buf, sizeof(buf));
-		uuid_unpack(buf, &uu);
-
-		uu.clock_seq = (uu.clock_seq & 0x3FFF) | 0x8000;
-		uu.time_hi_and_version = (uu.time_hi_and_version & 0x0FFF)
-			| 0x4000;
-		uuid_pack(&uu, out);
-		out += sizeof(uuid_t);
-	}
-}
-
-void uuid_generate_random(uuid_t out)
-{
-	int	num = 1;
-	/* No real reason to use the daemon for random uuid's -- yet */
-
-	uuid__generate_random(out, &num);
-}
-
-
-/*
- * This is the generic front-end to uuid_generate_random and
- * uuid_generate_time.  It uses uuid_generate_random only if
- * /dev/urandom is available, since otherwise we won't have
- * high-quality randomness.
- */
-void uuid_generate(uuid_t out)
-{
-	if (get_random_fd() >= 0)
-		uuid_generate_random(out);
-	else
-		uuid_generate_time(out);
-}
diff --git a/contrib/uuid/gen_uuid_nt.c b/contrib/uuid/gen_uuid_nt.c
deleted file mode 100644
index aa44bfd3d7d..00000000000
--- a/contrib/uuid/gen_uuid_nt.c
+++ /dev/null
@@ -1,92 +0,0 @@
-/*
- * gen_uuid_nt.c -- Use NT api to generate uuid
- *
- * Written by Andrey Shedel (andreys@ns.cr.cyco.com)
- */
-
-
-#include "uuidP.h"
-
-#pragma warning(push,4)
-
-#pragma comment(lib, "ntdll.lib")
-
-//
-// Here is a nice example why it's not a good idea
-// to use native API in ordinary applications.
-// Number of parameters in function below was changed from 3 to 4
-// for NT5.
-//
-//
-// NTSYSAPI
-// NTSTATUS
-// NTAPI
-// NtAllocateUuids(
-//     OUT PULONG p1,
-//     OUT PULONG p2,
-//     OUT PULONG p3,
-//     OUT PUCHAR Seed // 6 bytes
-//   );
-//
-//
-
-unsigned long
-__stdcall
-NtAllocateUuids(
-   void* p1,  // 8 bytes
-   void* p2,  // 4 bytes
-   void* p3   // 4 bytes
-   );
-
-typedef
-unsigned long
-(__stdcall*
-NtAllocateUuids_2000)(
-   void* p1,  // 8 bytes
-   void* p2,  // 4 bytes
-   void* p3,  // 4 bytes
-   void* seed // 6 bytes
-   );
-
-
-
-//
-// Nice, but instead of including ntddk.h ot winnt.h
-// I should define it here because they MISSED __stdcall in those headers.
-//
-
-__declspec(dllimport)
-struct _TEB*
-__stdcall
-NtCurrentTeb(void);
-
-
-//
-// The only way to get version information from the system is to examine
-// one stored in PEB. But it's pretty dangerouse because this value could
-// be altered in image header.
-//
-
-static
-int
-Nt5(void)
-{
-	//return NtCuttentTeb()->Peb->OSMajorVersion >= 5;
-	return (int)*(int*)((char*)(int)(*(int*)((char*)NtCurrentTeb() + 0x30)) + 0xA4) >= 5;
-}
-
-
-
-
-void uuid_generate(uuid_t out)
-{
-	if(Nt5())
-	{
-		unsigned char seed[6];
-		((NtAllocateUuids_2000)NtAllocateUuids)(out, ((char*)out)+8, ((char*)out)+12, &seed[0] );
-	}
-	else
-	{
-		NtAllocateUuids(out, ((char*)out)+8, ((char*)out)+12);
-	}
-}
diff --git a/contrib/uuid/isnull.c b/contrib/uuid/isnull.c
deleted file mode 100644
index 931e7e7dba2..00000000000
--- a/contrib/uuid/isnull.c
+++ /dev/null
@@ -1,48 +0,0 @@
-/*
- * isnull.c --- Check whether or not the UUID is null
- *
- * Copyright (C) 1996, 1997 Theodore Ts'o.
- *
- * %Begin-Header%
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, and the entire permission notice in its entirety,
- *    including the disclaimer of warranties.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- * 3. The name of the author may not be used to endorse or promote
- *    products derived from this software without specific prior
- *    written permission.
- *
- * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
- * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
- * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
- * WHICH ARE HEREBY DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
- * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
- * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
- * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
- * DAMAGE.
- * %End-Header%
- */
-
-#include "uuidP.h"
-
-/* Returns 1 if the uuid is the NULL uuid */
-int uuid_is_null(const uuid_t uu)
-{
-	const unsigned char 	*cp;
-	int			i;
-
-	for (i=0, cp = uu; i < 16; i++)
-		if (*cp++)
-			return 0;
-	return 1;
-}
-
diff --git a/contrib/uuid/pack.c b/contrib/uuid/pack.c
deleted file mode 100644
index 097516d2e2f..00000000000
--- a/contrib/uuid/pack.c
+++ /dev/null
@@ -1,69 +0,0 @@
-/*
- * Internal routine for packing UUID's
- *
- * Copyright (C) 1996, 1997 Theodore Ts'o.
- *
- * %Begin-Header%
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, and the entire permission notice in its entirety,
- *    including the disclaimer of warranties.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- * 3. The name of the author may not be used to endorse or promote
- *    products derived from this software without specific prior
- *    written permission.
- *
- * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
- * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
- * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
- * WHICH ARE HEREBY DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
- * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
- * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
- * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
- * DAMAGE.
- * %End-Header%
- */
-
-#include <string.h>
-#include "uuidP.h"
-
-void uuid_pack(const struct uuid *uu, uuid_t ptr)
-{
-	uint32_t	tmp;
-	unsigned char	*out = ptr;
-
-	tmp = uu->time_low;
-	out[3] = (unsigned char) tmp;
-	tmp >>= 8;
-	out[2] = (unsigned char) tmp;
-	tmp >>= 8;
-	out[1] = (unsigned char) tmp;
-	tmp >>= 8;
-	out[0] = (unsigned char) tmp;
-
-	tmp = uu->time_mid;
-	out[5] = (unsigned char) tmp;
-	tmp >>= 8;
-	out[4] = (unsigned char) tmp;
-
-	tmp = uu->time_hi_and_version;
-	out[7] = (unsigned char) tmp;
-	tmp >>= 8;
-	out[6] = (unsigned char) tmp;
-
-	tmp = uu->clock_seq;
-	out[9] = (unsigned char) tmp;
-	tmp >>= 8;
-	out[8] = (unsigned char) tmp;
-
-	memcpy(out+10, uu->node, 6);
-}
-
diff --git a/contrib/uuid/parse.c b/contrib/uuid/parse.c
deleted file mode 100644
index 074383efae7..00000000000
--- a/contrib/uuid/parse.c
+++ /dev/null
@@ -1,79 +0,0 @@
-/*
- * parse.c --- UUID parsing
- *
- * Copyright (C) 1996, 1997 Theodore Ts'o.
- *
- * %Begin-Header%
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, and the entire permission notice in its entirety,
- *    including the disclaimer of warranties.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- * 3. The name of the author may not be used to endorse or promote
- *    products derived from this software without specific prior
- *    written permission.
- *
- * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
- * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
- * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
- * WHICH ARE HEREBY DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
- * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
- * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
- * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
- * DAMAGE.
- * %End-Header%
- */
-
-#include <stdlib.h>
-#include <stdio.h>
-#include <ctype.h>
-#include <string.h>
-
-#include "uuidP.h"
-
-int uuid_parse(const char *in, uuid_t uu)
-{
-	struct uuid	uuid;
-	int 		i;
-	const char	*cp;
-	char		buf[3];
-
-	if (strlen(in) != 36)
-		return -1;
-	for (i=0, cp = in; i <= 36; i++,cp++) {
-		if ((i == 8) || (i == 13) || (i == 18) ||
-		    (i == 23)) {
-			if (*cp == '-')
-				continue;
-			else
-				return -1;
-		}
-		if (i== 36)
-			if (*cp == 0)
-				continue;
-		if (!isxdigit(*cp))
-			return -1;
-	}
-	uuid.time_low = strtoul(in, NULL, 16);
-	uuid.time_mid = strtoul(in+9, NULL, 16);
-	uuid.time_hi_and_version = strtoul(in+14, NULL, 16);
-	uuid.clock_seq = strtoul(in+19, NULL, 16);
-	cp = in+24;
-	buf[2] = 0;
-	for (i=0; i < 6; i++) {
-		buf[0] = *cp++;
-		buf[1] = *cp++;
-		uuid.node[i] = strtoul(buf, NULL, 16);
-	}
-
-	uuid_pack(&uuid, uu);
-	return 0;
-}
diff --git a/contrib/uuid/tst_uuid.c b/contrib/uuid/tst_uuid.c
deleted file mode 100644
index e03138f7d18..00000000000
--- a/contrib/uuid/tst_uuid.c
+++ /dev/null
@@ -1,180 +0,0 @@
-/*
- * tst_uuid.c --- test program from the UUID library
- *
- * Copyright (C) 1996, 1997, 1998 Theodore Ts'o.
- *
- * %Begin-Header%
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, and the entire permission notice in its entirety,
- *    including the disclaimer of warranties.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- * 3. The name of the author may not be used to endorse or promote
- *    products derived from this software without specific prior
- *    written permission.
- *
- * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
- * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
- * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
- * WHICH ARE HEREBY DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
- * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
- * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
- * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
- * DAMAGE.
- * %End-Header%
- */
-
-#ifdef _WIN32
-#define _WIN32_WINNT 0x0500
-#include <windows.h>
-#define UUID MYUUID
-#endif
-
-#include <stdio.h>
-#include <stdlib.h>
-
-#include "uuid.h"
-
-static int test_uuid(const char * uuid, int isValid)
-{
-	static const char * validStr[2] = {"invalid", "valid"};
-	uuid_t uuidBits;
-	int parsedOk;
-
-	parsedOk = uuid_parse(uuid, uuidBits) == 0;
-
-	printf("%s is %s", uuid, validStr[isValid]);
-	if (parsedOk != isValid) {
-		printf(" but uuid_parse says %s\n", validStr[parsedOk]);
-		return 1;
-	}
-	printf(", OK\n");
-	return 0;
-}
-
-#ifdef __GNUC__
-#define ATTR(x) __attribute__(x)
-#else
-#define ATTR(x)
-#endif
-
-int
-main(int argc ATTR((unused)) , char **argv ATTR((unused)))
-{
-	uuid_t		buf, tst;
-	char		str[100];
-	struct timeval	tv;
-	time_t		time_reg;
-	unsigned char	*cp;
-	int i;
-	int failed = 0;
-	int type, variant;
-
-	uuid_generate(buf);
-	uuid_unparse(buf, str);
-	printf("UUID generate = %s\n", str);
-	printf("UUID: ");
-	for (i=0, cp = (unsigned char *) &buf; i < 16; i++) {
-		printf("%02x", *cp++);
-	}
-	printf("\n");
-	type = uuid_type(buf); 	variant = uuid_variant(buf);
-	printf("UUID type = %d, UUID variant = %d\n", type, variant);
-	if (variant != UUID_VARIANT_DCE) {
-		printf("Incorrect UUID Variant; was expecting DCE!\n");
-		failed++;
-	}
-	printf("\n");
-
-	uuid_generate_random(buf);
-	uuid_unparse(buf, str);
-	printf("UUID random string = %s\n", str);
-	printf("UUID: ");
-	for (i=0, cp = (unsigned char *) &buf; i < 16; i++) {
-		printf("%02x", *cp++);
-	}
-	printf("\n");
-	type = uuid_type(buf); 	variant = uuid_variant(buf);
-	printf("UUID type = %d, UUID variant = %d\n", type, variant);
-	if (variant != UUID_VARIANT_DCE) {
-		printf("Incorrect UUID Variant; was expecting DCE!\n");
-		failed++;
-	}
-	if (type != 4) {
-		printf("Incorrect UUID type; was expecting "
-		       "4 (random type)!\n");
-		failed++;
-	}
-	printf("\n");
-
-	uuid_generate_time(buf);
-	uuid_unparse(buf, str);
-	printf("UUID string = %s\n", str);
-	printf("UUID time: ");
-	for (i=0, cp = (unsigned char *) &buf; i < 16; i++) {
-		printf("%02x", *cp++);
-	}
-	printf("\n");
-	type = uuid_type(buf); 	variant = uuid_variant(buf);
-	printf("UUID type = %d, UUID variant = %d\n", type, variant);
-	if (variant != UUID_VARIANT_DCE) {
-		printf("Incorrect UUID Variant; was expecting DCE!\n");
-		failed++;
-	}
-	if (type != 1) {
-		printf("Incorrect UUID type; was expecting "
-		       "1 (time-based type)!\\n");
-		failed++;
-	}
-	tv.tv_sec = 0;
-	tv.tv_usec = 0;
-	time_reg = uuid_time(buf, &tv);
-	printf("UUID time is: (%ld, %ld): %s\n", tv.tv_sec, tv.tv_usec,
-	       ctime(&time_reg));
-	uuid_parse(str, tst);
-	if (!uuid_compare(buf, tst))
-		printf("UUID parse and compare succeeded.\n");
-	else {
-		printf("UUID parse and compare failed!\n");
-		failed++;
-	}
-	uuid_clear(tst);
-	if (uuid_is_null(tst))
-		printf("UUID clear and is null succeeded.\n");
-	else {
-		printf("UUID clear and is null failed!\n");
-		failed++;
-	}
-	uuid_copy(buf, tst);
-	if (!uuid_compare(buf, tst))
-		printf("UUID copy and compare succeeded.\n");
-	else {
-		printf("UUID copy and compare failed!\n");
-		failed++;
-	}
-	failed += test_uuid("84949cc5-4701-4a84-895b-354c584a981b", 1);
-	failed += test_uuid("84949CC5-4701-4A84-895B-354C584A981B", 1);
-	failed += test_uuid("84949cc5-4701-4a84-895b-354c584a981bc", 0);
-	failed += test_uuid("84949cc5-4701-4a84-895b-354c584a981", 0);
-	failed += test_uuid("84949cc5x4701-4a84-895b-354c584a981b", 0);
-	failed += test_uuid("84949cc504701-4a84-895b-354c584a981b", 0);
-	failed += test_uuid("84949cc5-470104a84-895b-354c584a981b", 0);
-	failed += test_uuid("84949cc5-4701-4a840895b-354c584a981b", 0);
-	failed += test_uuid("84949cc5-4701-4a84-895b0354c584a981b", 0);
-	failed += test_uuid("g4949cc5-4701-4a84-895b-354c584a981b", 0);
-	failed += test_uuid("84949cc5-4701-4a84-895b-354c584a981g", 0);
-
-	if (failed) {
-		printf("%d failures.\n", failed);
-		exit(1);
-	}
-	return 0;
-}
diff --git a/contrib/uuid/unpack.c b/contrib/uuid/unpack.c
deleted file mode 100644
index beaaff3ca8a..00000000000
--- a/contrib/uuid/unpack.c
+++ /dev/null
@@ -1,63 +0,0 @@
-/*
- * Internal routine for unpacking UUID
- *
- * Copyright (C) 1996, 1997 Theodore Ts'o.
- *
- * %Begin-Header%
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, and the entire permission notice in its entirety,
- *    including the disclaimer of warranties.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- * 3. The name of the author may not be used to endorse or promote
- *    products derived from this software without specific prior
- *    written permission.
- *
- * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
- * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
- * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
- * WHICH ARE HEREBY DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
- * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
- * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
- * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
- * DAMAGE.
- * %End-Header%
- */
-
-#include <string.h>
-#include "uuidP.h"
-
-void uuid_unpack(const uuid_t in, struct uuid *uu)
-{
-	const uint8_t	*ptr = in;
-	uint32_t		tmp;
-
-	tmp = *ptr++;
-	tmp = (tmp << 8) | *ptr++;
-	tmp = (tmp << 8) | *ptr++;
-	tmp = (tmp << 8) | *ptr++;
-	uu->time_low = tmp;
-
-	tmp = *ptr++;
-	tmp = (tmp << 8) | *ptr++;
-	uu->time_mid = tmp;
-
-	tmp = *ptr++;
-	tmp = (tmp << 8) | *ptr++;
-	uu->time_hi_and_version = tmp;
-
-	tmp = *ptr++;
-	tmp = (tmp << 8) | *ptr++;
-	uu->clock_seq = tmp;
-
-	memcpy(uu->node, ptr, 6);
-}
-
diff --git a/contrib/uuid/unparse.c b/contrib/uuid/unparse.c
deleted file mode 100644
index a95bbb04258..00000000000
--- a/contrib/uuid/unparse.c
+++ /dev/null
@@ -1,76 +0,0 @@
-/*
- * unparse.c -- convert a UUID to string
- *
- * Copyright (C) 1996, 1997 Theodore Ts'o.
- *
- * %Begin-Header%
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, and the entire permission notice in its entirety,
- *    including the disclaimer of warranties.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- * 3. The name of the author may not be used to endorse or promote
- *    products derived from this software without specific prior
- *    written permission.
- *
- * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
- * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
- * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
- * WHICH ARE HEREBY DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
- * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
- * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
- * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
- * DAMAGE.
- * %End-Header%
- */
-
-#include <stdio.h>
-
-#include "uuidP.h"
-
-static const char *fmt_lower =
-	"%08x-%04x-%04x-%02x%02x-%02x%02x%02x%02x%02x%02x";
-
-static const char *fmt_upper =
-	"%08X-%04X-%04X-%02X%02X-%02X%02X%02X%02X%02X%02X";
-
-#ifdef UUID_UNPARSE_DEFAULT_UPPER
-#define FMT_DEFAULT fmt_upper
-#else
-#define FMT_DEFAULT fmt_lower
-#endif
-
-static void uuid_unparse_x(const uuid_t uu, char *out, const char *fmt)
-{
-	struct uuid uuid;
-
-	uuid_unpack(uu, &uuid);
-	sprintf(out, fmt,
-		uuid.time_low, uuid.time_mid, uuid.time_hi_and_version,
-		uuid.clock_seq >> 8, uuid.clock_seq & 0xFF,
-		uuid.node[0], uuid.node[1], uuid.node[2],
-		uuid.node[3], uuid.node[4], uuid.node[5]);
-}
-
-void uuid_unparse_lower(const uuid_t uu, char *out)
-{
-	uuid_unparse_x(uu, out,	fmt_lower);
-}
-
-void uuid_unparse_upper(const uuid_t uu, char *out)
-{
-	uuid_unparse_x(uu, out,	fmt_upper);
-}
-
-void uuid_unparse(const uuid_t uu, char *out)
-{
-	uuid_unparse_x(uu, out, FMT_DEFAULT);
-}
diff --git a/contrib/uuid/uuid.h b/contrib/uuid/uuid.h
deleted file mode 100644
index ab006652fc0..00000000000
--- a/contrib/uuid/uuid.h
+++ /dev/null
@@ -1,104 +0,0 @@
-/*
- * Public include file for the UUID library
- *
- * Copyright (C) 1996, 1997, 1998 Theodore Ts'o.
- *
- * %Begin-Header%
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, and the entire permission notice in its entirety,
- *    including the disclaimer of warranties.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- * 3. The name of the author may not be used to endorse or promote
- *    products derived from this software without specific prior
- *    written permission.
- *
- * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
- * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
- * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
- * WHICH ARE HEREBY DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
- * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
- * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
- * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
- * DAMAGE.
- * %End-Header%
- */
-
-#ifndef _UUID_UUID_H
-#define _UUID_UUID_H
-
-#include "config.h"
-#include <sys/types.h>
-#ifndef _WIN32
-#include <sys/time.h>
-#endif
-#include <time.h>
-
-typedef unsigned char uuid_t[16];
-
-/* UUID Variant definitions */
-#define UUID_VARIANT_NCS 	0
-#define UUID_VARIANT_DCE 	1
-#define UUID_VARIANT_MICROSOFT	2
-#define UUID_VARIANT_OTHER	3
-
-/* UUID Type definitions */
-#define UUID_TYPE_DCE_TIME   1
-#define UUID_TYPE_DCE_RANDOM 4
-
-/* Allow UUID constants to be defined */
-#ifdef __GNUC__
-#define UUID_DEFINE(name,u0,u1,u2,u3,u4,u5,u6,u7,u8,u9,u10,u11,u12,u13,u14,u15) \
-	static const uuid_t name __attribute__ ((unused)) = {u0,u1,u2,u3,u4,u5,u6,u7,u8,u9,u10,u11,u12,u13,u14,u15}
-#else
-#define UUID_DEFINE(name,u0,u1,u2,u3,u4,u5,u6,u7,u8,u9,u10,u11,u12,u13,u14,u15) \
-	static const uuid_t name = {u0,u1,u2,u3,u4,u5,u6,u7,u8,u9,u10,u11,u12,u13,u14,u15}
-#endif
-
-#ifdef __cplusplus
-extern "C" {
-#endif
-
-/* clear.c */
-void uuid_clear(uuid_t uu);
-
-/* compare.c */
-int uuid_compare(const uuid_t uu1, const uuid_t uu2);
-
-/* copy.c */
-void uuid_copy(uuid_t dst, const uuid_t src);
-
-/* gen_uuid.c */
-void uuid_generate(uuid_t out);
-void uuid_generate_random(uuid_t out);
-void uuid_generate_time(uuid_t out);
-
-/* isnull.c */
-int uuid_is_null(const uuid_t uu);
-
-/* parse.c */
-int uuid_parse(const char *in, uuid_t uu);
-
-/* unparse.c */
-void uuid_unparse(const uuid_t uu, char *out);
-void uuid_unparse_lower(const uuid_t uu, char *out);
-void uuid_unparse_upper(const uuid_t uu, char *out);
-
-/* uuid_time.c */
-time_t uuid_time(const uuid_t uu, struct timeval *ret_tv);
-int uuid_type(const uuid_t uu);
-int uuid_variant(const uuid_t uu);
-
-#ifdef __cplusplus
-}
-#endif
-
-#endif /* _UUID_UUID_H */
diff --git a/contrib/uuid/uuidP.h b/contrib/uuid/uuidP.h
deleted file mode 100644
index 9a2de6132fe..00000000000
--- a/contrib/uuid/uuidP.h
+++ /dev/null
@@ -1,63 +0,0 @@
-/*
- * uuid.h -- private header file for uuids
- *
- * Copyright (C) 1996, 1997 Theodore Ts'o.
- *
- * %Begin-Header%
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, and the entire permission notice in its entirety,
- *    including the disclaimer of warranties.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- * 3. The name of the author may not be used to endorse or promote
- *    products derived from this software without specific prior
- *    written permission.
- *
- * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
- * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
- * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
- * WHICH ARE HEREBY DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
- * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
- * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
- * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
- * DAMAGE.
- * %End-Header%
- */
-
-#include "uuid.h"
-#ifdef HAVE_INTTYPES_H
-#include <inttypes.h>
-#else
-#include "uuid_types.h"
-#endif
-#include <sys/types.h>
-
-
-/*
- * Offset between 15-Oct-1582 and 1-Jan-70
- */
-#define TIME_OFFSET_HIGH 0x01B21DD2
-#define TIME_OFFSET_LOW  0x13814000
-
-struct uuid {
-	uint32_t	time_low;
-	uint16_t	time_mid;
-	uint16_t	time_hi_and_version;
-	uint16_t	clock_seq;
-	uint8_t	node[6];
-};
-
-
-/*
- * prototypes
- */
-void uuid_pack(const struct uuid *uu, uuid_t ptr);
-void uuid_unpack(const uuid_t in, struct uuid *uu);
diff --git a/contrib/uuid/uuid_time.c b/contrib/uuid/uuid_time.c
deleted file mode 100644
index ccaa542fed0..00000000000
--- a/contrib/uuid/uuid_time.c
+++ /dev/null
@@ -1,171 +0,0 @@
-/*
- * uuid_time.c --- Interpret the time field from a uuid.  This program
- * 	violates the UUID abstraction barrier by reaching into the guts
- *	of a UUID and interpreting it.
- *
- * Copyright (C) 1998, 1999 Theodore Ts'o.
- *
- * %Begin-Header%
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, and the entire permission notice in its entirety,
- *    including the disclaimer of warranties.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- * 3. The name of the author may not be used to endorse or promote
- *    products derived from this software without specific prior
- *    written permission.
- *
- * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
- * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
- * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
- * WHICH ARE HEREBY DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
- * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
- * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
- * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
- * DAMAGE.
- * %End-Header%
- */
-
-#ifdef _WIN32
-#define _WIN32_WINNT 0x0500
-#include <windows.h>
-#define UUID MYUUID
-#endif
-
-#include <stdio.h>
-#ifdef HAVE_UNISTD_H
-#include <unistd.h>
-#endif
-#include <stdlib.h>
-#include <sys/types.h>
-#ifdef HAVE_SYS_TIME_H
-#include <sys/time.h>
-#endif
-#include <time.h>
-
-#include "uuidP.h"
-#include "logging.h"
-
-time_t uuid_time(const uuid_t uu, struct timeval *ret_tv)
-{
-	struct timeval		tv;
-	struct uuid		uuid;
-	uint32_t		high;
-	uint64_t		clock_reg;
-
-	uuid_unpack(uu, &uuid);
-
-	high = uuid.time_mid | ((uuid.time_hi_and_version & 0xFFF) << 16);
-	clock_reg = uuid.time_low | ((uint64_t) high << 32);
-
-	clock_reg -= (((uint64_t) 0x01B21DD2) << 32) + 0x13814000;
-	tv.tv_sec = clock_reg / 10000000;
-	tv.tv_usec = (clock_reg % 10000000) / 10;
-
-	if (ret_tv)
-		*ret_tv = tv;
-
-	return tv.tv_sec;
-}
-
-int uuid_type(const uuid_t uu)
-{
-	struct uuid		uuid;
-
-	uuid_unpack(uu, &uuid);
-	return ((uuid.time_hi_and_version >> 12) & 0xF);
-}
-
-int uuid_variant(const uuid_t uu)
-{
-	struct uuid		uuid;
-	int			var;
-
-	uuid_unpack(uu, &uuid);
-	var = uuid.clock_seq;
-
-	if ((var & 0x8000) == 0)
-		return UUID_VARIANT_NCS;
-	if ((var & 0x4000) == 0)
-		return UUID_VARIANT_DCE;
-	if ((var & 0x2000) == 0)
-		return UUID_VARIANT_MICROSOFT;
-	return UUID_VARIANT_OTHER;
-}
-
-#ifdef DEBUG
-static const char *variant_string(int variant)
-{
-	switch (variant) {
-	case UUID_VARIANT_NCS:
-		return "NCS";
-	case UUID_VARIANT_DCE:
-		return "DCE";
-	case UUID_VARIANT_MICROSOFT:
-		return "Microsoft";
-	default:
-		return "Other";
-	}
-}
-
-
-int
-main(int argc, char **argv)
-{
-	uuid_t		buf;
-	time_t		time_reg;
-	struct timeval	tv;
-	int		type, variant;
-
-	if (argc != 2) {
-		fprintf(stderr, "Usage: %s uuid\n", argv[0]);
-		exit(1);
-	}
-	if (uuid_parse(argv[1], buf)) {
-		fprintf(stderr, "Invalid UUID: %s\n", argv[1]);
-		exit(1);
-	}
-	variant = uuid_variant(buf);
-	type = uuid_type(buf);
-	time_reg = uuid_time(buf, &tv);
-
-	printf("UUID variant is %d (%s)\n", variant, variant_string(variant));
-	if (variant != UUID_VARIANT_DCE) {
-		printf("Warning: This program only knows how to interpret "
-		       "DCE UUIDs.\n\tThe rest of the output is likely "
-		       "to be incorrect!!\n");
-	}
-	printf("UUID type is %d", type);
-	switch (type) {
-	case 1:
-		printf(" (time based)\n");
-		break;
-	case 2:
-		printf(" (DCE)\n");
-		break;
-	case 3:
-		printf(" (name-based)\n");
-		break;
-	case 4:
-		printf(" (random)\n");
-		break;
-	default:
-		printf("\n");
-	}
-	if (type != 1) {
-		printf("Warning: not a time-based UUID, so UUID time "
-		       "decoding will likely not work!\n");
-	}
-	printf("UUID time is: (%" GF_PRI_SECOND ", %" GF_PRI_USEC "): %s\n", tv.tv_sec, tv.tv_usec, ctime(&time_reg));
-
-	return 0;
-}
-#endif
diff --git a/contrib/uuid/uuid_types.h.in b/contrib/uuid/uuid_types.h.in
deleted file mode 100644
index f21ff4ee183..00000000000
--- a/contrib/uuid/uuid_types.h.in
+++ /dev/null
@@ -1,50 +0,0 @@
-/* 
- * If linux/types.h is already been included, assume it has defined
- * everything we need.  (cross fingers)  Other header files may have 
- * also defined the types that we need.
- */
-#if (!defined(_STDINT_H) && !defined(_UUID_STDINT_H))
-#define _UUID_STDINT_H
-
-typedef unsigned char uint8_t;
-typedef signed char int8_t;
-
-#if (@SIZEOF_INT@ == 8)
-typedef int		int64_t;
-typedef unsigned int	uint64_t;
-#elif (@SIZEOF_LONG@ == 8)
-typedef long		int64_t;
-typedef unsigned long	uint64_t;
-#elif (@SIZEOF_LONG_LONG@ == 8)
-#if defined(__GNUC__)
-typedef __signed__ long long 	int64_t;
-#else
-typedef signed long long 	int64_t;
-#endif
-typedef unsigned long long	uint64_t;
-#endif
-
-#if (@SIZEOF_INT@ == 2)
-typedef	int		int16_t;
-typedef	unsigned int	uint16_t;
-#elif (@SIZEOF_SHORT@ == 2)
-typedef	short		int16_t;
-typedef	unsigned short	uint16_t;
-#else
-  ?==error: undefined 16 bit type
-#endif
-
-#if (@SIZEOF_INT@ == 4)
-typedef	int		int32_t;
-typedef	unsigned int	uint32_t;
-#elif (@SIZEOF_LONG@ == 4)
-typedef	long		int32_t;
-typedef	unsigned long	uint32_t;
-#elif (@SIZEOF_SHORT@ == 4)
-typedef	short		int32_t;
-typedef	unsigned short	uint32_t;
-#else
- ?== error: undefined 32 bit type
-#endif
-
-#endif
diff --git a/contrib/uuid/uuidd.h b/contrib/uuid/uuidd.h
deleted file mode 100644
index c71f4b78835..00000000000
--- a/contrib/uuid/uuidd.h
+++ /dev/null
@@ -1,54 +0,0 @@
-/*
- * Definitions used by the uuidd daemon
- *
- * Copyright (C) 2007 Theodore Ts'o.
- *
- * %Begin-Header%
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, and the entire permission notice in its entirety,
- *    including the disclaimer of warranties.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- * 3. The name of the author may not be used to endorse or promote
- *    products derived from this software without specific prior
- *    written permission.
- *
- * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
- * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
- * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF
- * WHICH ARE HEREBY DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
- * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
- * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
- * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH
- * DAMAGE.
- * %End-Header%
- */
-
-#ifndef _UUID_UUIDD_H
-#define _UUID_UUIDD_H
-
-#define UUIDD_DIR		"/var/lib/libuuid"
-#define UUIDD_SOCKET_PATH	UUIDD_DIR "/request"
-#define UUIDD_PIDFILE_PATH	UUIDD_DIR "/uuidd.pid"
-#define UUIDD_PATH		"/usr/sbin/uuidd"
-
-#define UUIDD_OP_GETPID			0
-#define UUIDD_OP_GET_MAXOP		1
-#define UUIDD_OP_TIME_UUID		2
-#define UUIDD_OP_RANDOM_UUID		3
-#define UUIDD_OP_BULK_TIME_UUID		4
-#define UUIDD_OP_BULK_RANDOM_UUID	5
-#define UUIDD_MAX_OP			UUIDD_OP_BULK_RANDOM_UUID
-
-extern void uuid__generate_time(uuid_t out, int *num);
-extern void uuid__generate_random(uuid_t out, int *num);
-
-#endif /* _UUID_UUID_H */
diff --git a/contrib/xxhash/xxhash.c b/contrib/xxhash/xxhash.c
new file mode 100644
index 00000000000..56f80f8811d
--- /dev/null
+++ b/contrib/xxhash/xxhash.c
@@ -0,0 +1,1029 @@
+/*
+*  xxHash - Fast Hash algorithm
+*  Copyright (C) 2012-2016, Yann Collet
+*
+*  BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
+*
+*  Redistribution and use in source and binary forms, with or without
+*  modification, are permitted provided that the following conditions are
+*  met:
+*
+*  * Redistributions of source code must retain the above copyright
+*  notice, this list of conditions and the following disclaimer.
+*  * Redistributions in binary form must reproduce the above
+*  copyright notice, this list of conditions and the following disclaimer
+*  in the documentation and/or other materials provided with the
+*  distribution.
+*
+*  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+*  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+*  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+*  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+*  OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+*  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+*  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+*  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+*  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+*  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+*  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+*
+*  You can contact the author at :
+*  - xxHash homepage: http://www.xxhash.com
+*  - xxHash source repository : https://github.com/Cyan4973/xxHash
+*/
+
+
+/* *************************************
+*  Tuning parameters
+***************************************/
+/*!XXH_FORCE_MEMORY_ACCESS :
+ * By default, access to unaligned memory is controlled by `memcpy()`, which is safe and portable.
+ * Unfortunately, on some target/compiler combinations, the generated assembly is sub-optimal.
+ * The below switch allow to select different access method for improved performance.
+ * Method 0 (default) : use `memcpy()`. Safe and portable.
+ * Method 1 : `__packed` statement. It depends on compiler extension (ie, not portable).
+ *            This method is safe if your compiler supports it, and *generally* as fast or faster than `memcpy`.
+ * Method 2 : direct access. This method doesn't depend on compiler but violate C standard.
+ *            It can generate buggy code on targets which do not support unaligned memory accesses.
+ *            But in some circumstances, it's the only known way to get the most performance (ie GCC + ARMv6)
+ * See http://stackoverflow.com/a/32095106/646947 for details.
+ * Prefer these methods in priority order (0 > 1 > 2)
+ */
+#ifndef XXH_FORCE_MEMORY_ACCESS   /* can be defined externally, on command line for example */
+#  if defined(__GNUC__) && ( defined(__ARM_ARCH_6__) || defined(__ARM_ARCH_6J__) \
+                        || defined(__ARM_ARCH_6K__) || defined(__ARM_ARCH_6Z__) \
+                        || defined(__ARM_ARCH_6ZK__) || defined(__ARM_ARCH_6T2__) )
+#    define XXH_FORCE_MEMORY_ACCESS 2
+#  elif (defined(__INTEL_COMPILER) && !defined(_WIN32)) || \
+  (defined(__GNUC__) && ( defined(__ARM_ARCH_7__) || defined(__ARM_ARCH_7A__) \
+                    || defined(__ARM_ARCH_7R__) || defined(__ARM_ARCH_7M__) \
+                    || defined(__ARM_ARCH_7S__) ))
+#    define XXH_FORCE_MEMORY_ACCESS 1
+#  endif
+#endif
+
+/*!XXH_ACCEPT_NULL_INPUT_POINTER :
+ * If input pointer is NULL, xxHash default behavior is to dereference it, triggering a segfault.
+ * When this macro is enabled, xxHash actively checks input for null pointer.
+ * It it is, result for null input pointers is the same as a null-length input.
+ */
+#ifndef XXH_ACCEPT_NULL_INPUT_POINTER   /* can be defined externally */
+#  define XXH_ACCEPT_NULL_INPUT_POINTER 0
+#endif
+
+/*!XXH_FORCE_NATIVE_FORMAT :
+ * By default, xxHash library provides endian-independent Hash values, based on little-endian convention.
+ * Results are therefore identical for little-endian and big-endian CPU.
+ * This comes at a performance cost for big-endian CPU, since some swapping is required to emulate little-endian format.
+ * Should endian-independence be of no importance for your application, you may set the #define below to 1,
+ * to improve speed for Big-endian CPU.
+ * This option has no impact on Little_Endian CPU.
+ */
+#ifndef XXH_FORCE_NATIVE_FORMAT   /* can be defined externally */
+#  define XXH_FORCE_NATIVE_FORMAT 0
+#endif
+
+/*!XXH_FORCE_ALIGN_CHECK :
+ * This is a minor performance trick, only useful with lots of very small keys.
+ * It means : check for aligned/unaligned input.
+ * The check costs one initial branch per hash;
+ * set it to 0 when the input is guaranteed to be aligned,
+ * or when alignment doesn't matter for performance.
+ */
+#ifndef XXH_FORCE_ALIGN_CHECK /* can be defined externally */
+#  if defined(__i386) || defined(_M_IX86) || defined(__x86_64__) || defined(_M_X64)
+#    define XXH_FORCE_ALIGN_CHECK 0
+#  else
+#    define XXH_FORCE_ALIGN_CHECK 1
+#  endif
+#endif
+
+
+/* *************************************
+*  Includes & Memory related functions
+***************************************/
+/*! Modify the local functions below should you wish to use some other memory routines
+*   for malloc(), free() */
+#include <stdlib.h>
+static void* XXH_malloc(size_t s) { return malloc(s); }
+static void  XXH_free  (void* p)  { free(p); }
+/*! and for memcpy() */
+#include <string.h>
+static void* XXH_memcpy(void* dest, const void* src, size_t size) { return memcpy(dest,src,size); }
+
+#include <assert.h>   /* assert */
+
+#define XXH_STATIC_LINKING_ONLY
+#include "xxhash.h"
+
+
+/* *************************************
+*  Compiler Specific Options
+***************************************/
+#ifdef _MSC_VER    /* Visual Studio */
+#  pragma warning(disable : 4127)      /* disable: C4127: conditional expression is constant */
+#  define FORCE_INLINE static __forceinline
+#else
+#  if defined (__cplusplus) || defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L   /* C99 */
+#    ifdef __GNUC__
+#      define FORCE_INLINE static inline __attribute__((always_inline))
+#    else
+#      define FORCE_INLINE static inline
+#    endif
+#  else
+#    define FORCE_INLINE static
+#  endif /* __STDC_VERSION__ */
+#endif
+
+
+/* *************************************
+*  Basic Types
+***************************************/
+#ifndef MEM_MODULE
+# if !defined (__VMS) \
+  && (defined (__cplusplus) \
+  || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) )
+#   include <stdint.h>
+    typedef uint8_t  BYTE;
+    typedef uint16_t U16;
+    typedef uint32_t U32;
+# else
+    typedef unsigned char      BYTE;
+    typedef unsigned short     U16;
+    typedef unsigned int       U32;
+# endif
+#endif
+
+#if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2))
+
+/* Force direct memory access. Only works on CPU which support unaligned memory access in hardware */
+static U32 XXH_read32(const void* memPtr) { return *(const U32*) memPtr; }
+
+#elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1))
+
+/* __pack instructions are safer, but compiler specific, hence potentially problematic for some compilers */
+/* currently only defined for gcc and icc */
+typedef union { U32 u32; } __attribute__((packed)) unalign;
+static U32 XXH_read32(const void* ptr) { return ((const unalign*)ptr)->u32; }
+
+#else
+
+/* portable and safe solution. Generally efficient.
+ * see : http://stackoverflow.com/a/32095106/646947
+ */
+static U32 XXH_read32(const void* memPtr)
+{
+    U32 val;
+    memcpy(&val, memPtr, sizeof(val));
+    return val;
+}
+
+#endif   /* XXH_FORCE_DIRECT_MEMORY_ACCESS */
+
+
+/* ****************************************
+*  Compiler-specific Functions and Macros
+******************************************/
+#define GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__)
+
+/* Note : although _rotl exists for minGW (GCC under windows), performance seems poor */
+#if defined(_MSC_VER)
+#  define XXH_rotl32(x,r) _rotl(x,r)
+#  define XXH_rotl64(x,r) _rotl64(x,r)
+#else
+#  define XXH_rotl32(x,r) ((x << r) | (x >> (32 - r)))
+#  define XXH_rotl64(x,r) ((x << r) | (x >> (64 - r)))
+#endif
+
+#if defined(_MSC_VER)     /* Visual Studio */
+#  define XXH_swap32 _byteswap_ulong
+#elif GCC_VERSION >= 403
+#  define XXH_swap32 __builtin_bswap32
+#else
+static U32 XXH_swap32 (U32 x)
+{
+    return  ((x << 24) & 0xff000000 ) |
+            ((x <<  8) & 0x00ff0000 ) |
+            ((x >>  8) & 0x0000ff00 ) |
+            ((x >> 24) & 0x000000ff );
+}
+#endif
+
+
+/* *************************************
+*  Architecture Macros
+***************************************/
+typedef enum { XXH_bigEndian=0, XXH_littleEndian=1 } XXH_endianess;
+
+/* XXH_CPU_LITTLE_ENDIAN can be defined externally, for example on the compiler command line */
+#ifndef XXH_CPU_LITTLE_ENDIAN
+static int XXH_isLittleEndian(void)
+{
+    const union { U32 u; BYTE c[4]; } one = { 1 };   /* don't use static : performance detrimental  */
+    return one.c[0];
+}
+#   define XXH_CPU_LITTLE_ENDIAN   XXH_isLittleEndian()
+#endif
+
+
+/* ***************************
+*  Memory reads
+*****************************/
+typedef enum { XXH_aligned, XXH_unaligned } XXH_alignment;
+
+FORCE_INLINE U32 XXH_readLE32_align(const void* ptr, XXH_endianess endian, XXH_alignment align)
+{
+    if (align==XXH_unaligned)
+        return endian==XXH_littleEndian ? XXH_read32(ptr) : XXH_swap32(XXH_read32(ptr));
+    else
+        return endian==XXH_littleEndian ? *(const U32*)ptr : XXH_swap32(*(const U32*)ptr);
+}
+
+FORCE_INLINE U32 XXH_readLE32(const void* ptr, XXH_endianess endian)
+{
+    return XXH_readLE32_align(ptr, endian, XXH_unaligned);
+}
+
+static U32 XXH_readBE32(const void* ptr)
+{
+    return XXH_CPU_LITTLE_ENDIAN ? XXH_swap32(XXH_read32(ptr)) : XXH_read32(ptr);
+}
+
+
+/* *************************************
+*  Macros
+***************************************/
+#define XXH_STATIC_ASSERT(c)  { enum { XXH_sa = 1/(int)(!!(c)) }; }  /* use after variable declarations */
+XXH_PUBLIC_API unsigned XXH_versionNumber (void) { return XXH_VERSION_NUMBER; }
+
+
+/* *******************************************************************
+*  32-bit hash functions
+*********************************************************************/
+static const U32 PRIME32_1 = 2654435761U;
+static const U32 PRIME32_2 = 2246822519U;
+static const U32 PRIME32_3 = 3266489917U;
+static const U32 PRIME32_4 =  668265263U;
+static const U32 PRIME32_5 =  374761393U;
+
+static U32 XXH32_round(U32 seed, U32 input)
+{
+    seed += input * PRIME32_2;
+    seed  = XXH_rotl32(seed, 13);
+    seed *= PRIME32_1;
+    return seed;
+}
+
+/* mix all bits */
+static U32 XXH32_avalanche(U32 h32)
+{
+    h32 ^= h32 >> 15;
+    h32 *= PRIME32_2;
+    h32 ^= h32 >> 13;
+    h32 *= PRIME32_3;
+    h32 ^= h32 >> 16;
+    return(h32);
+}
+
+#define XXH_get32bits(p) XXH_readLE32_align(p, endian, align)
+
+static U32
+XXH32_finalize(U32 h32, const void* ptr, size_t len,
+                XXH_endianess endian, XXH_alignment align)
+
+{
+    const BYTE* p = (const BYTE*)ptr;
+#define PROCESS1             \
+    h32 += (*p) * PRIME32_5; \
+    p++;                     \
+    h32 = XXH_rotl32(h32, 11) * PRIME32_1 ;
+
+#define PROCESS4                         \
+    h32 += XXH_get32bits(p) * PRIME32_3; \
+    p+=4;                                \
+    h32  = XXH_rotl32(h32, 17) * PRIME32_4 ;
+
+    switch(len&15)  /* or switch(bEnd - p) */
+    {
+      case 12:      PROCESS4;
+                    /* fallthrough */
+      case 8:       PROCESS4;
+                    /* fallthrough */
+      case 4:       PROCESS4;
+                    return XXH32_avalanche(h32);
+
+      case 13:      PROCESS4;
+                    /* fallthrough */
+      case 9:       PROCESS4;
+                    /* fallthrough */
+      case 5:       PROCESS4;
+                    PROCESS1;
+                    return XXH32_avalanche(h32);
+
+      case 14:      PROCESS4;
+                    /* fallthrough */
+      case 10:      PROCESS4;
+                    /* fallthrough */
+      case 6:       PROCESS4;
+                    PROCESS1;
+                    PROCESS1;
+                    return XXH32_avalanche(h32);
+
+      case 15:      PROCESS4;
+                    /* fallthrough */
+      case 11:      PROCESS4;
+                    /* fallthrough */
+      case 7:       PROCESS4;
+                    /* fallthrough */
+      case 3:       PROCESS1;
+                    /* fallthrough */
+      case 2:       PROCESS1;
+                    /* fallthrough */
+      case 1:       PROCESS1;
+                    /* fallthrough */
+      case 0:       return XXH32_avalanche(h32);
+    }
+    assert(0);
+    return h32;   /* reaching this point is deemed impossible */
+}
+
+
+FORCE_INLINE U32
+XXH32_endian_align(const void* input, size_t len, U32 seed,
+                    XXH_endianess endian, XXH_alignment align)
+{
+    const BYTE* p = (const BYTE*)input;
+    const BYTE* bEnd = p + len;
+    U32 h32;
+
+#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1)
+    if (p==NULL) {
+        len=0;
+        bEnd=p=(const BYTE*)(size_t)16;
+    }
+#endif
+
+    if (len>=16) {
+        const BYTE* const limit = bEnd - 15;
+        U32 v1 = seed + PRIME32_1 + PRIME32_2;
+        U32 v2 = seed + PRIME32_2;
+        U32 v3 = seed + 0;
+        U32 v4 = seed - PRIME32_1;
+
+        do {
+            v1 = XXH32_round(v1, XXH_get32bits(p)); p+=4;
+            v2 = XXH32_round(v2, XXH_get32bits(p)); p+=4;
+            v3 = XXH32_round(v3, XXH_get32bits(p)); p+=4;
+            v4 = XXH32_round(v4, XXH_get32bits(p)); p+=4;
+        } while (p < limit);
+
+        h32 = XXH_rotl32(v1, 1)  + XXH_rotl32(v2, 7)
+            + XXH_rotl32(v3, 12) + XXH_rotl32(v4, 18);
+    } else {
+        h32  = seed + PRIME32_5;
+    }
+
+    h32 += (U32)len;
+
+    return XXH32_finalize(h32, p, len&15, endian, align);
+}
+
+
+XXH_PUBLIC_API unsigned int XXH32 (const void* input, size_t len, unsigned int seed)
+{
+#if 0
+    /* Simple version, good for code maintenance, but unfortunately slow for small inputs */
+    XXH32_state_t state;
+    XXH32_reset(&state, seed);
+    XXH32_update(&state, input, len);
+    return XXH32_digest(&state);
+#else
+    XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN;
+
+    if (XXH_FORCE_ALIGN_CHECK) {
+        if ((((size_t)input) & 3) == 0) {   /* Input is 4-bytes aligned, leverage the speed benefit */
+            if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT)
+                return XXH32_endian_align(input, len, seed, XXH_littleEndian, XXH_aligned);
+            else
+                return XXH32_endian_align(input, len, seed, XXH_bigEndian, XXH_aligned);
+    }   }
+
+    if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT)
+        return XXH32_endian_align(input, len, seed, XXH_littleEndian, XXH_unaligned);
+    else
+        return XXH32_endian_align(input, len, seed, XXH_bigEndian, XXH_unaligned);
+#endif
+}
+
+
+
+/*======   Hash streaming   ======*/
+
+XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void)
+{
+    return (XXH32_state_t*)XXH_malloc(sizeof(XXH32_state_t));
+}
+XXH_PUBLIC_API XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr)
+{
+    XXH_free(statePtr);
+    return XXH_OK;
+}
+
+XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* dstState, const XXH32_state_t* srcState)
+{
+    memcpy(dstState, srcState, sizeof(*dstState));
+}
+
+XXH_PUBLIC_API XXH_errorcode XXH32_reset(XXH32_state_t* statePtr, unsigned int seed)
+{
+    XXH32_state_t state;   /* using a local state to memcpy() in order to avoid strict-aliasing warnings */
+    memset(&state, 0, sizeof(state));
+    state.v1 = seed + PRIME32_1 + PRIME32_2;
+    state.v2 = seed + PRIME32_2;
+    state.v3 = seed + 0;
+    state.v4 = seed - PRIME32_1;
+    /* do not write into reserved, planned to be removed in a future version */
+    memcpy(statePtr, &state, sizeof(state) - sizeof(state.reserved));
+    return XXH_OK;
+}
+
+
+FORCE_INLINE
+XXH_errorcode XXH32_update_endian (XXH32_state_t* state, const void* input, size_t len, XXH_endianess endian)
+{
+    const BYTE* p = (const BYTE*)input;
+    const BYTE* const bEnd = p + len;
+
+    if (input==NULL)
+#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1)
+        return XXH_OK;
+#else
+        return XXH_ERROR;
+#endif
+
+    state->total_len_32 += (unsigned)len;
+    state->large_len |= (len>=16) | (state->total_len_32>=16);
+
+    if (state->memsize + len < 16)  {   /* fill in tmp buffer */
+        XXH_memcpy((BYTE*)(state->mem32) + state->memsize, input, len);
+        state->memsize += (unsigned)len;
+        return XXH_OK;
+    }
+
+    if (state->memsize) {   /* some data left from previous update */
+        XXH_memcpy((BYTE*)(state->mem32) + state->memsize, input, 16-state->memsize);
+        {   const U32* p32 = state->mem32;
+            state->v1 = XXH32_round(state->v1, XXH_readLE32(p32, endian)); p32++;
+            state->v2 = XXH32_round(state->v2, XXH_readLE32(p32, endian)); p32++;
+            state->v3 = XXH32_round(state->v3, XXH_readLE32(p32, endian)); p32++;
+            state->v4 = XXH32_round(state->v4, XXH_readLE32(p32, endian));
+        }
+        p += 16-state->memsize;
+        state->memsize = 0;
+    }
+
+    if (p <= bEnd-16) {
+        const BYTE* const limit = bEnd - 16;
+        U32 v1 = state->v1;
+        U32 v2 = state->v2;
+        U32 v3 = state->v3;
+        U32 v4 = state->v4;
+
+        do {
+            v1 = XXH32_round(v1, XXH_readLE32(p, endian)); p+=4;
+            v2 = XXH32_round(v2, XXH_readLE32(p, endian)); p+=4;
+            v3 = XXH32_round(v3, XXH_readLE32(p, endian)); p+=4;
+            v4 = XXH32_round(v4, XXH_readLE32(p, endian)); p+=4;
+        } while (p<=limit);
+
+        state->v1 = v1;
+        state->v2 = v2;
+        state->v3 = v3;
+        state->v4 = v4;
+    }
+
+    if (p < bEnd) {
+        XXH_memcpy(state->mem32, p, (size_t)(bEnd-p));
+        state->memsize = (unsigned)(bEnd-p);
+    }
+
+    return XXH_OK;
+}
+
+
+XXH_PUBLIC_API XXH_errorcode XXH32_update (XXH32_state_t* state_in, const void* input, size_t len)
+{
+    XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN;
+
+    if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT)
+        return XXH32_update_endian(state_in, input, len, XXH_littleEndian);
+    else
+        return XXH32_update_endian(state_in, input, len, XXH_bigEndian);
+}
+
+
+FORCE_INLINE U32
+XXH32_digest_endian (const XXH32_state_t* state, XXH_endianess endian)
+{
+    U32 h32;
+
+    if (state->large_len) {
+        h32 = XXH_rotl32(state->v1, 1)
+            + XXH_rotl32(state->v2, 7)
+            + XXH_rotl32(state->v3, 12)
+            + XXH_rotl32(state->v4, 18);
+    } else {
+        h32 = state->v3 /* == seed */ + PRIME32_5;
+    }
+
+    h32 += state->total_len_32;
+
+    return XXH32_finalize(h32, state->mem32, state->memsize, endian, XXH_aligned);
+}
+
+
+XXH_PUBLIC_API unsigned int XXH32_digest (const XXH32_state_t* state_in)
+{
+    XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN;
+
+    if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT)
+        return XXH32_digest_endian(state_in, XXH_littleEndian);
+    else
+        return XXH32_digest_endian(state_in, XXH_bigEndian);
+}
+
+
+/*======   Canonical representation   ======*/
+
+/*! Default XXH result types are basic unsigned 32 and 64 bits.
+*   The canonical representation follows human-readable write convention, aka big-endian (large digits first).
+*   These functions allow transformation of hash result into and from its canonical format.
+*   This way, hash values can be written into a file or buffer, remaining comparable across different systems.
+*/
+
+XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash)
+{
+    XXH_STATIC_ASSERT(sizeof(XXH32_canonical_t) == sizeof(XXH32_hash_t));
+    if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap32(hash);
+    memcpy(dst, &hash, sizeof(*dst));
+}
+
+XXH_PUBLIC_API XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src)
+{
+    return XXH_readBE32(src);
+}
+
+
+#ifndef XXH_NO_LONG_LONG
+
+/* *******************************************************************
+*  64-bit hash functions
+*********************************************************************/
+
+/*======   Memory access   ======*/
+
+#ifndef MEM_MODULE
+# define MEM_MODULE
+# if !defined (__VMS) \
+  && (defined (__cplusplus) \
+  || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) )
+#   include <stdint.h>
+    typedef uint64_t U64;
+# else
+    /* if compiler doesn't support unsigned long long, replace by another 64-bit type */
+    typedef unsigned long long U64;
+# endif
+#endif
+
+
+#if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2))
+
+/* Force direct memory access. Only works on CPU which support unaligned memory access in hardware */
+static U64 XXH_read64(const void* memPtr) { return *(const U64*) memPtr; }
+
+#elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1))
+
+/* __pack instructions are safer, but compiler specific, hence potentially problematic for some compilers */
+/* currently only defined for gcc and icc */
+typedef union { U32 u32; U64 u64; } __attribute__((packed)) unalign64;
+static U64 XXH_read64(const void* ptr) { return ((const unalign64*)ptr)->u64; }
+
+#else
+
+/* portable and safe solution. Generally efficient.
+ * see : http://stackoverflow.com/a/32095106/646947
+ */
+
+static U64 XXH_read64(const void* memPtr)
+{
+    U64 val;
+    memcpy(&val, memPtr, sizeof(val));
+    return val;
+}
+
+#endif   /* XXH_FORCE_DIRECT_MEMORY_ACCESS */
+
+#if defined(_MSC_VER)     /* Visual Studio */
+#  define XXH_swap64 _byteswap_uint64
+#elif GCC_VERSION >= 403
+#  define XXH_swap64 __builtin_bswap64
+#else
+static U64 XXH_swap64 (U64 x)
+{
+    return  ((x << 56) & 0xff00000000000000ULL) |
+            ((x << 40) & 0x00ff000000000000ULL) |
+            ((x << 24) & 0x0000ff0000000000ULL) |
+            ((x << 8)  & 0x000000ff00000000ULL) |
+            ((x >> 8)  & 0x00000000ff000000ULL) |
+            ((x >> 24) & 0x0000000000ff0000ULL) |
+            ((x >> 40) & 0x000000000000ff00ULL) |
+            ((x >> 56) & 0x00000000000000ffULL);
+}
+#endif
+
+FORCE_INLINE U64 XXH_readLE64_align(const void* ptr, XXH_endianess endian, XXH_alignment align)
+{
+    if (align==XXH_unaligned)
+        return endian==XXH_littleEndian ? XXH_read64(ptr) : XXH_swap64(XXH_read64(ptr));
+    else
+        return endian==XXH_littleEndian ? *(const U64*)ptr : XXH_swap64(*(const U64*)ptr);
+}
+
+FORCE_INLINE U64 XXH_readLE64(const void* ptr, XXH_endianess endian)
+{
+    return XXH_readLE64_align(ptr, endian, XXH_unaligned);
+}
+
+static U64 XXH_readBE64(const void* ptr)
+{
+    return XXH_CPU_LITTLE_ENDIAN ? XXH_swap64(XXH_read64(ptr)) : XXH_read64(ptr);
+}
+
+
+/*======   xxh64   ======*/
+
+static const U64 PRIME64_1 = 11400714785074694791ULL;
+static const U64 PRIME64_2 = 14029467366897019727ULL;
+static const U64 PRIME64_3 =  1609587929392839161ULL;
+static const U64 PRIME64_4 =  9650029242287828579ULL;
+static const U64 PRIME64_5 =  2870177450012600261ULL;
+
+static U64 XXH64_round(U64 acc, U64 input)
+{
+    acc += input * PRIME64_2;
+    acc  = XXH_rotl64(acc, 31);
+    acc *= PRIME64_1;
+    return acc;
+}
+
+static U64 XXH64_mergeRound(U64 acc, U64 val)
+{
+    val  = XXH64_round(0, val);
+    acc ^= val;
+    acc  = acc * PRIME64_1 + PRIME64_4;
+    return acc;
+}
+
+static U64 XXH64_avalanche(U64 h64)
+{
+    h64 ^= h64 >> 33;
+    h64 *= PRIME64_2;
+    h64 ^= h64 >> 29;
+    h64 *= PRIME64_3;
+    h64 ^= h64 >> 32;
+    return h64;
+}
+
+
+#define XXH_get64bits(p) XXH_readLE64_align(p, endian, align)
+
+static U64
+XXH64_finalize(U64 h64, const void* ptr, size_t len,
+               XXH_endianess endian, XXH_alignment align)
+{
+    const BYTE* p = (const BYTE*)ptr;
+
+#define PROCESS1_64          \
+    h64 ^= (*p) * PRIME64_5; \
+    p++;                     \
+    h64 = XXH_rotl64(h64, 11) * PRIME64_1;
+
+#define PROCESS4_64          \
+    h64 ^= (U64)(XXH_get32bits(p)) * PRIME64_1; \
+    p+=4;                    \
+    h64 = XXH_rotl64(h64, 23) * PRIME64_2 + PRIME64_3;
+
+#define PROCESS8_64 {        \
+    U64 const k1 = XXH64_round(0, XXH_get64bits(p)); \
+    p+=8;                    \
+    h64 ^= k1;               \
+    h64  = XXH_rotl64(h64,27) * PRIME64_1 + PRIME64_4; \
+}
+
+    switch(len&31) {
+      case 24: PROCESS8_64;
+                    /* fallthrough */
+      case 16: PROCESS8_64;
+                    /* fallthrough */
+      case  8: PROCESS8_64;
+               return XXH64_avalanche(h64);
+
+      case 28: PROCESS8_64;
+                    /* fallthrough */
+      case 20: PROCESS8_64;
+                    /* fallthrough */
+      case 12: PROCESS8_64;
+                    /* fallthrough */
+      case  4: PROCESS4_64;
+               return XXH64_avalanche(h64);
+
+      case 25: PROCESS8_64;
+                    /* fallthrough */
+      case 17: PROCESS8_64;
+                    /* fallthrough */
+      case  9: PROCESS8_64;
+               PROCESS1_64;
+               return XXH64_avalanche(h64);
+
+      case 29: PROCESS8_64;
+                    /* fallthrough */
+      case 21: PROCESS8_64;
+                    /* fallthrough */
+      case 13: PROCESS8_64;
+                    /* fallthrough */
+      case  5: PROCESS4_64;
+               PROCESS1_64;
+               return XXH64_avalanche(h64);
+
+      case 26: PROCESS8_64;
+                    /* fallthrough */
+      case 18: PROCESS8_64;
+                    /* fallthrough */
+      case 10: PROCESS8_64;
+               PROCESS1_64;
+               PROCESS1_64;
+               return XXH64_avalanche(h64);
+
+      case 30: PROCESS8_64;
+                    /* fallthrough */
+      case 22: PROCESS8_64;
+                    /* fallthrough */
+      case 14: PROCESS8_64;
+                    /* fallthrough */
+      case  6: PROCESS4_64;
+               PROCESS1_64;
+               PROCESS1_64;
+               return XXH64_avalanche(h64);
+
+      case 27: PROCESS8_64;
+                    /* fallthrough */
+      case 19: PROCESS8_64;
+                    /* fallthrough */
+      case 11: PROCESS8_64;
+               PROCESS1_64;
+               PROCESS1_64;
+               PROCESS1_64;
+               return XXH64_avalanche(h64);
+
+      case 31: PROCESS8_64;
+                    /* fallthrough */
+      case 23: PROCESS8_64;
+                    /* fallthrough */
+      case 15: PROCESS8_64;
+                    /* fallthrough */
+      case  7: PROCESS4_64;
+                    /* fallthrough */
+      case  3: PROCESS1_64;
+                    /* fallthrough */
+      case  2: PROCESS1_64;
+                    /* fallthrough */
+      case  1: PROCESS1_64;
+                    /* fallthrough */
+      case  0: return XXH64_avalanche(h64);
+    }
+
+    /* impossible to reach */
+    assert(0);
+    return 0;  /* unreachable, but some compilers complain without it */
+}
+
+FORCE_INLINE U64
+XXH64_endian_align(const void* input, size_t len, U64 seed,
+                XXH_endianess endian, XXH_alignment align)
+{
+    const BYTE* p = (const BYTE*)input;
+    const BYTE* bEnd = p + len;
+    U64 h64;
+
+#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1)
+    if (p==NULL) {
+        len=0;
+        bEnd=p=(const BYTE*)(size_t)32;
+    }
+#endif
+
+    if (len>=32) {
+        const BYTE* const limit = bEnd - 32;
+        U64 v1 = seed + PRIME64_1 + PRIME64_2;
+        U64 v2 = seed + PRIME64_2;
+        U64 v3 = seed + 0;
+        U64 v4 = seed - PRIME64_1;
+
+        do {
+            v1 = XXH64_round(v1, XXH_get64bits(p)); p+=8;
+            v2 = XXH64_round(v2, XXH_get64bits(p)); p+=8;
+            v3 = XXH64_round(v3, XXH_get64bits(p)); p+=8;
+            v4 = XXH64_round(v4, XXH_get64bits(p)); p+=8;
+        } while (p<=limit);
+
+        h64 = XXH_rotl64(v1, 1) + XXH_rotl64(v2, 7) + XXH_rotl64(v3, 12) + XXH_rotl64(v4, 18);
+        h64 = XXH64_mergeRound(h64, v1);
+        h64 = XXH64_mergeRound(h64, v2);
+        h64 = XXH64_mergeRound(h64, v3);
+        h64 = XXH64_mergeRound(h64, v4);
+
+    } else {
+        h64  = seed + PRIME64_5;
+    }
+
+    h64 += (U64) len;
+
+    return XXH64_finalize(h64, p, len, endian, align);
+}
+
+
+XXH_PUBLIC_API unsigned long long XXH64 (const void* input, size_t len, unsigned long long seed)
+{
+#if 0
+    /* Simple version, good for code maintenance, but unfortunately slow for small inputs */
+    XXH64_state_t state;
+    XXH64_reset(&state, seed);
+    XXH64_update(&state, input, len);
+    return XXH64_digest(&state);
+#else
+    XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN;
+
+    if (XXH_FORCE_ALIGN_CHECK) {
+        if ((((size_t)input) & 7)==0) {  /* Input is aligned, let's leverage the speed advantage */
+            if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT)
+                return XXH64_endian_align(input, len, seed, XXH_littleEndian, XXH_aligned);
+            else
+                return XXH64_endian_align(input, len, seed, XXH_bigEndian, XXH_aligned);
+    }   }
+
+    if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT)
+        return XXH64_endian_align(input, len, seed, XXH_littleEndian, XXH_unaligned);
+    else
+        return XXH64_endian_align(input, len, seed, XXH_bigEndian, XXH_unaligned);
+#endif
+}
+
+/*======   Hash Streaming   ======*/
+
+XXH_PUBLIC_API XXH64_state_t* XXH64_createState(void)
+{
+    return (XXH64_state_t*)XXH_malloc(sizeof(XXH64_state_t));
+}
+XXH_PUBLIC_API XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr)
+{
+    XXH_free(statePtr);
+    return XXH_OK;
+}
+
+XXH_PUBLIC_API void XXH64_copyState(XXH64_state_t* dstState, const XXH64_state_t* srcState)
+{
+    memcpy(dstState, srcState, sizeof(*dstState));
+}
+
+XXH_PUBLIC_API XXH_errorcode XXH64_reset(XXH64_state_t* statePtr, unsigned long long seed)
+{
+    XXH64_state_t state;   /* using a local state to memcpy() in order to avoid strict-aliasing warnings */
+    memset(&state, 0, sizeof(state));
+    state.v1 = seed + PRIME64_1 + PRIME64_2;
+    state.v2 = seed + PRIME64_2;
+    state.v3 = seed + 0;
+    state.v4 = seed - PRIME64_1;
+     /* do not write into reserved, planned to be removed in a future version */
+    memcpy(statePtr, &state, sizeof(state) - sizeof(state.reserved));
+    return XXH_OK;
+}
+
+FORCE_INLINE
+XXH_errorcode XXH64_update_endian (XXH64_state_t* state, const void* input, size_t len, XXH_endianess endian)
+{
+    const BYTE* p = (const BYTE*)input;
+    const BYTE* const bEnd = p + len;
+
+    if (input==NULL)
+#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1)
+        return XXH_OK;
+#else
+        return XXH_ERROR;
+#endif
+
+    state->total_len += len;
+
+    if (state->memsize + len < 32) {  /* fill in tmp buffer */
+        XXH_memcpy(((BYTE*)state->mem64) + state->memsize, input, len);
+        state->memsize += (U32)len;
+        return XXH_OK;
+    }
+
+    if (state->memsize) {   /* tmp buffer is full */
+        XXH_memcpy(((BYTE*)state->mem64) + state->memsize, input, 32-state->memsize);
+        state->v1 = XXH64_round(state->v1, XXH_readLE64(state->mem64+0, endian));
+        state->v2 = XXH64_round(state->v2, XXH_readLE64(state->mem64+1, endian));
+        state->v3 = XXH64_round(state->v3, XXH_readLE64(state->mem64+2, endian));
+        state->v4 = XXH64_round(state->v4, XXH_readLE64(state->mem64+3, endian));
+        p += 32-state->memsize;
+        state->memsize = 0;
+    }
+
+    if (p+32 <= bEnd) {
+        const BYTE* const limit = bEnd - 32;
+        U64 v1 = state->v1;
+        U64 v2 = state->v2;
+        U64 v3 = state->v3;
+        U64 v4 = state->v4;
+
+        do {
+            v1 = XXH64_round(v1, XXH_readLE64(p, endian)); p+=8;
+            v2 = XXH64_round(v2, XXH_readLE64(p, endian)); p+=8;
+            v3 = XXH64_round(v3, XXH_readLE64(p, endian)); p+=8;
+            v4 = XXH64_round(v4, XXH_readLE64(p, endian)); p+=8;
+        } while (p<=limit);
+
+        state->v1 = v1;
+        state->v2 = v2;
+        state->v3 = v3;
+        state->v4 = v4;
+    }
+
+    if (p < bEnd) {
+        XXH_memcpy(state->mem64, p, (size_t)(bEnd-p));
+        state->memsize = (unsigned)(bEnd-p);
+    }
+
+    return XXH_OK;
+}
+
+XXH_PUBLIC_API XXH_errorcode XXH64_update (XXH64_state_t* state_in, const void* input, size_t len)
+{
+    XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN;
+
+    if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT)
+        return XXH64_update_endian(state_in, input, len, XXH_littleEndian);
+    else
+        return XXH64_update_endian(state_in, input, len, XXH_bigEndian);
+}
+
+FORCE_INLINE U64 XXH64_digest_endian (const XXH64_state_t* state, XXH_endianess endian)
+{
+    U64 h64;
+
+    if (state->total_len >= 32) {
+        U64 const v1 = state->v1;
+        U64 const v2 = state->v2;
+        U64 const v3 = state->v3;
+        U64 const v4 = state->v4;
+
+        h64 = XXH_rotl64(v1, 1) + XXH_rotl64(v2, 7) + XXH_rotl64(v3, 12) + XXH_rotl64(v4, 18);
+        h64 = XXH64_mergeRound(h64, v1);
+        h64 = XXH64_mergeRound(h64, v2);
+        h64 = XXH64_mergeRound(h64, v3);
+        h64 = XXH64_mergeRound(h64, v4);
+    } else {
+        h64  = state->v3 /*seed*/ + PRIME64_5;
+    }
+
+    h64 += (U64) state->total_len;
+
+    return XXH64_finalize(h64, state->mem64, (size_t)state->total_len, endian, XXH_aligned);
+}
+
+XXH_PUBLIC_API unsigned long long XXH64_digest (const XXH64_state_t* state_in)
+{
+    XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN;
+
+    if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT)
+        return XXH64_digest_endian(state_in, XXH_littleEndian);
+    else
+        return XXH64_digest_endian(state_in, XXH_bigEndian);
+}
+
+
+/*====== Canonical representation   ======*/
+
+XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH64_canonical_t* dst, XXH64_hash_t hash)
+{
+    XXH_STATIC_ASSERT(sizeof(XXH64_canonical_t) == sizeof(XXH64_hash_t));
+    if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap64(hash);
+    memcpy(dst, &hash, sizeof(*dst));
+}
+
+XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src)
+{
+    return XXH_readBE64(src);
+}
+
+#endif  /* XXH_NO_LONG_LONG */
diff --git a/contrib/xxhash/xxhash.h b/contrib/xxhash/xxhash.h
new file mode 100644
index 00000000000..d6bad943358
--- /dev/null
+++ b/contrib/xxhash/xxhash.h
@@ -0,0 +1,328 @@
+/*
+   xxHash - Extremely Fast Hash algorithm
+   Header File
+   Copyright (C) 2012-2016, Yann Collet.
+
+   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
+
+   Redistribution and use in source and binary forms, with or without
+   modification, are permitted provided that the following conditions are
+   met:
+
+       * Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer.
+       * Redistributions in binary form must reproduce the above
+   copyright notice, this list of conditions and the following disclaimer
+   in the documentation and/or other materials provided with the
+   distribution.
+
+   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+   You can contact the author at :
+   - xxHash source repository : https://github.com/Cyan4973/xxHash
+*/
+
+/* Notice extracted from xxHash homepage :
+
+xxHash is an extremely fast Hash algorithm, running at RAM speed limits.
+It also successfully passes all tests from the SMHasher suite.
+
+Comparison (single thread, Windows Seven 32 bits, using SMHasher on a Core 2 Duo @3GHz)
+
+Name            Speed       Q.Score   Author
+xxHash          5.4 GB/s     10
+CrapWow         3.2 GB/s      2       Andrew
+MumurHash 3a    2.7 GB/s     10       Austin Appleby
+SpookyHash      2.0 GB/s     10       Bob Jenkins
+SBox            1.4 GB/s      9       Bret Mulvey
+Lookup3         1.2 GB/s      9       Bob Jenkins
+SuperFastHash   1.2 GB/s      1       Paul Hsieh
+CityHash64      1.05 GB/s    10       Pike & Alakuijala
+FNV             0.55 GB/s     5       Fowler, Noll, Vo
+CRC32           0.43 GB/s     9
+MD5-32          0.33 GB/s    10       Ronald L. Rivest
+SHA1-32         0.28 GB/s    10
+
+Q.Score is a measure of quality of the hash function.
+It depends on successfully passing SMHasher test set.
+10 is a perfect score.
+
+A 64-bit version, named XXH64, is available since r35.
+It offers much better speed, but for 64-bit applications only.
+Name     Speed on 64 bits    Speed on 32 bits
+XXH64       13.8 GB/s            1.9 GB/s
+XXH32        6.8 GB/s            6.0 GB/s
+*/
+
+#ifndef XXHASH_H_5627135585666179
+#define XXHASH_H_5627135585666179 1
+
+#if defined (__cplusplus)
+extern "C" {
+#endif
+
+
+/* ****************************
+*  Definitions
+******************************/
+#include <stddef.h>   /* size_t */
+typedef enum { XXH_OK=0, XXH_ERROR } XXH_errorcode;
+
+
+/* ****************************
+ *  API modifier
+ ******************************/
+/** XXH_INLINE_ALL (and XXH_PRIVATE_API)
+ *  This is useful to include xxhash functions in `static` mode
+ *  in order to inline them, and remove their symbol from the public list.
+ *  Inlining can offer dramatic performance improvement on small keys.
+ *  Methodology :
+ *     #define XXH_INLINE_ALL
+ *     #include "xxhash.h"
+ * `xxhash.c` is automatically included.
+ *  It's not useful to compile and link it as a separate module.
+ */
+#if defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API)
+#  ifndef XXH_STATIC_LINKING_ONLY
+#    define XXH_STATIC_LINKING_ONLY
+#  endif
+#  if defined(__GNUC__)
+#    define XXH_PUBLIC_API static __inline __attribute__((unused))
+#  elif defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */)
+#    define XXH_PUBLIC_API static inline
+#  elif defined(_MSC_VER)
+#    define XXH_PUBLIC_API static __inline
+#  else
+     /* this version may generate warnings for unused static functions */
+#    define XXH_PUBLIC_API static
+#  endif
+#else
+#  define XXH_PUBLIC_API   /* do nothing */
+#endif /* XXH_INLINE_ALL || XXH_PRIVATE_API */
+
+/*! XXH_NAMESPACE, aka Namespace Emulation :
+ *
+ * If you want to include _and expose_ xxHash functions from within your own library,
+ * but also want to avoid symbol collisions with other libraries which may also include xxHash,
+ *
+ * you can use XXH_NAMESPACE, to automatically prefix any public symbol from xxhash library
+ * with the value of XXH_NAMESPACE (therefore, avoid NULL and numeric values).
+ *
+ * Note that no change is required within the calling program as long as it includes `xxhash.h` :
+ * regular symbol name will be automatically translated by this header.
+ */
+#ifdef XXH_NAMESPACE
+#  define XXH_CAT(A,B) A##B
+#  define XXH_NAME2(A,B) XXH_CAT(A,B)
+#  define XXH_versionNumber XXH_NAME2(XXH_NAMESPACE, XXH_versionNumber)
+#  define XXH32 XXH_NAME2(XXH_NAMESPACE, XXH32)
+#  define XXH32_createState XXH_NAME2(XXH_NAMESPACE, XXH32_createState)
+#  define XXH32_freeState XXH_NAME2(XXH_NAMESPACE, XXH32_freeState)
+#  define XXH32_reset XXH_NAME2(XXH_NAMESPACE, XXH32_reset)
+#  define XXH32_update XXH_NAME2(XXH_NAMESPACE, XXH32_update)
+#  define XXH32_digest XXH_NAME2(XXH_NAMESPACE, XXH32_digest)
+#  define XXH32_copyState XXH_NAME2(XXH_NAMESPACE, XXH32_copyState)
+#  define XXH32_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH32_canonicalFromHash)
+#  define XXH32_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH32_hashFromCanonical)
+#  define XXH64 XXH_NAME2(XXH_NAMESPACE, XXH64)
+#  define XXH64_createState XXH_NAME2(XXH_NAMESPACE, XXH64_createState)
+#  define XXH64_freeState XXH_NAME2(XXH_NAMESPACE, XXH64_freeState)
+#  define XXH64_reset XXH_NAME2(XXH_NAMESPACE, XXH64_reset)
+#  define XXH64_update XXH_NAME2(XXH_NAMESPACE, XXH64_update)
+#  define XXH64_digest XXH_NAME2(XXH_NAMESPACE, XXH64_digest)
+#  define XXH64_copyState XXH_NAME2(XXH_NAMESPACE, XXH64_copyState)
+#  define XXH64_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH64_canonicalFromHash)
+#  define XXH64_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH64_hashFromCanonical)
+#endif
+
+
+/* *************************************
+*  Version
+***************************************/
+#define XXH_VERSION_MAJOR    0
+#define XXH_VERSION_MINOR    6
+#define XXH_VERSION_RELEASE  5
+#define XXH_VERSION_NUMBER  (XXH_VERSION_MAJOR *100*100 + XXH_VERSION_MINOR *100 + XXH_VERSION_RELEASE)
+XXH_PUBLIC_API unsigned XXH_versionNumber (void);
+
+
+/*-**********************************************************************
+*  32-bit hash
+************************************************************************/
+typedef unsigned int XXH32_hash_t;
+
+/*! XXH32() :
+    Calculate the 32-bit hash of sequence "length" bytes stored at memory address "input".
+    The memory between input & input+length must be valid (allocated and read-accessible).
+    "seed" can be used to alter the result predictably.
+    Speed on Core 2 Duo @ 3 GHz (single thread, SMHasher benchmark) : 5.4 GB/s */
+XXH_PUBLIC_API XXH32_hash_t XXH32 (const void* input, size_t length, unsigned int seed);
+
+/*======   Streaming   ======*/
+typedef struct XXH32_state_s XXH32_state_t;   /* incomplete type */
+XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void);
+XXH_PUBLIC_API XXH_errorcode  XXH32_freeState(XXH32_state_t* statePtr);
+XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* dst_state, const XXH32_state_t* src_state);
+
+XXH_PUBLIC_API XXH_errorcode XXH32_reset  (XXH32_state_t* statePtr, unsigned int seed);
+XXH_PUBLIC_API XXH_errorcode XXH32_update (XXH32_state_t* statePtr, const void* input, size_t length);
+XXH_PUBLIC_API XXH32_hash_t  XXH32_digest (const XXH32_state_t* statePtr);
+
+/*
+ * Streaming functions generate the xxHash of an input provided in multiple segments.
+ * Note that, for small input, they are slower than single-call functions, due to state management.
+ * For small inputs, prefer `XXH32()` and `XXH64()`, which are better optimized.
+ *
+ * XXH state must first be allocated, using XXH*_createState() .
+ *
+ * Start a new hash by initializing state with a seed, using XXH*_reset().
+ *
+ * Then, feed the hash state by calling XXH*_update() as many times as necessary.
+ * The function returns an error code, with 0 meaning OK, and any other value meaning there is an error.
+ *
+ * Finally, a hash value can be produced anytime, by using XXH*_digest().
+ * This function returns the nn-bits hash as an int or long long.
+ *
+ * It's still possible to continue inserting input into the hash state after a digest,
+ * and generate some new hashes later on, by calling again XXH*_digest().
+ *
+ * When done, free XXH state space if it was allocated dynamically.
+ */
+
+/*======   Canonical representation   ======*/
+
+typedef struct { unsigned char digest[4]; } XXH32_canonical_t;
+XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash);
+XXH_PUBLIC_API XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src);
+
+/* Default result type for XXH functions are primitive unsigned 32 and 64 bits.
+ * The canonical representation uses human-readable write convention, aka big-endian (large digits first).
+ * These functions allow transformation of hash result into and from its canonical format.
+ * This way, hash values can be written into a file / memory, and remain comparable on different systems and programs.
+ */
+
+
+#ifndef XXH_NO_LONG_LONG
+/*-**********************************************************************
+*  64-bit hash
+************************************************************************/
+typedef unsigned long long XXH64_hash_t;
+
+/*! XXH64() :
+    Calculate the 64-bit hash of sequence of length "len" stored at memory address "input".
+    "seed" can be used to alter the result predictably.
+    This function runs faster on 64-bit systems, but slower on 32-bit systems (see benchmark).
+*/
+XXH_PUBLIC_API XXH64_hash_t XXH64 (const void* input, size_t length, unsigned long long seed);
+
+/*======   Streaming   ======*/
+typedef struct XXH64_state_s XXH64_state_t;   /* incomplete type */
+XXH_PUBLIC_API XXH64_state_t* XXH64_createState(void);
+XXH_PUBLIC_API XXH_errorcode  XXH64_freeState(XXH64_state_t* statePtr);
+XXH_PUBLIC_API void XXH64_copyState(XXH64_state_t* dst_state, const XXH64_state_t* src_state);
+
+XXH_PUBLIC_API XXH_errorcode XXH64_reset  (XXH64_state_t* statePtr, unsigned long long seed);
+XXH_PUBLIC_API XXH_errorcode XXH64_update (XXH64_state_t* statePtr, const void* input, size_t length);
+XXH_PUBLIC_API XXH64_hash_t  XXH64_digest (const XXH64_state_t* statePtr);
+
+/*======   Canonical representation   ======*/
+typedef struct { unsigned char digest[8]; } XXH64_canonical_t;
+XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH64_canonical_t* dst, XXH64_hash_t hash);
+XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src);
+#endif  /* XXH_NO_LONG_LONG */
+
+
+
+#ifdef XXH_STATIC_LINKING_ONLY
+
+/* ================================================================================================
+   This section contains declarations which are not guaranteed to remain stable.
+   They may change in future versions, becoming incompatible with a different version of the library.
+   These declarations should only be used with static linking.
+   Never use them in association with dynamic linking !
+=================================================================================================== */
+
+/* These definitions are only present to allow
+ * static allocation of XXH state, on stack or in a struct for example.
+ * Never **ever** use members directly. */
+
+#if !defined (__VMS) \
+  && (defined (__cplusplus) \
+  || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) )
+#   include <stdint.h>
+
+struct XXH32_state_s {
+   uint32_t total_len_32;
+   uint32_t large_len;
+   uint32_t v1;
+   uint32_t v2;
+   uint32_t v3;
+   uint32_t v4;
+   uint32_t mem32[4];
+   uint32_t memsize;
+   uint32_t reserved;   /* never read nor write, might be removed in a future version */
+};   /* typedef'd to XXH32_state_t */
+
+struct XXH64_state_s {
+   uint64_t total_len;
+   uint64_t v1;
+   uint64_t v2;
+   uint64_t v3;
+   uint64_t v4;
+   uint64_t mem64[4];
+   uint32_t memsize;
+   uint32_t reserved[2];          /* never read nor write, might be removed in a future version */
+};   /* typedef'd to XXH64_state_t */
+
+# else
+
+struct XXH32_state_s {
+   unsigned total_len_32;
+   unsigned large_len;
+   unsigned v1;
+   unsigned v2;
+   unsigned v3;
+   unsigned v4;
+   unsigned mem32[4];
+   unsigned memsize;
+   unsigned reserved;   /* never read nor write, might be removed in a future version */
+};   /* typedef'd to XXH32_state_t */
+
+#   ifndef XXH_NO_LONG_LONG  /* remove 64-bit support */
+struct XXH64_state_s {
+   unsigned long long total_len;
+   unsigned long long v1;
+   unsigned long long v2;
+   unsigned long long v3;
+   unsigned long long v4;
+   unsigned long long mem64[4];
+   unsigned memsize;
+   unsigned reserved[2];     /* never read nor write, might be removed in a future version */
+};   /* typedef'd to XXH64_state_t */
+#    endif
+
+# endif
+
+
+#if defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API)
+#  include "xxhash.c"   /* include xxhash function bodies as `static`, for inlining */
+#endif
+
+#endif /* XXH_STATIC_LINKING_ONLY */
+
+
+#if defined (__cplusplus)
+}
+#endif
+
+#endif /* XXHASH_H_5627135585666179 */
diff --git a/contrib/xxhash/xxhsum.c b/contrib/xxhash/xxhsum.c
new file mode 100644
index 00000000000..69931f727f0
--- /dev/null
+++ b/contrib/xxhash/xxhsum.c
@@ -0,0 +1,1301 @@
+/*
+*  xxhsum - Command line interface for xxhash algorithms
+*  Copyright (C) Yann Collet 2012-2016
+*
+*  GPL v2 License
+*
+*  This program is free software; you can redistribute it and/or modify
+*  it under the terms of the GNU General Public License as published by
+*  the Free Software Foundation; either version 2 of the License, or
+*  (at your option) any later version.
+*
+*  This program is distributed in the hope that it will be useful,
+*  but WITHOUT ANY WARRANTY; without even the implied warranty of
+*  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+*  GNU General Public License for more details.
+*
+*  You should have received a copy of the GNU General Public License along
+*  with this program; if not, write to the Free Software Foundation, Inc.,
+*  51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+*
+*  You can contact the author at :
+*  - xxHash homepage : http://www.xxhash.com
+*  - xxHash source repository : https://github.com/Cyan4973/xxHash
+*/
+
+/* xxhsum :
+ * Provides hash value of a file content, or a list of files, or stdin
+ * Display convention is Big Endian, for both 32 and 64 bits algorithms
+ */
+
+#ifndef XXHASH_C_2097394837
+#define XXHASH_C_2097394837
+
+/* ************************************
+ *  Compiler Options
+ **************************************/
+/* MS Visual */
+#if defined(_MSC_VER) || defined(_WIN32)
+#  define _CRT_SECURE_NO_WARNINGS   /* removes visual warnings */
+#endif
+
+/* Under Linux at least, pull in the *64 commands */
+#ifndef _LARGEFILE64_SOURCE
+#  define _LARGEFILE64_SOURCE
+#endif
+
+
+/* ************************************
+ *  Includes
+ **************************************/
+#include <stdlib.h>     /* malloc, calloc, free, exit */
+#include <stdio.h>      /* fprintf, fopen, ftello64, fread, stdin, stdout, _fileno (when present) */
+#include <string.h>     /* strcmp */
+#include <sys/types.h>  /* stat, stat64, _stat64 */
+#include <sys/stat.h>   /* stat, stat64, _stat64 */
+#include <time.h>       /* clock_t, clock, CLOCKS_PER_SEC */
+#include <assert.h>     /* assert */
+
+#define XXH_STATIC_LINKING_ONLY   /* *_state_t */
+#include "xxhash.h"
+
+
+/* ************************************
+ *  OS-Specific Includes
+ **************************************/
+#if defined(MSDOS) || defined(OS2) || defined(WIN32) || defined(_WIN32) || defined(__CYGWIN__)
+#  include <fcntl.h>    /* _O_BINARY */
+#  include <io.h>       /* _setmode, _isatty */
+#  define SET_BINARY_MODE(file) _setmode(_fileno(file), _O_BINARY)
+#  define IS_CONSOLE(stdStream) _isatty(_fileno(stdStream))
+#else
+#  include <unistd.h>   /* isatty, STDIN_FILENO */
+#  define SET_BINARY_MODE(file)
+#  define IS_CONSOLE(stdStream) isatty(STDIN_FILENO)
+#endif
+
+#if !defined(S_ISREG)
+#  define S_ISREG(x) (((x) & S_IFMT) == S_IFREG)
+#endif
+
+
+/* ************************************
+*  Basic Types
+**************************************/
+#ifndef MEM_MODULE
+# define MEM_MODULE
+# if defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L   /* C99 */
+#   include <stdint.h>
+    typedef uint8_t  BYTE;
+    typedef uint16_t U16;
+    typedef uint32_t U32;
+    typedef  int32_t S32;
+    typedef uint64_t U64;
+#  else
+    typedef unsigned char      BYTE;
+    typedef unsigned short     U16;
+    typedef unsigned int       U32;
+    typedef   signed int       S32;
+    typedef unsigned long long U64;
+#  endif
+#endif
+
+static unsigned BMK_isLittleEndian(void)
+{
+    const union { U32 u; BYTE c[4]; } one = { 1 };   /* don't use static : performance detrimental  */
+    return one.c[0];
+}
+
+
+/* *************************************
+ *  Constants
+ ***************************************/
+#define LIB_VERSION XXH_VERSION_MAJOR.XXH_VERSION_MINOR.XXH_VERSION_RELEASE
+#define QUOTE(str) #str
+#define EXPAND_AND_QUOTE(str) QUOTE(str)
+#define PROGRAM_VERSION EXPAND_AND_QUOTE(LIB_VERSION)
+static const int g_nbBits = (int)(sizeof(void*)*8);
+static const char g_lename[] = "little endian";
+static const char g_bename[] = "big endian";
+#define ENDIAN_NAME (BMK_isLittleEndian() ? g_lename : g_bename)
+static const char author[] = "Yann Collet";
+#define WELCOME_MESSAGE(exename) "%s %s (%i-bits %s), by %s \n", \
+                    exename, PROGRAM_VERSION, g_nbBits, ENDIAN_NAME, author
+
+#define KB *( 1<<10)
+#define MB *( 1<<20)
+#define GB *(1U<<30)
+
+static size_t XXH_DEFAULT_SAMPLE_SIZE = 100 KB;
+#define NBLOOPS    3                              /* Default number of benchmark iterations */
+#define TIMELOOP_S 1
+#define TIMELOOP  (TIMELOOP_S * CLOCKS_PER_SEC)   /* Minimum timing per iteration */
+#define XXHSUM32_DEFAULT_SEED 0                   /* Default seed for algo_xxh32 */
+#define XXHSUM64_DEFAULT_SEED 0                   /* Default seed for algo_xxh64 */
+
+#define MAX_MEM    (2 GB - 64 MB)
+
+static const char stdinName[] = "-";
+typedef enum { algo_xxh32, algo_xxh64 } algoType;
+static const algoType g_defaultAlgo = algo_xxh64;    /* required within main() & usage() */
+
+/* <16 hex char> <SPC> <SPC> <filename> <'\0'>
+ * '4096' is typical Linux PATH_MAX configuration. */
+#define DEFAULT_LINE_LENGTH (sizeof(XXH64_hash_t) * 2 + 2 + 4096 + 1)
+
+/* Maximum acceptable line length. */
+#define MAX_LINE_LENGTH (32 KB)
+
+
+/* ************************************
+ *  Display macros
+ **************************************/
+#define DISPLAY(...)         fprintf(stderr, __VA_ARGS__)
+#define DISPLAYRESULT(...)   fprintf(stdout, __VA_ARGS__)
+#define DISPLAYLEVEL(l, ...) do { if (g_displayLevel>=l) DISPLAY(__VA_ARGS__); } while (0)
+static int g_displayLevel = 2;
+
+
+/* ************************************
+ *  Local variables
+ **************************************/
+static U32 g_nbIterations = NBLOOPS;
+
+
+/* ************************************
+ *  Benchmark Functions
+ **************************************/
+static clock_t BMK_clockSpan( clock_t start )
+{
+    return clock() - start;   /* works even if overflow; Typical max span ~ 30 mn */
+}
+
+
+static size_t BMK_findMaxMem(U64 requiredMem)
+{
+    size_t const step = 64 MB;
+    void* testmem = NULL;
+
+    requiredMem = (((requiredMem >> 26) + 1) << 26);
+    requiredMem += 2*step;
+    if (requiredMem > MAX_MEM) requiredMem = MAX_MEM;
+
+    while (!testmem) {
+        if (requiredMem > step) requiredMem -= step;
+        else requiredMem >>= 1;
+        testmem = malloc ((size_t)requiredMem);
+    }
+    free (testmem);
+
+    /* keep some space available */
+    if (requiredMem > step) requiredMem -= step;
+    else requiredMem >>= 1;
+
+    return (size_t)requiredMem;
+}
+
+
+static U64 BMK_GetFileSize(const char* infilename)
+{
+    int r;
+#if defined(_MSC_VER)
+    struct _stat64 statbuf;
+    r = _stat64(infilename, &statbuf);
+#else
+    struct stat statbuf;
+    r = stat(infilename, &statbuf);
+#endif
+    if (r || !S_ISREG(statbuf.st_mode)) return 0;   /* No good... */
+    return (U64)statbuf.st_size;
+}
+
+typedef U32 (*hashFunction)(const void* buffer, size_t bufferSize, U32 seed);
+
+static U32 localXXH32(const void* buffer, size_t bufferSize, U32 seed) { return XXH32(buffer, bufferSize, seed); }
+
+static U32 localXXH64(const void* buffer, size_t bufferSize, U32 seed) { return (U32)XXH64(buffer, bufferSize, seed); }
+
+static void BMK_benchHash(hashFunction h, const char* hName, const void* buffer, size_t bufferSize)
+{
+    U32 nbh_perIteration = ((300 MB) / (bufferSize+1)) + 1;  /* first loop conservatively aims for 300 MB/s */
+    U32 iterationNb;
+    double fastestH = 100000000.;
+
+    DISPLAYLEVEL(2, "\r%70s\r", "");       /* Clean display line */
+    if (g_nbIterations<1) g_nbIterations=1;
+    for (iterationNb = 1; iterationNb <= g_nbIterations; iterationNb++) {
+        U32 r=0;
+        clock_t cStart;
+
+        DISPLAYLEVEL(2, "%1i-%-17.17s : %10u ->\r", iterationNb, hName, (U32)bufferSize);
+        cStart = clock();
+        while (clock() == cStart);   /* starts clock() at its exact beginning */
+        cStart = clock();
+
+        {   U32 i;
+            for (i=0; i<nbh_perIteration; i++)
+                r += h(buffer, bufferSize, i);
+        }
+        if (r==0) DISPLAYLEVEL(3,".\r");  /* do something with r to avoid compiler "optimizing" away hash function */
+        {   double const timeS = ((double)BMK_clockSpan(cStart) / CLOCKS_PER_SEC) / nbh_perIteration;
+            if (timeS < fastestH) fastestH = timeS;
+            DISPLAYLEVEL(2, "%1i-%-17.17s : %10u -> %8.0f it/s (%7.1f MB/s) \r",
+                    iterationNb, hName, (U32)bufferSize,
+                    (double)1 / fastestH,
+                    ((double)bufferSize / (1<<20)) / fastestH );
+        }
+        assert(fastestH > 1./2000000000);  /* avoid U32 overflow */
+        nbh_perIteration = (U32)(1 / fastestH) + 1;  /* adjust nbh_perIteration to last roughtly one second */
+    }
+    DISPLAYLEVEL(1, "%-19.19s : %10u -> %8.0f it/s (%7.1f MB/s) \n", hName, (U32)bufferSize,
+        (double)1 / fastestH,
+        ((double)bufferSize / (1<<20)) / fastestH);
+    if (g_displayLevel<1)
+        DISPLAYLEVEL(0, "%u, ", (U32)((double)1 / fastestH));
+}
+
+
+/* BMK_benchMem():
+ * specificTest : 0 == run all tests, 1+ run only specific test
+ * buffer : is supposed 8-bytes aligned (if malloc'ed, it should be)
+ * the real allocated size of buffer is supposed to be >= (bufferSize+3).
+ * @return : 0 on success, 1 if error (invalid mode selected) */
+static int BMK_benchMem(const void* buffer, size_t bufferSize, U32 specificTest)
+{
+    assert((((size_t)buffer) & 8) == 0);  /* ensure alignment */
+
+    /* XXH32 bench */
+    if ((specificTest==0) | (specificTest==1))
+        BMK_benchHash(localXXH32, "XXH32", buffer, bufferSize);
+
+    /* Bench XXH32 on Unaligned input */
+    if ((specificTest==0) | (specificTest==2))
+        BMK_benchHash(localXXH32, "XXH32 unaligned", ((const char*)buffer)+1, bufferSize);
+
+    /* Bench XXH64 */
+    if ((specificTest==0) | (specificTest==3))
+        BMK_benchHash(localXXH64, "XXH64", buffer, bufferSize);
+
+    /* Bench XXH64 on Unaligned input */
+    if ((specificTest==0) | (specificTest==4))
+        BMK_benchHash(localXXH64, "XXH64 unaligned", ((const char*)buffer)+3, bufferSize);
+
+    if (specificTest > 4) {
+        DISPLAY("benchmark mode invalid \n");
+        return 1;
+    }
+    return 0;
+}
+
+
+static size_t BMK_selectBenchedSize(const char* fileName)
+{   U64 const inFileSize = BMK_GetFileSize(fileName);
+    size_t benchedSize = (size_t) BMK_findMaxMem(inFileSize);
+    if ((U64)benchedSize > inFileSize) benchedSize = (size_t)inFileSize;
+    if (benchedSize < inFileSize) {
+        DISPLAY("Not enough memory for '%s' full size; testing %i MB only...\n", fileName, (int)(benchedSize>>20));
+    }
+    return benchedSize;
+}
+
+
+static int BMK_benchFiles(const char** fileNamesTable, int nbFiles, U32 specificTest)
+{
+    int result = 0;
+    int fileIdx;
+
+    for (fileIdx=0; fileIdx<nbFiles; fileIdx++) {
+        const char* const inFileName = fileNamesTable[fileIdx];
+        FILE* const inFile = fopen( inFileName, "rb" );
+        size_t const benchedSize = BMK_selectBenchedSize(inFileName);
+        char* const buffer = (char*)calloc(benchedSize+16+3, 1);
+        void* const alignedBuffer = (buffer+15) - (((size_t)(buffer+15)) & 0xF);  /* align on next 16 bytes */
+
+        /* Checks */
+        if ((inFile==NULL) || (inFileName==NULL)) {
+            DISPLAY("Pb opening %s\n", inFileName);
+            free(buffer);
+            return 11;
+        }
+        if(!buffer) {
+            DISPLAY("\nError: not enough memory!\n");
+            fclose(inFile);
+            return 12;
+        }
+
+        /* Fill input buffer */
+        DISPLAYLEVEL(1, "\rLoading %s...        \n", inFileName);
+        {   size_t const readSize = fread(alignedBuffer, 1, benchedSize, inFile);
+            fclose(inFile);
+            if(readSize != benchedSize) {
+                DISPLAY("\nError: problem reading file '%s' !!    \n", inFileName);
+                free(buffer);
+                return 13;
+        }   }
+
+        /* bench */
+        result |= BMK_benchMem(alignedBuffer, benchedSize, specificTest);
+
+        free(buffer);
+    }
+
+    return result;
+}
+
+
+
+static int BMK_benchInternal(size_t keySize, int specificTest)
+{
+    void* const buffer = calloc(keySize+16+3, 1);
+    void* const alignedBuffer = ((char*)buffer+15) - (((size_t)((char*)buffer+15)) & 0xF);  /* align on next 16 bytes */
+    if(!buffer) {
+        DISPLAY("\nError: not enough memory!\n");
+        return 12;
+    }
+
+    /* bench */
+    DISPLAYLEVEL(1, "Sample of ");
+    if (keySize > 10 KB) {
+        DISPLAYLEVEL(1, "%u KB", (U32)(keySize >> 10));
+    } else {
+        DISPLAYLEVEL(1, "%u bytes", (U32)keySize);
+    }
+    DISPLAYLEVEL(1, "...        \n");
+
+    {   int const result = BMK_benchMem(alignedBuffer, keySize, specificTest);
+        free(buffer);
+        return result;
+    }
+}
+
+
+static void BMK_checkResult(U32 r1, U32 r2)
+{
+    static int nbTests = 1;
+    if (r1==r2) {
+        DISPLAYLEVEL(3, "\rTest%3i : %08X == %08X   ok   ", nbTests, r1, r2);
+    } else {
+        DISPLAY("\rERROR : Test%3i : %08X <> %08X   !!!!!   \n", nbTests, r1, r2);
+        exit(1);
+    }
+    nbTests++;
+}
+
+
+static void BMK_checkResult64(U64 r1, U64 r2)
+{
+    static int nbTests = 1;
+    if (r1!=r2) {
+        DISPLAY("\rERROR : Test%3i : 64-bit values non equals   !!!!!   \n", nbTests);
+        DISPLAY("\r %08X%08X != %08X%08X \n", (U32)(r1>>32), (U32)r1, (U32)(r2>>32), (U32)r2);
+        exit(1);
+    }
+    nbTests++;
+}
+
+
+static void BMK_testSequence64(void* sentence, size_t len, U64 seed, U64 Nresult)
+{
+    XXH64_state_t state;
+    U64 Dresult;
+    size_t pos;
+
+    Dresult = XXH64(sentence, len, seed);
+    BMK_checkResult64(Dresult, Nresult);
+
+    XXH64_reset(&state, seed);
+    XXH64_update(&state, sentence, len);
+    Dresult = XXH64_digest(&state);
+    BMK_checkResult64(Dresult, Nresult);
+
+    XXH64_reset(&state, seed);
+    for (pos=0; pos<len; pos++)
+        XXH64_update(&state, ((char*)sentence)+pos, 1);
+    Dresult = XXH64_digest(&state);
+    BMK_checkResult64(Dresult, Nresult);
+}
+
+
+static void BMK_testSequence(const void* sequence, size_t len, U32 seed, U32 Nresult)
+{
+    XXH32_state_t state;
+    U32 Dresult;
+    size_t pos;
+
+    Dresult = XXH32(sequence, len, seed);
+    BMK_checkResult(Dresult, Nresult);
+
+    XXH32_reset(&state, seed);
+    XXH32_update(&state, sequence, len);
+    Dresult = XXH32_digest(&state);
+    BMK_checkResult(Dresult, Nresult);
+
+    XXH32_reset(&state, seed);
+    for (pos=0; pos<len; pos++)
+        XXH32_update(&state, ((const char*)sequence)+pos, 1);
+    Dresult = XXH32_digest(&state);
+    BMK_checkResult(Dresult, Nresult);
+}
+
+
+#define SANITY_BUFFER_SIZE 101
+static void BMK_sanityCheck(void)
+{
+    static const U32 prime = 2654435761U;
+    BYTE sanityBuffer[SANITY_BUFFER_SIZE];
+    U32 byteGen = prime;
+
+    int i;
+    for (i=0; i<SANITY_BUFFER_SIZE; i++) {
+        sanityBuffer[i] = (BYTE)(byteGen>>24);
+        byteGen *= byteGen;
+    }
+
+    BMK_testSequence(NULL,          0, 0,     0x02CC5D05);
+    BMK_testSequence(NULL,          0, prime, 0x36B78AE7);
+    BMK_testSequence(sanityBuffer,  1, 0,     0xB85CBEE5);
+    BMK_testSequence(sanityBuffer,  1, prime, 0xD5845D64);
+    BMK_testSequence(sanityBuffer, 14, 0,     0xE5AA0AB4);
+    BMK_testSequence(sanityBuffer, 14, prime, 0x4481951D);
+    BMK_testSequence(sanityBuffer, SANITY_BUFFER_SIZE, 0,     0x1F1AA412);
+    BMK_testSequence(sanityBuffer, SANITY_BUFFER_SIZE, prime, 0x498EC8E2);
+
+    BMK_testSequence64(NULL        ,  0, 0,     0xEF46DB3751D8E999ULL);
+    BMK_testSequence64(NULL        ,  0, prime, 0xAC75FDA2929B17EFULL);
+    BMK_testSequence64(sanityBuffer,  1, 0,     0x4FCE394CC88952D8ULL);
+    BMK_testSequence64(sanityBuffer,  1, prime, 0x739840CB819FA723ULL);
+    BMK_testSequence64(sanityBuffer, 14, 0,     0xCFFA8DB881BC3A3DULL);
+    BMK_testSequence64(sanityBuffer, 14, prime, 0x5B9611585EFCC9CBULL);
+    BMK_testSequence64(sanityBuffer, SANITY_BUFFER_SIZE, 0,     0x0EAB543384F878ADULL);
+    BMK_testSequence64(sanityBuffer, SANITY_BUFFER_SIZE, prime, 0xCAA65939306F1E21ULL);
+
+    DISPLAYLEVEL(3, "\r%70s\r", "");       /* Clean display line */
+    DISPLAYLEVEL(3, "Sanity check -- all tests ok\n");
+}
+
+
+/* ********************************************************
+*  File Hashing
+**********************************************************/
+
+static void BMK_display_LittleEndian(const void* ptr, size_t length)
+{
+    const BYTE* p = (const BYTE*)ptr;
+    size_t idx;
+    for (idx=length-1; idx<length; idx--)    /* intentional underflow to negative to detect end */
+        DISPLAYRESULT("%02x", p[idx]);
+}
+
+static void BMK_display_BigEndian(const void* ptr, size_t length)
+{
+    const BYTE* p = (const BYTE*)ptr;
+    size_t idx;
+    for (idx=0; idx<length; idx++)
+        DISPLAYRESULT("%02x", p[idx]);
+}
+
+static void BMK_hashStream(void* xxhHashValue, const algoType hashType, FILE* inFile, void* buffer, size_t blockSize)
+{
+    XXH64_state_t state64;
+    XXH32_state_t state32;
+    size_t readSize;
+
+    /* Init */
+    XXH32_reset(&state32, XXHSUM32_DEFAULT_SEED);
+    XXH64_reset(&state64, XXHSUM64_DEFAULT_SEED);
+
+    /* Load file & update hash */
+    readSize = 1;
+    while (readSize) {
+        readSize = fread(buffer, 1, blockSize, inFile);
+        switch(hashType)
+        {
+        case algo_xxh32:
+            XXH32_update(&state32, buffer, readSize);
+            break;
+        case algo_xxh64:
+            XXH64_update(&state64, buffer, readSize);
+            break;
+        default:
+            break;
+        }
+    }
+
+    switch(hashType)
+    {
+    case algo_xxh32:
+        {   U32 const h32 = XXH32_digest(&state32);
+            memcpy(xxhHashValue, &h32, sizeof(h32));
+            break;
+        }
+    case algo_xxh64:
+        {   U64 const h64 = XXH64_digest(&state64);
+            memcpy(xxhHashValue, &h64, sizeof(h64));
+            break;
+        }
+    default:
+            break;
+    }
+}
+
+
+typedef enum { big_endian, little_endian} endianess;
+
+static int BMK_hash(const char* fileName,
+                    const algoType hashType,
+                    const endianess displayEndianess)
+{
+    FILE*  inFile;
+    size_t const blockSize = 64 KB;
+    void*  buffer;
+    U32    h32 = 0;
+    U64    h64 = 0;
+
+    /* Check file existence */
+    if (fileName == stdinName) {
+        inFile = stdin;
+        SET_BINARY_MODE(stdin);
+    }
+    else
+        inFile = fopen( fileName, "rb" );
+    if (inFile==NULL) {
+        DISPLAY( "Pb opening %s\n", fileName);
+        return 1;
+    }
+
+    /* Memory allocation & restrictions */
+    buffer = malloc(blockSize);
+    if(!buffer) {
+        DISPLAY("\nError: not enough memory!\n");
+        fclose(inFile);
+        return 1;
+    }
+
+    /* loading notification */
+    {   const size_t fileNameSize = strlen(fileName);
+        const char* const fileNameEnd = fileName + fileNameSize;
+        const int maxInfoFilenameSize = (int)(fileNameSize > 30 ? 30 : fileNameSize);
+        int infoFilenameSize = 1;
+        while ((infoFilenameSize < maxInfoFilenameSize)
+            && (fileNameEnd[-1-infoFilenameSize] != '/')
+            && (fileNameEnd[-1-infoFilenameSize] != '\\') )
+              infoFilenameSize++;
+        DISPLAY("\rLoading %s...  \r", fileNameEnd - infoFilenameSize);
+
+        /* Load file & update hash */
+        switch(hashType)
+        {
+        case algo_xxh32:
+            BMK_hashStream(&h32, algo_xxh32, inFile, buffer, blockSize);
+            break;
+        case algo_xxh64:
+            BMK_hashStream(&h64, algo_xxh64, inFile, buffer, blockSize);
+            break;
+        default:
+            break;
+        }
+
+        fclose(inFile);
+        free(buffer);
+        DISPLAY("%s             \r", fileNameEnd - infoFilenameSize);  /* erase line */
+    }
+
+    /* display Hash */
+    switch(hashType)
+    {
+    case algo_xxh32:
+        {   XXH32_canonical_t hcbe32;
+            XXH32_canonicalFromHash(&hcbe32, h32);
+            displayEndianess==big_endian ?
+                BMK_display_BigEndian(&hcbe32, sizeof(hcbe32)) : BMK_display_LittleEndian(&hcbe32, sizeof(hcbe32));
+            DISPLAYRESULT("  %s\n", fileName);
+            break;
+        }
+    case algo_xxh64:
+        {   XXH64_canonical_t hcbe64;
+            XXH64_canonicalFromHash(&hcbe64, h64);
+            displayEndianess==big_endian ?
+                BMK_display_BigEndian(&hcbe64, sizeof(hcbe64)) : BMK_display_LittleEndian(&hcbe64, sizeof(hcbe64));
+            DISPLAYRESULT("  %s\n", fileName);
+            break;
+        }
+    default:
+            break;
+    }
+
+    return 0;
+}
+
+
+static int BMK_hashFiles(const char** fnList, int fnTotal,
+                         algoType hashType, endianess displayEndianess)
+{
+    int fnNb;
+    int result = 0;
+
+    if (fnTotal==0)
+        return BMK_hash(stdinName, hashType, displayEndianess);
+
+    for (fnNb=0; fnNb<fnTotal; fnNb++)
+        result += BMK_hash(fnList[fnNb], hashType, displayEndianess);
+    DISPLAY("\r%70s\r", "");
+    return result;
+}
+
+
+typedef enum {
+    GetLine_ok,
+    GetLine_eof,
+    GetLine_exceedMaxLineLength,
+    GetLine_outOfMemory,
+} GetLineResult;
+
+typedef enum {
+    CanonicalFromString_ok,
+    CanonicalFromString_invalidFormat,
+} CanonicalFromStringResult;
+
+typedef enum {
+    ParseLine_ok,
+    ParseLine_invalidFormat,
+} ParseLineResult;
+
+typedef enum {
+    LineStatus_hashOk,
+    LineStatus_hashFailed,
+    LineStatus_failedToOpen,
+} LineStatus;
+
+typedef union {
+    XXH32_canonical_t xxh32;
+    XXH64_canonical_t xxh64;
+} Canonical;
+
+typedef struct {
+    Canonical   canonical;
+    const char* filename;
+    int         xxhBits;    /* canonical type : 32:xxh32, 64:xxh64 */
+} ParsedLine;
+
+typedef struct {
+    unsigned long   nProperlyFormattedLines;
+    unsigned long   nImproperlyFormattedLines;
+    unsigned long   nMismatchedChecksums;
+    unsigned long   nOpenOrReadFailures;
+    unsigned long   nMixedFormatLines;
+    int             xxhBits;
+    int             quit;
+} ParseFileReport;
+
+typedef struct {
+    const char*     inFileName;
+    FILE*           inFile;
+    int             lineMax;
+    char*           lineBuf;
+    size_t          blockSize;
+    char*           blockBuf;
+    int             strictMode;
+    int             statusOnly;
+    int             warn;
+    int             quiet;
+    ParseFileReport report;
+} ParseFileArg;
+
+
+/*  Read line from stream.
+    Returns GetLine_ok, if it reads line successfully.
+    Returns GetLine_eof, if stream reaches EOF.
+    Returns GetLine_exceedMaxLineLength, if line length is longer than MAX_LINE_LENGTH.
+    Returns GetLine_outOfMemory, if line buffer memory allocation failed.
+ */
+static GetLineResult getLine(char** lineBuf, int* lineMax, FILE* inFile)
+{
+    GetLineResult result = GetLine_ok;
+    int len = 0;
+
+    if ((*lineBuf == NULL) || (*lineMax<1)) {
+        free(*lineBuf);  /* in case it's != NULL */
+        *lineMax = 0;
+        *lineBuf = (char*)malloc(DEFAULT_LINE_LENGTH);
+        if(*lineBuf == NULL) return GetLine_outOfMemory;
+        *lineMax = DEFAULT_LINE_LENGTH;
+    }
+
+    for (;;) {
+        const int c = fgetc(inFile);
+        if (c == EOF) {
+            /* If we meet EOF before first character, returns GetLine_eof,
+             * otherwise GetLine_ok.
+             */
+            if (len == 0) result = GetLine_eof;
+            break;
+        }
+
+        /* Make enough space for len+1 (for final NUL) bytes. */
+        if (len+1 >= *lineMax) {
+            char* newLineBuf = NULL;
+            int newBufSize = *lineMax;
+
+            newBufSize += (newBufSize/2) + 1; /* x 1.5 */
+            if (newBufSize > MAX_LINE_LENGTH) newBufSize = MAX_LINE_LENGTH;
+            if (len+1 >= newBufSize) return GetLine_exceedMaxLineLength;
+
+            newLineBuf = (char*) realloc(*lineBuf, newBufSize);
+            if (newLineBuf == NULL) return GetLine_outOfMemory;
+
+            *lineBuf = newLineBuf;
+            *lineMax = newBufSize;
+        }
+
+        if (c == '\n') break;
+        (*lineBuf)[len++] = (char) c;
+    }
+
+    (*lineBuf)[len] = '\0';
+    return result;
+}
+
+
+/*  Converts one hexadecimal character to integer.
+ *  Returns -1, if given character is not hexadecimal.
+ */
+static int charToHex(char c)
+{
+    int result = -1;
+    if (c >= '0' && c <= '9') {
+        result = (int) (c - '0');
+    } else if (c >= 'A' && c <= 'F') {
+        result = (int) (c - 'A') + 0x0a;
+    } else if (c >= 'a' && c <= 'f') {
+        result = (int) (c - 'a') + 0x0a;
+    }
+    return result;
+}
+
+
+/*  Converts XXH32 canonical hexadecimal string hashStr to big endian unsigned char array dst.
+ *  Returns CANONICAL_FROM_STRING_INVALID_FORMAT, if hashStr is not well formatted.
+ *  Returns CANONICAL_FROM_STRING_OK, if hashStr is parsed successfully.
+ */
+static CanonicalFromStringResult canonicalFromString(unsigned char* dst,
+                                                     size_t dstSize,
+                                                     const char* hashStr)
+{
+    size_t i;
+    for (i = 0; i < dstSize; ++i) {
+        int h0, h1;
+
+        h0 = charToHex(hashStr[i*2 + 0]);
+        if (h0 < 0) return CanonicalFromString_invalidFormat;
+
+        h1 = charToHex(hashStr[i*2 + 1]);
+        if (h1 < 0) return CanonicalFromString_invalidFormat;
+
+        dst[i] = (unsigned char) ((h0 << 4) | h1);
+    }
+    return CanonicalFromString_ok;
+}
+
+
+/*  Parse single line of xxHash checksum file.
+ *  Returns PARSE_LINE_ERROR_INVALID_FORMAT, if line is not well formatted.
+ *  Returns PARSE_LINE_OK if line is parsed successfully.
+ *  And members of parseLine will be filled by parsed values.
+ *
+ *  - line must be ended with '\0'.
+ *  - Since parsedLine.filename will point within given argument `line`,
+ *    users must keep `line`s content during they are using parsedLine.
+ *
+ *  Given xxHash checksum line should have the following format:
+ *
+ *      <8 or 16 hexadecimal char> <space> <space> <filename...> <'\0'>
+ */
+static ParseLineResult parseLine(ParsedLine* parsedLine, const char* line)
+{
+    const char* const firstSpace = strchr(line, ' ');
+    const char* const secondSpace = firstSpace + 1;
+
+    parsedLine->filename = NULL;
+    parsedLine->xxhBits = 0;
+
+    if (firstSpace == NULL || *secondSpace != ' ') return ParseLine_invalidFormat;
+
+    switch (firstSpace - line)
+    {
+    case 8:
+        {   XXH32_canonical_t* xxh32c = &parsedLine->canonical.xxh32;
+            if (canonicalFromString(xxh32c->digest, sizeof(xxh32c->digest), line)
+                != CanonicalFromString_ok) {
+                return ParseLine_invalidFormat;
+            }
+            parsedLine->xxhBits = 32;
+            break;
+        }
+
+    case 16:
+        {   XXH64_canonical_t* xxh64c = &parsedLine->canonical.xxh64;
+            if (canonicalFromString(xxh64c->digest, sizeof(xxh64c->digest), line)
+                != CanonicalFromString_ok) {
+                return ParseLine_invalidFormat;
+            }
+            parsedLine->xxhBits = 64;
+            break;
+        }
+
+    default:
+            return ParseLine_invalidFormat;
+            break;
+    }
+
+    parsedLine->filename = secondSpace + 1;
+    return ParseLine_ok;
+}
+
+
+/*!  Parse xxHash checksum file.
+ */
+static void parseFile1(ParseFileArg* parseFileArg)
+{
+    const char* const inFileName = parseFileArg->inFileName;
+    ParseFileReport* const report = &parseFileArg->report;
+
+    unsigned long lineNumber = 0;
+    memset(report, 0, sizeof(*report));
+
+    while (!report->quit) {
+        FILE* fp = NULL;
+        LineStatus lineStatus = LineStatus_hashFailed;
+        GetLineResult getLineResult;
+        ParsedLine parsedLine;
+        memset(&parsedLine, 0, sizeof(parsedLine));
+
+        lineNumber++;
+        if (lineNumber == 0) {
+            /* This is unlikely happen, but md5sum.c has this
+             * error check. */
+            DISPLAY("%s : too many checksum lines\n", inFileName);
+            report->quit = 1;
+            break;
+        }
+
+        getLineResult = getLine(&parseFileArg->lineBuf, &parseFileArg->lineMax,
+                                parseFileArg->inFile);
+        if (getLineResult != GetLine_ok) {
+            if (getLineResult == GetLine_eof) break;
+
+            switch (getLineResult)
+            {
+            case GetLine_ok:
+            case GetLine_eof:
+                /* These cases never happen.  See above getLineResult related "if"s.
+                   They exist just for make gcc's -Wswitch-enum happy. */
+                break;
+
+            default:
+                DISPLAY("%s : %lu: unknown error\n", inFileName, lineNumber);
+                break;
+
+            case GetLine_exceedMaxLineLength:
+                DISPLAY("%s : %lu: too long line\n", inFileName, lineNumber);
+                break;
+
+            case GetLine_outOfMemory:
+                DISPLAY("%s : %lu: out of memory\n", inFileName, lineNumber);
+                break;
+            }
+            report->quit = 1;
+            break;
+        }
+
+        if (parseLine(&parsedLine, parseFileArg->lineBuf) != ParseLine_ok) {
+            report->nImproperlyFormattedLines++;
+            if (parseFileArg->warn) {
+                DISPLAY("%s : %lu: improperly formatted XXHASH checksum line\n"
+                    , inFileName, lineNumber);
+            }
+            continue;
+        }
+
+        if (report->xxhBits != 0 && report->xxhBits != parsedLine.xxhBits) {
+            /* Don't accept xxh32/xxh64 mixed file */
+            report->nImproperlyFormattedLines++;
+            report->nMixedFormatLines++;
+            if (parseFileArg->warn) {
+                DISPLAY("%s : %lu: improperly formatted XXHASH checksum line (XXH32/64)\n"
+                    , inFileName, lineNumber);
+            }
+            continue;
+        }
+
+        report->nProperlyFormattedLines++;
+        if (report->xxhBits == 0) {
+            report->xxhBits = parsedLine.xxhBits;
+        }
+
+        fp = fopen(parsedLine.filename, "rb");
+        if (fp == NULL) {
+            lineStatus = LineStatus_failedToOpen;
+        } else {
+            lineStatus = LineStatus_hashFailed;
+            switch (parsedLine.xxhBits)
+            {
+            case 32:
+                {   XXH32_hash_t xxh;
+                    BMK_hashStream(&xxh, algo_xxh32, fp, parseFileArg->blockBuf, parseFileArg->blockSize);
+                    if (xxh == XXH32_hashFromCanonical(&parsedLine.canonical.xxh32)) {
+                        lineStatus = LineStatus_hashOk;
+                }   }
+                break;
+
+            case 64:
+                {   XXH64_hash_t xxh;
+                    BMK_hashStream(&xxh, algo_xxh64, fp, parseFileArg->blockBuf, parseFileArg->blockSize);
+                    if (xxh == XXH64_hashFromCanonical(&parsedLine.canonical.xxh64)) {
+                        lineStatus = LineStatus_hashOk;
+                }   }
+                break;
+
+            default:
+                break;
+            }
+            fclose(fp);
+        }
+
+        switch (lineStatus)
+        {
+        default:
+            DISPLAY("%s : unknown error\n", inFileName);
+            report->quit = 1;
+            break;
+
+        case LineStatus_failedToOpen:
+            report->nOpenOrReadFailures++;
+            if (!parseFileArg->statusOnly) {
+                DISPLAYRESULT("%s : %lu: FAILED open or read %s\n"
+                    , inFileName, lineNumber, parsedLine.filename);
+            }
+            break;
+
+        case LineStatus_hashOk:
+        case LineStatus_hashFailed:
+            {   int b = 1;
+                if (lineStatus == LineStatus_hashOk) {
+                    /* If --quiet is specified, don't display "OK" */
+                    if (parseFileArg->quiet) b = 0;
+                } else {
+                    report->nMismatchedChecksums++;
+                }
+
+                if (b && !parseFileArg->statusOnly) {
+                    DISPLAYRESULT("%s: %s\n", parsedLine.filename
+                        , lineStatus == LineStatus_hashOk ? "OK" : "FAILED");
+            }   }
+            break;
+        }
+    }   /* while (!report->quit) */
+}
+
+
+/*  Parse xxHash checksum file.
+ *  Returns 1, if all procedures were succeeded.
+ *  Returns 0, if any procedures was failed.
+ *
+ *  If strictMode != 0, return error code if any line is invalid.
+ *  If statusOnly != 0, don't generate any output.
+ *  If warn != 0, print a warning message to stderr.
+ *  If quiet != 0, suppress "OK" line.
+ *
+ *  "All procedures are succeeded" means:
+ *    - Checksum file contains at least one line and less than SIZE_T_MAX lines.
+ *    - All files are properly opened and read.
+ *    - All hash values match with its content.
+ *    - (strict mode) All lines in checksum file are consistent and well formatted.
+ *
+ */
+static int checkFile(const char* inFileName,
+                     const endianess displayEndianess,
+                     U32 strictMode,
+                     U32 statusOnly,
+                     U32 warn,
+                     U32 quiet)
+{
+    int result = 0;
+    FILE* inFile = NULL;
+    ParseFileArg parseFileArgBody;
+    ParseFileArg* const parseFileArg = &parseFileArgBody;
+    ParseFileReport* const report = &parseFileArg->report;
+
+    if (displayEndianess != big_endian) {
+        /* Don't accept little endian */
+        DISPLAY( "Check file mode doesn't support little endian\n" );
+        return 0;
+    }
+
+    /* note : stdinName is special constant pointer.  It is not a string. */
+    if (inFileName == stdinName) {
+        /* note : Since we expect text input for xxhash -c mode,
+         * Don't set binary mode for stdin */
+        inFile = stdin;
+    } else {
+        inFile = fopen( inFileName, "rt" );
+    }
+
+    if (inFile == NULL) {
+        DISPLAY( "Pb opening %s\n", inFileName);
+        return 0;
+    }
+
+    parseFileArg->inFileName    = inFileName;
+    parseFileArg->inFile        = inFile;
+    parseFileArg->lineMax       = DEFAULT_LINE_LENGTH;
+    parseFileArg->lineBuf       = (char*) malloc((size_t) parseFileArg->lineMax);
+    parseFileArg->blockSize     = 64 * 1024;
+    parseFileArg->blockBuf      = (char*) malloc(parseFileArg->blockSize);
+    parseFileArg->strictMode    = strictMode;
+    parseFileArg->statusOnly    = statusOnly;
+    parseFileArg->warn          = warn;
+    parseFileArg->quiet         = quiet;
+
+    parseFile1(parseFileArg);
+
+    free(parseFileArg->blockBuf);
+    free(parseFileArg->lineBuf);
+
+    if (inFile != stdin) fclose(inFile);
+
+    /* Show error/warning messages.  All messages are copied from md5sum.c
+     */
+    if (report->nProperlyFormattedLines == 0) {
+        DISPLAY("%s: no properly formatted XXHASH checksum lines found\n", inFileName);
+    } else if (!statusOnly) {
+        if (report->nImproperlyFormattedLines) {
+            DISPLAYRESULT("%lu lines are improperly formatted\n"
+                , report->nImproperlyFormattedLines);
+        }
+        if (report->nOpenOrReadFailures) {
+            DISPLAYRESULT("%lu listed files could not be read\n"
+                , report->nOpenOrReadFailures);
+        }
+        if (report->nMismatchedChecksums) {
+            DISPLAYRESULT("%lu computed checksums did NOT match\n"
+                , report->nMismatchedChecksums);
+    }   }
+
+    /* Result (exit) code logic is copied from
+     * gnu coreutils/src/md5sum.c digest_check() */
+    result =   report->nProperlyFormattedLines != 0
+            && report->nMismatchedChecksums == 0
+            && report->nOpenOrReadFailures == 0
+            && (!strictMode || report->nImproperlyFormattedLines == 0)
+            && report->quit == 0;
+    return result;
+}
+
+
+static int checkFiles(const char** fnList, int fnTotal,
+                      const endianess displayEndianess,
+                      U32 strictMode,
+                      U32 statusOnly,
+                      U32 warn,
+                      U32 quiet)
+{
+    int ok = 1;
+
+    /* Special case for stdinName "-",
+     * note: stdinName is not a string.  It's special pointer. */
+    if (fnTotal==0) {
+        ok &= checkFile(stdinName, displayEndianess, strictMode, statusOnly, warn, quiet);
+    } else {
+        int fnNb;
+        for (fnNb=0; fnNb<fnTotal; fnNb++)
+            ok &= checkFile(fnList[fnNb], displayEndianess, strictMode, statusOnly, warn, quiet);
+    }
+    return ok ? 0 : 1;
+}
+
+
+/* ********************************************************
+*  Main
+**********************************************************/
+
+static int usage(const char* exename)
+{
+    DISPLAY( WELCOME_MESSAGE(exename) );
+    DISPLAY( "Usage :\n");
+    DISPLAY( "      %s [arg] [filenames]\n", exename);
+    DISPLAY( "When no filename provided, or - provided : use stdin as input\n");
+    DISPLAY( "Arguments :\n");
+    DISPLAY( " -H# : hash selection : 0=32bits, 1=64bits (default: %i)\n", (int)g_defaultAlgo);
+    DISPLAY( " -c  : read xxHash sums from the [filenames] and check them\n");
+    DISPLAY( " -h  : help \n");
+    return 0;
+}
+
+
+static int usage_advanced(const char* exename)
+{
+    usage(exename);
+    DISPLAY( "Advanced :\n");
+    DISPLAY( " --little-endian : hash printed using little endian convention (default: big endian)\n");
+    DISPLAY( " -V, --version   : display version\n");
+    DISPLAY( " -h, --help      : display long help and exit\n");
+    DISPLAY( " -b  : benchmark mode \n");
+    DISPLAY( " -i# : number of iterations (benchmark mode; default %i)\n", g_nbIterations);
+    DISPLAY( "\n");
+    DISPLAY( "The following four options are useful only when verifying checksums (-c):\n");
+    DISPLAY( "--strict : don't print OK for each successfully verified file\n");
+    DISPLAY( "--status : don't output anything, status code shows success\n");
+    DISPLAY( "--quiet  : exit non-zero for improperly formatted checksum lines\n");
+    DISPLAY( "--warn   : warn about improperly formatted checksum lines\n");
+    return 0;
+}
+
+static int badusage(const char* exename)
+{
+    DISPLAY("Wrong parameters\n");
+    usage(exename);
+    return 1;
+}
+
+/*! readU32FromChar() :
+   @return : unsigned integer value read from input in `char` format,
+             0 is no figure at *stringPtr position.
+    Interprets K, KB, KiB, M, MB and MiB suffix.
+    Modifies `*stringPtr`, advancing it to position where reading stopped.
+    Note : function result can overflow if digit string > MAX_UINT */
+static unsigned readU32FromChar(const char** stringPtr)
+{
+    unsigned result = 0;
+    while ((**stringPtr >='0') && (**stringPtr <='9'))
+        result *= 10, result += **stringPtr - '0', (*stringPtr)++ ;
+    if ((**stringPtr=='K') || (**stringPtr=='M')) {
+        result <<= 10;
+        if (**stringPtr=='M') result <<= 10;
+        (*stringPtr)++ ;
+        if (**stringPtr=='i') (*stringPtr)++;
+        if (**stringPtr=='B') (*stringPtr)++;
+    }
+    return result;
+}
+
+int main(int argc, const char** argv)
+{
+    int i, filenamesStart = 0;
+    const char* const exename = argv[0];
+    U32 benchmarkMode = 0;
+    U32 fileCheckMode = 0;
+    U32 strictMode    = 0;
+    U32 statusOnly    = 0;
+    U32 warn          = 0;
+    U32 quiet         = 0;
+    U32 specificTest  = 0;
+    size_t keySize    = XXH_DEFAULT_SAMPLE_SIZE;
+    algoType algo     = g_defaultAlgo;
+    endianess displayEndianess = big_endian;
+
+    /* special case : xxh32sum default to 32 bits checksum */
+    if (strstr(exename, "xxh32sum") != NULL) algo = algo_xxh32;
+
+    for(i=1; i<argc; i++) {
+        const char* argument = argv[i];
+
+        if(!argument) continue;   /* Protection, if argument empty */
+
+        if (!strcmp(argument, "--little-endian")) { displayEndianess = little_endian; continue; }
+        if (!strcmp(argument, "--check")) { fileCheckMode = 1; continue; }
+        if (!strcmp(argument, "--strict")) { strictMode = 1; continue; }
+        if (!strcmp(argument, "--status")) { statusOnly = 1; continue; }
+        if (!strcmp(argument, "--quiet")) { quiet = 1; continue; }
+        if (!strcmp(argument, "--warn")) { warn = 1; continue; }
+        if (!strcmp(argument, "--help")) { return usage_advanced(exename); }
+        if (!strcmp(argument, "--version")) { DISPLAY(WELCOME_MESSAGE(exename)); return 0; }
+
+        if (*argument!='-') {
+            if (filenamesStart==0) filenamesStart=i;   /* only supports a continuous list of filenames */
+            continue;
+        }
+
+        /* command selection */
+        argument++;   /* note : *argument=='-' */
+
+        while (*argument!=0) {
+            switch(*argument)
+            {
+            /* Display version */
+            case 'V':
+                DISPLAY(WELCOME_MESSAGE(exename)); return 0;
+
+            /* Display help on usage */
+            case 'h':
+                return usage_advanced(exename);
+
+            /* select hash algorithm */
+            case 'H':
+                algo = (algoType)(argument[1] - '0');
+                argument+=2;
+                break;
+
+            /* File check mode */
+            case 'c':
+                fileCheckMode=1;
+                argument++;
+                break;
+
+            /* Warning mode (file check mode only, alias of "--warning") */
+            case 'w':
+                warn=1;
+                argument++;
+                break;
+
+            /* Trigger benchmark mode */
+            case 'b':
+                argument++;
+                benchmarkMode = 1;
+                specificTest = readU32FromChar(&argument);   /* select one specific test (hidden option) */
+                break;
+
+            /* Modify Nb Iterations (benchmark only) */
+            case 'i':
+                argument++;
+                g_nbIterations = readU32FromChar(&argument);
+                break;
+
+            /* Modify Block size (benchmark only) */
+            case 'B':
+                argument++;
+                keySize = readU32FromChar(&argument);
+                break;
+
+            /* Modify verbosity of benchmark output (hidden option) */
+            case 'q':
+                argument++;
+                g_displayLevel--;
+                break;
+
+            default:
+                return badusage(exename);
+            }
+        }
+    }   /* for(i=1; i<argc; i++) */
+
+    /* Check benchmark mode */
+    if (benchmarkMode) {
+        DISPLAYLEVEL(2, WELCOME_MESSAGE(exename) );
+        BMK_sanityCheck();
+        if (filenamesStart==0) return BMK_benchInternal(keySize, specificTest);
+        return BMK_benchFiles(argv+filenamesStart, argc-filenamesStart, specificTest);
+    }
+
+    /* Check if input is defined as console; trigger an error in this case */
+    if ( (filenamesStart==0) && IS_CONSOLE(stdin) ) return badusage(exename);
+
+    if (filenamesStart==0) filenamesStart = argc;
+    if (fileCheckMode) {
+        return checkFiles(argv+filenamesStart, argc-filenamesStart,
+                          displayEndianess, strictMode, statusOnly, warn, quiet);
+    } else {
+        return BMK_hashFiles(argv+filenamesStart, argc-filenamesStart, algo, displayEndianess);
+    }
+}
+
+#endif /* XXHASH_C_2097394837 */
diff --git a/doc/Makefile.am b/doc/Makefile.am
index 1103b607dba..de68c20b4d7 100644
--- a/doc/Makefile.am
+++ b/doc/Makefile.am
@@ -1,6 +1,9 @@
 EXTRA_DIST = glusterfs.8 mount.glusterfs.8 gluster.8 \
 			glusterd.8 glusterfsd.8
 
-man8_MANS = glusterfs.8 mount.glusterfs.8 gluster.8 glusterd.8 glusterfsd.8
+man8_MANS = glusterfs.8 mount.glusterfs.8 gluster.8
+if WITH_SERVER
+man8_MANS += glusterd.8 glusterfsd.8
+endif
 
 CLEANFILES =
diff --git a/doc/README.md b/doc/README.md
new file mode 100644
index 00000000000..6aa28642ef4
--- /dev/null
+++ b/doc/README.md
@@ -0,0 +1,26 @@
+## Developer Guide
+
+Gluster's contributors can check about the internals by visiting [Developer Guide Section](developer-guide). While it is not 'comprehensive', it can help you to get started.
+
+Also while coding, keep [Coding Standard](developer-guide/coding-standard.md) in mind.
+
+When you are ready to commit the changes, make sure you meet our [Commit message standard](developer-guide/commit-guidelines.md).
+
+## Admin Guide ##
+
+The gluster administration guide is maintained at [github](https://github.com/gluster/glusterdocs). The browsable admin guide can be found [here](http://docs.gluster.org/en/latest/Administrator%20Guide/).
+
+The doc patch has to be sent against the above mentioned repository.
+
+## Features/Spec ##
+
+The Gluster features which are 'in progress' or implemented can be found at [github](https://github.com/gluster/glusterfs-specs).
+
+## Upgrade Guide ##
+
+The gluster upgrade guide is maintained at [github](https://github.com/gluster/glusterdocs). The browsable upgrade guide can be found [here](http://docs.gluster.org/en/latest/Upgrade-Guide)
+
+The doc patch has to be sent against the above mentioned repository.
+
+
+For more details about the docuemntation workflow please refer [this discussion](https://www.mail-archive.com/gluster-users@gluster.org/msg21168.html)
diff --git a/doc/TODO/quota.txt b/doc/TODO/quota.txt
deleted file mode 100644
index f8ee1b7512c..00000000000
--- a/doc/TODO/quota.txt
+++ /dev/null
@@ -1,14 +0,0 @@
-* Add the quota config xattr to newly added brick
-* Get directory size before enforcing quota limits on rename
-* Xattr cleanup on disabling quota
-* Heal limit-key while self-healing directories in dht
-* Correct the statfs calculation when set limit is beyond actual disk
-  space. 
-* Use STACK_WIND_TAIL when quota is turned off to avoid using a frame.
-* Parent gfid xattr self-healing in storage/posix.
-* Failure of regression test related to Quota anonymous fd handling
-  handling. mkdir -p in the test-script was failing with EIO error
-* Add GET_ANCESTRY test cases to posix pgfid regression test cases
-* regression tests related to quota accounting covering all possible
-  cases of dht rename 
-* Build Ancestry in quota_update_size if parent is found to be NULL
diff --git a/doc/admin-guide/en-US/images/640px-GlusterFS_Architecture.png b/doc/admin-guide/en-US/images/640px-GlusterFS_Architecture.png
deleted file mode 100644
index 95f89ec8286..00000000000
--- a/doc/admin-guide/en-US/images/640px-GlusterFS_Architecture.png
+++ /dev/null
diff --git a/doc/admin-guide/en-US/images/Distributed_Replicated_Volume.png b/doc/admin-guide/en-US/images/Distributed_Replicated_Volume.png
deleted file mode 100644
index 22daecdb903..00000000000
--- a/doc/admin-guide/en-US/images/Distributed_Replicated_Volume.png
+++ /dev/null
diff --git a/doc/admin-guide/en-US/images/Distributed_Striped_Replicated_Volume.png b/doc/admin-guide/en-US/images/Distributed_Striped_Replicated_Volume.png
deleted file mode 100644
index d286fa99e94..00000000000
--- a/doc/admin-guide/en-US/images/Distributed_Striped_Replicated_Volume.png
+++ /dev/null
diff --git a/doc/admin-guide/en-US/images/Distributed_Striped_Volume.png b/doc/admin-guide/en-US/images/Distributed_Striped_Volume.png
deleted file mode 100644
index 752fa982fa6..00000000000
--- a/doc/admin-guide/en-US/images/Distributed_Striped_Volume.png
+++ /dev/null
diff --git a/doc/admin-guide/en-US/images/Distributed_Volume.png b/doc/admin-guide/en-US/images/Distributed_Volume.png
deleted file mode 100644
index 4386ca935b9..00000000000
--- a/doc/admin-guide/en-US/images/Distributed_Volume.png
+++ /dev/null
diff --git a/doc/admin-guide/en-US/images/Geo-Rep03_Internet.png b/doc/admin-guide/en-US/images/Geo-Rep03_Internet.png
deleted file mode 100644
index 3cd0eaded02..00000000000
--- a/doc/admin-guide/en-US/images/Geo-Rep03_Internet.png
+++ /dev/null
diff --git a/doc/admin-guide/en-US/images/Geo-Rep04_Cascading.png b/doc/admin-guide/en-US/images/Geo-Rep04_Cascading.png
deleted file mode 100644
index 54bf9f05cff..00000000000
--- a/doc/admin-guide/en-US/images/Geo-Rep04_Cascading.png
+++ /dev/null
diff --git a/doc/admin-guide/en-US/images/Geo-Rep_LAN.png b/doc/admin-guide/en-US/images/Geo-Rep_LAN.png
deleted file mode 100644
index a74f6dbb50a..00000000000
--- a/doc/admin-guide/en-US/images/Geo-Rep_LAN.png
+++ /dev/null
diff --git a/doc/admin-guide/en-US/images/Geo-Rep_WAN.png b/doc/admin-guide/en-US/images/Geo-Rep_WAN.png
deleted file mode 100644
index d72d72768bc..00000000000
--- a/doc/admin-guide/en-US/images/Geo-Rep_WAN.png
+++ /dev/null
diff --git a/doc/admin-guide/en-US/images/GlusterFS_Architecture.png b/doc/admin-guide/en-US/images/GlusterFS_Architecture.png
deleted file mode 100644
index b506db1f4e7..00000000000
--- a/doc/admin-guide/en-US/images/GlusterFS_Architecture.png
+++ /dev/null
diff --git a/doc/admin-guide/en-US/images/Hadoop_Architecture.png b/doc/admin-guide/en-US/images/Hadoop_Architecture.png
deleted file mode 100644
index 8725bd330bb..00000000000
--- a/doc/admin-guide/en-US/images/Hadoop_Architecture.png
+++ /dev/null
diff --git a/doc/admin-guide/en-US/images/Replicated_Volume.png b/doc/admin-guide/en-US/images/Replicated_Volume.png
deleted file mode 100644
index 135a63f345a..00000000000
--- a/doc/admin-guide/en-US/images/Replicated_Volume.png
+++ /dev/null
diff --git a/doc/admin-guide/en-US/images/Striped_Replicated_Volume.png b/doc/admin-guide/en-US/images/Striped_Replicated_Volume.png
deleted file mode 100644
index adf1f8465eb..00000000000
--- a/doc/admin-guide/en-US/images/Striped_Replicated_Volume.png
+++ /dev/null
diff --git a/doc/admin-guide/en-US/images/Striped_Volume.png b/doc/admin-guide/en-US/images/Striped_Volume.png
deleted file mode 100644
index 63a84b242ab..00000000000
--- a/doc/admin-guide/en-US/images/Striped_Volume.png
+++ /dev/null
diff --git a/doc/admin-guide/en-US/images/UFO_Architecture.png b/doc/admin-guide/en-US/images/UFO_Architecture.png
deleted file mode 100644
index be85d7b2825..00000000000
--- a/doc/admin-guide/en-US/images/UFO_Architecture.png
+++ /dev/null
diff --git a/doc/admin-guide/en-US/images/VSA_Architecture.png b/doc/admin-guide/en-US/images/VSA_Architecture.png
deleted file mode 100644
index c3ab80cf3e8..00000000000
--- a/doc/admin-guide/en-US/images/VSA_Architecture.png
+++ /dev/null
diff --git a/doc/admin-guide/en-US/images/icon.svg b/doc/admin-guide/en-US/images/icon.svg
deleted file mode 100644
index b2f16d0f61d..00000000000
--- a/doc/admin-guide/en-US/images/icon.svg
+++ /dev/null
@@ -1,19 +0,0 @@
-<?xml version="1.0" encoding="UTF-8" standalone="no"?>
-<svg xmlns:svg="http://www.w3.org/2000/svg" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.0" width="32" height="32" id="svg3017">
-  <defs id="defs3019">
-    <linearGradient id="linearGradient2381">
-      <stop id="stop2383" style="stop-color:#ffffff;stop-opacity:1" offset="0"/>
-      <stop id="stop2385" style="stop-color:#ffffff;stop-opacity:0" offset="1"/>
-    </linearGradient>
-    <linearGradient x1="296.4996" y1="188.81061" x2="317.32471" y2="209.69398" id="linearGradient2371" xlink:href="#linearGradient2381" gradientUnits="userSpaceOnUse" gradientTransform="matrix(0.90776,0,0,0.90776,24.35648,49.24131)"/>
-  </defs>
-  <g transform="matrix(0.437808,-0.437808,0.437808,0.437808,-220.8237,43.55311)" id="g5089">
-    <path d="m 8.4382985,-6.28125 c -0.6073916,0 -4.3132985,5.94886271 -4.3132985,8.25 l 0,26.71875 c 0,0.846384 0.5818159,1.125 1.15625,1.125 l 25.5625,0 c 0.632342,0 1.125001,-0.492658 1.125,-1.125 l 0,-5.21875 0.28125,0 c 0.49684,0 0.906249,-0.409411 0.90625,-0.90625 l 0,-27.9375 c 0,-0.4968398 -0.40941,-0.90625 -0.90625,-0.90625 l -23.8117015,0 z" transform="translate(282.8327,227.1903)" id="path5091" style="fill:#5c5c4f;stroke:#000000;stroke-width:3.23021388;stroke-miterlimit:4;stroke-dasharray:none"/>
-    <rect width="27.85074" height="29.369793" rx="1.1414107" ry="1.1414107" x="286.96509" y="227.63805" id="rect5093" style="fill:#032c87"/>
-    <path d="m 288.43262,225.43675 25.2418,0 0,29.3698 -26.37615,0.0241 1.13435,-29.39394 z" id="rect5095" style="fill:#ffffff"/>
-    <path d="m 302.44536,251.73726 c 1.38691,7.85917 -0.69311,11.28365 -0.69311,11.28365 2.24384,-1.60762 3.96426,-3.47694 4.90522,-5.736 0.96708,2.19264 1.83294,4.42866 4.27443,5.98941 0,0 -1.59504,-7.2004 -1.71143,-11.53706 l -6.77511,0 z" id="path5097" style="fill:#a70000;fill-opacity:1;stroke-width:2"/>
-    <rect width="25.241802" height="29.736675" rx="0.89682275" ry="0.89682275" x="290.73544" y="220.92249" id="rect5099" style="fill:#809cc9"/>
-    <path d="m 576.47347,725.93939 6.37084,0.41502 0.4069,29.51809 c -1.89202,-1.31785 -6.85427,-3.7608 -8.26232,-1.68101 l 0,-26.76752 c 0,-0.82246 0.66212,-1.48458 1.48458,-1.48458 z" transform="matrix(0.499065,-0.866565,0,1,0,0)" id="rect5101" style="fill:#4573b3;fill-opacity:1"/>
-    <path d="m 293.2599,221.89363 20.73918,0 c 0.45101,0 0.8141,0.3631 0.8141,0.81411 0.21547,6.32836 -19.36824,21.7635 -22.36739,17.59717 l 0,-17.59717 c 0,-0.45101 0.3631,-0.81411 0.81411,-0.81411 z" id="path5103" style="opacity:0.65536726;fill:url(#linearGradient2371);fill-opacity:1"/>
-  </g>
-</svg>
diff --git a/doc/admin-guide/en-US/markdown/.gitignore b/doc/admin-guide/en-US/markdown/.gitignore
deleted file mode 100644
index 9eed460045f..00000000000
--- a/doc/admin-guide/en-US/markdown/.gitignore
+++ /dev/null
@@ -1,2 +0,0 @@
-output/*.pdf
-
diff --git a/doc/admin-guide/en-US/markdown/admin_ACLs.md b/doc/admin-guide/en-US/markdown/admin_ACLs.md
deleted file mode 100644
index eeb6d83afdd..00000000000
--- a/doc/admin-guide/en-US/markdown/admin_ACLs.md
+++ /dev/null
@@ -1,184 +0,0 @@
-#POSIX Access Control Lists
-
-POSIX Access Control Lists (ACLs) allows you to assign different
-permissions for different users or groups even though they do not
-correspond to the original owner or the owning group.
-
-For example: User john creates a file but does not want to allow anyone
-to do anything with this file, except another user, antony (even though
-there are other users that belong to the group john).
-
-This means, in addition to the file owner, the file group, and others,
-additional users and groups can be granted or denied access by using
-POSIX ACLs.
-
-##Activating POSIX ACLs Support
-
-To use POSIX ACLs for a file or directory, the partition of the file or
-directory must be mounted with POSIX ACLs support.
-
-###Activating POSIX ACLs Support on Sever
-
-To mount the backend export directories for POSIX ACLs support, use the
-following command:
-
-`# mount -o acl `
-
-For example:
-
-`# mount -o acl /dev/sda1 /export1 `
-
-Alternatively, if the partition is listed in the /etc/fstab file, add
-the following entry for the partition to include the POSIX ACLs option:
-
-`LABEL=/work /export1 ext3 rw, acl 14 `
-
-###Activating POSIX ACLs Support on Client
-
-To mount the glusterfs volumes for POSIX ACLs support, use the following
-command:
-
-`# mount –t glusterfs -o acl `
-
-For example:
-
-`# mount -t glusterfs -o acl 198.192.198.234:glustervolume /mnt/gluster`
-
-##Setting POSIX ACLs
-
-You can set two types of POSIX ACLs, that is, access ACLs and default
-ACLs. You can use access ACLs to grant permission for a specific file or
-directory. You can use default ACLs only on a directory but if a file
-inside that directory does not have an ACLs, it inherits the permissions
-of the default ACLs of the directory.
-
-You can set ACLs for per user, per group, for users not in the user
-group for the file, and via the effective right mask.
-
-##Setting Access ACLs
-
-You can apply access ACLs to grant permission for both files and
-directories.
-
-**To set or modify Access ACLs**
-
-You can set or modify access ACLs use the following command:
-
-`# setfacl –m  file `
-
-The ACL entry types are the POSIX ACLs representations of owner, group,
-and other.
-
-Permissions must be a combination of the characters `r` (read), `w`
-(write), and `x` (execute). You must specify the ACL entry in the
-following format and can specify multiple entry types separated by
-commas.
-
-  ACL Entry | Description
-  --- | ---
-  u:uid:\<permission\> | Sets the access ACLs for a user. You can specify user name or UID
-  g:gid:\<permission\> | Sets the access ACLs for a group. You can specify group name or GID.
-  m:\<permission\> | Sets the effective rights mask. The mask is the combination of all access permissions of the owning group and all of the user and group entries.
-  o:\<permission\> | Sets the access ACLs for users other than the ones in the group for the file.
-
-If a file or directory already has an POSIX ACLs, and the setfacl
-command is used, the additional permissions are added to the existing
-POSIX ACLs or the existing rule is modified.
-
-For example, to give read and write permissions to user antony:
-
-`# setfacl -m u:antony:rw /mnt/gluster/data/testfile `
-
-##Setting Default ACLs
-
-You can apply default ACLs only to directories. They determine the
-permissions of a file system objects that inherits from its parent
-directory when it is created.
-
-To set default ACLs
-
-You can set default ACLs for files and directories using the following
-command:
-
-`# setfacl –m –-set `
-
-For example, to set the default ACLs for the /data directory to read for
-users not in the user group:
-
-`# setfacl –m --set o::r /mnt/gluster/data `
-
-> **Note**
->
-> An access ACLs set for an individual file can override the default
-> ACLs permissions.
-
-**Effects of a Default ACLs**
-
-The following are the ways in which the permissions of a directory's
-default ACLs are passed to the files and subdirectories in it:
-
--   A subdirectory inherits the default ACLs of the parent directory
-    both as its default ACLs and as an access ACLs.
--   A file inherits the default ACLs as its access ACLs.
-
-##Retrieving POSIX ACLs
-
-You can view the existing POSIX ACLs for a file or directory.
-
-**To view existing POSIX ACLs**
-
--   View the existing access ACLs of a file using the following command:
-
-    `# getfacl `
-
-    For example, to view the existing POSIX ACLs for sample.jpg
-
-        # getfacl /mnt/gluster/data/test/sample.jpg
-        # owner: antony
-        # group: antony
-        user::rw-
-        group::rw-
-        other::r--
-
--   View the default ACLs of a directory using the following command:
-
-    `# getfacl `
-
-    For example, to view the existing ACLs for /data/doc
-
-        # getfacl /mnt/gluster/data/doc
-        # owner: antony
-        # group: antony
-        user::rw-
-        user:john:r--
-        group::r--
-        mask::r--
-        other::r--
-        default:user::rwx
-        default:user:antony:rwx
-        default:group::r-x
-        default:mask::rwx
-        default:other::r-x
-
-##Removing POSIX ACLs
-
-To remove all the permissions for a user, groups, or others, use the
-following command:
-
-`# setfacl -x `
-
-For example, to remove all permissions from the user antony:
-
-`# setfacl -x u:antony /mnt/gluster/data/test-file`
-
-##Samba and ACLs
-
-If you are using Samba to access GlusterFS FUSE mount, then POSIX ACLs
-are enabled by default. Samba has been compiled with the
-`--with-acl-support` option, so no special flags are required when
-accessing or mounting a Samba share.
-
-##NFS and ACLs
-
-Currently GlusterFS supports POSIX ACL configuration through NFS mount,
-i.e. setfacl and getfacl commands work through NFS mount.
diff --git a/doc/admin-guide/en-US/markdown/admin_Hadoop.md b/doc/admin-guide/en-US/markdown/admin_Hadoop.md
deleted file mode 100644
index 1f5e8d4ae49..00000000000
--- a/doc/admin-guide/en-US/markdown/admin_Hadoop.md
+++ /dev/null
@@ -1,31 +0,0 @@
-#Managing Hadoop Compatible Storage
-
-GlusterFS provides compatibility for Apache Hadoop and it uses the
-standard file system APIs available in Hadoop to provide a new storage
-option for Hadoop deployments. Existing MapReduce based applications can
-use GlusterFS seamlessly. This new functionality opens up data within
-Hadoop deployments to any file-based or object-based application.
-
-##Advantages
-
-The following are the advantages of Hadoop Compatible Storage with
-GlusterFS:
-
--   Provides simultaneous file-based and object-based access within
-    Hadoop.
--   Eliminates the centralized metadata server.
--   Provides compatibility with MapReduce applications and rewrite is
-    not required.
--   Provides a fault tolerant file system.
-
-###Pre-requisites
-
-The following are the pre-requisites to install Hadoop Compatible
-Storage :
-
--   Java Runtime Environment
--   getfattr - command line utility
-
-##Installing, and Configuring Hadoop Compatible Storage
-
-See the detailed instruction set at https://forge.gluster.org/hadoop/pages/ConfiguringHadoop2
diff --git a/doc/admin-guide/en-US/markdown/admin_console.md b/doc/admin-guide/en-US/markdown/admin_console.md
deleted file mode 100644
index 126b7e2064f..00000000000
--- a/doc/admin-guide/en-US/markdown/admin_console.md
+++ /dev/null
@@ -1,50 +0,0 @@
-##Using the Gluster Console Manager – Command Line Utility
-
-The Gluster Console Manager is a single command line utility that
-simplifies configuration and management of your storage environment. The
-Gluster Console Manager is similar to the LVM (Logical Volume Manager)
-CLI or ZFS Command Line Interface, but across multiple storage servers.
-You can use the Gluster Console Manager online, while volumes are
-mounted and active. Gluster automatically synchronizes volume
-configuration information across all Gluster servers.
-
-Using the Gluster Console Manager, you can create new volumes, start
-volumes, and stop volumes, as required. You can also add bricks to
-volumes, remove bricks from existing volumes, as well as change
-translator settings, among other operations.
-
-You can also use the commands to create scripts for automation, as well
-as use the commands as an API to allow integration with third-party
-applications.
-
-###Running the Gluster Console Manager
-
-You can run the Gluster Console Manager on any GlusterFS server either
-by invoking the commands or by running the Gluster CLI in interactive
-mode. You can also use the gluster command remotely using SSH.
-
--   To run commands directly:
-
-    ` # gluster peer `
-
-    For example:
-
-    ` # gluster peer status `
-
--   To run the Gluster Console Manager in interactive mode
-
-    `# gluster`
-
-    You can execute gluster commands from the Console Manager prompt:
-
-    ` gluster> `
-
-    For example, to view the status of the peer server:
-
-    \# `gluster `
-
-    `gluster > peer status `
-
-    Display the status of the peer.
-
-
diff --git a/doc/admin-guide/en-US/markdown/admin_directory_Quota.md b/doc/admin-guide/en-US/markdown/admin_directory_Quota.md
deleted file mode 100644
index 21e42c66902..00000000000
--- a/doc/admin-guide/en-US/markdown/admin_directory_Quota.md
+++ /dev/null
@@ -1,164 +0,0 @@
-#Managing Directory Quota
-
-Directory quotas in GlusterFS allow you to set limits on usage of disk
-space by directories or volumes. The storage administrators can control
-the disk space utilization at the directory and/or volume levels in
-GlusterFS by setting limits to allocatable disk space at any level in
-the volume and directory hierarchy. This is particularly useful in cloud
-deployments to facilitate utility billing model.
-
-> **Note**
->
-> For now, only Hard limit is supported. Here, the limit cannot be
-> exceeded and attempts to use more disk space or inodes beyond the set
-> limit will be denied.
-
-System administrators can also monitor the resource utilization to limit
-the storage for the users depending on their role in the organization.
-
-You can set the quota at the following levels:
-
--   **Directory level** – limits the usage at the directory level
--   **Volume level** – limits the usage at the volume level
-
-> **Note**
->
-> You can set the disk limit on the directory even if it is not created.
-> The disk limit is enforced immediately after creating that directory.
-> For more information on setting disk limit, see ?.
-
-##Enabling Quota
-
-You must enable Quota to set disk limits.
-
-**To enable quota**
-
--   Enable the quota using the following command:
-
-    `# gluster volume quota  enable `
-
-    For example, to enable quota on test-volume:
-
-        # gluster volume quota test-volume enable
-        Quota is enabled on /test-volume
-
-##Disabling Quota
-
-You can disable Quota, if needed.
-
-**To disable quota:**
-
--   Disable the quota using the following command:
-
-    `# gluster volume quota  disable `
-
-    For example, to disable quota translator on test-volume:
-
-        # gluster volume quota test-volume disable
-        Quota translator is disabled on /test-volume
-
-##Setting or Replacing Disk Limit
-
-You can create new directories in your storage environment and set the
-disk limit or set disk limit for the existing directories. The directory
-name should be relative to the volume with the export directory/mount
-being treated as "/".
-
-**To set or replace disk limit**
-
--   Set the disk limit using the following command:
-
-    `# gluster volume quota  limit-usage /`
-
-    For example, to set limit on data directory on test-volume where
-    data is a directory under the export directory:
-
-        # gluster volume quota test-volume limit-usage /data 10GB
-        Usage limit has been set on /data
-
-    > **Note**
-    >
-    > In a multi-level directory hierarchy, the strictest disk limit
-    > will be considered for enforcement.
-
-##Displaying Disk Limit Information
-
-You can display disk limit information on all the directories on which
-the limit is set.
-
-**To display disk limit information**
-
--   Display disk limit information of all the directories on which limit
-    is set, using the following command:
-
-    `# gluster volume quota  list`
-
-    For example, to see the set disks limit on test-volume:
-
-        # gluster volume quota test-volume list
-
-
-        /Test/data    10 GB       6 GB
-        /Test/data1   10 GB       4 GB
-
--   Display disk limit information on a particular directory on which
-    limit is set, using the following command:
-
-    `# gluster volume quota  list `
-
-    For example, to see the set limit on /data directory of test-volume:
-
-        # gluster volume quota test-volume list /data
-
-
-        /Test/data    10 GB       6 GB
-
-##Updating Memory Cache Size
-
-For performance reasons, quota caches the directory sizes on client. You
-can set timeout indicating the maximum valid duration of directory sizes
-in cache, from the time they are populated.
-
-For example: If there are multiple clients writing to a single
-directory, there are chances that some other client might write till the
-quota limit is exceeded. However, this new file-size may not get
-reflected in the client till size entry in cache has become stale
-because of timeout. If writes happen on this client during this
-duration, they are allowed even though they would lead to exceeding of
-quota-limits, since size in cache is not in sync with the actual size.
-When timeout happens, the size in cache is updated from servers and will
-be in sync and no further writes will be allowed. A timeout of zero will
-force fetching of directory sizes from server for every operation that
-modifies file data and will effectively disables directory size caching
-on client side.
-
-**To update the memory cache size**
-
--   Update the memory cache size using the following command:
-
-    `# gluster volume set  features.quota-timeout`
-
-    For example, to update the memory cache size for every 5 seconds on
-    test-volume:
-
-        # gluster volume set test-volume features.quota-timeout 5
-        Set volume successful
-
-##Removing Disk Limit
-
-You can remove set disk limit, if you do not want quota anymore.
-
-**To remove disk limit**
-
--   Remove disk limit set on a particular directory using the following
-    command:
-
-    `# gluster volume quota  remove `
-
-    For example, to remove the disk limit on /data directory of
-    test-volume:
-
-        # gluster volume quota test-volume remove /data
-        Usage limit set on /data is removed
-
-
diff --git a/doc/admin-guide/en-US/markdown/admin_distributed_geo_rep.md b/doc/admin-guide/en-US/markdown/admin_distributed_geo_rep.md
deleted file mode 100644
index 1ec4f28ae56..00000000000
--- a/doc/admin-guide/en-US/markdown/admin_distributed_geo_rep.md
+++ /dev/null
@@ -1,114 +0,0 @@
-# Distributed Geo-Replication in glusterfs-3.5
-
-This is a admin how-to guide for new dustributed-geo-replication being released as part of glusterfs-3.5
-
-##### Note:
-This article is targeted towards users/admins who want to try new geo-replication, without going much deeper into internals and technology used.
-
-### How is it different from earlier geo-replication?
-
-- Up until now, in geo-replication, only one of the nodes in master volume would participate in geo-replication. This meant that all the data syncing is taken care by only one node while other nodes in the cluster would sit idle (not participate in data syncing). With distributed-geo-replication, each node of the master volume takes the repsonsibility of syncing the data present in that node. In case of replicate configuration, one of them would 'Active'ly sync the data while other node of the replica pair would be 'Passive'. The 'Passive' node only becomes 'Active' when the 'Active' pair goes down. This way new geo-rep leverages all the nodes in the volume and remove the bottleneck of syncing from one single node. 
-- New change detection mechanism is the other thing which has been improved with new geo-rep. So far geo-rep used to crawl through glusterfs file system to figure out the files that need to synced. And because crawling filesystem can be an expensive operation, this used to be a major bottleneck for performance. With distributed geo-rep, all the files that need to be synced are identified through changelog xlator. Changelog xlator journals all the fops that modifes the file and these journals are then consumed by geo-rep to effectively identify the files that need to be synced.
-- A new syncing method tar+ssh, has been introduced to improve the performance of few specific data sets. You can switch between rsync and tar+ssh syncing method via CLI to suite your data set needs. This tar+ssh is better suited for data sets which have large number of small files.
-
-
-### Using Distributed geo-replication:
-
-#### Prerequisites:
-- There should be a password-less ssh setup between at least one node in master volume to one node in slave volume. The geo-rep create command should be executed from this node which has password-less ssh setup to slave.
-
-- Unlike previous version, slave **must** be a gluster volume. Slave can not be a directory. And both the master and slave volumes should have been created and started before creating geo-rep session.
-
-#### Creating secret pem pub file
-- Execute the below command from the node where you setup the password-less ssh to slave. This will create the secret pem pub file which would have information of RSA key of all the nodes in the master volume. And when geo-rep create command is executed, glusterd uses this file to establish a geo-rep specific ssh connections.
-```sh
-gluster system:: execute gsec_create
-```
-
-#### Creating geo-replication session.
-Create a geo-rep session between master and slave volume using the following command. The node in which this command is executed and the <slave_host> specified in the command should have password less ssh setup between them. The push-pem option actually uses the secret pem pub file created earlier and establishes geo-rep specific password less ssh between each node in master to each node of slave.
-```sh
-gluster volume geo-replication <master_volume> <slave_host>::<slave_volume> create push-pem [force]
-```
-
-If the total available size in slave volume is more than the total size of master, the command will throw error message. In such cases 'force' option can be used.
-
-#### Starting a geo-rep session
-There is no change in this command from previous versions to this version.
-```sh
-gluster volume geo-replication <master_volume> <slave_host>::<slave_volume> start
-```
-This command actually starts the session. Meaning the gsyncd monitor process will be started, which in turn spawns gsync worker processes whenever required. This also turns on changelog xlator (if not in ON state already), which starts recording all the changes on each of the glusterfs bricks. And if master is empty during geo-rep start, the change detection mechanism will be changelog. Else it’ll be xsync (the changes are identified by crawling through filesystem). Later when the initial data is syned to slave, change detection mechanism will be set to changelog
-
-#### Status of geo-replication
-
-gluster now has variants of status command.
-
-```sh
-gluster volume geo-replication <master_volume> <slave_volume>::<slave_volume> status
-```
-
-This displays the status of session from each brick of the master to each brick of the slave node.
-
-If you want more detailed status, then run 'status detail'
-
-```sh
-gluster volume geo-replication <master_volume> <slave_volume>::<slave_volume> status detail
-```
-
-This command displays extra information like, total files synced, files that needs to be synced, deletes pending etc.
-
-#### Stopping geo-replication session
-
-This command stops all geo-rep relates processes i.e. gsyncd monitor and works processes. Note that changelog will **not** be turned off with this command.
-
-```sh
-gluster volume geo-replication <master_volume> <slave_volume>::<slave_volume> stop [force]
-```
-Force option is to be used, when one of the node (or glusterd in one of the node) is down. Once stopped, the session can be restarted any time. Note that upon restarting of the session, the change detection mechanism falls back to xsync mode. This happens even though you have changelog generating journals, while the geo-rep session is stopped.
-
-#### Deleting geo-replication session
-
-Now you can delete the glusterfs geo-rep session. This will delete all the config data associated with the geo-rep session.
-
-```sh
-gluster volume geo-replication <master_volume> <slave_volume>::<slave_volume> delete
-```
-
-This deletes all the gsync conf files in each of the nodes. This returns failure, if any of the node is down. And unlike geo-rep stop, there is 'force' option with this.
-
-#### Changing the config values
-
-There are some configuration values which can be changed using the CLI. And you can see all the current config values with following command.
-
-```sh
-gluster volume geo-replication <master_volume> <slave_host>::<slave_volume> config
-```
-
-But you can check only one of them, like log_file or change-detector
-
-```sh
-gluster volume geo-replication <master_volume> <slave_host>::<slave_volume> config log-file
-```
-```sh
-gluster volume geo-replication <master_volume> <slave_host>::<slave_volume> config change-detector
-```
-```sh
-gluster volume geo-replication <master_volume> <slave_host>::<slave_volume> config working-dir
-```
-To set a new value to this, just provide a new value. Note that, not all the config values are allowed to change. Some can not be modified.
-
-```sh
-gluster volume geo-replication <master_volume> <slave_host>::<slave_volume> config change-detector xsync
-```
-Make sure you provide the proper value to the config value. And if you have large number of small files data set, then you can use tar+ssh as syncing method. Note that, if geo-rep session is running, this restarts the gsyncd.
-
-```sh
-gluster volume geo-replication <master_volume> <slave_host>::<slave_volume> config use-tarssh true
-```
-Resetting these value to default is also simple.
-
-```sh
-gluster volume geo-replication <master_volume> <slave_host>::<slave_volume> config \!use-tarssh
-```
-That makes the config key (tar-ssh in this case) to fall back to it’s default value.
diff --git a/doc/admin-guide/en-US/markdown/admin_geo-replication.md b/doc/admin-guide/en-US/markdown/admin_geo-replication.md
deleted file mode 100644
index 47a2f66283f..00000000000
--- a/doc/admin-guide/en-US/markdown/admin_geo-replication.md
+++ /dev/null
@@ -1,688 +0,0 @@
-#Managing Geo-replication
-
-Geo-replication provides a continuous, asynchronous, and incremental
-replication service from one site to another over Local Area Networks
-(LANs), Wide Area Network (WANs), and across the Internet.
-
-Geo-replication uses a master–slave model, whereby replication and
-mirroring occurs between the following partners:
-
--   **Master** – a GlusterFS volume
-
--   **Slave** – a slave which can be of the following types:
-
-    -   A local directory which can be represented as file URL like
-        `file:///path/to/dir`. You can use shortened form, for example,
-        ` /path/to/dir`.
-
-    -   A GlusterFS Volume - Slave volume can be either a local volume
-        like `gluster://localhost:volname` (shortened form - `:volname`)
-        or a volume served by different host like
-        `gluster://host:volname` (shortened form - `host:volname`).
-
-    > **Note**
-    >
-    > Both of the above types can be accessed remotely using SSH tunnel.
-    > To use SSH, add an SSH prefix to either a file URL or gluster type
-    > URL. For example, ` ssh://root@remote-host:/path/to/dir`
-    > (shortened form - `root@remote-host:/path/to/dir`) or
-    > `ssh://root@remote-host:gluster://localhost:volname` (shortened
-    > from - `root@remote-host::volname`).
-
-This section introduces Geo-replication, illustrates the various
-deployment scenarios, and explains how to configure the system to
-provide replication and mirroring in your environment.
-
-##Replicated Volumes vs Geo-replication
-
-The following table lists the difference between replicated volumes and
-geo-replication:
-
-  Replicated Volumes | Geo-replication
-  --- | ---
-  Mirrors data across clusters | Mirrors data across geographically distributed clusters
-  Provides high-availability | Ensures backing up of data for disaster recovery
-  Synchronous replication (each and every file operation is sent across all the bricks) | Asynchronous replication (checks for the changes in files periodically and syncs them on detecting differences)
-
-##Preparing to Deploy Geo-replication
-
-This section provides an overview of the Geo-replication deployment
-scenarios, describes how you can check the minimum system requirements,
-and explores common deployment scenarios.
-
-##Exploring Geo-replication Deployment Scenarios
-
-Geo-replication provides an incremental replication service over Local
-Area Networks (LANs), Wide Area Network (WANs), and across the Internet.
-This section illustrates the most common deployment scenarios for
-Geo-replication, including the following:
-
--   Geo-replication over LAN
--   Geo-replication over WAN
--   Geo-replication over the Internet
--   Multi-site cascading Geo-replication
-
-**Geo-replication over LAN**
-
-You can configure Geo-replication to mirror data over a Local Area
-Network.
-
-![ Geo-replication over LAN ][]
-
-**Geo-replication over WAN**
-
-You can configure Geo-replication to replicate data over a Wide Area
-Network.
-
-![ Geo-replication over WAN ][]
-
-**Geo-replication over Internet**
-
-You can configure Geo-replication to mirror data over the Internet.
-
-![ Geo-replication over Internet ][]
-
-**Multi-site cascading Geo-replication**
-
-You can configure Geo-replication to mirror data in a cascading fashion
-across multiple sites.
-
-![ Multi-site cascading Geo-replication ][]
-
-##Geo-replication Deployment Overview
-
-Deploying Geo-replication involves the following steps:
-
-1.  Verify that your environment matches the minimum system requirement.
-2.  Determine the appropriate deployment scenario.
-3.  Start Geo-replication on master and slave systems, as required.
-
-##Checking Geo-replication Minimum Requirements
-
-Before deploying GlusterFS Geo-replication, verify that your systems
-match the minimum requirements.
-
-The following table outlines the minimum requirements for both master
-and slave nodes within your environment:
-
-  Component                | Master                                                                | Slave
-  ---                      | ---                                                                   | ---
-  Operating System         | GNU/Linux                                                             | GNU/Linux
-  Filesystem               | GlusterFS 3.2 or higher                                               | GlusterFS 3.2 or higher (GlusterFS needs to be installed, but does not need to be running), ext3, ext4, or XFS (any other POSIX compliant file system would work, but has not been tested extensively)
-  Python                   | Python 2.4 (with ctypes external module), or Python 2.5 (or higher)   | Python 2.4 (with ctypes external module), or Python 2.5 (or higher)
-  Secure shell             | OpenSSH version 4.0 (or higher)                                       | SSH2-compliant daemon
-  Remote synchronization   | rsync 3.0.7 or higher                                                 | rsync 3.0.7 or higher
-  FUSE                     | GlusterFS supported versions                                          | GlusterFS supported versions
-
-##Setting Up the Environment for Geo-replication
-
-**Time Synchronization**
-
--   On bricks of a geo-replication master volume, all the servers' time
-    must be uniform. You are recommended to set up NTP (Network Time
-    Protocol) service to keep the bricks sync in time and avoid
-    out-of-time sync effect.
-
-    For example: In a Replicated volume where brick1 of the master is at
-    12.20 hrs and brick 2 of the master is at 12.10 hrs with 10 minutes
-    time lag, all the changes in brick2 between this period may go
-    unnoticed during synchronization of files with Slave.
-
-    For more information on setting up NTP, see [][].
-
-**To setup Geo-replication for SSH**
-
-Password-less login has to be set up between the host machine (where
-geo-replication Start command will be issued) and the remote machine
-(where slave process should be launched through SSH).
-
-1.  On the node where geo-replication sessions are to be set up, run the
-    following command:
-
-    `# ssh-keygen -f /var/lib/glusterd/geo-replication/secret.pem`
-
-    Press Enter twice to avoid passphrase.
-
-2.  Run the following command on master for all the slave hosts:
-
-    `# ssh-copy-id -i /var/lib/glusterd/geo-replication/secret.pem.pub @`
-
-##Setting Up the Environment for a Secure Geo-replication Slave
-
-You can configure a secure slave using SSH so that master is granted a
-restricted access. With GlusterFS, you need not specify configuration
-parameters regarding the slave on the master-side configuration. For
-example, the master does not require the location of the rsync program
-on slave but the slave must ensure that rsync is in the PATH of the user
-which the master connects using SSH. The only information that master
-and slave have to negotiate are the slave-side user account, slave's
-resources that master uses as slave resources, and the master's public
-key. Secure access to the slave can be established using the following
-options:
-
--   Restricting Remote Command Execution
-
--   Using `Mountbroker` for Slaves
-
--   Using IP based Access Control
-
-**Backward Compatibility**
-
-Your existing Ge-replication environment will work with GlusterFS,
-except for the following:
-
--   The process of secure reconfiguration affects only the glusterfs
-    instance on slave. The changes are transparent to master with the
-    exception that you may have to change the SSH target to an
-    unprivileged account on slave.
-
--   The following are the some exceptions where this might not work:
-
-    -   Geo-replication URLs which specify the slave resource when
-        configuring master will include the following special
-        characters: space, \*, ?, [;
-
-    -   Slave must have a running instance of glusterd, even if there is
-        no gluster volume among the mounted slave resources (that is,
-        file tree slaves are used exclusively) .
-
-### Restricting Remote Command Execution
-
-If you restrict remote command execution, then the Slave audits commands
-coming from the master and the commands related to the given
-geo-replication session is allowed. The Slave also provides access only
-to the files within the slave resource which can be read or manipulated
-by the Master.
-
-To restrict remote command execution:
-
-1.  Identify the location of the gsyncd helper utility on Slave. This
-    utility is installed in `PREFIX/libexec/glusterfs/gsyncd`, where
-    PREFIX is a compile-time parameter of glusterfs. For example,
-    `--prefix=PREFIX` to the configure script with the following common
-    values` /usr, /usr/local, and /opt/glusterfs/glusterfs_version`.
-
-2.  Ensure that command invoked from master to slave passed through the
-    slave's gsyncd utility.
-
-    You can use either of the following two options:
-
-    -   Set gsyncd with an absolute path as the shell for the account
-        which the master connects through SSH. If you need to use a
-        privileged account, then set it up by creating a new user with
-        UID 0.
-
-    -   Setup key authentication with command enforcement to gsyncd. You
-        must prefix the copy of master's public key in the Slave
-        account's `authorized_keys` file with the following command:
-
-        `command=<path to gsyncd>`.
-
-        For example,
-        `command="PREFIX/glusterfs/gsyncd" ssh-rsa AAAAB3Nza....`
-
-### Using Mountbroker for Slaves
-
-`mountbroker` is a new service of glusterd. This service allows an
-unprivileged process to own a GlusterFS mount by registering a label
-(and DSL (Domain-specific language) options ) with glusterd through a
-glusterd volfile. Using CLI, you can send a mount request to glusterd to
-receive an alias (symlink) of the mounted volume.
-
-A request from the agent , the unprivileged slave agents use the
-mountbroker service of glusterd to set up an auxiliary gluster mount for
-the agent in a special environment which ensures that the agent is only
-allowed to access with special parameters that provide administrative
-level access to the particular volume.
-
-**To setup an auxiliary gluster mount for the agent**:
-
-1.  Create a new group. For example, `geogroup`.
-
-2.  Create a unprivileged account. For example, ` geoaccount`. Make it a
-    member of ` geogroup`.
-
-3.  Create a new directory owned by root and with permissions *0711.*
-    For example, create a create mountbroker-root directory
-    `/var/mountbroker-root`.
-
-4.  Add the following options to the glusterd volfile, assuming the name
-    of the slave gluster volume as `slavevol`:
-
-    `option mountbroker-root /var/mountbroker-root `
-
-    `option mountbroker-geo-replication.geoaccount slavevol`
-
-    `option geo-replication-log-group geogroup`
-
-    If you are unable to locate the glusterd volfile at
-    `/etc/glusterfs/glusterd.vol`, you can create a volfile containing
-    both the default configuration and the above options and place it at
-    `/etc/glusterfs/`.
-
-    A sample glusterd volfile along with default options:
-
-        volume management
-            type mgmt/glusterd
-            option working-directory /var/lib/glusterd
-            option transport-type socket,rdma
-            option transport.socket.keepalive-time 10
-            option transport.socket.keepalive-interval 2
-            option transport.socket.read-fail-log off
-
-            option mountbroker-root /var/mountbroker-root 
-            option mountbroker-geo-replication.geoaccount slavevol
-            option geo-replication-log-group geogroup
-        end-volume
-
-    If you host multiple slave volumes on Slave, you can repeat step 2.
-    for each of them and add the following options to the `volfile`:
-
-        option mountbroker-geo-replication.geoaccount2 slavevol2
-        option mountbroker-geo-replication.geoaccount3 slavevol3
-
-5.  Setup Master to access Slave as `geoaccount@Slave`.
-
-    You can add multiple slave volumes within the same account
-    (geoaccount) by providing comma-separated list (without spaces) as
-    the argument of `mountbroker-geo-replication.geogroup`. You can also
-    have multiple options of the form `mountbroker-geo-replication.*`.
-    It is recommended to use one service account per Master machine. For
-    example, if there are multiple slave volumes on Slave for the master
-    machines Master1, Master2, and Master3, then create a dedicated
-    service user on Slave for them by repeating Step 2. for each (like
-    geogroup1, geogroup2, and geogroup3), and then add the following
-    corresponding options to the volfile:
-
-    `option mountbroker-geo-replication.geoaccount1 slavevol11,slavevol12,slavevol13`
-
-    `option mountbroker-geo-replication.geoaccount2 slavevol21,slavevol22`
-
-    `option mountbroker-geo-replication.geoaccount3 slavevol31`
-
-    Now set up Master1 to ssh to geoaccount1@Slave, etc.
-
-    You must restart glusterd after making changes in the configuration
-    to effect the updates.
-
-### Using IP based Access Control
-
-You can use IP based access control method to provide access control for
-the slave resources using IP address. You can use method for both Slave
-and file tree slaves, but in the section, we are focusing on file tree
-slaves using this method.
-
-To set access control based on IP address for file tree slaves:
-
-1.  Set a general restriction for accessibility of file tree resources:
-
-    `# gluster volume geo-replication '/*' config allow-network ::1,127.0.0.1 `
-
-    This will refuse all requests for spawning slave agents except for
-    requests initiated locally.
-
-2.  If you want the to lease file tree at `/data/slave-tree` to Master,
-    enter the following command:
-
-    `# gluster volume geo-replicationconfig allow-network `
-
-    `MasterIP` is the IP address of Master. The slave agent spawn
-    request from master will be accepted if it is executed at
-    `/data/slave-tree`.
-
-If the Master side network configuration does not enable the Slave to
-recognize the exact IP address of Master, you can use CIDR notation to
-specify a subnet instead of a single IP address as MasterIP or even
-comma-separated lists of CIDR subnets.
-
-If you want to extend IP based access control to gluster slaves, use the
-following command:
-
-`# gluster volume geo-replication '*' config allow-network ::1,127.0.0.1`
-
-##Starting Geo-replication
-
-This section describes how to configure and start Gluster
-Geo-replication in your storage environment, and verify that it is
-functioning correctly.
-
-###Starting Geo-replication
-
-To start Gluster Geo-replication
-
--   Start geo-replication between the hosts using the following command:
-
-    `# gluster volume geo-replication  start`
-
-    For example:
-
-        # gluster volume geo-replication Volume1 example.com:/data/remote_dir start
-        Starting geo-replication session between Volume1
-        example.com:/data/remote_dir has been successful
-
-    > **Note**
-    >
-    > You may need to configure the service before starting Gluster
-    > Geo-replication.
-
-###Verifying Successful Deployment
-
-You can use the gluster command to verify the status of Gluster
-Geo-replication in your environment.
-
-**To verify the status Gluster Geo-replication**
-
--   Verify the status by issuing the following command on host:
-
-    `# gluster volume geo-replication  status`
-
-    For example:
-
-    `# gluster volume geo-replication Volume1 example.com:/data/remote_dir status`
-
-        # gluster volume geo-replication Volume1 example.com:/data/remote_dir status
-
-        MASTER    SLAVE                            STATUS
-        ______    ______________________________   ____________
-        Volume1 root@example.com:/data/remote_dir  Starting....
-
-###Displaying Geo-replication Status Information
-
-You can display status information about a specific geo-replication
-master session, or a particular master-slave session, or all
-geo-replication sessions, as needed.
-
-**To display geo-replication status information**
-
--   Display information of all geo-replication sessions using the
-    following command:
-
-        # gluster volume geo-replication Volume1 example.com:/data/remote_dir status
-
-        MASTER    SLAVE                            STATUS
-        ______    ______________________________   ____________
-        Volume1 root@example.com:/data/remote_dir  Starting....
-
--   Display information of a particular master slave session using the
-    following command:
-
-    `# gluster volume geo-replication  status`
-
-    For example, to display information of Volume1 and
-    example.com:/data/remote\_dir
-
-    `# gluster volume geo-replication Volume1 example.com:/data/remote_dir status`
-
-    The status of the geo-replication between Volume1 and
-    example.com:/data/remote\_dir is displayed.
-
--   Display information of all geo-replication sessions belonging to a
-    master
-
-    `# gluster volume geo-replication MASTER status`
-
-    For example, to display information of Volume1
-
-        # gluster volume geo-replication Volume1 example.com:/data/remote_dir status
-
-        MASTER    SLAVE                            STATUS
-        ______    ______________________________   ____________
-        Volume1 ssh://example.com:gluster://127.0.0.1:remove_volume  OK
-
-        Volume1 ssh://example.com:file:///data/remote_dir  OK
-
-    The status of a session could be one of the following four:
-
--   **Starting**: This is the initial phase of the Geo-replication
-    session; it remains in this state for a minute, to make sure no
-    abnormalities are present.
-
--   **OK**: The geo-replication session is in a stable state.
-
--   **Faulty**: The geo-replication session has witnessed some
-    abnormality and the situation has to be investigated further.
-
--   **Corrupt**: The monitor thread which is monitoring the
-    geo-replication session has died. This situation should not occur
-    normally.
-
-##Configuring Geo-replication
-
-To configure Gluster Geo-replication
-
--   Use the following command at the Gluster command line:
-
-    `# gluster volume geo-replication  config [options]`
-
-    For example:
-
-    To view list of all option/value pair, use the following command:
-
-    `# gluster volume geo-replication Volume1 example.com:/data/remote_dir config`
-
-##Stopping Geo-replication
-
-You can use the gluster command to stop Gluster Geo-replication (syncing
-of data from Master to Slave) in your environment.
-
-**To stop Gluster Geo-replication**
-
--   Stop geo-replication between the hosts using the following command:
-
-    `# gluster volume geo-replication  stop `
-
-    For example:
-
-        # gluster volume geo-replication Volume1 example.com:/data/remote_dir stop
-        Stopping geo-replication session between Volume1 and
-        example.com:/data/remote_dir has been successful
-
-##Restoring Data from the Slave
-
-You can restore data from the slave to the master volume, whenever the
-master volume becomes faulty for reasons like hardware failure.
-
-The example in this section assumes that you are using the Master Volume
-(Volume1) with the following configuration:
-
-    machine1# gluster volume info
-    Type: Distribute
-    Status: Started
-    Number of Bricks: 2
-    Transport-type: tcp
-    Bricks:
-    Brick1: machine1:/export/dir16
-    Brick2: machine2:/export/dir16
-    Options Reconfigured:
-    geo-replication.indexing: on
-
-The data is syncing from master volume (Volume1) to slave directory
-(example.com:/data/remote\_dir). To view the status of this
-geo-replication session run the following command on Master:
-
-    # gluster volume geo-replication Volume1 root@example.com:/data/remote_dir status
-
-    MASTER    SLAVE                             STATUS
-    ______    ______________________________    ____________
-    Volume1  root@example.com:/data/remote_dir   OK
-
-**Before Failure**
-
-Assume that the Master volume had 100 files and was mounted at
-/mnt/gluster on one of the client machines (client). Run the following
-command on Client machine to view the list of files:
-
-    client# ls /mnt/gluster | wc –l
-    100
-
-The slave directory (example.com) will have same data as in the master
-volume and same can be viewed by running the following command on slave:
-
-    example.com# ls /data/remote_dir/ | wc –l
-    100
-
-**After Failure**
-
-If one of the bricks (machine2) fails, then the status of
-Geo-replication session is changed from "OK" to "Faulty". To view the
-status of this geo-replication session run the following command on
-Master:
-
-    # gluster volume geo-replication Volume1 root@example.com:/data/remote_dir status
-
-    MASTER    SLAVE                              STATUS
-    ______    ______________________________     ____________
-    Volume1   root@example.com:/data/remote_dir  Faulty
-
-Machine2 is failed and now you can see discrepancy in number of files
-between master and slave. Few files will be missing from the master
-volume but they will be available only on slave as shown below.
-
-Run the following command on Client:
-
-    client # ls /mnt/gluster | wc –l
-    52
-
-Run the following command on slave (example.com):
-
-    Example.com# # ls /data/remote_dir/ | wc –l
-    100
-
-**To restore data from the slave machine**
-
-1.  Stop all Master's geo-replication sessions using the following
-    command:
-
-    `# gluster volume geo-replication  stop`
-
-    For example:
-
-        machine1# gluster volume geo-replication Volume1
-        example.com:/data/remote_dir stop
-
-        Stopping geo-replication session between Volume1 &
-        example.com:/data/remote_dir has been successful
-
-    > **Note**
-    >
-    > Repeat `# gluster volume geo-replication  stop `command on all
-    > active geo-replication sessions of master volume.
-
-2.  Replace the faulty brick in the master by using the following
-    command:
-
-    `# gluster volume replace-brick  start`
-
-    For example:
-
-        machine1# gluster volume replace-brick Volume1 machine2:/export/dir16 machine3:/export/dir16 start
-        Replace-brick started successfully
-
-3.  Commit the migration of data using the following command:
-
-    `# gluster volume replace-brick  commit force `
-
-    For example:
-
-        machine1# gluster volume replace-brick Volume1 machine2:/export/dir16 machine3:/export/dir16 commit force
-        Replace-brick commit successful
-
-4.  Verify the migration of brick by viewing the volume info using the
-    following command:
-
-    `# gluster volume info `
-
-    For example:
-
-        machine1# gluster volume info
-        Volume Name: Volume1
-        Type: Distribute
-        Status: Started
-        Number of Bricks: 2
-        Transport-type: tcp
-        Bricks:
-        Brick1: machine1:/export/dir16
-        Brick2: machine3:/export/dir16
-        Options Reconfigured:
-        geo-replication.indexing: on
-
-5.  Run rsync command manually to sync data from slave to master
-    volume's client (mount point).
-
-    For example:
-
-    `example.com# rsync -PavhS --xattrs --ignore-existing /data/remote_dir/ client:/mnt/gluster`
-
-    Verify that the data is synced by using the following command:
-
-    On master volume, run the following command:
-
-        Client # ls | wc –l
-        100
-
-    On the Slave run the following command:
-
-        example.com# ls /data/remote_dir/ | wc –l
-        100
-
-    Now Master volume and Slave directory is synced.
-
-6.  Restart geo-replication session from master to slave using the
-    following command:
-
-    `# gluster volume geo-replication  start `
-
-    For example:
-
-        machine1# gluster volume geo-replication Volume1
-        example.com:/data/remote_dir start
-        Starting geo-replication session between Volume1 &
-        example.com:/data/remote_dir has been successful
-
-##Best Practices
-
-**Manually Setting Time**
-
-If you have to change the time on your bricks manually, then you must
-set uniform time on all bricks. Setting time backward corrupts the
-geo-replication index, so the recommended way to set the time manually is:
-
-1.  Stop geo-replication between the master and slave using the
-    following command:
-
-    `# gluster volume geo-replication  sto`p
-
-2.  Stop the geo-replication indexing using the following command:
-
-    `# gluster volume set  geo-replication.indexing of`f
-
-3.  Set uniform time on all bricks.s
-
-4.  Restart your geo-replication sessions by using the following
-    command:
-
-    `# gluster volume geo-replication start `
-
-**Running Geo-replication commands in one system**
-
-It is advisable to run the geo-replication commands in one of the bricks
-in the trusted storage pool. This is because, the log files for the
-geo-replication session would be stored in the \*Server\* where the
-Geo-replication start is initiated. Hence it would be easier to locate
-the log-files when required.
-
-**Isolation**
-
-Geo-replication slave operation is not sandboxed as of now and is ran as
-a privileged service. So for the security reason, it is advised to
-create a sandbox environment (dedicated machine / dedicated virtual
-machine / chroot/container type solution) by the administrator to run
-the geo-replication slave in it. Enhancement in this regard will be
-available in follow-up minor release.
-
-  [ Geo-replication over LAN ]: ../images/Geo-Rep_LAN.png
-  [ Geo-replication over WAN ]: ../images/Geo-Rep_WAN.png
-  [ Geo-replication over Internet ]: ../images/Geo-Rep03_Internet.png
-  [ Multi-site cascading Geo-replication ]: ../images/Geo-Rep04_Cascading.png
-  []: http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Migration_Planning_Guide/ch04s07.html
-  [1]: www.redhat.com/support/
diff --git a/doc/admin-guide/en-US/markdown/admin_managing_snapshots.md b/doc/admin-guide/en-US/markdown/admin_managing_snapshots.md
deleted file mode 100644
index 820dcb18863..00000000000
--- a/doc/admin-guide/en-US/markdown/admin_managing_snapshots.md
+++ /dev/null
@@ -1,251 +0,0 @@
-Managing GlusterFS Volume Snapshots
-==========================
-
-This section describes how to perform common GlusterFS volume snapshot
-management operations
-
-Pre-requisites
-=====================
-
-GlusterFS volume snapshot feature is based on thinly provisioned LVM snapshot.
-To make use of snapshot feature GlusterFS volume should fulfill following
-pre-requisites:
-
-* Each brick should be on an independent thinly provisioned LVM.
-* Brick LVM should not contain any other data other than brick.
-* None of the brick should be on a thick LVM.
-* gluster version should be 3.6 and above.
-
-Details of how to create thin volume can be found at the following link.
-https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Volume_Manager_Administration/thinly_provisioned_volume_creation.html
-
-
-Few features of snapshot are:
-=============================
-
-**Crash Consistency**
-
-when a snapshot is taken at a particular point-in-time, it is made sure that
-the taken snapshot is crash consistent. when the taken snapshot is restored,
-then the data is identical as it was at the time of taking a snapshot.
-
-
-**Online Snapshot**
-
-When the snapshot is being taken the file system and its associated data
-continue to be available for the clients.
-
-
-**Quorum Based**
-
-The quorum feature ensures that the volume is in good condition while the bricks
-are down. Quorum is not met if any bricks are down in a n-way replication where
-n <= 2. Quorum is met when m bricks are up, where m >= (n/2 + 1) where n is odd,
-and m >= n/2 and first brick is up where n is even. snapshot creation fails
-if quorum is not met.
-
-
-**Barrier**
-
-During snapshot creation some of the fops are blocked to guarantee crash
-consistency. There is a default time-out of 2 minutes, if snapshot creation
-is not complete within that span then fops are unbarried. If unbarrier happens
-before the snapshot creation is complete then the snapshot creation operation
-fails. This to ensure that the snapshot is in a consistent state.
-
-
-
-Snapshot Management
-=====================
-
-
-**Snapshot creation**
-
-Syntax :
-*gluster snapshot create <snapname\> <volname\(s\)\> \[description <description\>\] \[force\]*
-
-Details :
-Creates a snapshot of a GlusterFS volume. User can provide a snap-name and a
-description to identify the snap. The description cannot be more than 1024
-characters.
-
-NOTE : To be able to take a snapshot, volume should be present and it
-should be in started state.
-
------------------------------------------------------------------------------
-
-**Restoring snaps**
-
-Syntax :
-*gluster snapshot restore <snapname\>*
-
-Details :
-Restores an already taken snapshot of a GlusterFS volume.
-Snapshot restore is an offline activity therefore if the volume is
-online (in started state) then the restore operation will fail.
-
-Once the snapshot is restored it will not be available in the
-list of snapshots.
-
----------------------------------------------------------------------------
-
-**Deleting snaps**
-
-Syntax :
-*gluster snapshot delete \(all | <snapname\> | volume <volname\>\)*
-
-Details :
-If snapname is specified then mentioned snapshot is deleted.
-If volname is specified then all snapshots belonging to that particular
-volume is deleted. If keyword *all* is used then all snapshots belonging
-to the system is deleted.
-
---------------------------------------------------------------------------
-
-**Listing of available snaps**
-
-Syntax:
-*gluster snapshot list \[volname\]*
-
-Details:
-Lists  all  snapshots  taken.
-If volname is provided, then only the snapshots belonging to
-that particular volume is listed.
-
--------------------------------------------------------------------------
-
-**Information of available snaps**
-
-Syntax:
-*gluster snapshot info \[\(snapname | volume <volname\>\)\]*
-
-Details:
-This command gives information such as snapshot name, snapshot UUID,
-time at which snapshot was created, and it lists down the snap-volume-name,
-number of snapshots already taken and number of snapshots still available
-for that particular volume, and the state of the snapshot.
-
-------------------------------------------------------------------------
-
-**Status of snapshots**
-
-Syntax:
-*gluster snapshot status \[\(snapname | volume <volname\>\)\]*
-
-Details:
-This  command  gives  status of the snapshot.
-The details included are snapshot brick path, volume group(LVM details),
-status of the snapshot bricks, PID of the bricks, data percentage  filled for
-that particular volume group to which the snapshots belong to, and total size
-of the logical volume.
-
-If snapname is specified then status of the mentioned snapshot is displayed.
-If volname  is specified then status of all snapshots belonging to that volume
-is displayed. If both snapname and volname is not specified then status of all
-the snapshots present in the system are displayed.
-
-------------------------------------------------------------------------
-
-**Configuring the snapshot behavior**
-
-Syntax:
-*snapshot config \[volname\] \(\[snap-max-hard-limit <count\>\] \[snap-max-soft-limit <percent>\]\)
-                            | \(\[auto-delete <enable|disable\>\]\)
-                            | \(\[activate-on-create <enable|disable\>\]\)*
-
-Details:
-Displays and sets the snapshot config values.
-
-snapshot  config without any keywords displays the snapshot config values of
-all volumes in the system. If volname is provided, then the snapshot config
-values of that volume is  displayed.
-
-Snapshot  config command along with keywords can be used to change the existing
-config values. If volname is provided then config value of that volume is
-changed, else it  will set/change the system limit.
-
-snap-max-soft-limit  and auto-delete are global options, that will be
-inherited by all volumes in the system and cannot be set to individual volumes.
-
-The system limit takes precedence over the volume specific limit.
-
-When auto-delete feature is enabled, then upon reaching the soft-limit,
-with every successful snapshot creation, the oldest snapshot will be deleted.
-
-When  auto-delete  feature  is disabled, then upon reaching the soft-limit,
-the user gets a warning with every successful snapshot creation.
-
-Upon reaching the hard-limit, further snapshot creations will not be allowed.
-
-activate-on-create is disabled by default. If you enable activate-on-create,
-then further snapshot will be activated during the time of snapshot creation.
--------------------------------------------------------------------------
-
-**Activating a snapshot**
-
-Syntax:
-*gluster snapshot activate <snapname\>*
-
-Details:
-Activates the mentioned snapshot.
-
-Note : By default the snapshot is activated during snapshot creation.
-
--------------------------------------------------------------------------
-
-**Deactivate a snapshot**
-
-Syntax:
-*gluster snapshot deactivate <snapname\>*
-
-Details:
-Deactivates the mentioned snapshot.
-
--------------------------------------------------------------------------
-
-**Accessing the snapshot**
-
-Snapshots can be activated in 2 ways.
-
-1) Mounting the snapshot:
-
-The snapshot can be accessed via FUSE mount (only fuse). To do that it has to be
-mounted first. A snapshot can be mounted via fuse by below command
-
-*mount -t glusterfs <hostname>:/snaps/<snap-name>/<volume-name> <mount-path>*
-
-i.e. say "host1" is one of the peers. Let "vol" be the volume name and "my-snap"
-be the snapshot name. In this case a snapshot can be mounted via this command
-
-*mount -t glusterfs host1:/snaps/my-snap/vol /mnt/snapshot*
-
-
-2) User serviceability:
-
-Apart from the above method of mounting the snapshot, a list of available
-snapshots and the contents of each snapshot can be viewed from any of the mount
-points accessing the glusterfs volume (either FUSE or NFS or SMB). For having
-user serviceable snapshots, it has to be enabled for a volume first. User
-serviceability can be enabled for a volume using the below command.
-
-*gluster volume set <volname> features.uss enable*
-
-Once enabled, from any of the directory (including root of the filesystem) an
-access point will be created to the snapshot world. The access point is a hidden
-directory cding into which will make the user enter the snapshot world. By
-default the hidden directory is ".snaps". Once user serviceability is enabled,
-one will be able to cd into .snaps from any directory. Doing "ls" on that
-directory shows a list of directories which are nothing but the snapshots
-present for that volume. Say if there are 3 snapshots ("snap1", "snap2",
-"snap3"), then doing ls in .snaps directory will show those 3 names as the
-directory entries. They represent the state of the directory from which .snaps
-was entered, at different points in time.
-
-NOTE: The access to the snapshots are read-only.
-
-Also, the name of the hidden directory (or the access point to the snapshot
-world) can be changed using the below command.
-
-*gluster volume set <volname> snapshot-directory <new-name>*
-
---------------------------------------------------------------------------------------
diff --git a/doc/admin-guide/en-US/markdown/admin_managing_volumes.md b/doc/admin-guide/en-US/markdown/admin_managing_volumes.md
deleted file mode 100644
index 3ac571c9a7e..00000000000
--- a/doc/admin-guide/en-US/markdown/admin_managing_volumes.md
+++ /dev/null
@@ -1,638 +0,0 @@
-#Managing GlusterFS Volumes
-
-This section describes how to perform common GlusterFS management
-operations, including the following:
-
-- [Tuning Volume Options](#tuning-options)
-- [Expanding Volumes](#expanding-volumes)
-- [Shrinking Volumes](#shrinking-volumes)
-- [Migrating Volumes](#migrating-volumes)
-- [Rebalancing Volumes](#rebalancing-volumes)
-- [Stopping Volumes](#stopping-volumes)
-- [Deleting Volumes](#deleting-volumes)
-- [Triggering Self-Heal on Replicate](#self-heal)
-
-<a name="tuning-options" />
-##Tuning Volume Options
-
-You can tune volume options, as needed, while the cluster is online and
-available.
-
-> **Note**
->
-> It is recommended that you to set server.allow-insecure option to ON if
-> there are too many bricks in each volume or if there are too many
-> services which have already utilized all the privileged ports in the
-> system. Turning this option ON allows ports to accept/reject messages
-> from insecure ports. So, use this option only if your deployment
-> requires it.
-
-Tune volume options using the following command:
-
-    `# gluster volume set `
-
-For example, to specify the performance cache size for test-volume:
-
-    # gluster volume set test-volume performance.cache-size 256MB
-    Set volume successful
-
-The following table lists the Volume options along with its
-description and default value:
-
-> **Note**
->
-> The default options given here are subject to modification at any
-> given time and may not be the same for all versions.
-
-
-Option | Description | Default Value | Available Options
---- | --- | --- | ---
-auth.allow | IP addresses of the clients which should be allowed to access the volume. | \* (allow all) | Valid IP address which includes wild card patterns including \*, such as 192.168.1.\*
-auth.reject | IP addresses of the clients which should be denied to access the volume. | NONE (reject none)  | Valid IP address which includes wild card patterns including \*, such as 192.168.2.\*
-client.grace-timeout | Specifies the duration for the lock state to be maintained on the client after a network disconnection. | 10 | 10 - 1800 secs
-cluster.self-heal-window-size | Specifies the maximum number of blocks per file on which self-heal would happen simultaneously. | 16 | 0 - 1025 blocks 
-cluster.data-self-heal-algorithm | Specifies the type of self-heal. If you set the option as "full", the entire file is copied from source to destinations. If the option is set to "diff" the file blocks that are not in sync are copied to destinations. Reset uses a heuristic model. If the file does not exist on one of the subvolumes, or a zero-byte file exists (created by entry self-heal) the entire content has to be copied anyway, so there is no benefit from using the "diff" algorithm. If the file size is about the same as page size, the entire file can be read and written with a few operations, which will be faster than "diff" which has to read checksums and then read and write. | reset | full/diff/reset 
-cluster.min-free-disk | Specifies the percentage of disk space that must be kept free. Might be useful for non-uniform bricks | 10% | Percentage of required minimum free disk space 
-cluster.stripe-block-size | Specifies the size of the stripe unit that will be read from or written to. | 128 KB (for all files) | size in bytes
-cluster.self-heal-daemon | Allows you to turn-off proactive self-heal on replicated | On | On/Off
-cluster.ensure-durability | This option makes sure the data/metadata is durable across abrupt shutdown of the brick. | On | On/Off
-diagnostics.brick-log-level | Changes the log-level of the bricks. | INFO | DEBUG/WARNING/ERROR/CRITICAL/NONE/TRACE
-diagnostics.client-log-level  |  Changes the log-level of the clients. | INFO | DEBUG/WARNING/ERROR/CRITICAL/NONE/TRACE
-diagnostics.latency-measurement | Statistics related to the latency of each operation would be tracked. | Off | On/Off 
-diagnostics.dump-fd-stats | Statistics related to file-operations would be tracked. | Off | On 
-feature.read-only | Enables you to mount the entire volume as read-only for all the clients (including NFS clients) accessing it. | Off | On/Off 
-features.lock-heal | Enables self-healing of locks when the network disconnects. | On | On/Off 
-features.quota-timeout | For performance reasons, quota caches the directory sizes on client. You can set timeout indicating the maximum duration of directory sizes in cache, from the time they are populated, during which they are considered valid | 0 |  0 - 3600 secs
-geo-replication.indexing | Use this option to automatically sync the changes in the filesystem from Master to Slave. | Off | On/Off 
-network.frame-timeout | The time frame after which the operation has to be declared as dead, if the server does not respond for a particular operation. | 1800 (30 mins) | 1800 secs
-network.ping-timeout | The time duration for which the client waits to check if the server is responsive. When a ping timeout happens, there is a network disconnect between the client and server. All resources held by server on behalf of the client get cleaned up. When a reconnection happens, all resources will need to be re-acquired before the client can resume its operations on the server. Additionally, the locks will be acquired and the lock tables updated. This reconnect is a very expensive operation and should be avoided. | 42 Secs | 42 Secs
-nfs.enable-ino32 | For 32-bit nfs clients or applications that do not support 64-bit inode numbers or large files, use this option from the CLI to make Gluster NFS return 32-bit inode numbers instead of 64-bit inode numbers. | Off | On/Off
-nfs.volume-access | Set the access type for the specified sub-volume. | read-write |  read-write/read-only
-nfs.trusted-write | If there is an UNSTABLE write from the client, STABLE flag will be returned to force the client to not send a COMMIT request. In some environments, combined with a replicated GlusterFS setup, this option can improve write performance. This flag allows users to trust Gluster replication logic to sync data to the disks and recover when required. COMMIT requests if received will be handled in a default manner by fsyncing. STABLE writes are still handled in a sync manner. | Off | On/Off 
-nfs.trusted-sync | All writes and COMMIT requests are treated as async. This implies that no write requests are guaranteed to be on server disks when the write reply is received at the NFS client. Trusted sync includes trusted-write behavior. | Off | On/Off
-nfs.export-dir | This option can be used to export specified comma separated subdirectories in the volume. The path must be an absolute path. Along with path allowed list of IPs/hostname can be associated with each subdirectory. If provided connection will allowed only from these IPs. Format: \<dir\>[(hostspec[hostspec...])][,...]. Where hostspec can be an IP address, hostname or an IP range in CIDR notation. **Note**: Care must be taken while configuring this option as invalid entries and/or unreachable DNS servers can introduce unwanted delay in all the mount calls. | No sub directory exported. | Absolute path with allowed list of IP/hostname
-nfs.export-volumes | Enable/Disable exporting entire volumes, instead if used in conjunction with nfs3.export-dir, can allow setting up only subdirectories as exports. | On | On/Off 
-nfs.rpc-auth-unix | Enable/Disable the AUTH\_UNIX authentication type. This option is enabled by default for better interoperability. However, you can disable it if required. | On | On/Off 
-nfs.rpc-auth-null | Enable/Disable the AUTH\_NULL authentication type. It is not recommended to change the default value for this option. | On | On/Off 
-nfs.rpc-auth-allow\<IP- Addresses\> | Allow a comma separated list of addresses and/or hostnames to connect to the server. By default, all clients are disallowed. This allows you to define a general rule for all exported volumes. | Reject All | IP address or Host name
-nfs.rpc-auth-reject\<IP- Addresses\> | Reject a comma separated list of addresses and/or hostnames from connecting to the server. By default, all connections are disallowed. This allows you to define a general rule for all exported volumes. | Reject All | IP address or Host name
-nfs.ports-insecure | Allow client connections from unprivileged ports. By default only privileged ports are allowed. This is a global setting in case insecure ports are to be enabled for all exports using a single option. | Off | On/Off 
-nfs.addr-namelookup | Turn-off name lookup for incoming client connections using this option. In some setups, the name server can take too long to reply to DNS queries resulting in timeouts of mount requests. Use this option to turn off name lookups during address authentication. Note, turning this off will prevent you from using hostnames in rpc-auth.addr.\* filters. | On | On/Off 
-nfs.register-with-portmap | For systems that need to run multiple NFS servers, you need to prevent more than one from registering with portmap service. Use this option to turn off portmap registration for Gluster NFS. | On | On/Off 
-nfs.port \<PORT- NUMBER\> | Use this option on systems that need Gluster NFS to be associated with a non-default port number. | NA | 38465- 38467
-nfs.disable | Turn-off volume being exported by NFS | Off | On/Off
-performance.write-behind-window-size | Size of the per-file write-behind buffer. | 1MB | Write-behind cache size 
-performance.io-thread-count | The number of threads in IO threads translator. | 16 | 0-65 
-performance.flush-behind | If this option is set ON, instructs write-behind translator to perform flush in background, by returning success (or any errors, if any of previous writes were failed) to application even before flush is sent to backend filesystem. | On | On/Off 
-performance.cache-max-file-size | Sets the maximum file size cached by the io-cache translator. Can use the normal size descriptors of KB, MB, GB,TB or PB (for example, 6GB). Maximum size uint64. | 2 \^ 64 -1 bytes | size in bytes
-performance.cache-min-file-size | Sets the minimum file size cached by the io-cache translator. Values same as "max" above | 0B | size in bytes
-performance.cache-refresh-timeout | The cached data for a file will be retained till 'cache-refresh-timeout' seconds, after which data re-validation is performed. | 1s | 0-61  
-performance.cache-size | Size of the read cache. | 32 MB |  size in bytes
-server.allow-insecure | Allow client connections from unprivileged ports. By default only privileged ports are allowed. This is a global setting in case insecure ports are to be enabled for all exports using a single option. | On | On/Off 
-server.grace-timeout | Specifies the duration for the lock state to be maintained on the server after a network disconnection. | 10 | 10 - 1800 secs 
-server.statedump-path | Location of the state dump file. | tmp directory of the brick |  New directory path
-storage.health-check-interval | Number of seconds between health-checks done on the filesystem that is used for the brick(s). Defaults to 30 seconds, set to 0 to disable. | tmp directory of the brick |  New directory path
-
-You can view the changed volume options using command:
- 
-    ` # gluster volume info `
-
-<a name="expanding-volumes" />
-##Expanding Volumes
-
-You can expand volumes, as needed, while the cluster is online and
-available. For example, you might want to add a brick to a distributed
-volume, thereby increasing the distribution and adding to the capacity
-of the GlusterFS volume.
-
-Similarly, you might want to add a group of bricks to a distributed
-replicated volume, increasing the capacity of the GlusterFS volume.
-
-> **Note**
->
-> When expanding distributed replicated and distributed striped volumes,
-> you need to add a number of bricks that is a multiple of the replica
-> or stripe count. For example, to expand a distributed replicated
-> volume with a replica count of 2, you need to add bricks in multiples
-> of 2 (such as 4, 6, 8, etc.).
-
-**To expand a volume**
-
-1.  On the first server in the cluster, probe the server to which you
-    want to add the new brick using the following command:
-
-    `# gluster peer probe `
-
-    For example:
-
-        # gluster peer probe server4
-        Probe successful
-
-2.  Add the brick using the following command:
-
-    `# gluster volume add-brick `
-
-    For example:
-
-        # gluster volume add-brick test-volume server4:/exp4
-        Add Brick successful
-
-3.  Check the volume information using the following command:
-
-    `# gluster volume info `
-
-    The command displays information similar to the following:
-
-        Volume Name: test-volume
-        Type: Distribute
-        Status: Started
-        Number of Bricks: 4
-        Bricks:
-        Brick1: server1:/exp1
-        Brick2: server2:/exp2
-        Brick3: server3:/exp3
-        Brick4: server4:/exp4
-
-4.  Rebalance the volume to ensure that all files are distributed to the
-    new brick.
-
-    You can use the rebalance command as described in ?.
-
-<a name="shrinking-volumes" />
-##Shrinking Volumes
-
-You can shrink volumes, as needed, while the cluster is online and
-available. For example, you might need to remove a brick that has become
-inaccessible in a distributed volume due to hardware or network failure.
-
-> **Note**
->
-> Data residing on the brick that you are removing will no longer be
-> accessible at the Gluster mount point. Note however that only the
-> configuration information is removed - you can continue to access the
-> data directly from the brick, as necessary.
-
-When shrinking distributed replicated and distributed striped volumes,
-you need to remove a number of bricks that is a multiple of the replica
-or stripe count. For example, to shrink a distributed striped volume
-with a stripe count of 2, you need to remove bricks in multiples of 2
-(such as 4, 6, 8, etc.). In addition, the bricks you are trying to
-remove must be from the same sub-volume (the same replica or stripe
-set).
-
-**To shrink a volume**
-
-1.  Remove the brick using the following command:
-
-    `# gluster volume remove-brick ` `start`
-
-    For example, to remove server2:/exp2:
-
-        # gluster volume remove-brick test-volume server2:/exp2 force
-
-        Removing brick(s) can result in data loss. Do you want to Continue? (y/n)
-
-2.  Enter "y" to confirm the operation. The command displays the
-    following message indicating that the remove brick operation is
-    successfully started:
-
-        Remove Brick successful 
-
-3.  (Optional) View the status of the remove brick operation using the
-    following command:
-
-    `# gluster volume remove-brick `` status`
-
-    For example, to view the status of remove brick operation on
-    server2:/exp2 brick:
-
-        # gluster volume remove-brick test-volume server2:/exp2 status
-                                        Node  Rebalanced-files  size  scanned       status
-                                   ---------  ----------------  ----  -------  -----------
-        617c923e-6450-4065-8e33-865e28d9428f               34   340      162   in progress
-
-4.  Check the volume information using the following command:
-
-    `# gluster volume info `
-
-    The command displays information similar to the following:
-
-        # gluster volume info
-        Volume Name: test-volume
-        Type: Distribute
-        Status: Started
-        Number of Bricks: 3
-        Bricks:
-        Brick1: server1:/exp1
-        Brick3: server3:/exp3
-        Brick4: server4:/exp4
-
-5.  Rebalance the volume to ensure that all files are distributed to the
-    new brick.
-
-    You can use the rebalance command as described in ?.
-
-<a name="migrating-volumes" />
-##Migrating Volumes
-
-You can migrate the data from one brick to another, as needed, while the
-cluster is online and available.
-
-**To migrate a volume**
-
-1.  Make sure the new brick, server5 in this example, is successfully
-    added to the cluster.
-
-2.  Migrate the data from one brick to another using the following
-    command:
-
-    ` # gluster volume replace-brick  start`
-
-    For example, to migrate the data in server3:/exp3 to server5:/exp5
-    in test-volume:
-
-        # gluster volume replace-brick test-volume server3:/exp3  server5:exp5 start
-        Replace brick start operation successful
-
-    > **Note**
-    >
-    > You need to have the FUSE package installed on the server on which
-    > you are running the replace-brick command for the command to work.
-
-3.  To pause the migration operation, if needed, use the following
-    command:
-
-    `# gluster volume replace-brick  pause `
-
-    For example, to pause the data migration from server3:/exp3 to
-    server5:/exp5 in test-volume:
-
-        # gluster volume replace-brick test-volume server3:/exp3 server5:/exp5 pause
-        Replace brick pause operation successful
-
-4.  To abort the migration operation, if needed, use the following
-    command:
-
-    ` # gluster volume replace-brick abort `
-
-    For example, to abort the data migration from server3:/exp3 to
-    server5:/exp5 in test-volume:
-
-        # gluster volume replace-brick test-volume server3:/exp3 server5:exp5 abort
-        Replace brick abort operation successful
-
-5.  Check the status of the migration operation using the following
-    command:
-
-    ` # gluster volume replace-brick status `
-
-    For example, to check the data migration status from server3:/exp3
-    to server5:/exp5 in test-volume:
-
-        # gluster volume replace-brick test-volume server3:/exp3 server5:/exp5 status
-        Current File = /usr/src/linux-headers-2.6.31-14/block/Makefile 
-        Number of files migrated = 10567
-        Migration complete
-
-    The status command shows the current file being migrated along with
-    the current total number of files migrated. After completion of
-    migration, it displays Migration complete.
-
-6.  Commit the migration of data from one brick to another using the
-    following command:
-
-    ` # gluster volume replace-brick commit `
-
-    For example, to commit the data migration from server3:/exp3 to
-    server5:/exp5 in test-volume:
-
-        # gluster volume replace-brick test-volume server3:/exp3 server5:/exp5 commit
-        replace-brick commit successful
-
-7.  Verify the migration of brick by viewing the volume info using the
-    following command:
-
-    `# gluster volume info `
-
-    For example, to check the volume information of new brick
-    server5:/exp5 in test-volume:
-
-        # gluster volume info test-volume
-        Volume Name: testvolume
-        Type: Replicate
-        Status: Started
-        Number of Bricks: 4
-        Transport-type: tcp
-        Bricks:
-        Brick1: server1:/exp1
-        Brick2: server2:/exp2
-        Brick3: server4:/exp4
-        Brick4: server5:/exp5
-
-        The new volume details are displayed.
-
-    The new volume details are displayed.
-
-    In the above example, previously, there were bricks; 1,2,3, and 4
-    and now brick 3 is replaced by brick 5.
-
-<a name="rebalancing-volumes" />
-##Rebalancing Volumes
-
-After expanding or shrinking a volume (using the add-brick and
-remove-brick commands respectively), you need to rebalance the data
-among the servers. New directories created after expanding or shrinking
-of the volume will be evenly distributed automatically. For all the
-existing directories, the distribution can be fixed by rebalancing the
-layout and/or data.
-
-This section describes how to rebalance GlusterFS volumes in your
-storage environment, using the following common scenarios:
-
--   **Fix Layout** - Fixes the layout changes so that the files can actually
-    go to newly added nodes.
-
--   **Fix Layout and Migrate Data** - Rebalances volume by fixing the layout
-    changes and migrating the existing data.
-
-###Rebalancing Volume to Fix Layout Changes
-
-Fixing the layout is necessary because the layout structure is static
-for a given directory. In a scenario where new bricks have been added to
-the existing volume, newly created files in existing directories will
-still be distributed only among the old bricks. The
-`# gluster volume rebalance fix-layout start `command will fix the
-layout information so that the files can also go to newly added nodes.
-When this command is issued, all the file stat information which is
-already cached will get revalidated.
-
-As of GlusterFS 3.6, the assignment of files to bricks will take into account
-the sizes of the bricks.  For example, a 20TB brick will be assigned twice as
-many files as a 10TB brick.  In versions before 3.6, the two bricks were
-treated as equal regardless of size, and would have been assigned an equal
-share of files.
-
-A fix-layout rebalance will only fix the layout changes and does not
-migrate data. If you want to migrate the existing data,
-use`# gluster volume rebalance  start ` command to rebalance data among
-the servers.
-
-**To rebalance a volume to fix layout changes**
-
--   Start the rebalance operation on any one of the server using the
-    following command:
-
-    `# gluster volume rebalance fix-layout start`
-
-    For example:
-
-        # gluster volume rebalance test-volume fix-layout start
-        Starting rebalance on volume test-volume has been successful
-
-###Rebalancing Volume to Fix Layout and Migrate Data
-
-After expanding or shrinking a volume (using the add-brick and
-remove-brick commands respectively), you need to rebalance the data
-among the servers.
-
-**To rebalance a volume to fix layout and migrate the existing data**
-
--   Start the rebalance operation on any one of the server using the
-    following command:
-
-    `# gluster volume rebalance start`
-
-    For example:
-
-        # gluster volume rebalance test-volume start
-        Starting rebalancing on volume test-volume has been successful
-
--   Start the migration operation forcefully on any one of the server
-    using the following command:
-
-    `# gluster volume rebalance start force`
-
-    For example:
-
-        # gluster volume rebalance test-volume start force
-        Starting rebalancing on volume test-volume has been successful
-
-###Displaying Status of Rebalance Operation
-
-You can display the status information about rebalance volume operation,
-as needed.
-
--   Check the status of the rebalance operation, using the following
-    command:
-
-    `# gluster volume rebalance  status`
-
-    For example:
-
-        # gluster volume rebalance test-volume status
-                                        Node  Rebalanced-files  size  scanned       status
-                                   ---------  ----------------  ----  -------  -----------
-        617c923e-6450-4065-8e33-865e28d9428f               416  1463      312  in progress
-
-    The time to complete the rebalance operation depends on the number
-    of files on the volume along with the corresponding file sizes.
-    Continue checking the rebalance status, verifying that the number of
-    files rebalanced or total files scanned keeps increasing.
-
-    For example, running the status command again might display a result
-    similar to the following:
-
-        # gluster volume rebalance test-volume status
-                                        Node  Rebalanced-files  size  scanned       status
-                                   ---------  ----------------  ----  -------  -----------
-        617c923e-6450-4065-8e33-865e28d9428f               498  1783      378  in progress
-
-    The rebalance status displays the following when the rebalance is
-    complete:
-
-        # gluster volume rebalance test-volume status
-                                        Node  Rebalanced-files  size  scanned       status
-                                   ---------  ----------------  ----  -------  -----------
-        617c923e-6450-4065-8e33-865e28d9428f               502  1873      334   completed
-
-###Stopping Rebalance Operation
-
-You can stop the rebalance operation, as needed.
-
--   Stop the rebalance operation using the following command:
-
-    `# gluster volume rebalance  stop`
-
-    For example:
-
-        # gluster volume rebalance test-volume stop
-                                        Node  Rebalanced-files  size  scanned       status
-                                   ---------  ----------------  ----  -------  -----------
-        617c923e-6450-4065-8e33-865e28d9428f               59   590      244       stopped
-        Stopped rebalance process on volume test-volume 
-
-<a name="stopping-volumes" />
-##Stopping Volumes
-
-1.  Stop the volume using the following command:
-
-    `# gluster volume stop `
-
-    For example, to stop test-volume:
-
-        # gluster volume stop test-volume
-        Stopping volume will make its data inaccessible. Do you want to continue? (y/n)
-
-2.  Enter `y` to confirm the operation. The output of the command
-    displays the following:
-
-        Stopping volume test-volume has been successful
-
-<a name="" />
-##Deleting Volumes
-
-1.  Delete the volume using the following command:
-
-    `# gluster volume delete `
-
-    For example, to delete test-volume:
-
-        # gluster volume delete test-volume
-        Deleting volume will erase all information about the volume. Do you want to continue? (y/n)
-
-2.  Enter `y` to confirm the operation. The command displays the
-    following:
-
-        Deleting volume test-volume has been successful
-
-<a name="self-heal" />
-##Triggering Self-Heal on Replicate
-
-In replicate module, previously you had to manually trigger a self-heal
-when a brick goes offline and comes back online, to bring all the
-replicas in sync. Now the pro-active self-heal daemon runs in the
-background, diagnoses issues and automatically initiates self-healing
-every 10 minutes on the files which requires*healing*.
-
-You can view the list of files that need *healing*, the list of files
-which are currently/previously *healed*, list of files which are in
-split-brain state, and you can manually trigger self-heal on the entire
-volume or only on the files which need *healing*.
-
--   Trigger self-heal only on the files which requires *healing*:
-
-    `# gluster volume heal `
-
-    For example, to trigger self-heal on files which requires *healing*
-    of test-volume:
-
-        # gluster volume heal test-volume
-        Heal operation on volume test-volume has been successful
-
--   Trigger self-heal on all the files of a volume:
-
-    `# gluster volume heal ` `full`
-
-    For example, to trigger self-heal on all the files of of
-    test-volume:
-
-        # gluster volume heal test-volume full
-        Heal operation on volume test-volume has been successful
-
--   View the list of files that needs *healing*:
-
-    `# gluster volume heal ` `info`
-
-    For example, to view the list of files on test-volume that needs
-    *healing*:
-
-        # gluster volume heal test-volume info
-        Brick :/gfs/test-volume_0
-        Number of entries: 0
-         
-        Brick :/gfs/test-volume_1
-        Number of entries: 101 
-        /95.txt
-        /32.txt
-        /66.txt
-        /35.txt
-        /18.txt
-        /26.txt
-        /47.txt
-        /55.txt
-        /85.txt
-        ...
-
--   View the list of files that are self-healed:
-
-    `# gluster volume heal ` `info healed`
-
-    For example, to view the list of files on test-volume that are
-    self-healed:
-
-        # gluster volume heal test-volume info healed
-        Brick :/gfs/test-volume_0 
-        Number of entries: 0
-
-        Brick :/gfs/test-volume_1 
-        Number of entries: 69
-        /99.txt
-        /93.txt
-        /76.txt
-        /11.txt
-        /27.txt
-        /64.txt
-        /80.txt
-        /19.txt
-        /41.txt
-        /29.txt
-        /37.txt
-        /46.txt
-        ...
-
--   View the list of files of a particular volume on which the self-heal
-    failed:
-
-    `# gluster volume heal ` `info failed`
-
-    For example, to view the list of files of test-volume that are not
-    self-healed:
-
-        # gluster volume heal test-volume info failed
-        Brick :/gfs/test-volume_0
-        Number of entries: 0 
-
-        Brick server2:/gfs/test-volume_3 
-        Number of entries: 72
-        /90.txt
-        /95.txt
-        /77.txt
-        /71.txt
-        /87.txt
-        /24.txt
-        ...
-
--   View the list of files of a particular volume which are in
-    split-brain state:
-
-    `# gluster volume heal ` `info split-brain`
-
-    For example, to view the list of files of test-volume which are in
-    split-brain state:
-
-        # gluster volume heal test-volume info split-brain
-        Brick server1:/gfs/test-volume_2 
-        Number of entries: 12
-        /83.txt
-        /28.txt
-        /69.txt
-        ...
-
-        Brick :/gfs/test-volume_2
-        Number of entries: 12
-        /83.txt
-        /28.txt
-        /69.txt
-        ...
-
-
diff --git a/doc/admin-guide/en-US/markdown/admin_monitoring_workload.md b/doc/admin-guide/en-US/markdown/admin_monitoring_workload.md
deleted file mode 100644
index c3ac0609b99..00000000000
--- a/doc/admin-guide/en-US/markdown/admin_monitoring_workload.md
+++ /dev/null
@@ -1,893 +0,0 @@
-#Monitoring your GlusterFS Workload
-
-You can monitor the GlusterFS volumes on different parameters.
-Monitoring volumes helps in capacity planning and performance tuning
-tasks of the GlusterFS volume. Using these information, you can identify
-and troubleshoot issues.
-
-You can use Volume Top and Profile commands to view the performance and
-identify bottlenecks/hotspots of each brick of a volume. This helps
-system administrators to get vital performance information whenever
-performance needs to be probed.
-
-You can also perform statedump of the brick processes and nfs server
-process of a volume, and also view volume status and volume information.
-
-##Running GlusterFS Volume Profile Command
-
-GlusterFS Volume Profile command provides an interface to get the
-per-brick I/O information for each File Operation (FOP) of a volume. The
-per brick information helps in identifying bottlenecks in the storage
-system.
-
-This section describes how to run GlusterFS Volume Profile command by
-performing the following operations:
-
--   [Start Profiling](#start-profiling)
--   [Displaying the I/0 Information](#displaying-io)
--   [Stop Profiling](#stop-profiling)
-
-<a name="start-profiling" />
-###Start Profiling
-
-You must start the Profiling to view the File Operation information for
-each brick.
-
-To start profiling, use following command:
-
-`# gluster volume profile  start `
-
-For example, to start profiling on test-volume:
-
-    # gluster volume profile test-volume start
-    Profiling started on test-volume
-
-When profiling on the volume is started, the following additional
-options are displayed in the Volume Info:
-
-    diagnostics.count-fop-hits: on
-    diagnostics.latency-measurement: on
-
-<a name="displaying-io" />
-###Displaying the I/0 Information
-
-You can view the I/O information of each brick by using the following command:
-
-`# gluster volume profile  info`
-
-For example, to see the I/O information on test-volume:
-
-    # gluster volume profile test-volume info
-    Brick: Test:/export/2
-    Cumulative Stats:
-
-    Block                     1b+           32b+           64b+
-    Size:
-           Read:                0              0              0
-           Write:             908             28              8
-
-    Block                   128b+           256b+         512b+
-    Size:
-           Read:                0               6             4
-           Write:               5              23            16
-
-    Block                  1024b+          2048b+        4096b+
-    Size:
-           Read:                 0              52           17
-           Write:               15             120          846
-
-    Block                   8192b+         16384b+      32768b+
-    Size:
-           Read:                52               8           34
-           Write:              234             134          286
-
-    Block                                  65536b+     131072b+
-    Size:
-           Read:                               118          622
-           Write:                             1341          594
-
-
-    %-latency  Avg-      Min-       Max-       calls     Fop
-              latency   Latency    Latency  
-    ___________________________________________________________
-    4.82      1132.28   21.00      800970.00   4575    WRITE
-    5.70       156.47    9.00      665085.00   39163   READDIRP
-    11.35      315.02    9.00     1433947.00   38698   LOOKUP
-    11.88     1729.34   21.00     2569638.00    7382   FXATTROP
-    47.35   104235.02 2485.00     7789367.00     488   FSYNC
-
-    ------------------
-
-    ------------------
-
-    Duration     : 335
-
-    BytesRead    : 94505058
-
-    BytesWritten : 195571980
-
-<a name="stop-profiling" />
-###Stop Profiling
-
-You can stop profiling the volume, if you do not need profiling
-information anymore.
-
-Stop profiling using the following command:
-
-    `# gluster volume profile  stop`
-
-For example, to stop profiling on test-volume:
-
-    `# gluster volume profile  stop`
-
-    `Profiling stopped on test-volume`
-
-##Running GlusterFS Volume TOP Command
-
-GlusterFS Volume Top command allows you to view the glusterfs bricks’
-performance metrics like read, write, file open calls, file read calls,
-file write calls, directory open calls, and directory real calls. The
-top command displays up to 100 results.
-
-This section describes how to run and view the results for the following
-GlusterFS Top commands:
-
--   [Viewing Open fd Count and Maximum fd Count](#open-fd-count)
--   [Viewing Highest File Read Calls](#file-read)
--   [Viewing Highest File Write Calls](#file-write)
--   [Viewing Highest Open Calls on Directories](#open-dir)
--   [Viewing Highest Read Calls on Directory](#read-dir)
--   [Viewing List of Read Performance on each Brick](#read-perf)
--   [Viewing List of Write Performance on each Brick](#write-perf)
-
-<a name="open-fd-count" />
-###Viewing Open fd Count and Maximum fd Count
-
-You can view both current open fd count (list of files that are
-currently the most opened and the count) on the brick and the maximum
-open fd count (count of files that are the currently open and the count
-of maximum number of files opened at any given point of time, since the
-servers are up and running). If the brick name is not specified, then
-open fd metrics of all the bricks belonging to the volume will be
-displayed.
-
--   View open fd count and maximum fd count using the following command:
-
-    `# gluster volume top  open [brick ] [list-cnt ]`
-
-    For example, to view open fd count and maximum fd count on brick
-    server:/export of test-volume and list top 10 open calls:
-
-    `# gluster volume top  open brick  list-cnt `
-
-    `Brick: server:/export/dir1 `
-
-    `Current open fd's: 34 Max open fd's: 209 `
-
-                     ==========Open file stats========
-
-        open            file name
-        call count     
-
-        2               /clients/client0/~dmtmp/PARADOX/
-                        COURSES.DB
-
-        11              /clients/client0/~dmtmp/PARADOX/
-                        ENROLL.DB
-
-        11              /clients/client0/~dmtmp/PARADOX/
-                        STUDENTS.DB
-
-        10              /clients/client0/~dmtmp/PWRPNT/
-                        TIPS.PPT
-
-        10              /clients/client0/~dmtmp/PWRPNT/
-                        PCBENCHM.PPT
-
-        9               /clients/client7/~dmtmp/PARADOX/
-                        STUDENTS.DB
-
-        9               /clients/client1/~dmtmp/PARADOX/
-                        STUDENTS.DB
-
-        9               /clients/client2/~dmtmp/PARADOX/
-                        STUDENTS.DB
-
-        9               /clients/client0/~dmtmp/PARADOX/
-                        STUDENTS.DB
-
-        9               /clients/client8/~dmtmp/PARADOX/
-                        STUDENTS.DB
-
-<a name="file-read" />
-###Viewing Highest File Read Calls
-
-You can view highest read calls on each brick. If brick name is not
-specified, then by default, list of 100 files will be displayed.
-
--   View highest file Read calls using the following command:
-
-    `# gluster volume top  read [brick ] [list-cnt ] `
-
-    For example, to view highest Read calls on brick server:/export of
-    test-volume:
-
-    `# gluster volume top  read brick  list-cnt `
-
-    `Brick:` server:/export/dir1
-
-                  ==========Read file stats========
-
-        read              filename
-        call count
-
-        116              /clients/client0/~dmtmp/SEED/LARGE.FIL
-
-        64               /clients/client0/~dmtmp/SEED/MEDIUM.FIL
-
-        54               /clients/client2/~dmtmp/SEED/LARGE.FIL
-
-        54               /clients/client6/~dmtmp/SEED/LARGE.FIL
-
-        54               /clients/client5/~dmtmp/SEED/LARGE.FIL
-
-        54               /clients/client0/~dmtmp/SEED/LARGE.FIL
-
-        54               /clients/client3/~dmtmp/SEED/LARGE.FIL
-
-        54               /clients/client4/~dmtmp/SEED/LARGE.FIL
-
-        54               /clients/client9/~dmtmp/SEED/LARGE.FIL
-
-        54               /clients/client8/~dmtmp/SEED/LARGE.FIL
-
-<a name="file-write" />
-###Viewing Highest File Write Calls
-
-You can view list of files which has highest file write calls on each
-brick. If brick name is not specified, then by default, list of 100
-files will be displayed.
-
--   View highest file Write calls using the following command:
-
-    `# gluster volume top  write [brick ] [list-cnt ] `
-
-    For example, to view highest Write calls on brick server:/export of
-    test-volume:
-
-    `# gluster volume top  write brick  list-cnt `
-
-    `Brick: server:/export/dir1 `
-
-                       ==========Write file stats========
-        write call count   filename
-
-        83                /clients/client0/~dmtmp/SEED/LARGE.FIL
-
-        59                /clients/client7/~dmtmp/SEED/LARGE.FIL
-
-        59                /clients/client1/~dmtmp/SEED/LARGE.FIL
-
-        59                /clients/client2/~dmtmp/SEED/LARGE.FIL
-
-        59                /clients/client0/~dmtmp/SEED/LARGE.FIL
-
-        59                /clients/client8/~dmtmp/SEED/LARGE.FIL
-
-        59                /clients/client5/~dmtmp/SEED/LARGE.FIL
-
-        59                /clients/client4/~dmtmp/SEED/LARGE.FIL
-
-        59                /clients/client6/~dmtmp/SEED/LARGE.FIL
-
-        59                /clients/client3/~dmtmp/SEED/LARGE.FIL
-
-<a name="open-dir" />
-###Viewing Highest Open Calls on Directories
-
-You can view list of files which has highest open calls on directories
-of each brick. If brick name is not specified, then the metrics of all
-the bricks belonging to that volume will be displayed.
-
--   View list of open calls on each directory using the following
-    command:
-
-    `# gluster volume top  opendir [brick ] [list-cnt ] `
-
-    For example, to view open calls on brick server:/export/ of
-    test-volume:
-
-    `# gluster volume top  opendir brick  list-cnt `
-
-    `Brick: server:/export/dir1 `
-
-                 ==========Directory open stats========
-
-        Opendir count     directory name
-
-        1001              /clients/client0/~dmtmp
-
-        454               /clients/client8/~dmtmp
-
-        454               /clients/client2/~dmtmp
-         
-        454               /clients/client6/~dmtmp
-
-        454               /clients/client5/~dmtmp
-
-        454               /clients/client9/~dmtmp
-
-        443               /clients/client0/~dmtmp/PARADOX
-
-        408               /clients/client1/~dmtmp
-
-        408               /clients/client7/~dmtmp
-
-        402               /clients/client4/~dmtmp
-
-<a name="read-dir" />
-###Viewing Highest Read Calls on Directory
-
-You can view list of files which has highest directory read calls on
-each brick. If brick name is not specified, then the metrics of all the
-bricks belonging to that volume will be displayed.
-
--   View list of highest directory read calls on each brick using the
-    following command:
-
-    `# gluster volume top  readdir [brick ] [list-cnt ] `
-
-    For example, to view highest directory read calls on brick
-    server:/export of test-volume:
-
-    `# gluster volume top  readdir brick  list-cnt `
-
-    `Brick: `
-
-        ==========Directory readdirp stats========
-
-        readdirp count           directory name
-
-        1996                    /clients/client0/~dmtmp
-
-        1083                    /clients/client0/~dmtmp/PARADOX
-
-        904                     /clients/client8/~dmtmp
-
-        904                     /clients/client2/~dmtmp
-
-        904                     /clients/client6/~dmtmp
-
-        904                     /clients/client5/~dmtmp
-
-        904                     /clients/client9/~dmtmp
-
-        812                     /clients/client1/~dmtmp
-
-        812                     /clients/client7/~dmtmp
-
-        800                     /clients/client4/~dmtmp
-
-<a name="read-perf" />
-###Viewing List of Read Performance on each Brick
-
-You can view the read throughput of files on each brick. If brick name
-is not specified, then the metrics of all the bricks belonging to that
-volume will be displayed. The output will be the read throughput.
-
-           ==========Read throughput file stats========
-
-    read         filename                         Time
-    through
-    put(MBp
-    s)
-     
-    2570.00    /clients/client0/~dmtmp/PWRPNT/      -2011-01-31
-               TRIDOTS.POT                      15:38:36.894610  
-    2570.00    /clients/client0/~dmtmp/PWRPNT/      -2011-01-31
-               PCBENCHM.PPT                     15:38:39.815310                                                             
-    2383.00    /clients/client2/~dmtmp/SEED/        -2011-01-31
-               MEDIUM.FIL                       15:52:53.631499
-                                                  
-    2340.00    /clients/client0/~dmtmp/SEED/        -2011-01-31
-               MEDIUM.FIL                       15:38:36.926198
-                                                  
-    2299.00   /clients/client0/~dmtmp/SEED/         -2011-01-31
-              LARGE.FIL                         15:38:36.930445
-                                                 
-    2259.00   /clients/client0/~dmtmp/PARADOX/      -2011-01-31
-              COURSES.X04                       15:38:40.549919
-
-    2221.00   /clients/client0/~dmtmp/PARADOX/      -2011-01-31
-              STUDENTS.VAL                      15:52:53.298766
-                                                  
-    2221.00   /clients/client3/~dmtmp/SEED/         -2011-01-31
-              COURSES.DB                        15:39:11.776780
-
-    2184.00   /clients/client3/~dmtmp/SEED/         -2011-01-31
-              MEDIUM.FIL                        15:39:10.251764
-                                                  
-    2184.00   /clients/client5/~dmtmp/WORD/         -2011-01-31
-              BASEMACH.DOC                      15:39:09.336572                                          
-
-This command will initiate a dd for the specified count and block size
-and measures the corresponding throughput.
-
--   View list of read performance on each brick using the following
-    command:
-
-    `# gluster volume top  read-perf [bs  count ] [brick ] [list-cnt ]`
-
-    For example, to view read performance on brick server:/export/ of
-    test-volume, 256 block size of count 1, and list count 10:
-
-    `# gluster volume top  read-perf bs 256 count 1 brick list-cnt `
-
-    `Brick: server:/export/dir1 256 bytes (256 B) copied, Throughput: 4.1 MB/s `
-
-               ==========Read throughput file stats========
-
-        read         filename                         Time
-        through
-        put(MBp
-        s)
-
-        2912.00   /clients/client0/~dmtmp/PWRPNT/    -2011-01-31
-                   TRIDOTS.POT                   15:38:36.896486
-                                                   
-        2570.00   /clients/client0/~dmtmp/PWRPNT/    -2011-01-31
-                   PCBENCHM.PPT                  15:38:39.815310
-                                                   
-        2383.00   /clients/client2/~dmtmp/SEED/      -2011-01-31
-                   MEDIUM.FIL                    15:52:53.631499
-                                                   
-        2340.00   /clients/client0/~dmtmp/SEED/      -2011-01-31
-                   MEDIUM.FIL                    15:38:36.926198
-
-        2299.00   /clients/client0/~dmtmp/SEED/      -2011-01-31
-                   LARGE.FIL                     15:38:36.930445
-                                                              
-        2259.00  /clients/client0/~dmtmp/PARADOX/    -2011-01-31
-                  COURSES.X04                    15:38:40.549919
-                                                   
-        2221.00  /clients/client9/~dmtmp/PARADOX/    -2011-01-31
-                  STUDENTS.VAL                   15:52:53.298766
-                                                   
-        2221.00  /clients/client8/~dmtmp/PARADOX/    -2011-01-31
-                 COURSES.DB                      15:39:11.776780
-                                                   
-        2184.00  /clients/client3/~dmtmp/SEED/       -2011-01-31
-                  MEDIUM.FIL                     15:39:10.251764
-                                                   
-        2184.00  /clients/client5/~dmtmp/WORD/       -2011-01-31
-                 BASEMACH.DOC                    15:39:09.336572
-                                                   
-<a name="write-perf" />
-###Viewing List of Write Performance on each Brick
-
-You can view list of write throughput of files on each brick. If brick
-name is not specified, then the metrics of all the bricks belonging to
-that volume will be displayed. The output will be the write throughput.
-
-This command will initiate a dd for the specified count and block size
-and measures the corresponding throughput. To view list of write
-performance on each brick:
-
--   View list of write performance on each brick using the following
-    command:
-
-    `# gluster volume top  write-perf [bs  count ] [brick ] [list-cnt ] `
-
-    For example, to view write performance on brick server:/export/ of
-    test-volume, 256 block size of count 1, and list count 10:
-
-    `# gluster volume top  write-perf bs 256 count 1 brick list-cnt `
-
-    `Brick`: server:/export/dir1
-
-    `256 bytes (256 B) copied, Throughput: 2.8 MB/s `
-
-               ==========Write throughput file stats========
-
-        write                filename                 Time
-        throughput
-        (MBps)
-         
-        1170.00    /clients/client0/~dmtmp/SEED/     -2011-01-31
-                   SMALL.FIL                     15:39:09.171494
-
-        1008.00    /clients/client6/~dmtmp/SEED/     -2011-01-31
-                   LARGE.FIL                      15:39:09.73189
-
-        949.00    /clients/client0/~dmtmp/SEED/      -2011-01-31
-                  MEDIUM.FIL                     15:38:36.927426
-
-        936.00   /clients/client0/~dmtmp/SEED/       -2011-01-31
-                 LARGE.FIL                        15:38:36.933177    
-        897.00   /clients/client5/~dmtmp/SEED/       -2011-01-31
-                 MEDIUM.FIL                       15:39:09.33628
-
-        897.00   /clients/client6/~dmtmp/SEED/       -2011-01-31
-                 MEDIUM.FIL                       15:39:09.27713
-
-        885.00   /clients/client0/~dmtmp/SEED/       -2011-01-31
-                  SMALL.FIL                      15:38:36.924271
-
-        528.00   /clients/client5/~dmtmp/SEED/       -2011-01-31
-                 LARGE.FIL                        15:39:09.81893
-
-        516.00   /clients/client6/~dmtmp/ACCESS/    -2011-01-31
-                 FASTENER.MDB                    15:39:01.797317
-
-##Displaying Volume Information
-
-You can display information about a specific volume, or all volumes, as
-needed.
-
--   Display information about a specific volume using the following
-    command:
-
-    `# gluster volume info ``VOLNAME`
-
-    For example, to display information about test-volume:
-
-        # gluster volume info test-volume
-        Volume Name: test-volume
-        Type: Distribute
-        Status: Created
-        Number of Bricks: 4
-        Bricks:
-        Brick1: server1:/exp1
-        Brick2: server2:/exp2
-        Brick3: server3:/exp3
-        Brick4: server4:/exp4
-
--   Display information about all volumes using the following command:
-
-    `# gluster volume info all`
-
-        # gluster volume info all
-
-        Volume Name: test-volume
-        Type: Distribute
-        Status: Created
-        Number of Bricks: 4
-        Bricks:
-        Brick1: server1:/exp1
-        Brick2: server2:/exp2
-        Brick3: server3:/exp3
-        Brick4: server4:/exp4
-
-        Volume Name: mirror
-        Type: Distributed-Replicate
-        Status: Started
-        Number of Bricks: 2 X 2 = 4
-        Bricks:
-        Brick1: server1:/brick1
-        Brick2: server2:/brick2
-        Brick3: server3:/brick3
-        Brick4: server4:/brick4
-
-        Volume Name: Vol
-        Type: Distribute
-        Status: Started
-        Number of Bricks: 1
-        Bricks:
-        Brick: server:/brick6
-
-##Performing Statedump on a Volume
-
-Statedump is a mechanism through which you can get details of all
-internal variables and state of the glusterfs process at the time of
-issuing the command.You can perform statedumps of the brick processes
-and nfs server process of a volume using the statedump command. The
-following options can be used to determine what information is to be
-dumped:
-
--   **mem** - Dumps the memory usage and memory pool details of the
-    bricks.
-
--   **iobuf** - Dumps iobuf details of the bricks.
-
--   **priv** - Dumps private information of loaded translators.
-
--   **callpool** - Dumps the pending calls of the volume.
-
--   **fd** - Dumps the open fd tables of the volume.
-
--   **inode** - Dumps the inode tables of the volume.
-
-**To display volume statedump**
-
--   Display statedump of a volume or NFS server using the following
-    command:
-
-    `# gluster volume statedump  [nfs] [all|mem|iobuf|callpool|priv|fd|inode]`
-
-    For example, to display statedump of test-volume:
-
-        # gluster volume statedump test-volume
-        Volume statedump successful
-
-    The statedump files are created on the brick servers in the` /tmp`
-    directory or in the directory set using `server.statedump-path`
-    volume option. The naming convention of the dump file is
-    `<brick-path>.<brick-pid>.dump`.
-
--   By defult, the output of the statedump is stored at
-    ` /tmp/<brickname.PID.dump>` file on that particular server. Change
-    the directory of the statedump file using the following command:
-
-    `# gluster volume set  server.statedump-path `
-
-    For example, to change the location of the statedump file of
-    test-volume:
-
-        # gluster volume set test-volume server.statedump-path /usr/local/var/log/glusterfs/dumps/
-        Set volume successful
-
-    You can view the changed path of the statedump file using the
-    following command:
-
-    `# gluster volume info `
-
-##Displaying Volume Status
-
-You can display the status information about a specific volume, brick or
-all volumes, as needed. Status information can be used to understand the
-current status of the brick, nfs processes, and overall file system.
-Status information can also be used to monitor and debug the volume
-information. You can view status of the volume along with the following
-details:
-
--   **detail** - Displays additional information about the bricks.
-
--   **clients** - Displays the list of clients connected to the volume.
-
--   **mem** - Displays the memory usage and memory pool details of the
-    bricks.
-
--   **inode** - Displays the inode tables of the volume.
-
--   **fd** - Displays the open fd (file descriptors) tables of the
-    volume.
-
--   **callpool** - Displays the pending calls of the volume.
-
-**To display volume status**
-
--   Display information about a specific volume using the following
-    command:
-
-    `# gluster volume status [all| []] [detail|clients|mem|inode|fd|callpool]`
-
-    For example, to display information about test-volume:
-
-        # gluster volume status test-volume
-        STATUS OF VOLUME: test-volume
-        BRICK                           PORT   ONLINE   PID
-        --------------------------------------------------------
-        arch:/export/1                  24009   Y       22445
-        --------------------------------------------------------
-        arch:/export/2                  24010   Y       22450
-
--   Display information about all volumes using the following command:
-
-    `# gluster volume status all`
-
-        # gluster volume status all
-        STATUS OF VOLUME: volume-test
-        BRICK                           PORT   ONLINE   PID
-        --------------------------------------------------------
-        arch:/export/4                  24010   Y       22455
-
-        STATUS OF VOLUME: test-volume
-        BRICK                           PORT   ONLINE   PID
-        --------------------------------------------------------
-        arch:/export/1                  24009   Y       22445
-        --------------------------------------------------------
-        arch:/export/2                  24010   Y       22450
-
--   Display additional information about the bricks using the following
-    command:
-
-    `# gluster volume status  detail`
-
-    For example, to display additional information about the bricks of
-    test-volume:
-
-        # gluster volume status test-volume details
-        STATUS OF VOLUME: test-volume
-        -------------------------------------------
-        Brick                : arch:/export/1      
-        Port                 : 24009              
-        Online               : Y                  
-        Pid                  : 16977              
-        File System          : rootfs              
-        Device               : rootfs              
-        Mount Options        : rw                  
-        Disk Space Free      : 13.8GB              
-        Total Disk Space     : 46.5GB              
-        Inode Size           : N/A                
-        Inode Count          : N/A                
-        Free Inodes          : N/A
-
-        Number of Bricks: 1
-        Bricks:
-        Brick: server:/brick6
-
--   Display the list of clients accessing the volumes using the
-    following command:
-
-    `# gluster volume status  clients`
-
-    For example, to display the list of clients connected to
-    test-volume:
-
-        # gluster volume status test-volume clients
-        Brick : arch:/export/1
-        Clients connected : 2
-        Hostname          Bytes Read   BytesWritten
-        --------          ---------    ------------
-        127.0.0.1:1013    776          676
-        127.0.0.1:1012    50440        51200
-
--   Display the memory usage and memory pool details of the bricks using
-    the following command:
-
-    `# gluster volume status  mem`
-
-    For example, to display the memory usage and memory pool details of
-    the bricks of test-volume:
-
-        Memory status for volume : test-volume
-        ----------------------------------------------
-        Brick : arch:/export/1
-        Mallinfo
-        --------
-        Arena    : 434176
-        Ordblks  : 2
-        Smblks   : 0
-        Hblks    : 12
-        Hblkhd   : 40861696
-        Usmblks  : 0
-        Fsmblks  : 0
-        Uordblks : 332416
-        Fordblks : 101760
-        Keepcost : 100400
-
-        Mempool Stats
-        -------------
-        Name                               HotCount ColdCount PaddedSizeof AllocCount MaxAlloc
-        ----                               -------- --------- ------------ ---------- --------
-        test-volume-server:fd_t                0     16384           92         57        5
-        test-volume-server:dentry_t           59       965           84         59       59
-        test-volume-server:inode_t            60       964          148         60       60
-        test-volume-server:rpcsvc_request_t    0       525         6372        351        2
-        glusterfs:struct saved_frame           0      4096          124          2        2
-        glusterfs:struct rpc_req               0      4096         2236          2        2
-        glusterfs:rpcsvc_request_t             1       524         6372          2        1
-        glusterfs:call_stub_t                  0      1024         1220        288        1
-        glusterfs:call_stack_t                 0      8192         2084        290        2
-        glusterfs:call_frame_t                 0     16384          172       1728        6
-
--   Display the inode tables of the volume using the following command:
-
-    `# gluster volume status  inode`
-
-    For example, to display the inode tables of the test-volume:
-
-        # gluster volume status test-volume inode
-        inode tables for volume test-volume
-        ----------------------------------------------
-        Brick : arch:/export/1
-        Active inodes:
-        GFID                                            Lookups            Ref   IA type
-        ----                                            -------            ---   -------
-        6f3fe173-e07a-4209-abb6-484091d75499                  1              9         2
-        370d35d7-657e-44dc-bac4-d6dd800ec3d3                  1              1         2
-
-        LRU inodes: 
-        GFID                                            Lookups            Ref   IA type
-        ----                                            -------            ---   -------
-        80f98abe-cdcf-4c1d-b917-ae564cf55763                  1              0         1
-        3a58973d-d549-4ea6-9977-9aa218f233de                  1              0         1
-        2ce0197d-87a9-451b-9094-9baa38121155                  1              0         2
-
--   Display the open fd tables of the volume using the following
-    command:
-
-    `# gluster volume status  fd`
-
-    For example, to display the open fd tables of the test-volume:
-
-        # gluster volume status test-volume fd
-
-        FD tables for volume test-volume
-        ----------------------------------------------
-        Brick : arch:/export/1
-        Connection 1:
-        RefCount = 0  MaxFDs = 128  FirstFree = 4
-        FD Entry            PID                 RefCount            Flags              
-        --------            ---                 --------            -----              
-        0                   26311               1                   2                  
-        1                   26310               3                   2                  
-        2                   26310               1                   2                  
-        3                   26311               3                   2                  
-         
-        Connection 2:
-        RefCount = 0  MaxFDs = 128  FirstFree = 0
-        No open fds
-         
-        Connection 3:
-        RefCount = 0  MaxFDs = 128  FirstFree = 0
-        No open fds
-
--   Display the pending calls of the volume using the following command:
-
-    `# gluster volume status  callpool`
-
-    Each call has a call stack containing call frames.
-
-    For example, to display the pending calls of test-volume:
-
-        # gluster volume status test-volume
-
-        Pending calls for volume test-volume
-        ----------------------------------------------
-        Brick : arch:/export/1
-        Pending calls: 2
-        Call Stack1
-         UID    : 0
-         GID    : 0
-         PID    : 26338
-         Unique : 192138
-         Frames : 7
-         Frame 1
-          Ref Count   = 1
-          Translator  = test-volume-server
-          Completed   = No
-         Frame 2
-          Ref Count   = 0
-          Translator  = test-volume-posix
-          Completed   = No
-          Parent      = test-volume-access-control
-          Wind From   = default_fsync
-          Wind To     = FIRST_CHILD(this)->fops->fsync
-         Frame 3
-          Ref Count   = 1
-          Translator  = test-volume-access-control
-          Completed   = No
-          Parent      = repl-locks
-          Wind From   = default_fsync
-          Wind To     = FIRST_CHILD(this)->fops->fsync
-         Frame 4
-          Ref Count   = 1
-          Translator  = test-volume-locks
-          Completed   = No
-          Parent      = test-volume-io-threads
-          Wind From   = iot_fsync_wrapper
-          Wind To     = FIRST_CHILD (this)->fops->fsync
-         Frame 5
-          Ref Count   = 1
-          Translator  = test-volume-io-threads
-          Completed   = No
-          Parent      = test-volume-marker
-          Wind From   = default_fsync
-          Wind To     = FIRST_CHILD(this)->fops->fsync
-         Frame 6
-          Ref Count   = 1
-          Translator  = test-volume-marker
-          Completed   = No
-          Parent      = /export/1
-          Wind From   = io_stats_fsync
-          Wind To     = FIRST_CHILD(this)->fops->fsync
-         Frame 7
-          Ref Count   = 1
-          Translator  = /export/1
-          Completed   = No
-          Parent      = test-volume-server
-          Wind From   = server_fsync_resume
-          Wind To     = bound_xl->fops->fsync
-
-
diff --git a/doc/admin-guide/en-US/markdown/admin_object_storage.md b/doc/admin-guide/en-US/markdown/admin_object_storage.md
deleted file mode 100644
index 71edab64536..00000000000
--- a/doc/admin-guide/en-US/markdown/admin_object_storage.md
+++ /dev/null
@@ -1,26 +0,0 @@
-# SwiftOnFile
-
-SwiftOnFile project enables GlusterFS volume to be used as backend for Openstack
-Swift - a distributed object store. This allows objects PUT over Swift's RESTful
-API to be accessed as files over filesystem interface and vice versa i.e files
-created over filesystem interface (NFS/FUSE/native) can be accessed as objects
-over Swift's RESTful API.
-
-SwiftOnFile project was formerly known as `gluster-swift` and also as `UFO
-(Unified File and Object)` before that. More information about SwiftOnFile can
-be found [here](https://github.com/swiftonfile/swiftonfile/blob/master/doc/markdown/quick_start_guide.md).
-There are differences in working of gluster-swift (now obsolete) and swiftonfile
-projects. The older gluster-swift code and relevant documentation can be found
-in [icehouse branch](https://github.com/swiftonfile/swiftonfile/tree/icehouse)
-of swiftonfile repo.
-
-## SwiftOnFile vs gluster-swift
-
-|                                                                                      Gluster-Swift                                                                                      |                                                                                               SwiftOnFile                                                                                               |
-|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
-| One GlusterFS volume maps to and stores only one Swift account. Mountpoint Hierarchy: `container/object`                                                                                | One GlusterFS volume or XFS partition can have multiple accounts. Mountpoint Hierarchy: `acc/container/object`                                                                                          |
-| Over-rides account server, container server and object server. We need to keep in sync with upstream Swift and often may need code changes or workarounds to support new Swift features | Implements only object-server. Very less need to catch-up to Swift as new features at proxy,container and account level would very likely be compatible with SwiftOnFile as it's just a storage policy. |
-| Does not use DBs for accounts and container.A container listing involves a filesystem crawl.A HEAD on account/container gives inaccurate or stale results without FS crawl.             | Uses Swift's DBs to store account and container information. An account or container listing does not involve FS crawl. Accurate info on HEAD to account/container – ability to support account quotas. |
-| GET on a container and account lists actual files in filesystem.                                                                                                                        | GET on a container and account only lists objects PUT over Swift. Files created over filesystem interface do not appear in container and object listings.                                               |
-| Standalone deployment required and does not integrate with existing Swift cluster.                                                                                                      | Integrates with any existing Swift deployment as a Storage Policy.                                                                                                                                      |
-
diff --git a/doc/admin-guide/en-US/markdown/admin_puppet.md b/doc/admin-guide/en-US/markdown/admin_puppet.md
deleted file mode 100644
index 103449be0e7..00000000000
--- a/doc/admin-guide/en-US/markdown/admin_puppet.md
+++ /dev/null
@@ -1,499 +0,0 @@
-#Puppet-Gluster
-<!---
-GlusterFS module by James
-Copyright (C) 2010-2013+ James Shubin
-Written by James Shubin <james@shubin.ca>
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU Affero General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU Affero General Public License for more details.
-
-You should have received a copy of the GNU Affero General Public License
-along with this program.  If not, see <http://www.gnu.org/licenses/>.
--->
-##A GlusterFS Puppet module by [James](https://ttboj.wordpress.com/)
-####Available from:
-####[https://github.com/purpleidea/puppet-gluster/](https://github.com/purpleidea/puppet-gluster/)
-
-####Also available from:
-####[https://forge.gluster.org/puppet-gluster/](https://forge.gluster.org/puppet-gluster/)
-
-####Table of Contents
-
-1. [Overview](#overview)
-2. [Module description - What the module does](#module-description)
-3. [Setup - Getting started with Puppet-Gluster](#setup)
-	* [What can Puppet-Gluster manage?](#what-can-puppet-gluster-manage)
-	* [Simple setup](#simple-setup)
-	* [Elastic setup](#elastic-setup)
-	* [Advanced setup](#advanced-setup)
-4. [Usage/FAQ - Notes on management and frequently asked questions](#usage-and-frequently-asked-questions)
-5. [Reference - Class and type reference](#reference)
-	* [gluster::simple](#glustersimple)
-	* [gluster::elastic](#glusterelastic)
-	* [gluster::server](#glusterserver)
-	* [gluster::host](#glusterhost)
-	* [gluster::brick](#glusterbrick)
-	* [gluster::volume](#glustervolume)
-	* [gluster::volume::property](#glustervolumeproperty)
-6. [Examples - Example configurations](#examples)
-7. [Limitations - Puppet versions, OS compatibility, etc...](#limitations)
-8. [Development - Background on module development](#development)
-9. [Author - Author and contact information](#author)
-
-##Overview
-
-The Puppet-Gluster module installs, configures, and manages a GlusterFS cluster.
-
-##Module Description
-
-This Puppet-Gluster module handles installation, configuration, and management
-of GlusterFS across all of the hosts in the cluster.
-
-##Setup
-
-###What can Puppet-Gluster manage?
-
-Puppet-Gluster is designed to be able to manage as much or as little of your
-GlusterFS cluster as you wish. All features are optional. If there is a feature
-that doesn't appear to be optional, and you believe it should be, please let me
-know. Having said that, it makes good sense to me to have Puppet-Gluster manage
-as much of your GlusterFS infrastructure as it can. At the moment, it cannot
-rack new servers, but I am accepting funding to explore this feature ;) At the
-moment it can manage:
-
-* GlusterFS packages (rpm)
-* GlusterFS configuration files (/var/lib/glusterd/)
-* GlusterFS host peering (gluster peer probe)
-* GlusterFS storage partitioning (fdisk)
-* GlusterFS storage formatting (mkfs)
-* GlusterFS brick creation (mkdir)
-* GlusterFS services (glusterd)
-* GlusterFS firewalling (whitelisting)
-* GlusterFS volume creation (gluster volume create)
-* GlusterFS volume state (started/stopped)
-* GlusterFS volume properties (gluster volume set)
-* And much more...
-
-###Simple setup
-
-include '::gluster::simple' is enough to get you up and running. When using the
-gluster::simple class, or with any other Puppet-Gluster configuration,
-identical definitions must be used on all hosts in the cluster. The simplest
-way to accomplish this is with a single shared puppet host definition like:
-
-```puppet
-node /^annex\d+$/ {        # annex{1,2,..N}
-        class { '::gluster::simple':
-        }
-}
-```
-
-If you wish to pass in different parameters, you can specify them in the class
-before you provision your hosts:
-
-```puppet
-class { '::gluster::simple':
-	replica => 2,
-	volume => ['volume1', 'volume2', 'volumeN'],
-}
-```
-
-###Elastic setup
-
-The gluster::elastic class is not yet available. Stay tuned!
-
-###Advanced setup
-
-Some system administrators may wish to manually itemize each of the required
-components for the Puppet-Gluster deployment. This happens automatically with
-the higher level modules, but may still be a desirable feature, particularly
-for non-elastic storage pools where the configuration isn't expected to change
-very often (if ever).
-
-To put together your cluster piece by piece, you must manually include and
-define each class and type that you wish to use. If there are certain aspects
-that you wish to manage yourself, you can omit them from your configuration.
-See the [reference](#reference) section below for the specifics. Here is one
-possible example:
-
-```puppet
-class { '::gluster::server':
-	shorewall => true,
-}
-
-gluster::host { 'annex1.example.com':
-	# use uuidgen to make these
-	uuid => '1f660ca2-2c78-4aa0-8f4d-21608218c69c',
-}
-
-# note that this is using a folder on your existing file system...
-# this can be useful for prototyping gluster using virtual machines
-# if this isn't a separate partition, remember that your root fs will
-# run out of space when your gluster volume does!
-gluster::brick { 'annex1.example.com:/data/gluster-storage1':
-	areyousure => true,
-}
-
-gluster::host { 'annex2.example.com':
-	# NOTE: specifying a host uuid is now optional!
-	# if you don't choose one, one will be assigned
-	#uuid => '2fbe6e2f-f6bc-4c2d-a301-62fa90c459f8',
-}
-
-gluster::brick { 'annex2.example.com:/data/gluster-storage2':
-	areyousure => true,
-}
-
-$brick_list = [
-	'annex1.example.com:/data/gluster-storage1',
-	'annex2.example.com:/data/gluster-storage2',
-]
-
-gluster::volume { 'examplevol':
-	replica => 2,
-	bricks => $brick_list,
-	start => undef,	# i'll start this myself
-}
-
-# namevar must be: <VOLNAME>#<KEY>
-gluster::volume::property { 'examplevol#auth.reject':
-	value => ['192.0.2.13', '198.51.100.42', '203.0.113.69'],
-}
-```
-
-##Usage and frequently asked questions
-
-All management should be done by manipulating the arguments on the appropriate
-Puppet-Gluster classes and types. Since certain manipulations are either not
-yet possible with Puppet-Gluster, or are not supported by GlusterFS, attempting
-to manipulate the Puppet configuration in an unsupported way will result in
-undefined behaviour, and possible even data loss, however this is unlikely.
-
-###How do I change the replica count?
-
-You must set this before volume creation. This is a limitation of GlusterFS.
-There are certain situations where you can change the replica count by adding
-a multiple of the existing brick count to get this desired effect. These cases
-are not yet supported by Puppet-Gluster. If you want to use Puppet-Gluster
-before and / or after this transition, you can do so, but you'll have to do the
-changes manually.
-
-###Do I need to use a virtual IP?
-
-Using a virtual IP (VIP) is strongly recommended as a distributed lock manager
-(DLM) and also to provide a highly-available (HA) IP address for your clients
-to connect to. For a more detailed explanation of the reasoning please see:
-
-[https://ttboj.wordpress.com/2012/08/23/how-to-avoid-cluster-race-conditions-or-how-to-implement-a-distributed-lock-manager-in-puppet/](https://ttboj.wordpress.com/2012/08/23/how-to-avoid-cluster-race-conditions-or-how-to-implement-a-distributed-lock-manager-in-puppet/)
-
-Remember that even if you're using a hosted solution (such as AWS) that doesn't
-provide an additional IP address, or you want to avoid using an additional IP,
-and you're okay not having full HA client mounting, you can use an unused
-private RFC1918 IP address as the DLM VIP. Remember that a layer 3 IP can
-co-exist on the same layer 2 network with the layer 3 network that is used by
-your cluster.
-
-###Is it possible to have Puppet-Gluster complete in a single run?
-
-No. This is a limitation of Puppet, and is related to how GlusterFS operates.
-For example, it is not reliably possible to predict which ports a particular
-GlusterFS volume will run on until after the volume is started. As a result,
-this module will initially whitelist connections from GlusterFS host IP
-addresses, and then further restrict this to only allow individual ports once
-this information is known. This is possible in conjunction with the
-[puppet-shorewall](https://github.com/purpleidea/puppet-shorewall) module.
-You should notice that each run should complete without error. If you do see an
-error, it means that either something is wrong with your system and / or
-configuration, or because there is a bug in Puppet-Gluster.
-
-###Can you integrate this with vagrant?
-
-Not until vagrant properly supports libvirt/KVM. I have no desire to use
-VirtualBox for fun.
-
-###Awesome work, but it's missing support for a feature and/or platform!
-
-Since this is an Open Source / Free Software project that I also give away for
-free (as in beer, free as in gratis, free as in libre), I'm unable to provide
-unlimited support. Please consider donating funds, hardware, virtual machines,
-and other resources. For specific needs, you could perhaps sponsor a feature!
-
-###You didn't answer my question, or I have a question!
-
-Contact me through my [technical blog](https://ttboj.wordpress.com/contact/)
-and I'll do my best to help. If you have a good question, please remind me to
-add my answer to this documentation!
-
-##Reference
-Please note that there are a number of undocumented options. For more
-information on these options, please view the source at:
-[https://github.com/purpleidea/puppet-gluster/](https://github.com/purpleidea/puppet-gluster/).
-If you feel that a well used option needs documenting here, please contact me.
-
-###Overview of classes and types
-
-* [gluster::simple](#glustersimple): Simple Puppet-Gluster deployment.
-* [gluster::elastic](#glusterelastic): Under construction.
-* [gluster::server](#glusterserver): Base class for server hosts.
-* [gluster::host](#glusterhost): Host type for each participating host.
-* [gluster::brick](#glusterbrick): Brick type for each defined brick, per host.
-* [gluster::volume](#glustervolume): Volume type for each defined volume.
-* [gluster::volume::property](#glustervolumeproperty): Manages properties for each volume.
-
-###gluster::simple
-This is gluster::simple. It should probably take care of 80% of all use cases.
-It is particularly useful for deploying quick test clusters. It uses a
-finite-state machine (FSM) to decide when the cluster has settled and volume
-creation can begin. For more information on the FSM in Puppet-Gluster see:
-[https://ttboj.wordpress.com/2013/09/28/finite-state-machines-in-puppet/](https://ttboj.wordpress.com/2013/09/28/finite-state-machines-in-puppet/)
-
-####`replica`
-The replica count. Can't be changed automatically after initial deployment.
-
-####`volume`
-The volume name or list of volume names to create.
-
-####`path`
-The valid brick path for each host. Defaults to local file system. If you need
-a different path per host, then Gluster::Simple will not meet your needs.
-
-####`vip`
-The virtual IP address to be used for the cluster distributed lock manager.
-
-####`shorewall`
-Boolean to specify whether puppet-shorewall integration should be used or not.
-
-###gluster::elastic
-Under construction.
-
-###gluster::server
-Main server class for the cluster. Must be included when building the GlusterFS
-cluster manually. Wrapper classes such as [gluster::simple](#glustersimple)
-include this automatically.
-
-####`vip`
-The virtual IP address to be used for the cluster distributed lock manager.
-
-####`shorewall`
-Boolean to specify whether puppet-shorewall integration should be used or not.
-
-###gluster::host
-Main host type for the cluster. Each host participating in the GlusterFS
-cluster must define this type on itself, and on every other host. As a result,
-this is not a singleton like the [gluster::server](#glusterserver) class.
-
-####`ip`
-Specify which IP address this host is using. This defaults to the
-_$::ipaddress_ variable. Be sure to set this manually if you're declaring this
-yourself on each host without using exported resources. If each host thinks the
-other hosts should have the same IP address as itself, then Puppet-Gluster and
-GlusterFS won't work correctly.
-
-####`uuid`
-Universally unique identifier (UUID) for the host. If empty, Puppet-Gluster
-will generate this automatically for the host. You can generate your own
-manually with _uuidgen_, and set them yourself. I found this particularly
-useful for testing, because I would pick easy to recognize UUID's like:
-_aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa_,
-_bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb_, and so on. If you set a UUID manually,
-and Puppet-Gluster has a chance to run, then it will remember your choice, and
-store it locally to be used again if you no longer specify the UUID. This is
-particularly useful for upgrading an existing un-managed GlusterFS installation
-to a Puppet-Gluster managed one, without changing any UUID's.
-
-###gluster::brick
-Main brick type for the cluster. Each brick is an individual storage segment to
-be used on a host. Each host must have at least one brick to participate in the
-cluster, but usually a host will have multiple bricks. A brick can be as simple
-as a file system folder, or it can be a separate file system. Please read the
-official GlusterFS documentation, if you aren't entirely comfortable with the
-concept of a brick.
-
-For most test clusters, and for experimentation, it is easiest to use a
-directory on the root file system. You can even use a _/tmp_ sub folder if you
-don't care about the persistence of your data. For more serious clusters, you
-might want to create separate file systems for your data. On self-hosted iron,
-it is not uncommon to create multiple RAID-6 drive pools, and to then create a
-separate file system per virtual drive. Each file system can then be used as a
-single brick.
-
-So that each volume in GlusterFS has the maximum ability to grow, without
-having to partition storage separately, the bricks in Puppet-Gluster are
-actually folders (on whatever backing store you wish) which then contain
-sub folders-- one for each volume. As a result, all the volumes on a given
-GlusterFS cluster can share the total available storage space. If you wish to
-limit the storage used by each volume, you can setup quotas. Alternatively, you
-can buy more hardware, and elastically grow your GlusterFS volumes, since the
-price per GB will be significantly less than any proprietary storage system.
-The one downside to this brick sharing, is that if you have chosen the brick
-per host count specifically to match your performance requirements, and
-each GlusterFS volume on the same cluster has drastically different brick per
-host performance requirements, then this won't suit your needs. I doubt that
-anyone actually has such requirements, but if you do insist on needing this
-compartmentalization, then you can probably use the Puppet-Gluster grouping
-feature to accomplish this goal. Please let me know about your use-case, and
-be warned that the grouping feature hasn't been extensively tested.
-
-To prove to you that I care about automation, this type offers the ability to
-automatically partition and format your file systems. This means you can plug
-in new iron, boot, provision and configure the entire system automatically.
-Regrettably, I don't have a lot of test hardware to routinely use this feature.
-If you'd like to donate some, I'd be happy to test this thoroughly. Having said
-that, I have used this feature, I consider it to be extremely safe, and it has
-never caused me to lose data. If you're uncertain, feel free to look at the
-code, or avoid using this feature entirely. If you think there's a way to make
-it even safer, then feel free to let me know.
-
-####`dev`
-Block device, such as _/dev/sdc_ or _/dev/disk/by-id/scsi-0123456789abcdef_. By
-default, Puppet-Gluster will assume you're using a folder to store the brick
-data, if you don't specify this parameter.
-
-####`fsuuid`
-File system UUID. This ensures we can distinctly identify a file system. You
-can set this to be used with automatic file system creation, or you can specify
-the file system UUID that you'd like to use.
-
-####`labeltype`
-Only _gpt_ is supported. Other options include _msdos_, but this has never been
-used because of it's size limitations.
-
-####`fstype`
-This should be _xfs_ or _ext4_. Using _xfs_ is recommended, but _ext4_ is also
-quite common. This only affects a file system that is getting created by this
-module. If you provision a new machine, with a root file system of _ext4_, and
-the brick you create is a root file system path, then this option does nothing.
-
-####`xfs_inode64`
-Set _inode64_ mount option when using the _xfs_ fstype. Choose _true_ to set.
-
-####`xfs_nobarrier`
-Set _nobarrier_ mount option when using the _xfs_ fstype. Choose _true_ to set.
-
-####`ro`
-Whether the file system should be mounted read only. For emergencies only.
-
-####`force`
-If _true_, this will overwrite any xfs file system it sees. This is useful for
-rebuilding GlusterFS repeatedly and wiping data. There are other safeties in
-place to stop this. In general, you probably don't ever want to touch this.
-
-####`areyousure`
-Do you want to allow Puppet-Gluster to do dangerous things? You have to set
-this to _true_ to allow Puppet-Gluster to _fdisk_ and _mkfs_ your file system.
-
-###gluster::volume
-Main volume type for the cluster. This is where a lot of the magic happens.
-Remember that changing some of these parameters after the volume has been
-created won't work, and you'll experience undefined behaviour. There could be
-FSM based error checking to verify that no changes occur, but it has been left
-out so that this code base can eventually support such changes, and so that the
-user can manually change a parameter if they know that it is safe to do so.
-
-####`bricks`
-List of bricks to use for this volume. If this is left at the default value of
-_true_, then this list is built automatically. The algorithm that determines
-this order does not support all possible situations, and most likely can't
-handle certain corner cases. It is possible to examine the FSM to view the
-selected brick order before it has a chance to create the volume. The volume
-creation script won't run until there is a stable brick list as seen by the FSM
-running on the host that has the DLM. If you specify this list of bricks
-manually, you must choose the order to match your desired volume layout. If you
-aren't sure about how to order the bricks, you should review the GlusterFS
-documentation first.
-
-####`transport`
-Only _tcp_ is supported. Possible values can include _rdma_, but this won't get
-any testing if I don't have access to infiniband hardware. Donations welcome.
-
-####`replica`
-Replica count. Usually you'll want to set this to _2_. Some users choose _3_.
-Other values are seldom seen. A value of _1_ can be used for simply testing a
-distributed setup, when you don't care about your data or high availability. A
-value greater than _4_ is probably wasteful and unnecessary. It might even
-cause performance issues if a synchronous write is waiting on a slow fourth
-server.
-
-####`stripe`
-Stripe count. Thoroughly unsupported and untested option. Not recommended for
-use by GlusterFS.
-
-####`ping`
-Do we want to include ping checks with _fping_?
-
-####`settle`
-Do we want to run settle checks?
-
-####`start`
-Requested state for the volume. Valid values include: _true_ (start), _false_
-(stop), or _undef_ (un-managed start/stop state).
-
-###gluster::volume::property
-Main volume property type for the cluster. This allows you to manage GlusterFS
-volume specific properties. There are a wide range of properties that volumes
-support. For the full list of properties, you should consult the GlusterFS
-documentation, or run the _gluster volume set help_ command. To set a property
-you must use the special name pattern of: _volume_#_key_. The value argument is
-used to set the associated value. It is smart enough to accept values in the
-most logical format for that specific property. Some properties aren't yet
-supported, so please report any problems you have with this functionality.
-Because this feature is an awesome way to _document as code_ the volume
-specific optimizations that you've made, make sure you use this feature even if
-you don't use all the others.
-
-####`value`
-The value to be used for this volume property.
-
-##Examples
-For example configurations, please consult the [examples/](https://github.com/purpleidea/puppet-gluster/tree/master/examples) directory in the git
-source repository. It is available from:
-
-[https://github.com/purpleidea/puppet-gluster/tree/master/examples](https://github.com/purpleidea/puppet-gluster/tree/master/examples)
-
-It is also available from:
-
-[https://forge.gluster.org/puppet-gluster/puppet-gluster/trees/master/examples](https://forge.gluster.org/puppet-gluster/puppet-gluster/trees/master/examples/)
-
-##Limitations
-
-This module has been tested against open source Puppet 3.2.4 and higher.
-
-The module has been tested on:
-
-* CentOS 6.4
-
-It will probably work without incident or without major modification on:
-
-* CentOS 5.x/6.x
-* RHEL 5.x/6.x
-
-It will most likely work with other Puppet versions and on other platforms, but
-testing under other conditions has been light due to lack of resources. It will
-most likely not work on Debian/Ubuntu systems without modification. I would
-really love to add support for these operating systems, but I do not have any
-test resources to do so. Please sponsor this if you'd like to see it happen.
-
-##Development
-
-This is my personal project that I work on in my free time.
-Donations of funding, hardware, virtual machines, and other resources are
-appreciated. Please contact me if you'd like to sponsor a feature, invite me to
-talk/teach or for consulting.
-
-You can follow along [on my technical blog](https://ttboj.wordpress.com/).
-
-##Author
-
-Copyright (C) 2010-2013+ James Shubin
-
-* [github](https://github.com/purpleidea/)
-* [@purpleidea](https://twitter.com/#!/purpleidea)
-* [https://ttboj.wordpress.com/](https://ttboj.wordpress.com/)
-
diff --git a/doc/admin-guide/en-US/markdown/admin_setting_volumes.md b/doc/admin-guide/en-US/markdown/admin_setting_volumes.md
deleted file mode 100644
index 395aa2c79a9..00000000000
--- a/doc/admin-guide/en-US/markdown/admin_setting_volumes.md
+++ /dev/null
@@ -1,580 +0,0 @@
-#Setting up GlusterFS Server Volumes
-
-A volume is a logical collection of bricks where each brick is an export
-directory on a server in the trusted storage pool. Most of the gluster
-management operations are performed on the volume.
-
-To create a new volume in your storage environment, specify the bricks
-that comprise the volume. After you have created a new volume, you must
-start it before attempting to mount it.
-
--   Volumes of the following types can be created in your storage
-    environment:
-
-    -   **Distributed** - Distributed volumes distributes files throughout
-        the bricks in the volume. You can use distributed volumes where
-        the requirement is to scale storage and the redundancy is either
-        not important or is provided by other hardware/software layers.
-
-    -   **Replicated** – Replicated volumes replicates files across bricks
-        in the volume. You can use replicated volumes in environments
-        where high-availability and high-reliability are critical.
-
-    -   **Striped** – Striped volumes stripes data across bricks in the
-        volume. For best results, you should use striped volumes only in
-        high concurrency environments accessing very large files.
-
-    -   **Distributed Striped** - Distributed striped volumes stripe data
-        across two or more nodes in the cluster. You should use
-        distributed striped volumes where the requirement is to scale
-        storage and in high concurrency environments accessing very
-        large files is critical.
-
-    -   **Distributed Replicated** - Distributed replicated volumes
-        distributes files across replicated bricks in the volume. You
-        can use distributed replicated volumes in environments where the
-        requirement is to scale storage and high-reliability is
-        critical. Distributed replicated volumes also offer improved
-        read performance in most environments.
-
-    -   **Distributed Striped Replicated** – Distributed striped replicated
-        volumes distributes striped data across replicated bricks in the
-        cluster. For best results, you should use distributed striped
-        replicated volumes in highly concurrent environments where
-        parallel access of very large files and performance is critical.
-        In this release, configuration of this volume type is supported
-        only for Map Reduce workloads.
-
-    -   **Striped Replicated** – Striped replicated volumes stripes data
-        across replicated bricks in the cluster. For best results, you
-        should use striped replicated volumes in highly concurrent
-        environments where there is parallel access of very large files
-        and performance is critical. In this release, configuration of
-        this volume type is supported only for Map Reduce workloads.
-
-    -   **Dispersed** - Dispersed volumes are based on erasure codes,
-        providing space-efficient protection against disk or server failures.
-        It stores an encoded fragment of the original file to each brick in
-        a way that only a subset of the fragments is needed to recover the
-        original file. The number of bricks that can be missing without
-        losing access to data is configured by the administrator on volume
-        creation time.
-
-    -   **Distributed Dispersed** - Distributed dispersed volumes distribute
-        files across dispersed subvolumes. This has the same advantages of
-        distribute replicate volumes, but using disperse to store the data
-        into the bricks.
-
-**To create a new volume**
-
--   Create a new volume :
-
-    `# gluster volume create [stripe  | replica | disperse] [transport tcp | rdma | tcp, rdma] `
-
-    For example, to create a volume called test-volume consisting of
-    server3:/exp3 and server4:/exp4:
-
-        # gluster volume create test-volume server3:/exp3 server4:/exp4
-        Creation of test-volume has been successful
-        Please start the volume to access data.
-
-##Creating Distributed Volumes
-
-In a distributed volumes files are spread randomly across the bricks in
-the volume. Use distributed volumes where you need to scale storage and
-redundancy is either not important or is provided by other
-hardware/software layers.
-
-> **Note**:
-> Disk/server failure in distributed volumes can result in a serious
-> loss of data because directory contents are spread randomly across the
-> bricks in the volume.
-
-![][]
-
-**To create a distributed volume**
-
-1.  Create a trusted storage pool.
-
-2.  Create the distributed volume:
-
-    `# gluster volume create  [transport tcp | rdma | tcp,rdma] `
-
-    For example, to create a distributed volume with four storage
-    servers using tcp:
-
-        # gluster volume create test-volume server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4
-        Creation of test-volume has been successful
-        Please start the volume to access data.
-
-    (Optional) You can display the volume information:
-
-        # gluster volume info
-        Volume Name: test-volume
-        Type: Distribute
-        Status: Created
-        Number of Bricks: 4
-        Transport-type: tcp
-        Bricks:
-        Brick1: server1:/exp1
-        Brick2: server2:/exp2
-        Brick3: server3:/exp3
-        Brick4: server4:/exp4
-
-    For example, to create a distributed volume with four storage
-    servers over InfiniBand:
-
-        # gluster volume create test-volume transport rdma server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4
-        Creation of test-volume has been successful
-        Please start the volume to access data.
-
-    If the transport type is not specified, *tcp* is used as the
-    default. You can also set additional options if required, such as
-    auth.allow or auth.reject.
-
-    > **Note**:
-    > Make sure you start your volumes before you try to mount them or
-    > else client operations after the mount will hang.
-
-##Creating Replicated Volumes
-
-Replicated volumes create copies of files across multiple bricks in the
-volume. You can use replicated volumes in environments where
-high-availability and high-reliability are critical.
-
-> **Note**:
-> The number of bricks should be equal to of the replica count for a
-> replicated volume. To protect against server and disk failures, it is
-> recommended that the bricks of the volume are from different servers.
-
-![][1]
-
-**To create a replicated volume**
-
-1.  Create a trusted storage pool.
-
-2.  Create the replicated volume:
-
-    `# gluster volume create  [replica ] [transport tcp | rdma tcp,rdma] `
-
-    For example, to create a replicated volume with two storage servers:
-
-        # gluster volume create test-volume replica 2 transport tcp server1:/exp1 server2:/exp2
-        Creation of test-volume has been successful
-        Please start the volume to access data.
-
-    If the transport type is not specified, *tcp* is used as the
-    default. You can also set additional options if required, such as
-    auth.allow or auth.reject.
-
-    > **Note**:
-
-    > - Make sure you start your volumes before you try to mount them or
-    > else client operations after the mount will hang.
-
-    > - GlusterFS will fail to create a replicate volume if more than one brick of a replica set is present on the same peer. For eg. four node replicated volume with a more that one brick of a replica set is present on the same peer.
-    > ```
-    # gluster volume create <volname> replica 4 server1:/brick1 server1:/brick2 server2:/brick3 server4:/brick4
-    volume create: <volname>: failed: Multiple bricks of a replicate volume are present on the same server. This setup is not optimal. Use 'force' at the end of the command if you want to override this behavior.```
-
-    >  Use the `force` option at the end of command if you want to create the volume in this case.
-
-##Creating Striped Volumes
-
-Striped volumes stripes data across bricks in the volume. For best
-results, you should use striped volumes only in high concurrency
-environments accessing very large files.
-
-> **Note**:
-> The number of bricks should be a equal to the stripe count for a
-> striped volume.
-
-![][2]
-
-**To create a striped volume**
-
-1.  Create a trusted storage pool.
-
-2.  Create the striped volume:
-
-    `# gluster volume create  [stripe ] [transport tcp | rdma | tcp,rdma] `
-
-    For example, to create a striped volume across two storage servers:
-
-        # gluster volume create test-volume stripe 2 transport tcp server1:/exp1 server2:/exp2
-        Creation of test-volume has been successful
-        Please start the volume to access data.
-
-    If the transport type is not specified, *tcp* is used as the
-    default. You can also set additional options if required, such as
-    auth.allow or auth.reject.
-
-    > **Note**:
-    > Make sure you start your volumes before you try to mount them or
-    > else client operations after the mount will hang.
-
-##Creating Distributed Striped Volumes
-
-Distributed striped volumes stripes files across two or more nodes in
-the cluster. For best results, you should use distributed striped
-volumes where the requirement is to scale storage and in high
-concurrency environments accessing very large files is critical.
-
-> **Note**:
-> The number of bricks should be a multiple of the stripe count for a
-> distributed striped volume.
-
-![][3]
-
-**To create a distributed striped volume**
-
-1.  Create a trusted storage pool.
-
-2.  Create the distributed striped volume:
-
-    `# gluster volume create  [stripe ] [transport tcp | rdma | tcp,rdma] `
-
-    For example, to create a distributed striped volume across eight
-    storage servers:
-
-        # gluster volume create test-volume stripe 4 transport tcp server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4 server5:/exp5 server6:/exp6 server7:/exp7 server8:/exp8
-        Creation of test-volume has been successful
-        Please start the volume to access data.
-
-    If the transport type is not specified, *tcp* is used as the
-    default. You can also set additional options if required, such as
-    auth.allow or auth.reject.
-
-    > **Note**:
-    > Make sure you start your volumes before you try to mount them or
-    > else client operations after the mount will hang.
-
-##Creating Distributed Replicated Volumes
-
-Distributes files across replicated bricks in the volume. You can use
-distributed replicated volumes in environments where the requirement is
-to scale storage and high-reliability is critical. Distributed
-replicated volumes also offer improved read performance in most
-environments.
-
-> **Note**:
-> The number of bricks should be a multiple of the replica count for a
-> distributed replicated volume. Also, the order in which bricks are
-> specified has a great effect on data protection. Each replica\_count
-> consecutive bricks in the list you give will form a replica set, with
-> all replica sets combined into a volume-wide distribute set. To make
-> sure that replica-set members are not placed on the same node, list
-> the first brick on every server, then the second brick on every server
-> in the same order, and so on.
-
-![][4]
-
-**To create a distributed replicated volume**
-
-1.  Create a trusted storage pool.
-
-2.  Create the distributed replicated volume:
-
-    `# gluster volume create  [replica ] [transport tcp | rdma | tcp,rdma] `
-
-    For example, four node distributed (replicated) volume with a
-    two-way mirror:
-
-        # gluster volume create test-volume replica 2 transport tcp server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4
-        Creation of test-volume has been successful
-        Please start the volume to access data.
-
-    For example, to create a six node distributed (replicated) volume
-    with a two-way mirror:
-
-        # gluster volume create test-volume replica 2 transport tcp server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4 server5:/exp5 server6:/exp6
-        Creation of test-volume has been successful
-        Please start the volume to access data.
-
-    If the transport type is not specified, *tcp* is used as the
-    default. You can also set additional options if required, such as
-    auth.allow or auth.reject.
-
-    > **Note**:
-    > - Make sure you start your volumes before you try to mount them or
-    > else client operations after the mount will hang.
-
-    > - GlusterFS will fail to create a distribute replicate volume if more than one brick of a replica set is present on the same peer. For eg. four node distribute (replicated) volume with a more than one brick of a replica set is present on the same peer.
-    > ```
-    # gluster volume create <volname> replica 2 server1:/brick1 server1:/brick2 server2:/brick3 server4:/brick4
-    volume create: <volname>: failed: Multiple bricks of a replicate volume are present on the same server. This setup is not optimal. Use 'force' at the end of the command if you want to override this behavior.```
-
-    >  Use the `force` option at the end of command if you want to create the volume in this case.
-
-
-##Creating Distributed Striped Replicated Volumes
-
-Distributed striped replicated volumes distributes striped data across
-replicated bricks in the cluster. For best results, you should use
-distributed striped replicated volumes in highly concurrent environments
-where parallel access of very large files and performance is critical.
-In this release, configuration of this volume type is supported only for
-Map Reduce workloads.
-
-> **Note**:
-> The number of bricks should be a multiples of number of stripe count
-> and replica count for a distributed striped replicated volume.
-
-**To create a distributed striped replicated volume**
-
-1.  Create a trusted storage pool.
-
-2.  Create a distributed striped replicated volume using the following
-    command:
-
-    `# gluster volume create  [stripe ] [replica ] [transport tcp | rdma | tcp,rdma] `
-
-    For example, to create a distributed replicated striped volume
-    across eight storage servers:
-
-        # gluster volume create test-volume stripe 2 replica 2 transport tcp server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4 server5:/exp5 server6:/exp6 server7:/exp7 server8:/exp8
-        Creation of test-volume has been successful
-        Please start the volume to access data.
-
-    If the transport type is not specified, *tcp* is used as the
-    default. You can also set additional options if required, such as
-    auth.allow or auth.reject.
-
-    > **Note**:
-    > - Make sure you start your volumes before you try to mount them or
-    > else client operations after the mount will hang.
-
-    > - GlusterFS will fail to create a distribute replicate volume if more than one brick of a replica set is present on the same peer. For eg. four node distribute (replicated) volume with a more than one brick of a replica set is present on the same peer.
-    > ```
-    # gluster volume create <volname> stripe 2 replica 2 server1:/brick1 server1:/brick2 server2:/brick3 server4:/brick4
-    volume create: <volname>: failed: Multiple bricks of a replicate volume are present on the same server. This setup is not optimal. Use 'force' at the end of the command if you want to override this behavior.```
-
-    >  Use the `force` option at the end of command if you want to create the volume in this case.
-
-##Creating Striped Replicated Volumes
-
-Striped replicated volumes stripes data across replicated bricks in the
-cluster. For best results, you should use striped replicated volumes in
-highly concurrent environments where there is parallel access of very
-large files and performance is critical. In this release, configuration
-of this volume type is supported only for Map Reduce workloads.
-
-> **Note**:
-> The number of bricks should be a multiple of the replicate count and
-> stripe count for a striped replicated volume.
-
-![][5]
-
-**To create a striped replicated volume**
-
-1.  Create a trusted storage pool consisting of the storage servers that
-    will comprise the volume.
-
-2.  Create a striped replicated volume :
-
-    `# gluster volume create  [stripe ] [replica ] [transport tcp | rdma | tcp,rdma] `
-
-    For example, to create a striped replicated volume across four
-    storage servers:
-
-        # gluster volume create test-volume stripe 2 replica 2 transport tcp server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4
-        Creation of test-volume has been successful
-        Please start the volume to access data.
-
-    To create a striped replicated volume across six storage servers:
-
-        # gluster volume create test-volume stripe 3 replica 2 transport tcp server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4 server5:/exp5 server6:/exp6
-        Creation of test-volume has been successful
-        Please start the volume to access data.
-
-    If the transport type is not specified, *tcp* is used as the
-    default. You can also set additional options if required, such as
-    auth.allow or auth.reject.
-
-    > **Note**:
-    > - Make sure you start your volumes before you try to mount them or
-    > else client operations after the mount will hang.
-
-    > - GlusterFS will fail to create a distribute replicate volume if more than one brick of a replica set is present on the same peer. For eg. four node distribute (replicated) volume with a more than one brick of replica set is present on the same peer.
-    > ```
-    # gluster volume create <volname> stripe 2 replica 2 server1:/brick1 server1:/brick2 server2:/brick3 server4:/brick4
-    volume create: <volname>: failed: Multiple bricks of a replicate volume are present on the same server. This setup is not optimal. Use 'force' at the end of the command if you want to override this behavior.```
-
-    >  Use the `force` option at the end of command if you want to create the volume in this case.
-
-##Creating Dispersed Volumes
-
-Dispersed volumes are based on erasure codes. It stripes the encoded data of
-files, with some redundancy addedd, across multiple bricks in the volume. You
-can use dispersed volumes to have a configurable level of reliability with a
-minimum space waste.
-
-**Redundancy**
-
-Each dispersed volume has a redundancy value defined when the volume is
-created. This value determines how many bricks can be lost without
-interrupting the operation of the volume. It also determines the amount of
-usable space of the volume using this formula:
-
-    <Usable size> = <Brick size> * (#Bricks - Redundancy)
-
-All bricks of a disperse set should have the same capacity otherwise, when
-the smaller brick becomes full, no additional data will be allowed in the
-disperse set.
-
-It's important to note that a configuration with 3 bricks and redundancy 1
-will have less usable space (66.7% of the total physical space) than a
-configuration with 10 bricks and redundancy 1 (90%). However the first one
-will be safer than the second one (roughly the probability of failure of
-the second configuration if more than 4.5 times bigger than the first one).
-
-For example, a dispersed volume composed by 6 bricks of 4TB and a redundancy
-of 2 will be completely operational even with two bricks inaccessible. However
-a third inaccessible brick will bring the volume down because it won't be
-possible to read or write to it. The usable space of the volume will be equal
-to 16TB.
-
-The implementation of erasure codes in GlusterFS limits the redundancy to a
-value smaller than #Bricks / 2 (or equivalently, redundancy * 2 < #Bricks).
-Having a redundancy equal to half of the number of bricks would be almost
-equivalent to a replica-2 volume, and probably a replicated volume will
-perform better in this case.
-
-**Optimal volumes**
-
-One of the worst things erasure codes have in terms of performance is the
-RMW (Read-Modify-Write) cycle. Erasure codes operate in blocks of a certain
-size and it cannot work with smaller ones. This means that if a user issues
-a write of a portion of a file that doesn't fill a full block, it needs to
-read the remaining portion from the current contents of the file, merge them,
-compute the updated encoded block and, finally, writing the resulting data.
-
-This adds latency, reducing performance when this happens. Some GlusterFS
-performance xlators can help to reduce or even eliminate this problem for
-some workloads, but it should be taken into account when using dispersed
-volumes for a specific use case.
-
-Current implementation of dispersed volumes use blocks of a size that depends
-on the number of bricks and redundancy: 512 * (#Bricks - redundancy) bytes.
-This value is also known as the stripe size.
-
-Using combinations of #Bricks/redundancy that give a power of two for the
-stripe size will make the disperse volume perform better in most workloads
-because it's more typical to write information in blocks that are multiple of
-two (for example databases, virtual machines and many applications).
-
-These combinations are considered *optimal*.
-
-For example, a configuration with 6 bricks and redundancy 2 will have a stripe
-size of 512 * (6 - 2) = 2048 bytes, so it's considered optimal. A configuration
-with 7 bricks and redundancy 2 would have a stripe size of 2560 bytes, needing
-a RMW cycle for many writes (of course this always depends on the use case).
-
-**To create a dispersed volume**
-
-1.  Create a trusted storage pool.
-
-2.  Create the dispersed volume:
-
-    `# gluster volume create [disperse [<count>]] [redundancy <count>] [transport tcp | rdma | tcp,rdma]`
-
-    A dispersed volume can be created by specifying the number of bricks in a
-    disperse set, by specifying the number of redundancy bricks, or both.
-
-    If *disperse* is not specified, or the _&lt;count&gt;_ is missing, the
-    entire volume will be treated as a single disperse set composed by all
-    bricks enumerated in the command line.
-
-    If *redundancy* is not specified, it is computed automatically to be the
-    optimal value. If this value does not exist, it's assumed to be '1' and a
-    warning message is shown:
-
-        # gluster volume create test-volume disperse 4 server{1..4}:/bricks/test-volume
-        There isn't an optimal redundancy value for this configuration. Do you want to create the volume with redundancy 1 ? (y/n)
-
-    In all cases where *redundancy* is automatically computed and it's not
-    equal to '1', a warning message is displayed:
-
-        # gluster volume create test-volume disperse 6 server{1..6}:/bricks/test-volume
-        The optimal redundancy for this configuration is 2. Do you want to create the volume with this value ? (y/n)
-
-    _redundancy_ must be greater than 0, and the total number of bricks must
-    be greater than 2 * _redundancy_. This means that a dispersed volume must
-    have a minimum of 3 bricks.
-
-    If the transport type is not specified, *tcp* is used as the default. You
-    can also set additional options if required, like in the other volume
-    types.
-
-    > **Note**:
-
-    > - Make sure you start your volumes before you try to mount them or
-    > else client operations after the mount will hang.
-
-    > - GlusterFS will fail to create a dispersed volume if more than one brick of a disperse set is present on the same peer.
-
-    > ```
-    # gluster volume create <volname> disperse 3 server1:/brick{1..3}
-    volume create: <volname>: failed: Multiple bricks of a replicate volume are present on the same server. This setup is not optimal.
-    Do you still want to continue creating the volume? (y/n)```
-
-    >  Use the `force` option at the end of command if you want to create the volume in this case.
-
-##Creating Distributed Dispersed Volumes
-
-Distributed dispersed volumes are the equivalent to distributed replicated
-volumes, but using dispersed subvolumes instead of replicated ones.
-
-**To create a distributed dispersed volume**
-
-1.  Create a trusted storage pool.
-
-2.  Create the distributed dispersed volume:
-
-    `# gluster volume create disperse <count> [redundancy <count>] [transport tcp | rdma tcp,rdma]`
-
-    To create a distributed dispersed volume, the *disperse* keyword and
-    &lt;count&gt; is mandatory, and the number of bricks specified in the
-    command line must must be a multiple of the disperse count.
-
-    *redundancy* is exactly the same as in the dispersed volume.
-
-    If the transport type is not specified, *tcp* is used as the default. You
-    can also set additional options if required, like in the other volume
-    types.
-
-    > **Note**:
-
-    > - Make sure you start your volumes before you try to mount them or
-    > else client operations after the mount will hang.
-
-    > - GlusterFS will fail to create a distributed dispersed volume if more than one brick of a disperse set is present on the same peer.
-
-    > ```
-    # gluster volume create <volname> disperse 3 server1:/brick{1..6}
-    volume create: <volname>: failed: Multiple bricks of a replicate volume are present on the same server. This setup is not optimal.
-    Do you still want to continue creating the volume? (y/n)```
-
-    > Use the `force` option at the end of command if you want to create the volume in this case.
-
-##Starting Volumes
-
-You must start your volumes before you try to mount them.
-
-**To start a volume**
-
--   Start a volume:
-
-    `# gluster volume start `
-
-    For example, to start test-volume:
-
-        # gluster volume start test-volume
-        Starting test-volume has been successful
-
-  []: ../images/Distributed_Volume.png
-  [1]: ../images/Replicated_Volume.png
-  [2]: ../images/Striped_Volume.png
-  [3]: ../images/Distributed_Striped_Volume.png
-  [4]: ../images/Distributed_Replicated_Volume.png
-  [5]: ../images/Striped_Replicated_Volume.png
diff --git a/doc/admin-guide/en-US/markdown/admin_settingup_clients.md b/doc/admin-guide/en-US/markdown/admin_settingup_clients.md
deleted file mode 100644
index 909eca5ae0a..00000000000
--- a/doc/admin-guide/en-US/markdown/admin_settingup_clients.md
+++ /dev/null
@@ -1,600 +0,0 @@
-#Accessing Data - Setting Up GlusterFS Client
-
-You can access gluster volumes in multiple ways. You can use Gluster
-Native Client method for high concurrency, performance and transparent
-failover in GNU/Linux clients. You can also use NFS v3 to access gluster
-volumes. Extensive testing has be done on GNU/Linux clients and NFS
-implementation in other operating system, such as FreeBSD, and Mac OS X,
-as well as Windows 7 (Professional and Up) and Windows Server 2003.
-Other NFS client implementations may work with gluster NFS server.
-
-You can use CIFS to access volumes when using Microsoft Windows as well
-as SAMBA clients. For this access method, Samba packages need to be
-present on the client side.
-
-##Gluster Native Client
-
-The Gluster Native Client is a FUSE-based client running in user space.
-Gluster Native Client is the recommended method for accessing volumes
-when high concurrency and high write performance is required.
-
-This section introduces the Gluster Native Client and explains how to
-install the software on client machines. This section also describes how
-to mount volumes on clients (both manually and automatically) and how to
-verify that the volume has mounted successfully.
-
-###Installing the Gluster Native Client
-
-Before you begin installing the Gluster Native Client, you need to
-verify that the FUSE module is loaded on the client and has access to
-the required modules as follows:
-
-1.  Add the FUSE loadable kernel module (LKM) to the Linux kernel:
-
-    `# modprobe fuse`
-
-2.  Verify that the FUSE module is loaded:
-
-    `# dmesg | grep -i fuse `
-    `fuse init (API version 7.13)`
-
-### Installing on Red Hat Package Manager (RPM) Distributions
-
-To install Gluster Native Client on RPM distribution-based systems
-
-1.  Install required prerequisites on the client using the following
-    command:
-
-    `$ sudo yum -y install openssh-server wget fuse fuse-libs openib libibverbs`
-
-2.  Ensure that TCP and UDP ports 24007 and 24008 are open on all
-    Gluster servers. Apart from these ports, you need to open one port
-    for each brick starting from port 49152 (instead of 24009 onwards as
-    with previous releases). The brick ports assignment scheme is now
-    compliant with IANA guidelines. For example: if you have
-    five bricks, you need to have ports 49152 to 49156 open.
-
-    You can use the following chains with iptables:
-
-    `$ sudo iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 24007:24008 -j ACCEPT `
-    `$ sudo iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 49152:49156 -j ACCEPT`
-
-    > **Note**
-    >
-    > If you already have iptable chains, make sure that the above
-    > ACCEPT rules precede the DROP rules. This can be achieved by
-    > providing a lower rule number than the DROP rule.
-
-3.  Download the latest glusterfs, glusterfs-fuse, and glusterfs-rdma
-    RPM files to each client. The glusterfs package contains the Gluster
-    Native Client. The glusterfs-fuse package contains the FUSE
-    translator required for mounting on client systems and the
-    glusterfs-rdma packages contain OpenFabrics verbs RDMA module for
-    Infiniband.
-
-    You can download the software at [GlusterFS download page][1].
-
-4.  Install Gluster Native Client on the client.
-
-    `$ sudo rpm -i glusterfs-3.3.0qa30-1.x86_64.rpm `
-    `$ sudo rpm -i glusterfs-fuse-3.3.0qa30-1.x86_64.rpm `
-    `$ sudo rpm -i glusterfs-rdma-3.3.0qa30-1.x86_64.rpm`
-
-    > **Note**
-    >
-    > The RDMA module is only required when using Infiniband.
-
-### Installing on Debian-based Distributions
-
-To install Gluster Native Client on Debian-based distributions
-
-1.  Install OpenSSH Server on each client using the following command:
-
-    `$ sudo apt-get install openssh-server vim wget`
-
-2.  Download the latest GlusterFS .deb file and checksum to each client.
-
-    You can download the software at [GlusterFS download page][1].
-
-3.  For each .deb file, get the checksum (using the following command)
-    and compare it against the checksum for that file in the md5sum
-    file.
-
-    `$ md5sum GlusterFS_DEB_file.deb `
-
-    The md5sum of the packages is available at: [GlusterFS download page][2]
-
-4.  Uninstall GlusterFS v3.1 (or an earlier version) from the client
-    using the following command:
-
-    `$ sudo dpkg -r glusterfs `
-
-    (Optional) Run `$ sudo dpkg -purge glusterfs `to purge the
-    configuration files.
-
-5.  Install Gluster Native Client on the client using the following
-    command:
-
-    `$ sudo dpkg -i GlusterFS_DEB_file `
-
-    For example:
-
-    `$ sudo dpkg -i glusterfs-3.3.x.deb `
-
-6.  Ensure that TCP and UDP ports 24007 and 24008 are open on all
-    Gluster servers. Apart from these ports, you need to open one port
-    for each brick starting from port 49152 (instead of 24009 onwards as
-    with previous releases). The brick ports assignment scheme is now
-    compliant with IANA guidelines. For example: if you have
-    five bricks, you need to have ports 49152 to 49156 open.
-
-    You can use the following chains with iptables:
-
-    `$ sudo iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 24007:24008 -j ACCEPT `
-    `$ sudo iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 49152:49156 -j ACCEPT`
-
-    > **Note**
-    >
-    > If you already have iptable chains, make sure that the above
-    > ACCEPT rules precede the DROP rules. This can be achieved by
-    > providing a lower rule number than the DROP rule.
-
-### Performing a Source Installation
-
-To build and install Gluster Native Client from the source code
-
-1.  Create a new directory using the following commands:
-
-    `# mkdir glusterfs `
-    `# cd glusterfs`
-
-2.  Download the source code.
-
-    You can download the source at [][1].
-
-3.  Extract the source code using the following command:
-
-    `# tar -xvzf SOURCE-FILE `
-
-4.  Run the configuration utility using the following command:
-
-    `# ./configure `
-
-        GlusterFS configure summary
-        ===========================
-        FUSE client : yes
-        Infiniband verbs : yes
-        epoll IO multiplex : yes
-        argp-standalone : no
-        fusermount : no
-        readline : yes
-
-    The configuration summary shows the components that will be built
-    with Gluster Native Client.
-
-5.  Build the Gluster Native Client software using the following
-    commands:
-
-    `# make `
-    `# make install`
-
-6.  Verify that the correct version of Gluster Native Client is
-    installed, using the following command:
-
-    `# glusterfs –-version`
-
-##Mounting Volumes
-
-After installing the Gluster Native Client, you need to mount Gluster
-volumes to access data. There are two methods you can choose:
-
--   [Manually Mounting Volumes](#manual-mount)
--   [Automatically Mounting Volumes](#auto-mount)
-
-> **Note**
->
-> Server names selected during creation of Volumes should be resolvable
-> in the client machine. You can use appropriate /etc/hosts entries or
-> DNS server to resolve server names to IP addresses.
-
-<a name="manual-mount" />
-### Manually Mounting Volumes
-
--   To mount a volume, use the following command:
-
-    `# mount -t glusterfs HOSTNAME-OR-IPADDRESS:/VOLNAME MOUNTDIR`
-
-    For example:
-
-    `# mount -t glusterfs server1:/test-volume /mnt/glusterfs`
-
-    > **Note**
-    >
-    > The server specified in the mount command is only used to fetch
-    > the gluster configuration volfile describing the volume name.
-    > Subsequently, the client will communicate directly with the
-    > servers mentioned in the volfile (which might not even include the
-    > one used for mount).
-    >
-    > If you see a usage message like "Usage: mount.glusterfs", mount
-    > usually requires you to create a directory to be used as the mount
-    > point. Run "mkdir /mnt/glusterfs" before you attempt to run the
-    > mount command listed above.
-
-**Mounting Options**
-
-You can specify the following options when using the
-`mount -t glusterfs` command. Note that you need to separate all options
-with commas.
-
-backupvolfile-server=server-name
-
-volfile-max-fetch-attempts=number of attempts
-
-log-level=loglevel
-
-log-file=logfile
-
-transport=transport-type
-
-direct-io-mode=[enable|disable]
-
-use-readdirp=[yes|no]
-
-For example:
-
-`# mount -t glusterfs -o backupvolfile-server=volfile_server2,use-readdirp=no,volfile-max-fetch-attempts=2,log-level=WARNING,log-file=/var/log/gluster.log server1:/test-volume /mnt/glusterfs`
-
-If `backupvolfile-server` option is added while mounting fuse client,
-when the first volfile server fails, then the server specified in
-`backupvolfile-server` option is used as volfile server to mount the
-client.
-
-In `volfile-max-fetch-attempts=X` option, specify the number of
-attempts to fetch volume files while mounting a volume. This option is
-useful when you mount a server with multiple IP addresses or when
-round-robin DNS is configured for the server-name..
-
-If `use-readdirp` is set to ON, it forces the use of readdirp
-mode in fuse kernel module
-
-<a name="auto-mount" />
-### Automatically Mounting Volumes
-
-You can configure your system to automatically mount the Gluster volume
-each time your system starts.
-
-The server specified in the mount command is only used to fetch the
-gluster configuration volfile describing the volume name. Subsequently,
-the client will communicate directly with the servers mentioned in the
-volfile (which might not even include the one used for mount).
-
--   To mount a volume, edit the /etc/fstab file and add the following
-    line:
-
-    `HOSTNAME-OR-IPADDRESS:/VOLNAME MOUNTDIR glusterfs defaults,_netdev 0 0 `
-
-    For example:
-
-    `server1:/test-volume /mnt/glusterfs glusterfs defaults,_netdev 0 0`
-
-**Mounting Options**
-
-You can specify the following options when updating the /etc/fstab file.
-Note that you need to separate all options with commas.
-
-log-level=loglevel
-
-log-file=logfile
-
-transport=transport-type
-
-direct-io-mode=[enable|disable]
-
-use-readdirp=no
-
-For example:
-
-`HOSTNAME-OR-IPADDRESS:/VOLNAME MOUNTDIR glusterfs defaults,_netdev,log-level=WARNING,log-file=/var/log/gluster.log 0 0 `
-
-### Testing Mounted Volumes
-
-To test mounted volumes
-
--   Use the following command:
-
-    `# mount `
-
-    If the gluster volume was successfully mounted, the output of the
-    mount command on the client will be similar to this example:
-
-    `server1:/test-volume on /mnt/glusterfs type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072`
-
--   Use the following command:
-
-    `# df`
-
-    The output of df command on the client will display the aggregated
-    storage space from all the bricks in a volume similar to this
-    example:
-
-    `# df -h /mnt/glusterfs Filesystem Size Used Avail Use% Mounted on server1:/test-volume 28T 22T 5.4T 82% /mnt/glusterfs`
-
--   Change to the directory and list the contents by entering the
-    following:
-
-    `# cd MOUNTDIR `
-    `# ls`
-
--   For example,
-
-    `# cd /mnt/glusterfs `
-    `# ls`
-
-#NFS
-
-You can use NFS v3 to access to gluster volumes. Extensive testing has
-be done on GNU/Linux clients and NFS implementation in other operating
-system, such as FreeBSD, and Mac OS X, as well as Windows 7
-(Professional and Up), Windows Server 2003, and others, may work with
-gluster NFS server implementation.
-
-GlusterFS now includes network lock manager (NLM) v4. NLM enables
-applications on NFSv3 clients to do record locking on files on NFS
-server. It is started automatically whenever the NFS server is run.
-
-You must install nfs-common package on both servers and clients (only
-for Debian-based) distribution.
-
-This section describes how to use NFS to mount Gluster volumes (both
-manually and automatically) and how to verify that the volume has been
-mounted successfully.
-
-##Using NFS to Mount Volumes
---------------------------
-
-You can use either of the following methods to mount Gluster volumes:
-
--   [Manually Mounting Volumes Using NFS](#manual-nfs)
--   [Automatically Mounting Volumes Using NFS](#auto-nfs)
-
-**Prerequisite**: Install nfs-common package on both servers and clients
-(only for Debian-based distribution), using the following command:
-
-`$ sudo aptitude install nfs-common `
-
-<a name="manual-nfs" />
-### Manually Mounting Volumes Using NFS
-
-**To manually mount a Gluster volume using NFS**
-
--   To mount a volume, use the following command:
-
-    `# mount -t nfs -o vers=3 HOSTNAME-OR-IPADDRESS:/VOLNAME MOUNTDIR`
-
-    For example:
-
-    `# mount -t nfs -o vers=3 server1:/test-volume /mnt/glusterfs`
-
-    > **Note**
-    >
-    > Gluster NFS server does not support UDP. If the NFS client you are
-    > using defaults to connecting using UDP, the following message
-    > appears:
-    >
-    > `requested NFS version or transport protocol is not supported`.
-
-    **To connect using TCP**
-
--   Add the following option to the mount command:
-
-    `-o mountproto=tcp `
-
-    For example:
-
-    `# mount -o mountproto=tcp -t nfs server1:/test-volume /mnt/glusterfs`
-
-**To mount Gluster NFS server from a Solaris client**
-
--   Use the following command:
-
-    `# mount -o proto=tcp,vers=3 nfs://HOSTNAME-OR-IPADDRESS:38467/VOLNAME MOUNTDIR`
-
-    For example:
-
-    ` # mount -o proto=tcp,vers=3 nfs://server1:38467/test-volume /mnt/glusterfs`
-
-<a name="auto-nfs" />
-### Automatically Mounting Volumes Using NFS
-
-You can configure your system to automatically mount Gluster volumes
-using NFS each time the system starts.
-
-**To automatically mount a Gluster volume using NFS**
-
--   To mount a volume, edit the /etc/fstab file and add the following
-    line:
-
-    `HOSTNAME-OR-IPADDRESS:/VOLNAME MOUNTDIR nfs defaults,_netdev,vers=3 0 0`
-
-    For example,
-
-    `server1:/test-volume /mnt/glusterfs nfs defaults,_netdev,vers=3 0 0`
-
-    > **Note**
-    >
-    > Gluster NFS server does not support UDP. If the NFS client you are
-    > using defaults to connecting using UDP, the following message
-    > appears:
-    >
-    > `requested NFS version or transport protocol is not supported.`
-
-    To connect using TCP
-
--   Add the following entry in /etc/fstab file :
-
-    `HOSTNAME-OR-IPADDRESS:/VOLNAME MOUNTDIR nfs defaults,_netdev,mountproto=tcp 0 0`
-
-    For example,
-
-    `server1:/test-volume /mnt/glusterfs nfs defaults,_netdev,mountproto=tcp 0 0`
-
-**To automount NFS mounts**
-
-Gluster supports \*nix standard method of automounting NFS mounts.
-Update the /etc/auto.master and /etc/auto.misc and restart the autofs
-service. After that, whenever a user or process attempts to access the
-directory it will be mounted in the background.
-
-### Testing Volumes Mounted Using NFS
-
-You can confirm that Gluster directories are mounting successfully.
-
-**To test mounted volumes**
-
--   Use the mount command by entering the following:
-
-    `# mount`
-
-    For example, the output of the mount command on the client will
-    display an entry like the following:
-
-    `server1:/test-volume on /mnt/glusterfs type nfs (rw,vers=3,addr=server1)`
-
--   Use the df command by entering the following:
-
-    `# df`
-
-    For example, the output of df command on the client will display the
-    aggregated storage space from all the bricks in a volume.
-
-        # df -h /mnt/glusterfs 
-        Filesystem              Size Used Avail Use% Mounted on 
-        server1:/test-volume    28T  22T  5.4T  82%  /mnt/glusterfs
-
--   Change to the directory and list the contents by entering the
-    following:
-
-    `# cd MOUNTDIR`
-    `# ls`
-
-#CIFS
-
-You can use CIFS to access to volumes when using Microsoft Windows as
-well as SAMBA clients. For this access method, Samba packages need to be
-present on the client side. You can export glusterfs mount point as the
-samba export, and then mount it using CIFS protocol.
-
-This section describes how to mount CIFS shares on Microsoft
-Windows-based clients (both manually and automatically) and how to
-verify that the volume has mounted successfully.
-
-> **Note**
->
-> CIFS access using the Mac OS X Finder is not supported, however, you
-> can use the Mac OS X command line to access Gluster volumes using
-> CIFS.
-
-##Using CIFS to Mount Volumes
-
-You can use either of the following methods to mount Gluster volumes:
-
--   [Exporting Gluster Volumes Through Samba](#export-samba)
--   [Manually Mounting Volumes Using CIFS](#cifs-manual)
--   [Automatically Mounting Volumes Using CIFS](#cifs-auto)
-
-You can also use Samba for exporting Gluster Volumes through CIFS
-protocol.
-
-<a name="export-samba" />
-### Exporting Gluster Volumes Through Samba
-
-We recommend you to use Samba for exporting Gluster volumes through the
-CIFS protocol.
-
-**To export volumes through CIFS protocol**
-
-1.  Mount a Gluster volume.
-
-2.  Setup Samba configuration to export the mount point of the Gluster
-    volume.
-
-    For example, if a Gluster volume is mounted on /mnt/gluster, you
-    must edit smb.conf file to enable exporting this through CIFS. Open
-    smb.conf file in an editor and add the following lines for a simple
-    configuration:
-
-    [glustertest]
-
-    comment = For testing a Gluster volume exported through CIFS
-
-    path = /mnt/glusterfs
-
-    read only = no
-
-    guest ok = yes
-
-Save the changes and start the smb service using your systems init
-scripts (/etc/init.d/smb [re]start).
-
-> **Note**
->
-> To be able mount from any server in the trusted storage pool, you must
-> repeat these steps on each Gluster node. For more advanced
-> configurations, see Samba documentation.
-
-<a name="cifs-manual" />
-### Manually Mounting Volumes Using CIFS
-
-You can manually mount Gluster volumes using CIFS on Microsoft
-Windows-based client machines.
-
-**To manually mount a Gluster volume using CIFS**
-
-1.  Using Windows Explorer, choose **Tools \> Map Network Drive…** from
-    the menu. The **Map Network Drive**window appears.
-
-2.  Choose the drive letter using the **Drive** drop-down list.
-
-3.  Click **Browse**, select the volume to map to the network drive, and
-    click **OK**.
-
-4.  Click **Finish.**
-
-The network drive (mapped to the volume) appears in the Computer window.
-
-Alternatively, to manually mount a Gluster volume using CIFS by going to 
-**Start \> Run** and entering Network path manually.
-
-<a name="cifs-auto" />
-### Automatically Mounting Volumes Using CIFS
-
-You can configure your system to automatically mount Gluster volumes
-using CIFS on Microsoft Windows-based clients each time the system
-starts.
-
-**To automatically mount a Gluster volume using CIFS**
-
-The network drive (mapped to the volume) appears in the Computer window
-and is reconnected each time the system starts.
-
-1.  Using Windows Explorer, choose **Tools \> Map Network Drive…** from
-    the menu. The **Map Network Drive**window appears.
-
-2.  Choose the drive letter using the **Drive** drop-down list.
-
-3.  Click **Browse**, select the volume to map to the network drive, and
-    click **OK**.
-
-4.  Click the **Reconnect** at logon checkbox.
-
-5.  Click **Finish.**
-
-### Testing Volumes Mounted Using CIFS
-
-You can confirm that Gluster directories are mounting successfully by
-navigating to the directory using Windows Explorer.
-
-  []: http://bits.gluster.com/gluster/glusterfs/3.3.0qa30/x86_64/
-  [1]: http://www.gluster.org/download/
-  [2]: http://download.gluster.com/pub/gluster/glusterfs
diff --git a/doc/admin-guide/en-US/markdown/admin_ssl.md b/doc/admin-guide/en-US/markdown/admin_ssl.md
deleted file mode 100644
index 4522bcedf88..00000000000
--- a/doc/admin-guide/en-US/markdown/admin_ssl.md
+++ /dev/null
@@ -1,128 +0,0 @@
-# Setting up GlusterFS with SSL/TLS
-
-GlusterFS allows its communication to be secured using the [Transport Layer
-Security][tls] standard (which supersedes Secure Sockets Layer), using the
-[OpenSSL][ossl] library.  Setting this up requires a basic working knowledge of
-some SSL/TLS concepts, which can only be briefly summarized here.
-
- * "Authentication" is the process of one entity (e.g. a machine, process, or
-   person) proving its identity to a second entity.
-
- * "Authorization" is the process of checking whether an entity has permission
-   to perform an action.
-
- * TLS provides authentication and encryption.  It does not provide
-   authorization, though GlusterFS can use TLS-authenticated identities to
-   authorize client connections to bricks/volumes.
-
- * An entity X which must authenticate to a second entity Y does so by sharing
-   with Y a *certificate*, which contains information sufficient to prove X's
-   identity.  X's proof of identity also requires possession of a *private key*
-   which matches its certificate, but this key is never seen by Y or anyone
-   else.  Because the certificate is already public, anyone who has the key can
-   claim that identity.
-
- * Each certificate contains the identity of its principal (owner) along with
-   the identity of a *certifying authority* or CA who can verify the integrity
-   of the certificate's contents.  The principal and CA can be the same (a
-   "self-signed certificate").  If they are different, the CA must *sign* the
-   certificate by appending information derived from both the certificate
-   contents and the CA's own private key.
-
-  * Certificate-signing relationships can extend through multiple levels.  For
-    example, a company X could sign another company Y's certificate, which could
-    then be used to sign a third certificate Z for a specific user or purpose.
-    Anyone who trusts X (and is willing to extend that trust through a
-    *certificate depth* of two or more) would therefore be able to authenticate
-    Y and Z as well.
-
-  * Any entity willing to accept other entities' authentication attempts must
-    have some sort of database seeded with the certificates that already accept.
-
-In GlusterFS's case, a client or server X uses the following files to contain
-TLS-related information:
-
-  * /etc/ssl/glusterfs.pem	X's own certificate
-
-  * /etc/ssl/glusterfs.key	X's private key
-
-  * /etc/ssl/glusterfs.ca	concatenation of *others'* certificates
-
-GlusterFS always performs *mutual authentication*, though clients do not
-currently do anything with the authenticated server identity.  Thus, if client X
-wants to communicate with server Y, then X's certificate (or that of a signer)
-must be in Y's CA file, and vice versa.
-
-For all uses of TLS in GlusterFS, if one side of a connection is configured to
-use TLS then the other side must use it as well.  There is no automatic fallback
-to non-TLS communication, or allowance for concurrent TLS and non-TLS access to
-the same resource, because either would be insecure.  Instead, any such "mixed
-mode" connections will be rejected by the TLS-using side, sacrificing
-availability to maintain security.
-
-## Enabling TLS on the I/O Path
-
-To enable authentication and encryption between clients and brick servers, two
-options must be set:
-
-	gluster volume set MYVOLUME client.ssl on
-	gluster volume set MYVOLUME server.ssl on
-
-Note that the above options affect only the GlusterFS native protocol.  Foreign
-protocols such as NFS, SMB, or Swift will not be affected.
-
-## Using TLS Identities for Authorization
-
-Once TLS has been enabled on the I/O path, TLS identities can be used instead of
-IP addresses or plain usernames to control access to specific volumes.  For
-example:
-
-	gluster volume set MYVOLUME auth.ssl-allow Zaphod
-
-Here, we're allowing the TLS-authenticated identity "Zaphod" to access MYVOLUME.
-This is intentionally identical to the existing "auth.allow" option, except that
-the name is taken from a TLS certificate instead of a command-line string.  Note
-that infelicities in the gluster CLI preclude using names that include spaces,
-which would otherwise be allowed.
-
-## Enabling TLS on the Management Path
-
-Management-daemon traffic is not controlled by an option.  Instead, it is
-controlled by the presence of a file on each machine:
-
-	/var/lib/glusterd/secure-access
-
-Creating this file will cause glusterd connections made from that machine to use
-TLS.  Note that even clients must do this to communicate with a remote glusterd
-while mounting, but not thereafter.
-
-## Additional Options
-
-The GlusterFS TLS implementation supports two additional options related to TLS
-internals.
-
-The first option allows the user to set the certificate depth, as mentioned
-above.
-
-	gluster volume set MYVOLUME ssl.cert-depth 2
-
-Here, we're setting our certificate depth to two, as in the introductory
-example.  By default this value is zero, meaning that only certificates which
-are directly specified in the local CA file will be accepted (i.e. no signed
-certificates at all).
-
-The second option allows the user to specify the set of allowed TLS ciphers.
-
-	gluster volume set MYVOLUME ssl.cipher-list HIGH:!SSLv2
-
-Cipher lists are negotiated between the two parties to a TLS connection, so
-that both sides' security needs are satisfied.  In this example, we're setting
-the initial cipher list to HIGH, representing ciphers that the cryptography
-community still believes to be unbroken.  We are also explicitly disallowing
-ciphers specific to SSL version 2.  The default is based on this example but
-also excludes CBC-based cipher modes to provide extra mitigation against the
-[POODLE][poo] attack.
-
-[tls]: http://tools.ietf.org/html/rfc5246
-[ossl]: https://www.openssl.org/
-[poo]: http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-3566
diff --git a/doc/admin-guide/en-US/markdown/admin_start_stop_daemon.md b/doc/admin-guide/en-US/markdown/admin_start_stop_daemon.md
deleted file mode 100644
index a47ece8d95b..00000000000
--- a/doc/admin-guide/en-US/markdown/admin_start_stop_daemon.md
+++ /dev/null
@@ -1,58 +0,0 @@
-#Managing the glusterd Service
-
-After installing GlusterFS, you must start glusterd service. The
-glusterd service serves as the Gluster elastic volume manager,
-overseeing glusterfs processes, and co-ordinating dynamic volume
-operations, such as adding and removing volumes across multiple storage
-servers non-disruptively.
-
-This section describes how to start the glusterd service in the
-following ways:
-
-- [Starting and Stopping glusterd Manually](#manual)
-- [Starting glusterd Automatically](#auto)
-
-> **Note**: You must start glusterd on all GlusterFS servers.
-
-<a name="manual" />
-##Starting and Stopping glusterd Manually
-
-This section describes how to start and stop glusterd manually
-
--   To start glusterd manually, enter the following command:
-
-    `# /etc/init.d/glusterd start `
-
--   To stop glusterd manually, enter the following command:
-
-    `# /etc/init.d/glusterd stop`
-
-<a name="auto" />
-##Starting glusterd Automatically
-
-This section describes how to configure the system to automatically
-start the glusterd service every time the system boots.
-
-###Red Hat and Fedora distros
-
-To configure Red Hat-based systems to automatically start the glusterd
-service every time the system boots, enter the following from the
-command line:
-
-`# chkconfig glusterd on `
-
-###Debian and derivatives like Ubuntu
-
-To configure Debian-based systems to automatically start the glusterd
-service every time the system boots, enter the following from the
-command line:
-
-`# update-rc.d glusterd defaults`
-
-###Systems Other than Red Hat and Debain
-
-To configure systems other than Red Hat or Debian to automatically start
-the glusterd service every time the system boots, enter the following
-entry to the*/etc/rc.local* file:
-
-`# echo "glusterd" >> /etc/rc.local `
diff --git a/doc/admin-guide/en-US/markdown/admin_storage_pools.md b/doc/admin-guide/en-US/markdown/admin_storage_pools.md
deleted file mode 100644
index de181f58c18..00000000000
--- a/doc/admin-guide/en-US/markdown/admin_storage_pools.md
+++ /dev/null
@@ -1,91 +0,0 @@
-#Setting up Trusted Storage Pools
-
-Before you can configure a GlusterFS volume, you must create a trusted
-storage pool consisting of the storage servers that provides bricks to a
-volume.
-
-A storage pool is a trusted network of storage servers. When you start
-the first server, the storage pool consists of that server alone. To add
-additional storage servers to the storage pool, you can use the probe
-command from a storage server that is already trusted.
-
-> **Note**: Do not self-probe the first server/localhost from itself.
-
-The GlusterFS service must be running on all storage servers that you
-want to add to the storage pool. See ? for more information.
-
-##Adding Servers to Trusted Storage Pool
-
-To create a trusted storage pool, add servers to the trusted storage
-pool
-
-1.  **The servers used to create the storage pool must be resolvable by
-    hostname.**
-
-    To add a server to the storage pool:
-
-    `# gluster peer probe `
-
-    For example, to create a trusted storage pool of four servers, add
-    three servers to the storage pool from server1:
-
-        # gluster peer probe server2
-        Probe successful
-
-        # gluster peer probe server3
-        Probe successful
-
-        # gluster peer probe server4
-        Probe successful
-
-2.  **Verify the peer status from the first server using the following
-    commands:**
-
-        # gluster peer status
-        Number of Peers: 3
-
-        Hostname: server2
-        Uuid: 5e987bda-16dd-43c2-835b-08b7d55e94e5
-        State: Peer in Cluster (Connected)
-
-        Hostname: server3
-        Uuid: 1e0ca3aa-9ef7-4f66-8f15-cbc348f29ff7
-        State: Peer in Cluster (Connected)
-
-        Hostname: server4
-        Uuid: 3e0caba-9df7-4f66-8e5d-cbc348f29ff7
-        State: Peer in Cluster (Connected)
-
-    3.  **Assign the hostname to the first server by probing it from another server (not the server used in steps 1 and 2):**
-
-        server2# gluster peer probe server1
-        Probe successful
-
-4.  **Verify the peer status from the same server you used in step 3 using the following
-    command:**
-
-        server2# gluster peer status
-        Number of Peers: 3
-
-        Hostname: server1
-        Uuid: ceed91d5-e8d1-434d-9d47-63e914c93424
-        State: Peer in Cluster (Connected)
-
-        Hostname: server3
-        Uuid: 1e0ca3aa-9ef7-4f66-8f15-cbc348f29ff7
-        State: Peer in Cluster (Connected)
-
-        Hostname: server4
-        Uuid: 3e0caba-9df7-4f66-8e5d-cbc348f29ff7
-        State: Peer in Cluster (Connected)
-
-##Removing Servers from the Trusted Storage Pool
-
-To remove a server from the storage pool:
-
-`# gluster peer detach`
-
-For example, to remove server4 from the trusted storage pool:
-
-    # gluster peer detach server4
-    Detach successful
diff --git a/doc/admin-guide/en-US/markdown/admin_troubleshooting.md b/doc/admin-guide/en-US/markdown/admin_troubleshooting.md
deleted file mode 100644
index fa19a2f71de..00000000000
--- a/doc/admin-guide/en-US/markdown/admin_troubleshooting.md
+++ /dev/null
@@ -1,495 +0,0 @@
-#Troubleshooting GlusterFS
-
-This section describes how to manage GlusterFS logs and most common
-troubleshooting scenarios related to GlusterFS.
-
-##Contents
-* [Managing GlusterFS Logs](#logs)
-* [Troubleshooting Geo-replication](#georep)
-* [Troubleshooting POSIX ACLs](#posix-acls)
-* [Troubleshooting Hadoop Compatible Storage](#hadoop)
-* [Troubleshooting NFS](#nfs)
-* [Troubleshooting File Locks](#file-locks)
-
-<a name="logs" />
-##Managing GlusterFS Logs
-
-###Rotating Logs
-
-Administrators can rotate the log file in a volume, as needed.
-
-**To rotate a log file**
-
-    `# gluster volume log rotate `
-
-For example, to rotate the log file on test-volume:
-
-    # gluster volume log rotate test-volume
-    log rotate successful
-
-> **Note**
-> When a log file is rotated, the contents of the current log file
-> are moved to log-file- name.epoch-time-stamp.
-
-<a name="georep" />
-##Troubleshooting Geo-replication
-
-This section describes the most common troubleshooting scenarios related
-to GlusterFS Geo-replication.
-
-###Locating Log Files
-
-For every Geo-replication session, the following three log files are
-associated to it (four, if the slave is a gluster volume):
-
--   **Master-log-file** - log file for the process which monitors the Master
-    volume
--   **Slave-log-file** - log file for process which initiates the changes in
-    slave
--   **Master-gluster-log-file** - log file for the maintenance mount point
-    that Geo-replication module uses to monitor the master volume
--   **Slave-gluster-log-file** - is the slave's counterpart of it
-
-**Master Log File**
-
-To get the Master-log-file for geo-replication, use the following
-command:
-
-`gluster volume geo-replication  config log-file`
-
-For example:
-
-`# gluster volume geo-replication Volume1 example.com:/data/remote_dir config log-file `
-
-**Slave Log File**
-
-To get the log file for Geo-replication on slave (glusterd must be
-running on slave machine), use the following commands:
-
-1.  On master, run the following command:
-
-    `# gluster volume geo-replication Volume1 example.com:/data/remote_dir config session-owner 5f6e5200-756f-11e0-a1f0-0800200c9a66 `
-
-    Displays the session owner details.
-
-2.  On slave, run the following command:
-
-    `# gluster volume geo-replication /data/remote_dir config log-file /var/log/gluster/${session-owner}:remote-mirror.log `
-
-3.  Replace the session owner details (output of Step 1) to the output
-    of the Step 2 to get the location of the log file.
-
-    `/var/log/gluster/5f6e5200-756f-11e0-a1f0-0800200c9a66:remote-mirror.log`
-
-###Rotating Geo-replication Logs
-
-Administrators can rotate the log file of a particular master-slave
-session, as needed. When you run geo-replication's ` log-rotate`
-command, the log file is backed up with the current timestamp suffixed
-to the file name and signal is sent to gsyncd to start logging to a new
-log file.
-
-**To rotate a geo-replication log file**
-
--   Rotate log file for a particular master-slave session using the
-    following command:
-
-    `# gluster volume geo-replication  log-rotate`
-
-    For example, to rotate the log file of master `Volume1` and slave
-    `example.com:/data/remote_dir` :
-
-        # gluster volume geo-replication Volume1 example.com:/data/remote_dir log rotate
-        log rotate successful
-
--   Rotate log file for all sessions for a master volume using the
-    following command:
-
-    `# gluster volume geo-replication  log-rotate`
-
-    For example, to rotate the log file of master `Volume1`:
-
-        # gluster volume geo-replication Volume1 log rotate
-        log rotate successful
-
--   Rotate log file for all sessions using the following command:
-
-    `# gluster volume geo-replication log-rotate`
-
-    For example, to rotate the log file for all sessions:
-
-        # gluster volume geo-replication log rotate
-        log rotate successful
-
-###Synchronization is not complete
-
-**Description**: GlusterFS Geo-replication did not synchronize the data
-completely but still the geo- replication status displayed is OK.
-
-**Solution**: You can enforce a full sync of the data by erasing the
-index and restarting GlusterFS Geo- replication. After restarting,
-GlusterFS Geo-replication begins synchronizing all the data. All files
-are compared using checksum, which can be a lengthy and high resource
-utilization operation on large data sets.
-
-
-###Issues in Data Synchronization
-
-**Description**: Geo-replication display status as OK, but the files do
-not get synced, only directories and symlink gets synced with the
-following error message in the log:
-
-    [2011-05-02 13:42:13.467644] E [master:288:regjob] GMaster: failed to
-    sync ./some\_file\`
-
-**Solution**: Geo-replication invokes rsync v3.0.0 or higher on the host
-and the remote machine. You must verify if you have installed the
-required version.
-
-###Geo-replication status displays Faulty very often
-
-**Description**: Geo-replication displays status as faulty very often
-with a backtrace similar to the following:
-
-    2011-04-28 14:06:18.378859] E [syncdutils:131:log\_raise\_exception]
-    \<top\>: FAIL: Traceback (most recent call last): File
-    "/usr/local/libexec/glusterfs/python/syncdaemon/syncdutils.py", line
-    152, in twraptf(\*aa) File
-    "/usr/local/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in
-    listen rid, exc, res = recv(self.inf) File
-    "/usr/local/libexec/glusterfs/python/syncdaemon/repce.py", line 42, in
-    recv return pickle.load(inf) EOFError
-
-**Solution**: This error indicates that the RPC communication between
-the master gsyncd module and slave gsyncd module is broken and this can
-happen for various reasons. Check if it satisfies all the following
-pre-requisites:
-
--   Password-less SSH is set up properly between the host and the remote
-    machine.
--   If FUSE is installed in the machine, because geo-replication module
-    mounts the GlusterFS volume using FUSE to sync data.
--   If the **Slave** is a volume, check if that volume is started.
--   If the Slave is a plain directory, verify if the directory has been
-    created already with the required permissions.
--   If GlusterFS 3.2 or higher is not installed in the default location
-    (in Master) and has been prefixed to be installed in a custom
-    location, configure the `gluster-command` for it to point to the
-    exact location.
--   If GlusterFS 3.2 or higher is not installed in the default location
-    (in slave) and has been prefixed to be installed in a custom
-    location, configure the `remote-gsyncd-command` for it to point to
-    the exact place where gsyncd is located.
-
-###Intermediate Master goes to Faulty State
-
-**Description**: In a cascading set-up, the intermediate master goes to
-faulty state with the following log:
-
-    raise RuntimeError ("aborting on uuid change from %s to %s" % \\
-    RuntimeError: aborting on uuid change from af07e07c-427f-4586-ab9f-
-    4bf7d299be81 to de6b5040-8f4e-4575-8831-c4f55bd41154
-
-**Solution**: In a cascading set-up the Intermediate master is loyal to
-the original primary master. The above log means that the
-geo-replication module has detected change in primary master. If this is
-the desired behavior, delete the config option volume-id in the session
-initiated from the intermediate master.
-
-<a name="posix-acls" />
-##Troubleshooting POSIX ACLs
-
-This section describes the most common troubleshooting issues related to
-POSIX ACLs.
-
-    setfacl command fails with “setfacl: \<file or directory name\>: Operation not supported” error
-
-You may face this error when the backend file systems in one of the
-servers is not mounted with the "-o acl" option. The same can be
-confirmed by viewing the following error message in the log file of the
-server "Posix access control list is not supported".
-
-**Solution**: Remount the backend file system with "-o acl" option.
-
-<a name="hadoop" />
-##Troubleshooting Hadoop Compatible Storage
-
-###Time Sync
-
-**Problem**: Running MapReduce job may throw exceptions if the time is out-of-sync on
-the hosts in the cluster.
-
-**Solution**: Sync the time on all hosts using ntpd program.
-
-<a name="nfs" />
-##Troubleshooting NFS
-
-This section describes the most common troubleshooting issues related to
-NFS .
-
-###mount command on NFS client fails with “RPC Error: Program not registered”
-
-    Start portmap or rpcbind service on the NFS server.
-
-This error is encountered when the server has not started correctly.
-On most Linux distributions this is fixed by starting portmap:
-
-`$ /etc/init.d/portmap start`
-
-On some distributions where portmap has been replaced by rpcbind, the
-following command is required:
-
-`$ /etc/init.d/rpcbind start `
-
-After starting portmap or rpcbind, gluster NFS server needs to be
-restarted.
-
-###NFS server start-up fails with “Port is already in use” error in the log file.
-
-Another Gluster NFS server is running on the same machine.
-
-This error can arise in case there is already a Gluster NFS server
-running on the same machine. This situation can be confirmed from the
-log file, if the following error lines exist:
-
-    [2010-05-26 23:40:49] E [rpc-socket.c:126:rpcsvc_socket_listen] rpc-socket: binding socket failed:Address already in use
-    [2010-05-26 23:40:49] E [rpc-socket.c:129:rpcsvc_socket_listen] rpc-socket: Port is already in use 
-    [2010-05-26 23:40:49] E [rpcsvc.c:2636:rpcsvc_stage_program_register] rpc-service: could not create listening connection 
-    [2010-05-26 23:40:49] E [rpcsvc.c:2675:rpcsvc_program_register] rpc-service: stage registration of program failed 
-    [2010-05-26 23:40:49] E [rpcsvc.c:2695:rpcsvc_program_register] rpc-service: Program registration failed: MOUNT3, Num: 100005, Ver: 3, Port: 38465 
-    [2010-05-26 23:40:49] E [nfs.c:125:nfs_init_versions] nfs: Program init failed 
-    [2010-05-26 23:40:49] C [nfs.c:531:notify] nfs: Failed to initialize protocols
-
-To resolve this error one of the Gluster NFS servers will have to be
-shutdown. At this time, Gluster NFS server does not support running
-multiple NFS servers on the same machine.
-
-###mount command fails with “rpc.statd” related error message
-
-If the mount command fails with the following error message:
-
-    mount.nfs: rpc.statd is not running but is required for remote locking.
-    mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
-
-For NFS clients to mount the NFS server, rpc.statd service must be
-running on the clients. Start rpc.statd service by running the following command:
-
-`$ rpc.statd `
-
-###mount command takes too long to finish.
-
-**Start rpcbind service on the NFS client**
-
-The problem is that the rpcbind or portmap service is not running on the
-NFS client. The resolution for this is to start either of these services
-by running the following command:
-
-`$ /etc/init.d/portmap start`
-
-On some distributions where portmap has been replaced by rpcbind, the
-following command is required:
-
-`$ /etc/init.d/rpcbind start`
-
-###NFS server glusterfsd starts but initialization fails with “nfsrpc- service: portmap registration of program failed” error message in the log.
-
-NFS start-up can succeed but the initialization of the NFS service can
-still fail preventing clients from accessing the mount points. Such a
-situation can be confirmed from the following error messages in the log
-file:
-
-    [2010-05-26 23:33:47] E [rpcsvc.c:2598:rpcsvc_program_register_portmap] rpc-service: Could notregister with portmap 
-    [2010-05-26 23:33:47] E [rpcsvc.c:2682:rpcsvc_program_register] rpc-service: portmap registration of program failed
-    [2010-05-26 23:33:47] E [rpcsvc.c:2695:rpcsvc_program_register] rpc-service: Program registration failed: MOUNT3, Num: 100005, Ver: 3, Port: 38465
-    [2010-05-26 23:33:47] E [nfs.c:125:nfs_init_versions] nfs: Program init failed
-    [2010-05-26 23:33:47] C [nfs.c:531:notify] nfs: Failed to initialize protocols
-    [2010-05-26 23:33:49] E [rpcsvc.c:2614:rpcsvc_program_unregister_portmap] rpc-service: Could not unregister with portmap
-    [2010-05-26 23:33:49] E [rpcsvc.c:2731:rpcsvc_program_unregister] rpc-service: portmap unregistration of program failed
-    [2010-05-26 23:33:49] E [rpcsvc.c:2744:rpcsvc_program_unregister] rpc-service: Program unregistration failed: MOUNT3, Num: 100005, Ver: 3, Port: 38465
-
-1.  **Start portmap or rpcbind service on the NFS server**
-
-    On most Linux distributions, portmap can be started using the
-    following command:
-
-    `$ /etc/init.d/portmap start `
-
-    On some distributions where portmap has been replaced by rpcbind,
-    run the following command:
-
-    `$ /etc/init.d/rpcbind start `
-
-    After starting portmap or rpcbind, gluster NFS server needs to be
-    restarted.
-
-2.  **Stop another NFS server running on the same machine**
-
-    Such an error is also seen when there is another NFS server running
-    on the same machine but it is not the Gluster NFS server. On Linux
-    systems, this could be the kernel NFS server. Resolution involves
-    stopping the other NFS server or not running the Gluster NFS server
-    on the machine. Before stopping the kernel NFS server, ensure that
-    no critical service depends on access to that NFS server's exports.
-
-    On Linux, kernel NFS servers can be stopped by using either of the
-    following commands depending on the distribution in use:
-
-    `$ /etc/init.d/nfs-kernel-server stop`
-
-    `$ /etc/init.d/nfs stop`
-
-3.  **Restart Gluster NFS server**
-
-###mount command fails with NFS server failed error.
-
-mount command fails with following error
-
-    *mount: mount to NFS server '10.1.10.11' failed: timed out (retrying).*
-
-Perform one of the following to resolve this issue:
-
-1.  **Disable name lookup requests from NFS server to a DNS server**
-
-    The NFS server attempts to authenticate NFS clients by performing a
-    reverse DNS lookup to match hostnames in the volume file with the
-    client IP addresses. There can be a situation where the NFS server
-    either is not able to connect to the DNS server or the DNS server is
-    taking too long to responsd to DNS request. These delays can result
-    in delayed replies from the NFS server to the NFS client resulting
-    in the timeout error seen above.
-
-    NFS server provides a work-around that disables DNS requests,
-    instead relying only on the client IP addresses for authentication.
-    The following option can be added for successful mounting in such
-    situations:
-
-    `option rpc-auth.addr.namelookup off `
-
-    > **Note**: Remember that disabling the NFS server forces authentication
-    > of clients to use only IP addresses and if the authentication
-    > rules in the volume file use hostnames, those authentication rules
-    > will fail and disallow mounting for those clients.
-
-    **OR**
-
-2.  **NFS version used by the NFS client is other than version 3**
-
-    Gluster NFS server supports version 3 of NFS protocol. In recent
-    Linux kernels, the default NFS version has been changed from 3 to 4.
-    It is possible that the client machine is unable to connect to the
-    Gluster NFS server because it is using version 4 messages which are
-    not understood by Gluster NFS server. The timeout can be resolved by
-    forcing the NFS client to use version 3. The **vers** option to
-    mount command is used for this purpose:
-
-    `$ mount  -o vers=3 `
-
-###showmount fails with clnt\_create: RPC: Unable to receive
-
-Check your firewall setting to open ports 111 for portmap
-requests/replies and Gluster NFS server requests/replies. Gluster NFS
-server operates over the following port numbers: 38465, 38466, and
-38467.
-
-###Application fails with "Invalid argument" or "Value too large for defined data type" error.
-
-These two errors generally happen for 32-bit nfs clients or applications
-that do not support 64-bit inode numbers or large files. Use the
-following option from the CLI to make Gluster NFS return 32-bit inode
-numbers instead: nfs.enable-ino32 \<on|off\>
-
-Applications that will benefit are those that were either:
-
--   built 32-bit and run on 32-bit machines such that they do not
-    support large files by default
--   built 32-bit on 64-bit systems
-
-This option is disabled by default so NFS returns 64-bit inode numbers
-by default.
-
-Applications which can be rebuilt from source are recommended to rebuild
-using the following flag with gcc:
-
-` -D_FILE_OFFSET_BITS=64`
-
-<a name="file-locks" />
-##Troubleshooting File Locks
-
-In GlusterFS 3.3 you can use `statedump` command to list the locks held
-on files. The statedump output also provides information on each lock
-with its range, basename, PID of the application holding the lock, and
-so on. You can analyze the output to know about the locks whose
-owner/application is no longer running or interested in that lock. After
-ensuring that the no application is using the file, you can clear the
-lock using the following `clear lock` commands.
-
-1.  **Perform statedump on the volume to view the files that are locked
-    using the following command:**
-
-    `# gluster volume statedump  inode`
-
-    For example, to display statedump of test-volume:
-
-        # gluster volume statedump test-volume
-        Volume statedump successful
-
-    The statedump files are created on the brick servers in the` /tmp`
-    directory or in the directory set using `server.statedump-path`
-    volume option. The naming convention of the dump file is
-    `<brick-path>.<brick-pid>.dump`.
-
-    The following are the sample contents of the statedump file. It
-    indicates that GlusterFS has entered into a state where there is an
-    entry lock (entrylk) and an inode lock (inodelk). Ensure that those
-    are stale locks and no resources own them.
-
-        [xlator.features.locks.vol-locks.inode]
-        path=/
-        mandatory=0
-        entrylk-count=1
-        lock-dump.domain.domain=vol-replicate-0
-        xlator.feature.locks.lock-dump.domain.entrylk.entrylk[0](ACTIVE)=type=ENTRYLK_WRLCK on basename=file1, pid = 714782904, owner=ffffff2a3c7f0000, transport=0x20e0670, , granted at Mon Feb 27 16:01:01 2012
-
-        conn.2.bound_xl./gfs/brick1.hashsize=14057
-        conn.2.bound_xl./gfs/brick1.name=/gfs/brick1/inode
-        conn.2.bound_xl./gfs/brick1.lru_limit=16384
-        conn.2.bound_xl./gfs/brick1.active_size=2
-        conn.2.bound_xl./gfs/brick1.lru_size=0
-        conn.2.bound_xl./gfs/brick1.purge_size=0
-
-        [conn.2.bound_xl./gfs/brick1.active.1]
-        gfid=538a3d4a-01b0-4d03-9dc9-843cd8704d07
-        nlookup=1
-        ref=2
-        ia_type=1
-        [xlator.features.locks.vol-locks.inode]
-        path=/file1
-        mandatory=0
-        inodelk-count=1
-        lock-dump.domain.domain=vol-replicate-0
-        inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid = 714787072, owner=00ffff2a3c7f0000, transport=0x20e0670, , granted at Mon Feb 27 16:01:01 2012
-
-2.  **Clear the lock using the following command:**
-
-    `# gluster volume clear-locks`
-
-    For example, to clear the entry lock on `file1` of test-volume:
-
-        # gluster volume clear-locks test-volume / kind granted entry file1
-        Volume clear-locks successful
-        vol-locks: entry blocked locks=0 granted locks=1
-
-3.  **Clear the inode lock using the following command:**
-
-    `# gluster volume clear-locks`
-
-    For example, to clear the inode lock on `file1` of test-volume:
-
-        # gluster  volume clear-locks test-volume /file1 kind granted inode 0,0-0
-        Volume clear-locks successful
-        vol-locks: inode blocked locks=0 granted locks=1
-
-    You can perform statedump on test-volume again to verify that the
-    above inode and entry locks are cleared.
-
-
diff --git a/doc/admin-guide/en-US/markdown/gfs_introduction.md b/doc/admin-guide/en-US/markdown/gfs_introduction.md
deleted file mode 100644
index 9f9c058156e..00000000000
--- a/doc/admin-guide/en-US/markdown/gfs_introduction.md
+++ /dev/null
@@ -1,31 +0,0 @@
-Introducing Gluster File System
-===============================
-
-GlusterFS is an open source, clustered file system capable of scaling to
-several petabytes and handling thousands of clients. GlusterFS can be
-flexibly combined with commodity physical, virtual, and cloud resources
-to deliver highly available and performant enterprise storage at a
-fraction of the cost of traditional solutions.
-
-GlusterFS clusters together storage building blocks over Infiniband RDMA
-and/or TCP/IP interconnect, aggregating disk and memory resources and
-managing data in a single global namespace. GlusterFS is based on a
-stackable user space design, delivering exceptional performance for
-diverse workloads.
-
-![ Virtualized Cloud Environments ](../images/640px-GlusterFS_Architecture.png)
-
-GlusterFS is designed for today's high-performance, virtualized cloud
-environments. Unlike traditional data centers, cloud environments
-require multi-tenancy along with the ability to grow or shrink resources
-on demand. Enterprises can scale capacity, performance, and availability
-on demand, with no vendor lock-in, across on-premise, public cloud, and
-hybrid environments.
-
-GlusterFS is in production at thousands of enterprises spanning media,
-healthcare, government, education, web 2.0, and financial services. The
-following table lists the commercial offerings:
-
-* [Red Hat Storage](https://access.redhat.com/site/documentation/Red_Hat_Storage/)
-* Red Hat Storage Software Appliance
-* Red Hat Virtual Storage Appliance
diff --git a/doc/admin-guide/en-US/markdown/glossary.md b/doc/admin-guide/en-US/markdown/glossary.md
deleted file mode 100644
index d047622a93b..00000000000
--- a/doc/admin-guide/en-US/markdown/glossary.md
+++ /dev/null
@@ -1,198 +0,0 @@
-Glossary
-========
-
-**Brick**
-:   A Brick is the basic unit of storage in GlusterFS, represented by an
-    export directory on a server in the trusted storage pool. A Brick is
-    represented by combining a server name with an export directory in the
-    following format:
-
-    `SERVER:EXPORT`
-
-    For example:
-
-    `myhostname:/exports/myexportdir/`
-
-**Client**
-:   Any machine that mounts a GlusterFS volume.
-
-**Cluster**
-:   A cluster is a group of linked computers, working together closely
-    thus in many respects forming a single computer.
-
-**Distributed File System**
-:   A file system that allows multiple clients to concurrently access
-    data over a computer network.
-
-**Extended Attributes**
-:   Extended file attributes (abbreviated xattr) is a file system feature
-    that enables users/programs to associate files/dirs with metadata.
-
-**Filesystem**
-:   A method of storing and organizing computer files and their data.
-    Essentially, it organizes these files into a database for the
-    storage, organization, manipulation, and retrieval by the computer's
-    operating system.
-
-    Source: [Wikipedia][]
-
-**FUSE**
-:   Filesystem in Userspace (FUSE) is a loadable kernel module for
-    Unix-like computer operating systems that lets non-privileged users
-    create their own file systems without editing kernel code. This is
-    achieved by running file system code in user space while the FUSE
-    module provides only a "bridge" to the actual kernel interfaces.
-
-    Source: [Wikipedia][1]
-
-**Geo-Replication**
-:   Geo-replication provides a continuous, asynchronous, and incremental
-    replication service from site to another over Local Area Networks
-    (LAN), Wide Area Network (WAN), and across the Internet.
-
-**GFID**
-:   Each file/directory on a GlusterFS volume has a unique 128-bit number
-    associated with it called the GFID. This is analogous to inode in a
-    regular filesystem.
-
-**glusterd**
-:   The Gluster management daemon that needs to run on all servers in
-    the trusted storage pool.
-
-**Infiniband**
-    InfiniBand is a switched fabric computer network communications link
-    used in high-performance computing and enterprise data centers.
-
-**Metadata**
-:   Metadata is data providing information about one or more other
-    pieces of data.
-
-**Namespace**
-:   Namespace is an abstract container or environment created to hold a
-    logical grouping of unique identifiers or symbols. Each Gluster
-    volume exposes a single namespace as a POSIX mount point that
-    contains every file in the cluster.
-
-**Node**
-:   A server or computer that hosts one or more bricks.
-
-**Open Source**
-:   Open source describes practices in production and development that
-    promote access to the end product's source materials. Some consider
-    open source a philosophy, others consider it a pragmatic
-    methodology.
-
-    Before the term open source became widely adopted, developers and
-    producers used a variety of phrases to describe the concept; open
-    source gained hold with the rise of the Internet, and the attendant
-    need for massive retooling of the computing source code.
-
-    Opening the source code enabled a self-enhancing diversity of
-    production models, communication paths, and interactive communities.
-    Subsequently, a new, three-word phrase "open source software" was
-    born to describe the environment that the new copyright, licensing,
-    domain, and consumer issues created.
-
-    Source: [Wikipedia][2]
-
-**Petabyte**
-:   A petabyte (derived from the SI prefix peta- ) is a unit of
-    information equal to one quadrillion (short scale) bytes, or 1000
-    terabytes. The unit symbol for the petabyte is PB. The prefix peta-
-    (P) indicates a power of 1000:
-
-    1 PB = 1,000,000,000,000,000 B = 10005 B = 1015 B.
-
-    The term "pebibyte" (PiB), using a binary prefix, is used for the
-    corresponding power of 1024.
-
-    Source: [Wikipedia][3]
-
-**POSIX**
-:   Portable Operating System Interface (for Unix) is the name of a
-    family of related standards specified by the IEEE to define the
-    application programming interface (API), along with shell and
-    utilities interfaces for software compatible with variants of the
-    Unix operating system. Gluster exports a fully POSIX compliant file
-    system.
-
-**Quorum**
-:   The configuration of quorum in a trusted storage pool determines the
-    number of server failures that the trusted storage pool can sustain.
-    If an additional failure occurs, the trusted storage pool becomes
-    unavailable.
-
-**Quota**
-:   Quotas allow you to set limits on usage of disk space by directories or
-    by volumes.
-
-**RAID**
-:   Redundant Array of Inexpensive Disks (RAID) is a technology that
-    provides increased storage reliability through redundancy, combining
-    multiple low-cost, less-reliable disk drives components into a
-    logical unit where all drives in the array are interdependent.
-
-**RDMA**
-:   Remote direct memory access (RDMA) is a direct memory access from the
-    memory of one computer into that of another without involving either
-    one's operating system. This permits high-throughput, low-latency
-    networking, which is especially useful in massively parallel computer
-    clusters.
-
-**Rebalance**
-:   A process of fixing layout and resdistributing data in a volume when a
-    brick is added or removed.
-
-**RRDNS**
-:   Round Robin Domain Name Service (RRDNS) is a method to distribute
-    load across application servers. RRDNS is implemented by creating
-    multiple A records with the same name and different IP addresses in
-    the zone file of a DNS server.
-
-**Samba**
-:   Samba allows file and print sharing between computers running Windows and
-    computers running Linux. It is an implementation of several services and
-    protocols including SMB and CIFS.
-
-**Self-Heal**
-:   The self-heal daemon that runs in the background, identifies
-    inconsistencies in files/dirs in a replicated volume and then resolves
-    or heals them. This healing process is usually required when one or more
-    bricks of a volume goes down and then comes up later.
-
-**Split-brain**
-:   This is a situation where data on two or more bricks in a replicated
-    volume start to diverge in terms of content or metadata. In this state,
-    one cannot determine programitically which set of data is "right" and
-    which is "wrong".
-
-**Translator**
-:   Translators (also called xlators) are stackable modules where each
-    module has a very specific purpose. Translators are stacked in a
-    hierarchical structure called as graph. A translator receives data
-    from its parent translator, performs necessary operations and then
-    passes the data down to its child translator in hierarchy.
-
-**Trusted Storage Pool**
-:   A storage pool is a trusted network of storage servers. When you
-    start the first server, the storage pool consists of that server
-    alone.
-
-**Userspace**
-:   Applications running in user space don’t directly interact with
-    hardware, instead using the kernel to moderate access. Userspace
-    applications are generally more portable than applications in kernel
-    space. Gluster is a user space application.
-
-**Volfile**
-:   Volfile is a configuration file used by glusterfs process. Volfile
-    will be usually located at `/var/lib/glusterd/vols/VOLNAME`.
-
-**Volume**
-:   A volume is a logical collection of bricks. Most of the gluster
-    management operations happen on the volume.
-
-  [Wikipedia]: http://en.wikipedia.org/wiki/Filesystem
-  [1]: http://en.wikipedia.org/wiki/Filesystem_in_Userspace
-  [2]: http://en.wikipedia.org/wiki/Open_source
-  [3]: http://en.wikipedia.org/wiki/Petabyte
diff --git a/doc/admin-guide/en-US/markdown/pdfgen.sh b/doc/admin-guide/en-US/markdown/pdfgen.sh
deleted file mode 100755
index 68b320617b1..00000000000
--- a/doc/admin-guide/en-US/markdown/pdfgen.sh
+++ /dev/null
@@ -1,16 +0,0 @@
-#!/bin/bash
-# pdfgen.sh simple pdf generation helper script.
-# Copyright (C) 2012-2013  James Shubin
-# Written by James Shubin <james@shubin.ca>
-
-#dir='/tmp/pdf'
-dir=`pwd`'/output/'
-ln -s ../images images
-mkdir -p "$dir"
-
-for i in *.md; do
-	pandoc $i -o "$dir"`echo $i | sed 's/\.md$/\.pdf/'`
-done
-
-rm images	# remove symlink
-
diff --git a/doc/authentication.txt b/doc/authentication.txt
deleted file mode 100644
index 036a9df9908..00000000000
--- a/doc/authentication.txt
+++ /dev/null
@@ -1,112 +0,0 @@
-
-* Authentication is provided by two modules addr and login. Login based authentication uses username/password from client for authentication. Each module returns either ACCEPT, REJCET or DONT_CARE. DONT_CARE is returned if the input authentication information to the module is not concerned to its working. The theory behind authentication is that "none of the auth modules should return REJECT and atleast one of them should return ACCEPT"
-
-* Currently all the authentication related information is passed un-encrypted over the network from client to server.
-
-----------------------------------------------------------------------------------------------------
-* options provided in protocol/client:
-	* for username/password based authentication:
-	      option username <username>
-	      option password <password>
-	* client can have only one set of username/password
-	* for addr based authentication:
-	      * no options required in protocol/client. Client has to bind to privileged port (port < 1024 ) which means the process in which protocol/client is loaded has to be run as root.
-
-----------------------------------------------------------------------------------------------------
-* options provided in protocol/server:
-	* for username/password based authentication:
-	      option auth.login.<brick>.allow [comma separated list of usernames using which clients can connect to volume <brick>]
-	      option auth.login.<username>.password <password> #specify password <password> for username <username>
-	* for addr based authentication:
-	      option auth.addr.<brick>.allow [comma separated list of ip-addresses/unix-paths from which clients are allowed to connect to volume <brick>]
-	      option auth.addr.<brick>.reject [comma separated list of ip-addresses/unix-paths from which clients are not allowed to connect to volume <brick>]
-	* negation operator '!' is used to invert the sense of matching.
-	  Eg., option auth.addr.brick.allow !a.b.c.d #do not allow client from a.b.c.d to connect to volume brick
-	       option auth.addr.brick.reject !w.x.y.z #allow client from w.x.y.z to connect to volume brick
-	* wildcard '*' can be used to match any ip-address/unix-path
-
-----------------------------------------------------------------------------------------------------
-
-* Usecases:
-
-* username/password based authentication only
-      protocol/client:
-	option username foo
-	option password foo-password
-	option remote-subvolume foo-brick
-
-      protocol/server:
-	option auth.login.foo-brick.allow foo,who #,other users allowed to connect to foo-brick
-	option auth.login.foo.password foo-password
-	option auth.login.who.password who-password
-
-      * in protocol/server, dont specify ip from which client is connecting in auth.addr.foo-brick.reject list
-
-****************************************************************************************************
-
-* ip based authentication only
-      protocol/client:
-	option remote-subvolume foo-brick
-	* Client is connecting from a.b.c.d
-
-      protocol/server:
-	option auth.addr.foo-brick.allow a.b.c.d,e.f.g.h,i.j.k.l #, other ip addresses from which clients are allowed to connect to foo-brick
-
-****************************************************************************************************
-* ip and username/password based authentication
-  * allow only "user foo from a.b.c.d"
-    protocol/client:
-	option username foo
-	option password foo-password
-	option remote-subvolume foo-brick
-
-    protocol/server:
-	option auth.login.foo-brick.allow foo
-	option auth.login.foo.password foo-password
-	option auth.addr.foo-brick.reject !a.b.c.d
-
-  * allow only "user foo" from a.b.c.d i.e., only user foo is allowed from a.b.c.d, but anyone is allowed from ip addresses other than a.b.c.d
-    protocol/client:
-	option username foo
-	option password foo-password
-	option remote-subvolume foo-brick
-
-    protocol/server:
-	option auth.login.foo-brick.allow foo
-	option auth.login.foo.password foo-password
-	option auth.addr.foo-brick.allow !a.b.c.d
-
-  * reject only "user shoo from a.b.c.d"
-    protcol/client:
-	option remote-subvolume shoo-brick
-
-    protocol/server:
-	# observe that no "option auth.login.shoo-brick.allow shoo" given
-	# Also other users from a.b.c.d have to be explicitly allowed using auth.login.shoo-brick.allow ...
-	option auth.addr.shoo-brick.allow !a.b.c.d
-
-  * reject only "user shoo" from a.b.c.d i.e., user shoo from a.b.c.d has to be rejected.
-    * same as reject only "user shoo from a.b.c.d" above, but rules have to be added whether to allow ip addresses (and users from those ips) other than a.b.c.d
-
-****************************************************************************************************
-
-* ip or username/password based authentication
-
-  * allow user foo or clients from a.b.c.d
-    protocol/client:
-	option remote-subvolume foo-brick
-
-    protocol/server:
-	option auth.login.foo-brick.allow foo
-	option auth.login.foo.password foo-password
-	option auth.addr.foo-brick.allow a.b.c.d
-
-  * reject user shoo or clients from a.b.c.d
-    protocol/client:
-	option remote-subvolume shoo-brick
-
-    protocol/server:
-	option auth.login.shoo-brick.allow <usernames other than shoo>
-	#for each username mentioned in the above <usernames other than shoo> list, specify password as below
-	option auth.login.<username other than shoo>.password password
-	option auth.addr.shoo-brick.reject a.b.c.d
diff --git a/doc/coding-standard.pdf b/doc/coding-standard.pdf
deleted file mode 100644
index bc9cb56202c..00000000000
--- a/doc/coding-standard.pdf
+++ /dev/null
diff --git a/doc/coding-standard.tex b/doc/coding-standard.tex
deleted file mode 100644
index 80c11041e60..00000000000
--- a/doc/coding-standard.tex
+++ /dev/null
@@ -1,384 +0,0 @@
-\documentclass{article}[12pt]
-\usepackage{color}
-
-\begin{document}
-
-
-\hrule
-\begin{center}\textbf{\Large{GlusterFS Coding Standards}}\end{center}
-\begin{center}\textbf{\large{\textcolor{red}{Z} Research}}\end{center}
-\begin{center}{July 14, 2008}\end{center}
-\hrule
-
-\vspace{8ex}
-
-\section*{$\bullet$ Structure definitions should have a comment per member}
-
-Every member in a structure definition must have a comment about its
-purpose. The comment should be descriptive without being overly verbose.
-
-\vspace{2ex}
-\textsl{Bad}:
-
-\begin{verbatim}
-        gf_lock_t   lock;           /* lock */
-\end{verbatim}
-
-\textsl{Good}:
-
-\begin{verbatim}
-        DBTYPE      access_mode;    /* access mode for accessing
-                                     * the databases, can be
-                                     * DB_HASH, DB_BTREE
-                                     * (option access-mode <mode>)
-                                     */
-\end{verbatim}
-
-\section*{$\bullet$ Declare all variables at the beginning of the function}
-All local variables in a function must be declared immediately after the
-opening brace. This makes it easy to keep track of memory that needs to be freed
-during exit. It also helps debugging, since gdb cannot handle variables
-declared inside loops or other such blocks.
-
-\section*{$\bullet$ Always initialize local variables}
-Every local variable should be initialized to a sensible default value
-at the point of its declaration. All pointers should be initialized to NULL,
-and all integers should be zero or (if it makes sense) an error value.
-
-\vspace{2ex}
-
-\textsl{Good}:
-
-\begin{verbatim}
-        int ret       = 0;
-        char *databuf = NULL;
-        int _fd       = -1;
-\end{verbatim}
-
-\section*{$\bullet$ Initialization should always be done with a constant value}
-Never use a non-constant expression as the initialization value for a variable.
-
-\vspace{2ex}
-
-\textsl{Bad}:
-
-\begin{verbatim}
-        pid_t pid     = frame->root->pid;
-        char *databuf = malloc (1024);
-\end{verbatim}
-
-\section*{$\bullet$ Validate all arguments to a function}
-All pointer arguments to a function must be checked for \texttt{NULL}.
-A macro named \texttt{VALIDATE} (in \texttt{common-utils.h})
-takes one argument, and if it is \texttt{NULL}, writes a log message and
-jumps to a label called \texttt{err} after setting op\_ret and op\_errno
-appropriately. It is recommended to use this template.
-
-\vspace{2ex}
-
-\textsl{Good}:
-
-\begin{verbatim}
-        VALIDATE(frame);
-        VALIDATE(this);
-        VALIDATE(inode);
-\end{verbatim}
-
-\section*{$\bullet$ Never rely on precedence of operators}
-Never write code that relies on the precedence of operators to execute
-correctly.  Such code can be hard to read and someone else might not
-know the precedence of operators as accurately as you do.
-\vspace{2ex}
-
-\textsl{Bad}:
-
-\begin{verbatim}
-        if (op_ret == -1 && errno != ENOENT)
-\end{verbatim}
-
-\textsl{Good}:
-
-\begin{verbatim}
-        if ((op_ret == -1) && (errno != ENOENT))
-\end{verbatim}
-
-\section*{$\bullet$ Use exactly matching types}
-Use a variable of the exact type declared in the manual to hold the
-return value of a function. Do not use an ``equivalent'' type.
-
-\vspace{2ex}
-
-\textsl{Bad}:
-
-\begin{verbatim}
-        int len = strlen (path);
-\end{verbatim}
-
-\textsl{Good}:
-
-\begin{verbatim}
-        size_t len = strlen (path);
-\end{verbatim}
-
-\section*{$\bullet$ Never write code such as \texttt{foo->bar->baz}; check every pointer}
-Do not write code that blindly follows a chain of pointer
-references. Any pointer in the chain may be \texttt{NULL} and thus
-cause a crash. Verify that each pointer is non-null before following
-it.
-
-\section*{$\bullet$ Check return value of all functions and system calls}
-The return value of all system calls and API functions must be checked
-for success or failure.
-
-\vspace{2ex}
-\textsl{Bad}:
-
-\begin{verbatim}
-        close (fd);
-\end{verbatim}
-
-\textsl{Good}:
-
-\begin{verbatim}
-        op_ret = close (_fd);
-        if (op_ret == -1) {
-                gf_log (this->name, GF_LOG_ERROR,
-                        "close on file %s failed (%s)", real_path,
-                        strerror (errno));
-                op_errno = errno;
-                goto out;
-        }
-\end{verbatim}
-
-
-\section*{$\bullet$ Gracefully handle failure of malloc}
-GlusterFS should never crash or exit due to lack of memory. If a
-memory allocation fails, the call should be unwound and an error
-returned to the user.
-
-\section*{$\bullet$ Use result args and reserve the return value to indicate success or failure}
-The return value of every functions must indicate success or failure (unless
-it is impossible for the function to fail --- e.g., boolean functions). If
-the function needs to return additional data, it must be returned using a
-result (pointer) argument.
-
-\vspace{2ex}
-\textsl{Bad}:
-
-\begin{verbatim}
-        int32_t dict_get_int32 (dict_t *this, char *key);
-\end{verbatim}
-
-\textsl{Good}:
-
-\begin{verbatim}
-        int dict_get_int32 (dict_t *this, char *key, int32_t *val);
-\end{verbatim}
-
-\section*{$\bullet$ Always use the `n' versions of string functions}
-Unless impossible, use the length-limited versions of the string functions.
-
-\vspace{2ex}
-\textsl{Bad}:
-
-\begin{verbatim}
-        strcpy (entry_path, real_path);
-\end{verbatim}
-
-\textsl{Good}:
-
-\begin{verbatim}
-        strncpy (entry_path, real_path, entry_path_len);
-\end{verbatim}
-
-\section*{$\bullet$ No dead or commented code}
-There must be no dead code (code to which control can never be passed) or
-commented out code in the codebase.
-
-\section*{$\bullet$ Only one unwind and return per function}
-There must be only one exit out of a function. \texttt{UNWIND} and return
-should happen at only point in the function.
-
-\section*{$\bullet$ Function length or Keep functions small}
-We live in the UNIX-world where modules do one thing and do it well.
-This rule should apply to our functions also. If a function is very long, try splitting it
-into many little helper functions. The question is, in a coding
-spree, how do we know a function is long and unreadable. One rule of
-thumb given by Linus Torvalds is that, a function should be broken-up
-if you have 4 or more levels of indentation going on for more than 3-4
-lines.
-
-\vspace{2ex}
-\textsl{Example for a helper function}:
-\begin{verbatim}
-        static int
-        same_owner (posix_lock_t *l1, posix_lock_t *l2)
-        {
-                return ((l1->client_pid == l2->client_pid) &&
-                       (l1->transport  == l2->transport));
-        }
-\end{verbatim}
-
-\section*{$\bullet$ Defining functions as static}
-Define internal functions as static only if you're
-very sure that there will not be a crash(..of any kind..) emanating in
-that function. If there is even a remote possibility, perhaps due to
-pointer derefering, etc, declare the function as non-static. This
-ensures that when a crash does happen, the function name shows up the
-in the back-trace generated by libc. However, doing so has potential
-for polluting the function namespace, so to avoid conflicts with other
-components in other parts, ensure that the function names are
-prepended with a prefix that identify the component to which it
-belongs. For eg. non-static functions in io-threads translator start
-with iot\_.
-
-\section*{$\bullet$ Ensure function calls wrap around after
-80-columns.}
-Place remaining arguments on the next line if needed.
-
-\section*{$\bullet$ Functions arguments and function definition}
-Place all the arguments of a function definition on the same line
-until the line goes beyond 80-cols. Arguments that extend beyind
-80-cols should be placed on the next line.
-
-\section*{Style issues}
-
-\subsection*{Brace placement}
-Use K\&R/Linux style of brace placement for blocks.
-
-\textsl{Example}:
-\begin{verbatim}
-        int some_function (...)
-        {
-                if (...) {
-                        /* ... */
-                } else if (...) {
-                        /* ... */
-                } else {
-                        /* ... */
-                }
-
-                do {
-                        /* ... */
-                } while (cond);
-        }
-\end{verbatim}
-
-\subsection*{Indentation}
-Use \textbf{eight} spaces for indenting blocks. Ensure that your
-file contains only spaces and not tab characters. You can do this
-in Emacs by selecting the entire file (\texttt{C-x h}) and
-running \texttt{M-x untabify}.
-
-To make Emacs indent lines automatically by eight spaces, add this
-line to your \texttt{.emacs}:
-
-\begin{verbatim}
-        (add-hook 'c-mode-hook (lambda () (c-set-style "linux")))
-\end{verbatim}
-
-\subsection*{Comments}
-Write a comment before every function describing its purpose (one-line),
-its arguments, and its return value. Mention whether it is an internal
-function or an exported function.
-
-Write a comment before every structure describing its purpose, and
-write comments about each of its members.
-
-Follow the style shown below for comments, since such comments
-can then be automatically extracted by doxygen to generate
-documentation.
-
-\textsl{Example}:
-\begin{verbatim}
-/**
- * hash_name -hash function for filenames
- * @par:  parent inode number
- * @name: basename of inode
- * @mod:  number of buckets in the hashtable
- *
- * @return: success: bucket number
- *          failure: -1
- *
- * Not for external use.
- */
-\end{verbatim}
-
-\subsection*{Indicating critical sections}
-To clearly show regions of code which execute with locks held, use
-the following format:
-
-\begin{verbatim}
-        pthread_mutex_lock (&mutex);
-        {
-                /* code */
-        }
-        pthread_mutex_unlock (&mutex);
-\end{verbatim}
-
-\section*{A skeleton fop function}
-This is the recommended template for any fop. In the beginning come
-the initializations. After that, the `success' control flow should be
-linear.  Any error conditions should cause a \texttt{goto} to a single
-point, \texttt{out}.  At that point, the code should detect the error
-that has occurred and do appropriate cleanup.
-
-\begin{verbatim}
-int32_t
-sample_fop (call_frame_t *frame, xlator_t *this, ...)
-{
-        char *            var1     = NULL;
-        int32_t           op_ret   = -1;
-        int32_t           op_errno = 0;
-        DIR *             dir      = NULL;
-        struct posix_fd * pfd      = NULL;
-
-        VALIDATE_OR_GOTO (frame, out);
-        VALIDATE_OR_GOTO (this, out);
-
-        /* other validations */
-
-        dir = opendir (...);
-
-        if (dir == NULL) {
-                op_errno = errno;
-                gf_log (this->name, GF_LOG_ERROR,
-                        "opendir failed on %s (%s)", loc->path,
-                        strerror (op_errno));
-                goto out;
-        }
-
-        /* another system call */
-        if (...) {
-                op_errno = ENOMEM;
-                gf_log (this->name, GF_LOG_ERROR,
-                        "out of memory :(");
-                goto out;
-        }
-
-        /* ... */
-
- out:
-        if (op_ret == -1) {
-
-          /* check for all the cleanup that needs to be
-             done */
-
-                if (dir) {
-                        closedir (dir);
-                        dir = NULL;
-                }
-
-                if (pfd) {
-                        FREE (pfd->path);
-                        FREE (pfd);
-                        pfd = NULL;
-                }
-        }
-
-        STACK_UNWIND (frame, op_ret, op_errno, fd);
-        return 0;
-}
-\end{verbatim}
-
-\end{document}
diff --git a/doc/debugging/analyzing-regression-cores.md b/doc/debugging/analyzing-regression-cores.md
new file mode 100644
index 00000000000..5e10f41c6eb
--- /dev/null
+++ b/doc/debugging/analyzing-regression-cores.md
@@ -0,0 +1,54 @@
+# Analyzing Regression Cores
+This document explains how to analyze core-dumps obtained from regression machines, with examples.
+1. Download the core-tarball and extract it.
+2. `cd` into directory where the tarball is extracted.
+```
+[sh]# pwd
+/home/user/Downloads
+[sh]# ls
+build  build-install-20150625_05_42_39.tar.bz2  lib64  usr
+```
+3. Determine the core file you need to examine. There can be more than one core file. You can list them from './build/install/cores' directory.
+```
+[sh]# ls build/install/cores/
+core.9341  liblist.txt  liblist.txt.tmp
+```
+In case you are unsure which binary generated the core-file, executing 'file' command on it will help.
+```
+[sh]# file ./build/install/cores/core.9341 
+./build/install/cores/core.9341: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/build/install/sbin/glusterfsd -s slave26.cloud.gluster.org --volfile-id patchy'
+```
+As seen, the core file was generated by glusterfsd binary, and path to it is provided (/build/install/sbin/glusterfsd).
+
+4. Now, run the following command on the core:
+```
+gdb -ex 'set sysroot ./' -ex 'core-file ./build/install/cores/core.xxx' <target, say ./build/install/sbin/glusterd>
+In this case,
+gdb -ex 'set sysroot ./' -ex 'core-file ./build/install/cores/core.9341' ./build/install/sbin/glusterfsd
+```
+5. You can cross check if all shared libraries are available and loaded by using 'info sharedlibrary' command from inside gdb.
+6. Once verified, usual gdb commands based on requirement can be used to debug the core.
+   `bt` or `backtrace` from gdb of core used in examples:
+```
+Core was generated by `/build/install/sbin/glusterfsd -s slave26.cloud.gluster.org --volfile-id patchy'.
+Program terminated with signal SIGABRT, Aborted.
+#0  0x00007f512a54e625 in raise () from ./lib64/libc.so.6
+(gdb) bt
+#0  0x00007f512a54e625 in raise () from ./lib64/libc.so.6
+#1  0x00007f512a54fe05 in abort () from ./lib64/libc.so.6
+#2  0x00007f512a54774e in __assert_fail_base () from ./lib64/libc.so.6
+#3  0x00007f512a547810 in __assert_fail () from ./lib64/libc.so.6
+#4  0x00007f512b9fc434 in __gf_free (free_ptr=0x7f50f4000e50) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/mem-pool.c:304
+#5  0x00007f512b9b6657 in loc_wipe (loc=0x7f510c20d1a0) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/xlator.c:685
+#6  0x00007f511cb8201d in mq_start_quota_txn_v2 (this=0x7f5118019b60, loc=0x7f510c20d2b8, ctx=0x7f50f4000bf0, contri=0x7f50f4000d60)
+    at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/features/marker/src/marker-quota.c:2921
+#7  0x00007f511cb82c55 in mq_initiate_quota_task (opaque=0x7f510c20d2b0) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/features/marker/src/marker-quota.c:3199
+#8  0x00007f511cb81820 in mq_synctask (this=0x7f5118019b60, task=0x7f511cb829fa <mq_initiate_quota_task>, spawn=_gf_false, loc=0x7f510c20d430, dict=0x0, buf=0x0, contri=0)
+    at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/features/marker/src/marker-quota.c:2789
+#9  0x00007f511cb82f82 in mq_initiate_quota_blocking_txn (this=0x7f5118019b60, loc=0x7f510c20d430) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/features/marker/src/marker-quota.c:3230
+#10 0x00007f511cb82844 in mq_reduce_parent_size_task (opaque=0x7f510c000df0) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/features/marker/src/marker-quota.c:3117
+#11 0x00007f512ba0f9dc in synctask_wrap (old_task=0x7f510c0053e0) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/syncop.c:370
+#12 0x00007f512a55f8f0 in ?? () from ./lib64/libc.so.6
+#13 0x0000000000000000 in ?? ()
+(gdb) 
+```
diff --git a/doc/debugging/gfid-to-path.md b/doc/debugging/gfid-to-path.md
index 09c459e52c8..1917bf2cca1 100644
--- a/doc/debugging/gfid-to-path.md
+++ b/doc/debugging/gfid-to-path.md
@@ -1,37 +1,37 @@
-#Convert GFID to Path
+# Convert GFID to Path
 
 GlusterFS internal file identifier (GFID) is a uuid that is unique to each
 file across the entire cluster. This is analogous to inode number in a
 normal filesystem. The GFID of a file is stored in its xattr named
 `trusted.gfid`.
 
-####Special mount using [gfid-access translator][1]:
-~~~
+#### Special mount using [gfid-access translator][1]:
+```
 mount -t glusterfs -o aux-gfid-mount vm1:test /mnt/testvol
-~~~
+```
 
 Assuming, you have `GFID` of a file from changelog (or somewhere else).
 For trying this out, you can get `GFID` of a file from mountpoint:
-~~~
+```
 getfattr -n glusterfs.gfid.string /mnt/testvol/dir/file
-~~~
+```
 
 
 ---
-###Get file path from GFID (Method 1):
+### Get file path from GFID (Method 1):
 **(Lists hardlinks delimited by `:`, returns path as seen from mountpoint)**
 
-####Turn on build-pgfid option
-~~~
+#### Turn on build-pgfid option
+```
 gluster volume set test build-pgfid on
-~~~
+```
 Read virtual xattr `glusterfs.ancestry.path` which contains the file path
-~~~
+```
 getfattr -n glusterfs.ancestry.path -e text /mnt/testvol/.gfid/<GFID>
-~~~
+```
 
 **Example:**
-~~~
+```
 [root@vm1 glusterfs]# ls -il /mnt/testvol/dir/
 total 1
 10610563327990022372 -rw-r--r--. 2 root root 3 Jul 17 18:05 file
@@ -46,28 +46,23 @@ glusterfs.gfid.string="11118443-1894-4273-9340-4b212fa1c0e4"
 getfattr: Removing leading '/' from absolute path names
 # file: mnt/testvol/.gfid/11118443-1894-4273-9340-4b212fa1c0e4
 glusterfs.ancestry.path="/dir/file:/dir/file3"
-~~~
+```
 
 ---
-###Get file path from GFID (Method 2):
+### Get file path from GFID (Method 2):
 **(Does not list all hardlinks, returns backend brick path)**
-~~~
+```
 getfattr -n trusted.glusterfs.pathinfo -e text /mnt/testvol/.gfid/<GFID>
-~~~
+```
 
 **Example:**
-~~~
+```
 [root@vm1 glusterfs]# getfattr -n trusted.glusterfs.pathinfo -e text /mnt/testvol/.gfid/11118443-1894-4273-9340-4b212fa1c0e4
 getfattr: Removing leading '/' from absolute path names
 # file: mnt/testvol/.gfid/11118443-1894-4273-9340-4b212fa1c0e4
 trusted.glusterfs.pathinfo="(<DISTRIBUTE:test-dht> <POSIX(/mnt/brick-test/b):vm1:/mnt/brick-test/b/dir//file3>)"
-~~~
+```
 
 ---
-###Get file path from GFID (Method 3):
-https://gist.github.com/semiosis/4392640
-
----
-####References and links:
+#### References and links:
 [posix: placeholders for GFID to path conversion](http://review.gluster.org/5951)
-[1]: https://github.com/gluster/glusterfs/blob/master/doc/features/gfid-access.md
diff --git a/doc/debugging/mem-alloc-list.md b/doc/debugging/mem-alloc-list.md
new file mode 100644
index 00000000000..1c68e65d323
--- /dev/null
+++ b/doc/debugging/mem-alloc-list.md
@@ -0,0 +1,19 @@
+## Viewing Memory Allocations
+
+While statedumps provide stats of the number of allocations, size etc for a
+particular mem type, there is no easy way to examine all the allocated objects of that type
+in memory.Being able to view this information could help with determining how an object is used,
+and if there are any memory leaks.
+
+The mem_acct_rec structures have been updated to include lists to which the allocated object is
+added. These can be examined in gdb using simple scripts.
+
+`gdb> plist xl->mem_acct.rec[$type]->obj_list`
+
+will print out the pointers of all allocations of $type.
+
+These changes are primarily targeted at developers and need to enabled
+at compile-time using `configure --enable-debug`.
+
+
+
diff --git a/doc/debugging/split-brain.md b/doc/debugging/split-brain.md
index b0d938e26bc..6b122c40551 100644
--- a/doc/debugging/split-brain.md
+++ b/doc/debugging/split-brain.md
@@ -1,33 +1,36 @@
-Steps to recover from File split-brain.
-======================================
-
-Quick Start:
-============
-1. Get the path of the file that is in split-brain:  
->  It can be obtained either by  
->       a) The command `gluster volume heal info split-brain`.  
->       b) Identify the files for which file operations performed
-           from the client keep failing with Input/Output error.
-
-2. Close the applications that opened this file from the mount point.
+# Steps to recover from File split-brain
+This document contains steps to recover from a file split-brain.
+## Quick Start:
+### Step 1. Get the path of the file that is in split-brain:
+It can be obtained either by
+1. The command `gluster volume heal info split-brain`.
+2. Identify the files for which file operations performed from the client keep failing with Input/Output error.
+
+### Step 2. Close the applications that opened this file from the mount point.
 In case of VMs, they need to be powered-off.
 
-3. Decide on the correct copy:  
-> This is done by observing the afr changelog extended attributes of the file on
+### Step 3. Decide on the correct copy:  
+This is done by observing the afr changelog extended attributes of the file on
 the bricks using the getfattr command; then identifying the type of split-brain 
 (data split-brain, metadata split-brain, entry split-brain or split-brain due to
 gfid-mismatch); and finally determining which of the bricks contains the 'good copy'
 of the file.  
->   `getfattr -d -m . -e hex <file-path-on-brick>`.  
+```
+getfattr -d -m . -e hex <file-path-on-brick>
+```
+  
 It is also possible that one brick might contain the correct data while the
 other might contain the correct metadata.
 
-4. Reset the relevant extended attribute on the brick(s) that contains the
-'bad copy' of the file data/metadata using the setfattr command.  
->   `setfattr -n <attribute-name> -v <attribute-value> <file-path-on-brick>`
+### Step 4. Reset the relevant extended attribute on the brick(s) that contains the 'bad copy' of the file data/metadata using the setfattr command.  
+```
+setfattr -n <attribute-name> -v <attribute-value> <file-path-on-brick>
+```
 
-5. Trigger self-heal on the file by performing lookup from the client:  
->   `ls -l <file-path-on-gluster-mount>`
+### Step 5. Trigger self-heal on the file by performing lookup from the client:  
+```
+ls -l <file-path-on-gluster-mount>
+```
 
 Detailed Instructions for steps 3 through 5:  
 ===========================================
@@ -36,13 +39,15 @@ afr changelog extended attributes.
 
 Execute `getfattr -d -m . -e hex <file-path-on-brick>`
 
-* Example:  
+Example:  
+```
 [root@store3 ~]# getfattr -d -e hex -m. brick-a/file.txt  
 \#file: brick-a/file.txt  
 security.selinux=0x726f6f743a6f626a6563745f723a66696c655f743a733000  
 trusted.afr.vol-client-2=0x000000000000000000000000  
 trusted.afr.vol-client-3=0x000000000200000000000000  
 trusted.gfid=0x307a5c9efddd4e7c96e94fd4bcdcbd1b  
+```
 
 The extended attributes with `trusted.afr.<volname>-client-<subvolume-index>`
 are used by afr to maintain changelog of the file.The values of the
@@ -51,10 +56,11 @@ client (fuse or nfs-server) processes. When the glusterfs client modifies a file
 or directory, the client contacts each brick and updates the changelog extended 
 attribute according to the response of the brick.
 
-'subvolume-index' is nothing but (brick number - 1) in
+`subvolume-index` is nothing but (brick number - 1) in
 `gluster volume info <volname>` output.
 
-* Example:  
+Example:
+```
 [root@pranithk-laptop ~]# gluster volume info vol  
  Volume Name: vol  
  Type: Distributed-Replicate  
@@ -71,6 +77,7 @@ attribute according to the response of the brick.
  brick-f: pranithk-laptop:/gfs/brick-f  
  brick-g: pranithk-laptop:/gfs/brick-g  
  brick-h: pranithk-laptop:/gfs/brick-h  
+```
 
 In the example above:  
 ```
@@ -91,12 +98,15 @@ present in all the other bricks in it's replica set as seen by that brick.
 
 In the example volume given above, all files in brick-a will have 2 entries, 
 one for itself and the other for the file present in it's replica pair, i.e.brick-b:  
+```
 trusted.afr.vol-client-0=0x000000000000000000000000 -->changelog for itself (brick-a)  
 trusted.afr.vol-client-1=0x000000000000000000000000 -->changelog for brick-b as seen by brick-a  
-
+```
 Likewise, all files in brick-b will have:  
+```
 trusted.afr.vol-client-0=0x000000000000000000000000 -->changelog for brick-a as seen by brick-b  
 trusted.afr.vol-client-1=0x000000000000000000000000 -->changelog for itself (brick-b)  
+```
 
 The same can be extended for other replica pairs.  
 
@@ -122,7 +132,8 @@ When a file split-brain happens it could be either data split-brain or
 meta-data split-brain or both. When a split-brain happens the changelog of the
 file would be something like this:  
 
-* Example:(Lets consider both data, metadata split-brain on same file).  
+Example:(Lets consider both data, metadata split-brain on same file).  
+```
 [root@pranithk-laptop vol]# getfattr -d -m . -e hex /gfs/brick-?/a  
 getfattr: Removing leading '/' from absolute path names  
 \#file: gfs/brick-a/a  
@@ -133,10 +144,11 @@ trusted.gfid=0x80acdbd886524f6fbefa21fc356fed57
 trusted.afr.vol-client-0=0x000003b00000000100000000  
 trusted.afr.vol-client-1=0x000000000000000000000000  
 trusted.gfid=0x80acdbd886524f6fbefa21fc356fed57  
+```
 
-###Observations:
+### Observations:
 
-####According to changelog extended attributes on file /gfs/brick-a/a:  
+#### According to changelog extended attributes on file /gfs/brick-a/a:  
 The first 8 digits of trusted.afr.vol-client-0 are all
 zeros (0x00000000................), and the first 8 digits of
 trusted.afr.vol-client-1 are not all zeros (0x000003d7................).
@@ -149,7 +161,7 @@ trusted.afr.vol-client-1 are not all zeros (0x........00000001........).
 So the changelog on /gfs/brick-a/a implies that some metadata operations succeeded 
 on itself but failed on /gfs/brick-b/a.
 
-####According to Changelog extended attributes on file /gfs/brick-b/a:  
+#### According to Changelog extended attributes on file /gfs/brick-b/a:  
 The first 8 digits of trusted.afr.vol-client-0 are not all
 zeros (0x000003b0................), and the first 8 digits of
 trusted.afr.vol-client-1 are all zeros (0x00000000................).
@@ -205,6 +217,7 @@ Hence execute
 `setfattr -n trusted.afr.vol-client-1 -v 0x000003d70000000000000000 /gfs/brick-a/a`
 
 Thus after the above operations are done, the changelogs look like this:  
+```
 [root@pranithk-laptop vol]# getfattr -d -m . -e hex /gfs/brick-?/a  
 getfattr: Removing leading '/' from absolute path names  
 \#file: gfs/brick-a/a  
@@ -216,7 +229,7 @@ trusted.gfid=0x80acdbd886524f6fbefa21fc356fed57
 trusted.afr.vol-client-0=0x000000000000000100000000  
 trusted.afr.vol-client-1=0x000000000000000000000000  
 trusted.gfid=0x80acdbd886524f6fbefa21fc356fed57  
-
+```
 
 Triggering Self-heal:
 ---------------------
@@ -243,9 +256,9 @@ needs to be removed.The gfid-link files are present in the .glusterfs folder
 in the top-level directory of the brick. If the gfid of the file is
 0x307a5c9efddd4e7c96e94fd4bcdcbd1b (the trusted.gfid extended attribute got
 from the getfattr command earlier),the gfid-link file can be found at
-> /gfs/brick-a/.glusterfs/30/7a/307a5c9efddd4e7c96e94fd4bcdcbd1b
+`/gfs/brick-a/.glusterfs/30/7a/307a5c9efddd4e7c96e94fd4bcdcbd1b`
 
-####Word of caution:
+#### Word of caution:
 Before deleting the gfid-link, we have to ensure that there are no hard links
 to the file present on that brick. If hard-links exist,they must be deleted as
 well.
diff --git a/doc/debugging/statedump.md b/doc/debugging/statedump.md
index f34a5c3436a..9dfdce15fad 100644
--- a/doc/debugging/statedump.md
+++ b/doc/debugging/statedump.md
@@ -1,31 +1,53 @@
-#Statedump
+# Statedump
 Statedump is a file generated by glusterfs process with different data structure state which may contain the active inodes, fds, mempools, iobufs, memory allocation stats of different types of datastructures per xlator etc.
 
-##How to generate statedump
-We can find the directory where statedump files are created using 'gluster --print-statedumpdir' command.
+## How to generate statedump
+We can find the directory where statedump files are created using `gluster --print-statedumpdir` command.
 Create that directory if not already present based on the type of installation.
 Lets call this directory `statedump-directory`.
 
-We can generate statedump using 'kill -USR1 <pid-of-gluster-process>'.
+We can generate statedump using `kill -USR1 <pid-of-gluster-process>`.
 gluster-process is nothing but glusterd/glusterfs/glusterfsd process.
 
 There are also commands to generate statedumps for brick processes/nfs server/quotad
 
-For bricks:     `gluster volume statedump <volname>`
+For bricks: 
+```
+gluster volume statedump <volname>
+```
 
-For nfs server: `gluster volume statedump <volname> nfs`
+For nfs server:
+```
+gluster volume statedump <volname> nfs
+```
 
-For quotad:     `gluster volume statedump <volname> quotad`
+For quotad:
+```
+gluster volume statedump <volname> quotad
+```
 
 For brick-processes files will be created in `statedump-directory` with name of the file as `hyphenated-brick-path.<pid>.dump.timestamp`. For all other processes it will be `glusterdump.<pid>.dump.timestamp`.
 
-##How to read statedump
+For applications using libgfapi, `SIGUSR1` cannot be used, eg: smbd/libvirtd
+processes could have used the `SIGUSR1` signal already for other purposes.
+To generate statedump for the processes, using libgfapi, below command can be
+executed from one of the nodes in the gluster cluster to which the libgfapi
+application is connected to.
+```
+gluster volume statedump <volname> client <hostname>:<process id>
+```
+The statedumps can be found in the `statedump-directory`, the name of the
+statedumps being `glusterdump.<pid>.dump.timestamp`. For a process there can be
+multiple such files created depending on the number of times the volume is
+accessed by the process (related to the number of `glfs_init()` calls).
+
+## How to read statedump
 We shall see snippets of each type of statedump.
 
 First and last lines of the file have starting and ending time of writing the statedump file. Times will be in UTC timezone.
 
 mallinfo return status is printed in the following format. Please read man mallinfo for more information about what each field means.
-###Mallinfo
+### Mallinfo
 ```
 [mallinfo]
 mallinfo_arena=100020224 /* Non-mmapped space allocated (bytes) */
@@ -40,7 +62,7 @@ mallinfo_fordblks=3310112 /* Total free space (bytes) */
 mallinfo_keepcost=133712 /* Top-most, releasable space (bytes) */
 ```
 
-###Data structure allocation stats
+### Data structure allocation stats
 For every xlator data structure memory per translator loaded in the call-graph is displayed in the following format:
 
 For xlator with name: glusterfs
@@ -61,7 +83,7 @@ max_num_allocs=3 #Maximum number of active allocations at any point in the life
 total_allocs=7 #Number of times this data is allocated in the life of the process.
 ```
 
-###Mempools
+### Mempools
 
 Mempools are optimization to reduce the number of allocations of a data type. If we create a mem-pool of lets say 1024 elements for a data-type, new elements will be allocated from heap using syscalls like calloc, only if all the 1024 elements in the pool are in active use.
 
@@ -81,7 +103,7 @@ cur-stdalloc=0 #Denotes the number of allocations made from heap once cold-count
 max-stdalloc=0 #Maximum number of allocations from heap that are in active use at any point in the life of the process.
 ```
 
-###Iobufs
+### Iobufs
 ```
 [iobuf.global]
 iobuf_pool=0x1f0d970 #The memory pool for iobufs
@@ -92,7 +114,7 @@ iobuf_pool.arena_cnt=8 #Total number of arenas in the pool
 iobuf_pool.request_misses=0 #The number of iobufs that were stdalloc'd (as they exceeded the default max page size provided by iobuf_pool).
 ```
 
-There are 3 lists of arenas
+There are 3 lists of arenas:
 
 1. Arena list: arenas allocated during iobuf pool creation and the arenas that are in use(active_cnt != 0) will be part of this list.
 2. Purge list: arenas that can be purged(no active iobufs, active_cnt == 0).
@@ -129,7 +151,7 @@ arena.6.active_iobuf.2.ptr=0x7fdb92189000
 
 At any given point in time if there are lots of filled arenas then that could be a sign of iobuf leaks.
 
-###Call stack
+### Call stack
 All the fops received by gluster are handled using call-stacks. Call stack contains the information about uid/gid/pid etc of the process that is executing the fop. Each call-stack contains different call-frames per xlator which handles that fop.
 
 ```
@@ -144,7 +166,7 @@ op=LOOKUP #Fop
 type=1 #Type of the op i.e. FOP/MGMT-OP
 cnt=9 #Number of frames in this stack.
 ```
-###Call-frame
+### Call-frame
 Each frame will have information about which xlator the frame belongs to, what is the function it wound to/from and will be unwind to. It also mentions if the unwind happened or not. If we observe hangs in the system and want to find out which xlator is causing it. Take a statedump and see what is the final xlator which is yet to be unwound.
 
 ```
@@ -159,7 +181,7 @@ wind_to=priv->children[i]->fops->lookup
 unwind_to=afr_lookup_cbk #Parent xlator function to which unwind happened
 ```
 
-###History of operations in Fuse
+### History of operations in Fuse
 
 Fuse maintains history of operations that happened in fuse.
 
@@ -175,7 +197,7 @@ TIME=2014-07-09 16:44:57.523394
 message=[0] fuse_getattr_resume: 4591, STAT, path: (/iozone.tmp), gfid: (3afb4968-5100-478d-91e9-76264e634c9f)
 ```
 
-###Xlator configuration
+### Xlator configuration
 ```
 [cluster/replicate.r2-replicate-0] #Xlator type, name information
 child_count=2 #Number of children to the xlator
@@ -195,7 +217,7 @@ favorite_child=-1
 wait_count=1
 ```
 
-###Graph/inode table
+### Graph/inode table
 ```
 [active graph - 1]
 
@@ -207,7 +229,7 @@ conn.1.bound_xl./data/brick01a/homegfs.lru_size=183 #Number of inodes present in
 conn.1.bound_xl./data/brick01a/homegfs.purge_size=0 #Number of inodes present in purge list
 ```
 
-###Inode
+### Inode
 ```
 [conn.1.bound_xl./data/brick01a/homegfs.active.324] #324th inode in active inode list
 gfid=e6d337cf-97eb-44b3-9492-379ba3f6ad42 #Gfid of the inode
@@ -215,6 +237,7 @@ nlookup=13 #Number of times lookups happened from the client or from fuse kernel
 fd-count=4 #Number of fds opened on the inode
 ref=11 #Number of refs taken on the inode
 ia_type=1 #Type of the inode. This should be changed to some string :-(
+Ref by xl:.patchy-md-cache=11 #Further this there will be a list of xlators, and the ref count taken by each of them on this inode at the time of statedump
 
 [conn.1.bound_xl./data/brick01a/homegfs.lru.1] #1st inode in lru list. Note that ref count is zero for these inodes.
 gfid=5114574e-69bc-412b-9e52-f13ff087c6fc
@@ -222,8 +245,10 @@ nlookup=5
 fd-count=0
 ref=0
 ia_type=2
+Ref by xl:.fuse=1
+Ref by xl:.patchy-client-0=-1
 ```
-###Inode context
+### Inode context
 For each inode per xlator some context could be stored. This context can also be printed in the statedump. Here is the inode ctx of locks xlator
 ```
 [xlator.features.locks.homegfs-locks.inode]
@@ -240,12 +265,12 @@ lock-dump.domain.domain=homegfs-replicate-0 #Domain name where entry/data operat
 inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=11141120, len=131072, pid = 18446744073709551615, owner=080b1ada117f0000, client=0xb7fc30, connection-id=compute-30-029.com-3505-2014/06/29-14:46:12:477358-homegfs-client-0-0-1, granted at Sun Jun 29 11:10:36 2014 #Active lock information
 ```
 
-##FAQ
-###How to debug Memory leaks using statedump?
+## FAQ
+### How to debug Memory leaks using statedump?
 
-####Using memory accounting feature:
+#### Using memory accounting feature:
 
-`https://bugzilla.redhat.com/show_bug.cgi?id=1120151` is one of the bugs which was debugged using statedump to see which data-structure is leaking. Here is the process used to find what the leak is using statedump. According to the bug the observation is that the process memory usage is increasing whenever one of the bricks is wiped in a replicate volume and a `full` self-heal is invoked to heal the contents. Statedump of the process is taken using kill -USR1 `<pid-of-gluster-self-heal-daemon>`.
+[Bug 1120151](https://bugzilla.redhat.com/show_bug.cgi?id=1120151) is one of the bugs which was debugged using statedump to see which data-structure is leaking. Here is the process used to find what the leak is using statedump. According to the bug the observation is that the process memory usage is increasing whenever one of the bricks is wiped in a replicate volume and a `full` self-heal is invoked to heal the contents. Statedump of the process is taken using `kill -USR1 <pid-of-gluster-self-heal-daemon>`.
 ```
 grep -w num_allocs glusterdump.5225.dump.1405493251
 num_allocs=77078
@@ -268,10 +293,10 @@ grep of the statedump revealed too many allocations for the following data-types
 3. gf_common_mt_mem_pool.
 
 After checking afr-code for allocations with tag `gf_common_mt_char` found `data-self-heal` code path does not free one such allocated memory. `gf_common_mt_mem_pool` suggests that there is a leak in pool memory. `replicate-0:dict_t`, `glusterfs:data_t` and `glusterfs:data_pair_t` pools are using lot of memory, i.e. cold_count is `0` and too many allocations. Checking source code of dict.c revealed that `key` in `dict` is allocated with `gf_common_mt_char` i.e. `2.` tag and value is created using gf_asprintf which in-turn uses `gf_common_mt_asprintf` i.e. `1.`. Browsing the code for leak in self-heal code paths lead to a line which over-writes a variable with new dictionary even when it was already holding a reference to another dictionary. After fixing these leaks, ran the same test to verify that none of the `num_allocs` are increasing even after healing 10,000 files directory hierarchy in statedump of self-heal daemon.
-Please check http://review.gluster.org/8316 for more info about patch/code.
+Please check this [patch](http://review.gluster.org/8316) for more info about the fix.
 
-####Debugging leaks in memory pools:
-Statedump output of memory pools was used to test and verify the fixes to https://bugzilla.redhat.com/show_bug.cgi?id=1134221. On code analysis, dict_t objects were found to be leaking (in terms of not being unref'd enough number of times, during name self-heal. The test involved creating 100 files on plain replicate volume, removing them from one of the bricks's backend, and then triggering lookup on them from the mount point. Statedump of the mount process was taken before executing the test case and after it, after compiling glusterfs with -DDEBUG flags (to have cold count set to 0 by default).
+#### Debugging leaks in memory pools:
+Statedump output of memory pools was used to test and verify the fixes to [Bug 1134221](https://bugzilla.redhat.com/show_bug.cgi?id=1134221). On code analysis, dict_t objects were found to be leaking (in terms of not being unref'd enough number of times, during name self-heal. The test involved creating 100 files on plain replicate volume, removing them from one of the brick's backend, and then triggering lookup on them from the mount point. Statedump of the mount process was taken before executing the test case and after it, after compiling glusterfs with -DDEBUG flags (to have cold count set to 0 by default).
 
 Statedump output of the fuse mount process before the test case was executed:
 
@@ -303,7 +328,7 @@ cur-stdalloc=214
 max-stdalloc=220
 
 ```
-Here, with cold count being 0 by default, cur-stdalloc indicated the number of dict_t objects that were allocated in heap using mem_get(), and yet to be freed using mem_put() (refer to https://github.com/gluster/glusterfs/blob/master/doc/data-structures/mem-pool.md for more details on how mempool works). After the test case (name selfheal of 100 files), there was a rise in the cur-stdalloc value (from 14 to 214) for dict_t.
+Here, with cold count being 0 by default, `cur-stdalloc` indicated the number of `dict_t` objects that were allocated in heap using `mem_get()`, and yet to be freed using `mem_put()` (refer to this [page](../developer-guide/datastructure-mem-pool.md) for more details on how mempool works). After the test case (name selfheal of 100 files), there was a rise in the cur-stdalloc value (from 14 to 214) for `dict_t`.
 
 After these leaks were fixed, glusterfs was again compiled with -DDEBUG flags, and the same steps were performed again and statedump was taken before and after executing the test case, of the mount. This was done to ascertain the validity of the fix. And the following are the results:
 
@@ -337,8 +362,8 @@ max-stdalloc=119
 ```
 The value of cur-stdalloc remained 14 before and after the test, indicating that the fix indeed does what it's supposed to do.
 
-###How to debug hangs because of frame-loss?
-`https://bugzilla.redhat.com/show_bug.cgi?id=994959` is one of the bugs where statedump was helpful in finding where the frame was lost. Here is the process used to find where the hang is using statedump.
+### How to debug hangs because of frame-loss?
+[Bug 994959](https://bugzilla.redhat.com/show_bug.cgi?id=994959) is one of the bugs where statedump was helpful in finding where the frame was lost. Here is the process used to find where the hang is using statedump.
 When the hang was observed, statedumps are taken for all the processes. On mount's statedump the following stack is shown:
 ```
 [global.callpool.stack.1.frame.1]
@@ -386,4 +411,4 @@ unwind_to=qr_readdirp_cbk
 ```
 `unwind_to` shows that call was unwound to `afr_readdirp_cbk` from client xlator.
 Inspecting that function revealed that afr is not unwinding the stack when fop failed.
-Check http://review.gluster.org/5531 for more info about patch/code changes.
+Check this [patch](http://review.gluster.org/5531) for more info about the fix.
diff --git a/doc/developer-guide/Language-Bindings.md b/doc/developer-guide/Language-Bindings.md
new file mode 100644
index 00000000000..951f5fae2f6
--- /dev/null
+++ b/doc/developer-guide/Language-Bindings.md
@@ -0,0 +1,45 @@
+# Language Bindings
+GlusterFS 3.4 introduced the libgfapi client API for C programs. This
+page lists bindings to the libgfapi C library from other languages.
+
+Go
+--
+
+-   [gogfapi](https://github.com/gluster/gogfapi) - Go language bindings
+    for libgfapi, aiming to provide an api consistent with the default
+    Go file apis.
+
+Java
+----
+
+-   [libgfapi-jni](https://github.com/semiosis/libgfapi-jni/) - Low
+    level JNI binding for libgfapi
+-   [glusterfs-java-filesystem](https://github.com/semiosis/glusterfs-java-filesystem)
+    - High level NIO.2 FileSystem Provider implementation for the Java
+    platform
+-   [libgfapi-java-io](https://github.com/gluster/libgfapi-java-io) -
+    Java bindings for libgfapi, similar to java.io
+
+Python
+------
+
+-   [libgfapi-python](https://github.com/gluster/libgfapi-python) -
+    Libgfapi bindings for Python
+
+Ruby
+----
+
+-   [libgfapi-ruby](https://github.com/spajus/libgfapi-ruby) - Libgfapi
+    bindings for Ruby using FFI
+
+Rust
+----
+
+-   [gfapi-sys](https://github.com/cholcombe973/Gfapi-sys) - Libgfapi
+    bindings for Rust using FFI
+
+Perl
+----
+
+-   [libgfapi-perl](https://github.com/gluster/libgfapi-perl) - Libgfapi
+    bindings for Perl using FFI
diff --git a/doc/developer-guide/README.md b/doc/developer-guide/README.md
new file mode 100644
index 00000000000..aaf9c7476b0
--- /dev/null
+++ b/doc/developer-guide/README.md
@@ -0,0 +1,81 @@
+Developers
+==========
+
+### From GlusterDocumentation
+
+Contributing to the Gluster community
+-------------------------------------
+
+Are you itching to send in patches and participate as a developer in the
+Gluster community? Here are a number of starting points for getting
+involved. We don't require a signed contributor license agreement or
+copyright assignment, but we do require a "signed-off-by" line on each
+code check-in.
+
+-   [License
+    Change](http://www.gluster.org/2012/05/glusterfs-license-change/) -
+    we recently changed the client library code to a dual license under
+    the GPL v2 and the LGPL v3 or later
+-   [GlusterFS Coding Standards](./coding-standard.md)
+
+-   If you are not sure of where to start, and what to do, we have a small
+    write-up on what you can pick. [Check it out](./options-to-contribute.md)
+
+
+Adding File operations
+----------------------
+
+-   [Steps to be followed when adding a new FOP to GlusterFS ](./adding-fops.md)
+
+Automatic File Replication
+--------------------------
+
+-   [Cluster/afr translator](./afr.md)
+-   [History of Locking in AFR](./afr-locks-evolution.md)
+-   [Self heal Daemon](./afr-self-heal-daemon.md)
+
+Data Structures
+---------------
+
+-   [inode data structure](./datastructure-inode.md)
+-   [iobuf data structure](./datastructure-iobuf.md)
+-   [mem-pool data structure](./datastructure-mem-pool.md)
+
+Find the gfapi symbol versions [here](./gfapi-symbol-versions.md)
+
+Daemon Management Framework
+---------------------------
+
+-   [How to introduce new daemons using daemon management framework](./daemon-management-framework.md)
+
+Translators
+-----------
+
+-   [Performance/write-Behind Translator](./write-behind.md)
+-   [Translator Development](./translator-development.md)
+-   [Storage/posix Translator](./posix.md)
+
+
+Brick multiplex
+---------------
+
+-   [Brick mux resource reduction](./brickmux-thread-reduction.md)
+
+Fuse
+----
+
+-   [Interrupt handling](./fuse-interrupt.md)
+
+Testing/Debugging
+-----------------
+
+-   [Unit Tests in GlusterFS](./unittest.md)
+-   [Using the Gluster Test
+    Framework](./Using-Gluster-Test-Framework.md) - Step by
+    step instructions for running the Gluster Test Framework
+-   [Coredump Analysis](../debugging/analyzing-regression-cores.md) - Steps to analize coredumps generated by regression machines.
+-   [Identifying Resource Leaks](./identifying-resource-leaks.md)
+
+Release Process
+---------------
+-   [Versioning](./versioning.md)
diff --git a/doc/developer-guide/Using-Gluster-Test-Framework.md b/doc/developer-guide/Using-Gluster-Test-Framework.md
new file mode 100644
index 00000000000..d2bb1c391da
--- /dev/null
+++ b/doc/developer-guide/Using-Gluster-Test-Framework.md
@@ -0,0 +1,271 @@
+# USing Gluster Test Framwork
+Description
+-----------
+
+The Gluster Test Framework, is a suite of scripts used for regression
+testing of Gluster.
+
+It runs well on RHEL and CentOS (possibly Fedora too, presently being
+tested), and is automatically run against every patch submitted to
+Gluster [for review](http://review.gluster.org).
+
+The Gluster Test Framework is part of the main Gluster code base, living
+under the "tests" subdirectory:
+
+		http://git.gluster.org/?p=glusterfs.git;a=summary
+
+WARNING
+-------
+
+Running the Gluster Test Framework deletes “/var/lib/glusterd/\*”.
+
+**DO NOT run it on a server with any data.**
+
+Preparation steps for Ubuntu 14.04 LTS
+--------------------------------------
+
+1. \# apt-get install dbench git libacl1-dev mock nfs-common
+nfs-kernel-server libtest-harness-perl libyajl-dev xfsprogs psmisc attr
+acl lvm2 rpm
+
+2. \# apt-get install python-webob python-paste python-sphinx
+
+3. \# apt-get install autoconf automake bison dos2unix flex libfuse-dev
+libaio-dev libibverbs-dev librdmacm-dev libtool libxml2-dev
+libxml2-utils liblvm2-dev make libssl-dev pkg-config libpython-dev
+python-eventlet python-netifaces python-simplejson python-pyxattr
+libreadline-dev tar
+
+4) Install cmockery2 from github (https://github.com/lpabon/cmockery2)
+and compile and make install as in Readme
+
+5)
+
+		sudo groupadd mock
+		sudo useradd -g mock mock
+
+6) mkdir /var/run/gluster
+
+**Note**: redhat-rpm-config package is not found in ubuntu
+
+Preparation steps for CentOS 7 (only)
+-------------------------------------
+
+1. Install EPEL:
+
+		$ sudo yum install -y http://epel.mirror.net.in/epel/7/x86_64/e/epel-release-7-1.noarch.rpm
+
+2. Install the CentOS 7.x dependencies:
+
+		$ sudo yum install -y --enablerepo=epel cmockery2-devel dbench git libacl-devel mock nfs-utils perl-Test-Harness yajl xfsprogs psmisc
+
+		$ sudo yum install -y --enablerepo=epel python-webob1.0 python-paste-deploy1.5 python-sphinx10 redhat-rpm-config
+
+==\> Despite below missing packages it worked for me
+
+		No package python-webob1.0 available.
+		No package python-paste-deploy1.5 available.
+		No package python-sphinx10 available.
+
+		$ sudo yum install -y --enablerepo=epel autoconf automake bison dos2unix flex fuse-devel libaio-devel libibverbs-devel \
+		 librdmacm-devel libtool libxml2-devel lvm2-devel make openssl-devel pkgconfig \
+		 python-devel python-eventlet python-netifaces python-paste-deploy \
+		 python-simplejson python-sphinx python-webob pyxattr readline-devel rpm-build \
+		 tar
+
+3. Create the mock user
+
+		$ sudo useradd -g mock mock
+
+Preparation steps for CentOS 6.3+ (only)
+----------------------------------------
+
+1. Install EPEL:
+
+		$ sudo yum install -y http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
+
+2. Install the CentOS 6.x dependencies:
+
+		$ sudo yum install -y --enablerepo=epel cmockery2-devel dbench git libacl-devel mock nfs-utils perl-Test-Harness yajl xfsprogs
+		$ sudo yum install -y --enablerepo=epel python-webob1.0 python-paste-deploy1.5 python-sphinx10 redhat-rpm-config
+		$ sudo yum install -y --enablerepo=epel autoconf automake bison dos2unix flex fuse-devel libaio-devel libibverbs-devel \
+		  librdmacm-devel libtool libxml2-devel lvm2-devel make openssl-devel pkgconfig \
+		  python-devel python-eventlet python-netifaces python-paste-deploy \
+		  python-simplejson python-sphinx python-webob pyxattr readline-devel rpm-build \
+		  tar
+
+3. Create the mock user
+
+		$ sudo useradd -g mock mock
+
+Preparation steps for RHEL 6.3+ (only)
+--------------------------------------
+
+1. Ensure you have the "Scalable Filesystem Support" group installed
+
+This provides the xfsprogs package, which is required by the test
+framework.
+
+2. Install EPEL:
+
+		$ sudo yum install -y http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
+
+3. Install the CentOS 6.x dependencies:
+
+		$ sudo yum install -y --enablerepo=epel cmockery2-devel dbench git libacl-devel mock nfs-utils yajl perl-Test-Harness
+		$ sudo yum install -y --enablerepo=rhel-6-server-optional-rpms python-webob1.0 python-paste-deploy1.5 python-sphinx10 redhat-rpm-config
+		$ sudo yum install -y --disablerepo=rhs* --enablerepo=*optional-rpms autoconf \
+		  automake bison dos2unix flex fuse-devel libaio-devel libibverbs-devel \
+		  librdmacm-devel libtool libxml2-devel lvm2-devel make openssl-devel pkgconfig \
+		  python-devel python-eventlet python-netifaces python-paste-deploy \
+		  python-simplejson python-sphinx python-webob pyxattr readline-devel rpm-build \
+		  tar
+
+4. Create the mock user
+
+		$ sudo useradd -g mock mock
+
+Preparation steps for Fedora 16-19 (only)
+-----------------------------------------
+
+**Still in development**
+
+1. Install the Fedora dependencies:
+
+		$ sudo yum install -y attr cmockery2-devel dbench git mock nfs-utils perl-Test-Harness psmisc xfsprogs
+		$ sudo yum install -y python-webob1.0 python-paste-deploy1.5 python-sphinx10 redhat-rpm-config
+		$ sudo yum install -y autoconf automake bison dos2unix flex fuse-devel libaio-devel libibverbs-devel \
+		  librdmacm-devel libtool libxml2-devel lvm2-devel make openssl-devel pkgconfig \
+		  python-devel python-eventlet python-netifaces python-paste-deploy \
+		  python-simplejson python-sphinx python-webob pyxattr readline-devel rpm-build \
+		  tar
+
+3. Create the mock user
+
+		$ sudo useradd -g mock mock
+
+Common steps
+------------
+
+1. Ensure DNS for your server is working
+
+The Gluster Test Framework fails miserably if the full domain name for
+your server doesn't resolve back to itself.
+
+If you don't have a working DNS infrastructure in place, adding an entry
+for your server to its /etc/hosts file will work.
+
+2. Install the version of Gluster you are testing
+
+Either install an existing set of rpms:
+
+		$ sudo yum install [your gluster rpms here]
+
+Or compile your own ones (fairly easy):
+
+	http://www.gluster.org/community/documentation/index.php/CompilingRPMS
+
+3. Clone the GlusterFS git repository
+
+		$ git clone git://git.gluster.org/glusterfs
+		$ cd glusterfs
+
+Ensure mock can access the directory
+------------------------------------
+
+Some tests run as the user "mock". If the mock user can't access the
+tests subdirectory directory, these tests fail. (rpm.t is one such test)
+
+This is a known gotcha when the git repo is cloned to your home
+directory. Home directories generally don't have world readable
+permissions. You can fix this by adjusting your home directory
+permissions, or placing the git repo somewhere else (with access for the
+mock user).
+
+Running the tests
+-----------------
+
+The tests need to run as root, so they can mount volumes and manage
+gluster processes as needed.
+
+It's also best to run them directly as the root user, instead of through
+sudo. Strange things sporadicly happen (for me) when using the full test
+framework through sudo, that haven't happened (yet) when running
+directly as root. Hangs in dbench particularly, which are part of at
+least one test.
+
+		# ./run-tests.sh
+
+The test framework takes just over 45 minutes to run in a VM here (4
+cpu's assigned, 8GB ram, SSD storage). It may take significantly more or
+less time for you, depending on the hardware and software you're using.
+
+Showing debug information
+-------------------------
+
+To display verbose information while the tests are running, set the
+DEBUG environment variable to 1 prior to running the tests.
+
+		# DEBUG=1 ./run-tests.sh
+
+Log files
+---------
+
+Verbose output from the rpm.t test goes into "rpmbuild-mock.log",
+located in the same directory the test is run from.
+
+Reporting bugs
+--------------
+
+If you hit a bug when running the test framework, **please** create a
+bug report for it on Bugzilla so it gets fixed:
+
+	https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS&component=tests
+
+Creating your own tests
+-----------------------
+
+The test scripts are written in bash, with their filenames ending in .t
+instead of .sh.
+
+When creating your own test scripts, create them in an appropriate
+subdirectory under "tests" (eg "bugs" or "features") and use descriptive
+names like "bug-XXXXXXX-checking-feature-X.t"
+
+Also include the "include.rc" file, which defines the test types and
+host/brick/volume defaults:
+
+		. $(dirname $0)/../include.rc
+
+There are 5 test types available at present, but feel free to add more
+if you need something that doesn't yet exist. The test types are
+explained in more detail below.
+
+Also essential is the "cleanup" command, which removes any existing
+Gluster configuration (**without backing it up**), and also kills any
+running gluster processes.
+
+There is a basic test template you can copy, named bug-000000.t in the
+bugs subdirectory:
+
+		$ cp bugs/bug-000000.t somedir/descriptive-name.t
+
+### TEST
+
+-   Example of usage in basic/volume.t
+
+### TEST\_IN\_LOOP
+
+-   Example of usage in basic/rpm.t
+
+### EXPECT
+
+-   Example of usage in basic/volume.t
+
+### EXPECT\_WITHIN
+
+-   Example of usage in basic/volume-status.t
+
+### EXPECT\_KEYWORD
+
+-   Defined in include.rc, but seems to be unused?
diff --git a/doc/hacker-guide/en-US/markdown/adding-fops.md b/doc/developer-guide/adding-fops.md
index 3f72ed3e23a..3f72ed3e23a 100644
--- a/doc/hacker-guide/en-US/markdown/adding-fops.md
+++ b/doc/developer-guide/adding-fops.md
diff --git a/doc/code/xlators/cluster/afr/afr-locks-evolution.md b/doc/developer-guide/afr-locks-evolution.md
index 7d2a136d871..2dabbcfeb13 100644
--- a/doc/code/xlators/cluster/afr/afr-locks-evolution.md
+++ b/doc/developer-guide/afr-locks-evolution.md
@@ -32,10 +32,10 @@ AFR makes use of locks xlator extensively:
 * For Entry self-heal, it is `entrylk(NULL name, parent inode)`. Specifying NULL for the name takes full lock on the directory referred to by the inode.
 * For data self-heal, there is a bit of history as to how locks evolved:
 
-###Initial version (say version 1) :
+### Initial version (say version 1) :
 There was no concept of selfheal daemon (shd). Only client lookups triggered heals. so AFR always took `inodelk(0,0,DATA_DOMAIN)` for healing. The issue with this approach was that when heal was in progress, I/O from clients was blocked .
 
-###version 2:
+### version 2:
 shd was introduced. We needed to allow I/O to go through when heal was going,provided the ranges did not overlap. To that extent, the following approach was adopted:
 
 + 1.shd takes (full inodelk in DATA_DOMAIN). Thus client FOPS are blocked and cannot modify changelog-xattrs
@@ -79,7 +79,7 @@ It modifies data but the FOP succeeds only on brick 2. writev returns success, a
 and thus goes ahead and copies stale 128Kb from brick 1 to brick2. Thus as far as application is concerned, `writev` returned success but bricks have stale data.
 What needs to be done is `writev` must return success only if it succeeded on atleast one source brick (brick b1 in this case). Otherwise  The heal still happens in reverse direction but as far as the application is concerned, it received an error.  
 
-###Note on lock **domains**
+### Note on lock **domains**
 We have used conceptual names in this document like DATA_DOMAIN/ METADATA_DOMAIN/ SELF_HEAL_DOMAIN. In the code, these are mapped to strings that are based on the AFR xlator name like so:
 
 DATA_DOMAIN     --->"vol_name-replicate-n"
diff --git a/doc/developer-guide/afr-self-heal-daemon.md b/doc/developer-guide/afr-self-heal-daemon.md
new file mode 100644
index 00000000000..65940d420b7
--- /dev/null
+++ b/doc/developer-guide/afr-self-heal-daemon.md
@@ -0,0 +1,92 @@
+Self-Heal Daemon
+================
+The self-heal daemon (shd) is a glusterfs process that is responsible for healing files in a replicate/ disperse gluster volume.
+Every server (brick) node of the volume runs one instance of the shd. So even if one node contains replicate/ disperse bricks of
+multiple volumes, it would be healed by the same shd.
+
+This document only describes how the shd works for replicate (AFR) volumes.
+
+The shd is launched by glusterd when the volume starts (only if the volume includes a replicate configuration). The graph
+of the shd process in every node contains the following: The io-stats which is the top most xlator, its children being the
+replicate xlators (subvolumes) of *only* the bricks present in that particular node, and finally *all* the client xlators that are the children of the replicate xlators.
+
+The shd does two types of self-heal crawls: Index heal and Full heal. For both these types of crawls, the basic idea is the same:  
+For each file encountered while crawling, perform metadata, data and entry heals under appropriate locks.  
+* An overview of how each of these heals is performed is detailed in the 'Self-healing' section of *doc/features/afr-v1.md*
+* The different file locks which the shd takes for each of these heals is detailed in *doc/developer
+-guide/afr-locks-evolution.md*
+
+Metadata heal refers to healing extended attributes, mode and permissions of a file or directory.
+Data heal refers to healing the file contents.
+Entry self-heal refers to healing entries inside a directory.
+
+Index heal
+==========
+The index heal is done:  
+  a) Every 600 seconds (can be changed via the `cluster.heal-timeout` volume option)  
+  b) When it is explicitly triggered via the `gluster vol heal <VOLNAME>` command  
+  c) Whenever a replica brick that was down comes back up.  
+  
+Only one heal can be in progress at one time, irrespective of reason why it was triggered. If another heal is triggered before the first one completes, it will be queued.
+Only one heal can be queued while the first one is running. If an Index heal is queued, it can be overridden by queuing a Full heal and not vice-versa.  Also, before processing
+each entry in index heal, a check is made if a full heal is queued. If it is, then the index heal is aborted so that the full heal can proceed. 
+
+In index heal, each shd reads the entries present inside .glusterfs/indices/xattrop/ folder and triggers heal on each entry with appropriate locks.
+The .glusterfs/indices/xattrop/ directory contains a base entry of the name "xattrop-<virtual-gfid-string>". All other entries are hardlinks to the base entry. The
+*names* of the hardlinks are the gfid strings of the files that may need heal. 
+
+When a client (mount) performs an operation on the file, the index xlator present in each brick process adds the hardlinks in the pre-op phase of the FOP's transaction
+and removes it in post-op phase if the operation is successful. Thus if an entry is present inside the .glusterfs/indices/xattrop/ directory when there is no I/O 
+happening on the file, it means the file needs healing (or atleast an examination if the brick crashed after the post-op completed but just before the removal of the hardlink).
+
+#### Index heal steps:
+<pre><code>
+In shd process of *each node* {
+        opendir +readdir (.glusterfs/indices/xattrop/)
+        for each entry inside it {
+                self_heal_entry() //Explained below.
+        }
+}
+</code></pre>
+
+<pre><code>
+self_heal_entry() {
+        Call syncop_lookup(replicae subvolume) which eventually does {
+                take appropriate locks
+                determine source and sinks from AFR changelog xattrs	
+                perform whatever heal is needed (any of metadata, data and entry heal in that order)
+                clear changelog xattrs and hardlink inside .glusterfs/indices/xattrop/
+        }
+}
+</code></pre>
+
+Note:
+* If the gfid hardlink is present in the .glusterfs/indices/xattrop/ of both replica bricks, then each shd will try to heal the file but only one of them will be able to proceed due to the self-heal domain lock.
+
+* While processing entries inside .glusterfs/indices/xattrop/, if shd encounters an entry whose parent is yet to be healed, it will skip it and it will be picked up in the next crawl.
+
+* If a file is in data/ metadata split-brain, it will not be healed.
+
+* If a directory is in entry split-brain, a conservative merge will be performed, wherein after the merge, the entries of the directory will be a union of the entries in the replica pairs.
+
+Full heal
+=========
+A full heal is triggered by running `gluster vol heal <VOLNAME> full`. This command is usually run in disk replacement scenarios where the entire data is to be copied from one of the healthy bricks of the replica to the brick that was just replaced.
+
+Unlike the index heal which runs on the shd of every node in a replicate subvolume, the full heal is run only on the shd of one node per replicate subvolume: the node having the highest UUID.
+i.e In a 2x2 volume made of 4 nodes N1, N2, N3 and N4, If UUID of N1>N2 and UUID N4 >N3, then the full crawl is carried out by the shds of N1 and N4.(Node UUID can be found in `/var/lib/glusterd/glusterd.info`)
+
+The full heal steps are almost identical to the index heal, except the heal is performed on each replica starting from the root of the volume:
+<pre><code>
+In shd process of *highest UUID node per replica* {
+        opendir +readdir ("/")
+        for each entry inside it {
+                self_heal_entry()
+                if (entry == directory) {
+                        /* Recurse*/
+                        again opendir+readdir (directory) followed by self_heal_entry() of each entry.
+                }
+                
+        }
+}
+</code></pre>
diff --git a/doc/hacker-guide/en-US/markdown/afr.md b/doc/developer-guide/afr.md
index 566573a4e26..566573a4e26 100644
--- a/doc/hacker-guide/en-US/markdown/afr.md
+++ b/doc/developer-guide/afr.md
diff --git a/doc/developer-guide/brickmux-thread-reduction.md b/doc/developer-guide/brickmux-thread-reduction.md
new file mode 100644
index 00000000000..7d76e8ff579
--- /dev/null
+++ b/doc/developer-guide/brickmux-thread-reduction.md
@@ -0,0 +1,64 @@
+# Resource usage reduction in brick multiplexing
+
+Each brick is regresented with a graph of translators in a brick process.
+Each translator in the graph has its own set of threads and mem pools
+and other system resources allocations. Most of the times all these
+resources are not put to full use. Reducing the resource consumption
+of each brick is a problem in itself that needs to be addressed. The other
+aspect to it is, sharing of resources across brick graph, this becomes
+critical in brick multiplexing scenario. In this document we will be discussing
+only about the threads.
+
+If a brick mux process hosts 50 bricks there are atleast 600+ threads created
+in that process. Some of these are global threads that are shared by all the
+brick graphs, and others are per translator threads. The global threads like
+synctask threads, timer threads, sigwaiter, poller etc. are configurable and
+do not needs to be reduced. The per translator threads keeps growing as the
+number of bricks in the process increases. Each brick spawns atleast 10+
+threads:
+- io-threads
+- posix threads:
+     1. Janitor
+     2. Fsyncer
+     3. Helper
+     4. aio-thread
+- changelog and bitrot threads(even when the features are not enabled)
+
+## io-threads
+
+io-threads should be made global to the process, having 16+ threads for
+each brick does not make sense. But io-thread translator is loaded in
+the graph, and the position of io-thread translator decides from when
+the fops will be parallelised across threads. We cannot entirely move
+the io-threads to libglusterfs and say the multiplexing happens from
+the master translator or so. Hence, the io-thread orchestrator code
+is moved to libglusterfs, which ensures there is only one set of
+io-threads that is shared among the io-threads translator in each brick.
+This poses performance issues due to lock-contention in the io-threds
+layer. This also shall be addressed by having multiple locks instead of
+one global lock for io-threads.
+
+## Posix threads
+Most of the posix threads execute tasks in a timely manner, hence it can be
+replaced with a timer whose handler register a task to synctask framework, once
+the task is complete, the timer is registered again. With this we can eliminate
+the need of one thread for each task. The problem with using synctasks is
+the performance impact it will have due to make/swapcontext. For task that
+does not involve network wait, we need not do makecontext, instead the task
+function with arg can be stored and executed when a synctask thread is free.
+We need to implement an api in synctask to execute atomic tasks(no network wait)
+without the overhead of make/swapcontext. This will solve the performance
+impact associated with using synctask framework.
+
+And the other challenge, is to cancel all the tasks pending from a translator.
+This is important to cleanly detach brick. For this, we need to implement an
+api in synctask that can cancel all the tasks from a given translator.
+
+For future, this will be replced to use global thread-pool(once implemented).
+
+## Changelog and bitrot threads
+
+In the initial implementation, the threads are not created if the feature is
+not enabled. We need to share threads across changelog instances if we plan
+to enable these features in brick mux scenario.
+
diff --git a/doc/developer-guide/coding-standard.md b/doc/developer-guide/coding-standard.md
new file mode 100644
index 00000000000..031c6c0da99
--- /dev/null
+++ b/doc/developer-guide/coding-standard.md
@@ -0,0 +1,697 @@
+GlusterFS Coding Standards
+==========================
+
+Before you get started
+----------------------
+Before starting with other part of coding standard, install `clang-format`
+
+On Fedora:
+```
+$ dnf install clang
+```
+On debian/Ubuntu:
+```
+$ apt-get install clang
+```
+Once you are done with all the local changes, you need to run below set of commands,
+before submitting the patch for review.
+```
+$ git add $file # if any
+$ git commit -a -s -m "commit message"
+$ git show --pretty="format:" --name-only | grep -v "contrib/" | egrep "*\.[ch]$" | xargs clang-format -i
+$ git diff # see if there are any changes
+$ git commit -a --amend # get the format changes done
+$ ./submit-for-review.sh
+```
+
+
+Structure definitions should have a comment per member
+------------------------------------------------------
+
+Every member in a structure definition must have a comment about its purpose.
+The comment should be descriptive without being overly verbose.  For pointer
+members, lifecycle concerns for the pointed-to object should be noted.  For lock
+members, the relationship between the lock member and the other members it
+protects should be explicit.
+
+*Bad:*
+
+```
+gf_lock_t   lock;           /* lock */
+```
+
+*Good:*
+
+```
+DBTYPE      access_mode;    /* access mode for accessing
+                             * the databases, can be
+                             * DB_HASH, DB_BTREE
+                             * (option access-mode <mode>)
+                             */
+```
+
+Structure members should be aligned based on the padding requirements
+---------------------------------------------------------------------
+
+The compiler will make sure that structure members have optimum alignment,
+but at the expense of suboptimal padding. More important is to optimize the
+padding. The compiler won't do that for you.
+
+This also will help utilize the memory better
+
+*Bad:*
+```
+struct bad {
+    bool  b; /* 0 */
+             /* 1..7 pad */
+    void *p; /* 8..15 */
+    char  c; /* 16 */
+    char  a[16]; /* 17..33 */
+             /* 34..39 pad */
+    int64_t ii; /* 40..47 */
+    int32_t i; /* 48..51 */
+             /* 52..55 pad */
+    int64_t iii; /* 56..63 */
+};
+```
+
+*Good:*
+```
+struct good {
+   int64_t ii; /* explicit 64-bit types */
+   void *p; /* may be 64- or 32-bit */
+   long  l;   /* may be 64- or 32-bit */
+   int   i;   /* 32-bit */
+   short s;   /* 16-bit */
+   char  c;   /* 8-bit */
+   bool  b;   /* 8-bit */
+   char  a[1024];
+);
+```
+Make sure the items with the most stringent alignment requirements will need
+to come earliest (ie, pointers and  perhaps uint64_t etc), and those with less
+stringent alignment requirements at the end (uint16/uint8 and char). Also note
+that the long array (if any) should be at the end of the structure, regardless
+of the type.
+
+Also note, if your structure's overall size is crossing 1k-4k limit, it is
+recommended to mention the reason why the particular structure needs so much
+memory as a comment at the top.
+
+Use \_typename for struct tags and typename\_t for typedefs
+---------------------------------------------------------
+
+Being consistent here makes it possible to automate navigation from use of a
+type to its true definition (not just the typedef).
+
+*Bad:*
+
+```
+struct thing {...};
+struct thing_t {...};
+typedef struct _thing thing;
+```
+
+*Good:*
+
+```
+typedef struct _thing {...} thing_t;
+```
+
+No double underscores
+---------------------
+
+Identifiers beginning with double underscores are supposed to reserved for the
+compiler.
+
+http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf
+
+When you need to define inner/outer functions, use a different prefix/suffix.
+
+*Bad:*
+
+```
+void __do_something (void);
+
+void
+do_something (void)
+{
+        LOCK ();
+        __do_something ();
+        UNLOCK ();
+}
+```
+
+*Good:*
+
+```
+void do_something_locked (void);
+```
+
+Only use safe pointers in initializers
+----------------------------------------------------------
+
+Some pointers, such as `this` in a fop function, can be assumed to be non-NULL.
+However, other parameters and further-derived values might be NULL.
+
+*Good:*
+
+```
+pid_t pid     = frame->root->pid;
+```
+
+
+*Bad:*
+
+```
+data_t *my_data = dict_get (xdata, "fubar");
+```
+
+No giant stack allocations
+--------------------------
+
+Synctasks have small finite stacks.  To avoid overflowing these stacks, avoid
+allocating any large data structures on the stack.  Use dynamic allocation
+instead.
+
+*Bad:*
+
+```
+gf_boolean_t port_inuse[65536]; /* 256KB, this actually happened */
+```
+
+NOTE: Ideal is to limit the stack array to less than 256 bytes.
+
+
+Character array initializing
+----------------------------
+
+It is recommended to keep the character array initializing to empty string.
+
+*Good:*
+```
+char msg[1024] = "";
+```
+
+Not so much recommended, even though it means the same.
+
+```
+char msg[1024] = {0,};
+```
+
+We recommend above to structure initialization.
+
+
+
+Validate all arguments to a function
+------------------------------------
+
+All pointer arguments to a function must be checked for `NULL`.
+A macro named `GF_VALIDATE_OR_GOTO` (in `common-utils.h`)
+takes two arguments; if the first is `NULL`, it writes a log message and
+jumps to a label specified by the second aergument after setting errno
+appropriately. There are several variants of this function for more
+specific purposes, and their use is recommended.
+
+*Bad:*
+
+```
+/* top of function */
+ret = dict_get (xdata, ...)
+```
+
+*Good:*
+
+```
+/* top of function */
+GF_VALIDATE_OR_GOTO(xdata,out);
+ret = dict_get (xdata, ...)
+```
+
+Never rely on precedence of operators
+-------------------------------------
+
+Never write code that relies on the precedence of operators to execute
+correctly.  Such code can be hard to read and someone else might not
+know the precedence of operators as accurately as you do.  This includes
+precedence of increment/decrement vs. field/subscript.  The only exceptions are
+arithmetic operators (which have had defined precedence since before computers
+even existed) and boolean negation.
+
+*Bad:*
+
+```
+if (op_ret == -1 && errno != ENOENT)
+++foo->bar      /* incrementing foo, or incrementing foo->bar? */
+a && b || !c
+```
+
+*Good:*
+
+```
+if ((op_ret == -1) && (errno != ENOENT))
+(++foo)->bar
+++(foo->bar)
+(a && b) || !c
+a && (b || !c)
+```
+
+Use exactly matching types
+--------------------------
+
+Use a variable of the exact type declared in the manual to hold the
+return value of a function. Do not use an 'equivalent' type.
+
+
+*Bad:*
+
+```
+int len = strlen (path);
+```
+
+*Good:*
+
+```
+size_t len = strlen (path);
+```
+
+Avoid code such as `foo->bar->baz`; check every pointer
+-------------------------------------------------------------
+
+Do not write code that blindly follows a chain of pointer references. Any
+pointer in the chain may be `NULL` and thus cause a crash. Verify that each
+pointer is non-null before following it.  Even if `foo->bar` has been checked
+and is known safe, repeating it can make code more verbose and less clear.
+
+This rule includes `[]` as well as `->` because both dereference pointers.
+
+*Bad:*
+
+```
+foo->bar->field1 = value1;
+xyz = foo->bar->field2 + foo->bar->field3 * foo->bar->field4;
+foo->bar[5].baz
+```
+
+*Good:*
+
+```
+my_bar = foo->bar;
+if (!my_bar) ... return;
+my_bar->field1 = value1;
+xyz = my_bar->field2 + my_bar->field3 * my_bar->field4;
+```
+
+Document unchecked return values
+----------------------------------------------------
+
+In general, return values should be checked.  If a function is being called
+for its side effects and the return value really doesn't matter, an explicit
+cast to void is required (to keep static analyzers happy) and a comment is
+recommended.
+
+*Bad:*
+
+```
+close (fd);
+do_important_thing ();
+```
+
+*Good (or at least OK):*
+
+```
+(void) sleep (1);
+```
+
+Gracefully handle failure of malloc (and other allocation functions)
+--------------------------------------------------------------------
+
+GlusterFS should never crash or exit due to lack of memory. If a
+memory allocation fails, the call should be unwound and an error
+returned to the user.
+
+*Use result args and reserve the return value to indicate success or failure:*
+
+The return value of every functions must indicate success or failure (unless
+it is impossible for the function to fail --- e.g., boolean functions). If
+the function needs to return additional data, it must be returned using a
+result (pointer) argument.
+
+*Bad:*
+
+```
+int32_t dict_get_int32 (dict_t *this, char *key);
+```
+
+*Good:*
+
+```
+int dict_get_int32 (dict_t *this, char *key, int32_t *val);
+```
+
+Always use the 'n' versions of string functions
+-----------------------------------------------
+
+Unless impossible, use the length-limited versions of the string functions.
+
+*Bad:*
+
+```
+strcpy (entry_path, real_path);
+```
+
+*Good:*
+
+```
+strncpy (entry_path, real_path, entry_path_len);
+```
+
+Do not use memset prior to sprintf/snprintf/vsnprintf etc...
+------------------------------------------------------------
+snprintf(and other similar string functions) terminates the buffer with a
+'\0'(null character). Hence, there is no need to do a memset before using
+snprintf. (Of course you need to account one extra byte for the null character
+in your allocation).
+
+Note: Similarly if you are doing pre-memory allocation for the buffer, use
+GF_MALLOC instead of GF_CALLOC, since the later is bit costlier.
+
+*Bad:*
+
+```
+char buffer[x];
+memset (buffer, 0, x);
+bytes_read = snprintf (buffer, sizeof buffer, "bad standard");
+```
+
+*Good:*
+```
+char buffer[x];
+bytes_read = snprintf (buffer, sizeof (buffer), "good standard");
+```
+
+And it is always to good initialize the char array if the string is static.
+
+E.g.
+```
+char buffer[] = "good standard";
+```
+
+No dead or commented code
+-------------------------
+
+There must be no dead code (code to which control can never be passed) or
+commented out code in the codebase.
+
+Function length or Keep functions small
+---------------------------------------
+
+We live in the UNIX-world where modules do one thing and do it well.
+This rule should apply to our functions also. If a function is very long, try splitting it
+into many little helper functions. The question is, in a coding
+spree, how do we know a function is long and unreadable. One rule of
+thumb given by Linus Torvalds is that, a function should be broken-up
+if you have 4 or more levels of indentation going on for more than 3-4
+lines.
+
+*Example for a helper function:*
+```
+static int
+same_owner (posix_lock_t *l1, posix_lock_t *l2)
+{
+        return ((l1->client_pid == l2->client_pid) &&
+               (l1->transport  == l2->transport));
+}
+```
+
+Define functions as static
+--------------------------
+
+Declare functions as static unless they're exposed via a module-level API for
+use from other modules.
+
+No nested functions
+-------------------
+
+Nested functions have proven unreliable, e.g. as callbacks in code that uses
+ucontext (green) threads,
+
+Use inline functions instead of macros whenever possible
+--------------------------------------------------------
+
+Inline functions enforce type safety; macros do not.  Use macros only for things
+that explicitly need to be type-agnostic (e.g. cases where one might use
+generics or templates in other languages), or that use other preprocessor
+features such as `#` for stringification or `##` for token pasting.  In general,
+"static inline" is the preferred form.
+
+Avoid copypasta
+---------------
+
+Code that is copied and then pasted into multiple functions often creates
+maintenance problems later, e.g. updating all but one instance for a subsequent
+change.  If you find yourself copying the same "boilerplate" many places,
+consider refactoring to use helper functions (including inline) or macros, or
+code generation.
+
+Ensure function calls wrap around after 80-columns
+--------------------------------------------------
+
+Place remaining arguments on the next line if needed.
+
+Functions arguments and function definition
+-------------------------------------------
+
+Place all the arguments of a function definition on the same line
+until the line goes beyond 80-cols. Arguments that extend beyind
+80-cols should be placed on the next line.
+
+Style issues
+------------
+
+### Brace placement
+
+Use K&R/Linux style of brace placement for blocks.
+
+*Good:*
+
+```
+int some_function (...)
+{
+        if (...) {
+                /* ... */
+        } else if (...) {
+                /* ... */
+        } else {
+                /* ... */
+        }
+
+        do {
+                /* ... */
+        } while (cond);
+}
+```
+
+### Indentation
+
+Use *eight* spaces for indenting blocks. Ensure that your
+file contains only spaces and not tab characters. You can do this
+in Emacs by selecting the entire file (`C-x h`) and
+running `M-x untabify`.
+
+To make Emacs indent lines automatically by eight spaces, add this
+line to your `.emacs`:
+
+```
+(add-hook 'c-mode-hook (lambda () (c-set-style "linux")))
+```
+
+### Comments
+
+Write a comment before every function describing its purpose (one-line),
+its arguments, and its return value. Mention whether it is an internal
+function or an exported function.
+
+Write a comment before every structure describing its purpose, and
+write comments about each of its members.
+
+Follow the style shown below for comments, since such comments
+can then be automatically extracted by doxygen to generate
+documentation.
+
+*Good:*
+
+```
+/**
+* hash_name -hash function for filenames
+* @par:  parent inode number
+* @name: basename of inode
+* @mod:  number of buckets in the hashtable
+*
+* @return: success: bucket number
+*          failure: -1
+*
+* Not for external use.
+*/
+```
+
+### Indicating critical sections
+
+To clearly show regions of code which execute with locks held, use
+the following format:
+
+```
+pthread_mutex_lock (&mutex);
+{
+        /* code */
+}
+pthread_mutex_unlock (&mutex);
+```
+
+### Always use braces
+
+Even around single statements.
+
+*Bad:*
+
+```
+if (condition) action ();
+
+if (condition)
+        action ();
+```
+
+*Good:*
+
+```
+if (condition) {
+        action ();
+}
+```
+
+### Avoid multi-line conditionals
+
+These can be hard to read and even harder to modify later.  Predicate functions
+and helper variables are always better for maintainability.
+
+*Bad:*
+
+```
+if ((thing1 && other_complex_condition (thing1, lots, of, args))
+    || (!thing2 || even_more_complex_condition (thing2))
+    || all_sorts_of_stuff_with_thing3) {
+    return;
+}
+
+```
+
+*Better:*
+
+```
+thing1_ok = predicate1 (thing1, lots, of, args
+thing2_ok = predicate2 (thing2);
+thing3_ok = predicate3 (thing3);
+
+if (!thing1_ok || !thing2_ok || !thing3_ok) {
+        return;
+}
+```
+
+*Best:*
+
+```
+if (thing1 && other_complex_condition (thing1, lots, of, args)) {
+        return;
+}
+if (!thing2 || even_more_complex_condition (thing2)) {
+        /* Note potential for a different message here. */
+        return;
+}
+if (all_sorts_of_stuff_with_thing3) {
+        /* And here too. */
+        return;
+}
+```
+
+### Use 'const' liberally
+
+If a value isn't supposed/expected to change, there's no cost to adding a
+'const' keyword and it will help prevent violation of expectations.
+
+### Avoid global variables (including 'static' auto variables)
+Almost all state in Gluster is contextual and should be contained in the
+appropriate structure reflecting its scope (e.g. `call\_frame\_t`, `call\_stack\_t`,
+`xlator\_t`, `glusterfs\_ctx\_t`).  With dynamic loading and graph switches in play,
+each global requires careful consideration of when it should be initialized or
+reinitialized, when it might _accidentally_ be reinitialized, when its value
+might become stale, and so on.  A few global variables are needed to serve as
+'anchor points' for these structures, and more exceptions to the rule might be
+approved in the future, but new globals should not be added to the codebase
+without explicit approval.
+
+## A skeleton fop function
+
+This is the recommended template for any fop. In the beginning come the
+initializations. After that, the 'success' control flow should be linear.  Any
+error conditions should cause a `goto` to a label at the end.  By convention
+this is 'out' if there is only one such label, but a cascade of such labels is
+allowable to support multi-stage cleanup.  At that point, the code should detect
+the error that has occurred and do appropriate cleanup.
+
+```
+int32_t
+sample_fop (call_frame_t *frame, xlator_t *this, ...)
+{
+        char *            var1     = NULL;
+        int32_t           op_ret   = -1;
+        int32_t           op_errno = 0;
+        DIR *             dir      = NULL;
+        struct posix_fd * pfd      = NULL;
+
+        VALIDATE_OR_GOTO (frame, out);
+        VALIDATE_OR_GOTO (this, out);
+
+        /* other validations */
+
+        dir = opendir (...);
+
+        if (dir == NULL) {
+                op_errno = errno;
+                gf_log (this->name, GF_LOG_ERROR,
+                        "opendir failed on %s (%s)", loc->path,
+                        strerror (op_errno));
+                goto out;
+        }
+
+        /* another system call */
+        if (...) {
+                op_errno = ENOMEM;
+                gf_log (this->name, GF_LOG_ERROR,
+                        "out of memory :(");
+                goto out;
+        }
+
+        /* ... */
+
+ out:
+        if (op_ret == -1) {
+
+          /* check for all the cleanup that needs to be
+             done */
+
+                if (dir) {
+                        closedir (dir);
+                        dir = NULL;
+                }
+
+                if (pfd) {
+                        FREE (pfd->path);
+                        FREE (pfd);
+                        pfd = NULL;
+                }
+        }
+
+        STACK_UNWIND (frame, op_ret, op_errno, fd);
+        return 0;
+}
+```
diff --git a/doc/developer-guide/commit-guidelines.md b/doc/developer-guide/commit-guidelines.md
new file mode 100644
index 00000000000..38bbe525cbd
--- /dev/null
+++ b/doc/developer-guide/commit-guidelines.md
@@ -0,0 +1,136 @@
+## Git Commit Good Practice
+
+The following document is based on experience doing code development, bug troubleshooting and code review across a number of projects using Git. The document is mostly borrowed from [Open Stack](https://wiki.openstack.org/wiki/GitCommitMessages), but made more meaningful in the context of GlusterFS project.
+
+This topic can be split into two areas of concern
+
+* The structured set/split of the code changes
+* The information provided in the commit message
+
+### Executive Summary
+The points and examples that will be raised in this document ought to clearly demonstrate the value in splitting up changes into a sequence of individual commits, and the importance in writing good commit messages to go along with them. If these guidelines were widely applied it would result in a significant improvement in the quality of the GlusterFS Git history. Both a carrot & stick will be required to effect changes. This document intends to be the carrot by alerting people to the benefits, while anyone doing Gerrit code review can act as the stick ;-P
+
+In other words, when reviewing a change in Gerrit:
+* Do not simply look at the correctness of the code.
+* Review the commit message itself and request improvements to its content.
+* Look out for commits which are mixing multiple logical changes and require the submitter to split them into separate commits.
+* Ensure whitespace changes are not mixed in with functional changes.
+* Ensure no-op code refactoring is done separately from functional changes.
+
+And so on.
+
+It might be mentioned that Gerrit's handling of patch series is not entirely perfect. Let that not become a valid reason to avoid creating patch series. The tools being used should be subservient to developers needs, and since they are open source they can be fixed / improved. Software source code is "read mostly, write occassionally" and thus the most important criteria is to improve the long term maintainability by the large pool of developers in the community, and not to sacrifice too much for the sake of the single author who may never touch the code again.
+
+And now the long detailed guidelines & examples of good & bad practice
+
+### Structural split of changes
+The cardinal rule for creating good commits is to ensure there is only one "logical change" per commit. There are many reasons why this is an important rule:
+
+* The smaller the amount of code being changed, the quicker & easier it is to review & identify potential flaws.
+* If a change is found to be flawed later, it may be necessary to revert the broken commit. This is much easier to do if there are not other unrelated code changes entangled with the original commit.
+* When troubleshooting problems using Git's bisect capability, small well defined changes will aid in isolating exactly where the code problem was introduced.
+* When browsing history using Git annotate/blame, small well defined changes also aid in isolating exactly where & why a piece of code came from.
+
+#### Things to avoid when creating commits
+With the above points in mind, there are some commonly encountered examples of bad things to avoid
+
+* Mixing whitespace changes with functional code changes.
+
+The whitespace changes will obscure the important functional changes, making it harder for a reviewer to correctly determine whether the change is correct. Solution: Create 2 commits, one with the whitespace changes, one with the functional changes. Typically the whitespace change would be done first, but that need not be a hard rule.
+
+* Mixing two unrelated functional changes.
+
+Again the reviewer will find it harder to identify flaws if two unrelated changes are mixed together. If it becomes necessary to later revert a broken commit, the two unrelated changes will need to be untangled, with further risk of bug creation.
+
+* Sending large new features in a single giant commit.
+
+It may well be the case that the code for a new feature is only useful when all of it is present. This does not, however, imply that the entire feature should be provided in a single commit. New features often entail refactoring existing code. It is highly desirable that any refactoring is done in commits which are separate from those implementing the new feature. This helps reviewers and test suites validate that the refactoring has no unintentional functional changes.
+
+Even the newly written code can often be split up into multiple pieces that can be independently reviewed. For example, changes which add new internal fops or library functions, can be in self-contained commits. Again this leads to easier code review. It also allows other developers to cherry-pick small parts of the work, if the entire new feature is not immediately ready for merge. This will encourage the author & reviewers to think about the generic library functions' design, and not simply pick a design that is easier for their currently chosen internal implementation.
+
+The basic rule to follow is
+
+If a code change can be split into a sequence of patches/commits, then it should be split. Less is not more. More is more.
+
+##### Examples of bad practice
+
+TODO: Pick glusterfs specific example.
+
+
+##### Examples of good practice
+
+
+### Information in commit messages
+As important as the content of the change, is the content of the commit message describing it. When writing a commit message there are some important things to remember
+
+* Do not assume the reviewer understands what the original problem was.
+
+When reading bug reports, after a number of back & forth comments, it is often as clear as mud, what the root cause problem is. The commit message should have a clear statement as to what the original problem is. The bug is merely interesting historical background on /how/ the problem was identified. It should be possible to review a proposed patch for correctness without needing to read the bug ticket.
+
+* Do not assume the reviewer has access to external web services/site.
+
+In 6 months time when someone is on a train/plane/coach/beach/pub troubleshooting a problem & browsing Git history, there is no guarantee they will have access to the online bug tracker, or online blueprint documents. The great step forward with distributed SCM is that you no longer need to be "online" to have access to all information about the code repository. The commit message should be totally self-contained, to maintain that benefit.
+
+* Do not assume the code is self-evident/self-documenting.
+
+What is self-evident to one person, might be clear as mud to another person. Always document what the original problem was and how it is being fixed, for any change except the most obvious typos, or whitespace only commits.
+
+* Describe why a change is being made.
+
+A common mistake is to just document how the code has been written, without describing /why/ the developer chose to do it that way. By all means describe the overall code structure, particularly for large changes, but more importantly describe the intent/motivation behind the changes.
+
+* Read the commit message to see if it hints at improved code structure.
+
+Often when describing a large commit message, it becomes obvious that a commit should have in fact been split into 2 or more parts. Don't be afraid to go back and rebase the change to split it up into separate commits.
+
+* Ensure sufficient information to decide whether to review.
+
+When Gerrit sends out email alerts for new patch submissions there is minimal information included, principally the commit message and the list of files changes. Given the high volume of patches, it is not reasonable to expect all reviewers to examine the patches in detail. The commit message must thus contain sufficient information to alert the potential reviewers to the fact that this is a patch they need to look at.
+
+* The first commit line is the most important.
+
+In Git commits the first line of the commit message has special significance. It is used as email subject line, git annotate messages, gitk viewer annotations, merge commit messages and many more places where space is at a premium. As well as summarizing the change itself, it should take care to detail what part of the code is affected. eg if it is 'afr', 'dht' or any translator. Or in some cases, it can be touching all these components, but the commit message can be 'coverity:', 'txn-framework:', 'new-fop: ', etc.
+
+* Describe any limitations of the current code.
+
+If the code being changed still has future scope for improvements, or any known limitations then mention these in the commit message. This demonstrates to the reviewer that the broader picture has been considered and what tradeoffs have been done in terms of short term goals vs. long term wishes.
+
+* Do not include patch set-specific comments.
+
+In other words, if you rebase your change please don't add "Patch set 2: rebased" to your commit message. That isn't going to be relevant once your change has merged. Please do make a note of that in Gerrit as a comment on your change, however. It helps reviewers know what changed between patch sets. This also applies to comments such as "Added unit tests", "Fixed localization problems", or any other such patch set to patch set changes that don't affect the overall intent of your commit.
+
+**The main rule to follow is:**
+
+The commit message must contain all the information required to fully understand & review the patch for correctness. Less is not more. More is more.
+
+
+#### Including external references
+
+The commit message is primarily targeted towards human interpretation, but there is always some metadata provided for machine use. In the case of GlusterFS this includes at least the 'Change-id', "bug"/"feature" ID references and "Signed-off-by" tag (generated by 'git commit -s').
+
+The 'Change-id' line is a unique hash describing the change, which is generated by a Git commit hook. This should not be changed when rebasing a commit following review feedback, since it is used by Gerrit, to track versions of a patch.
+
+The 'bug' line can reference a bug in a few ways. Gerrit creates a link to the bug when viewing the patch on review.gluster.org so that reviewers can quickly access the bug/issue on Bugzilla or Github.
+
+**Fixes: bz#1601166** -- use 'Fixes: bz#NNNNN' if the commit is intended to fully fix and close the bug being referenced.
+**Fixes: #411** -- use 'Fixes: #NNN' if the patch fixes the github issue completely.
+
+**Updates: bz#1193929** -- use 'Updates: bz#NNNN' if the commit is only a partial fix and more work is needed.
+**Updates: #175** -- use 'Updates: #NNNN' if the commit is only a partial fix and more work is needed for the feature completion.
+
+We encourage the use of `Co-Authored-By: name <name@example.com>` in commit messages to indicate people who worked on a particular patch. It's a convention for recognizing multiple authors, and our projects would encourage the stats tools to observe it when collecting statistics.
+
+### Summary of Git commit message structure
+
+* Provide a brief description of the change in the first line.
+* The first line should be limited to 50 characters and should not end with a period.
+
+* Insert a single blank line after the first line.
+
+* Provide a detailed description of the change in the following lines, breaking paragraphs where needed.
+
+* Subsequent lines should be wrapped at 72 characters.
+
+Put the 'Change-id', 'Fixes bz#NNNNN' and 'Signed-off-by: <>' lines at the very end.
+
+TODO: Add good examples
diff --git a/doc/developer-guide/daemon-management-framework.md b/doc/developer-guide/daemon-management-framework.md
new file mode 100644
index 00000000000..592192e665d
--- /dev/null
+++ b/doc/developer-guide/daemon-management-framework.md
@@ -0,0 +1,38 @@
+
+How to introduce new daemons using daemon management framework
+==============================================================
+Glusterd manages GlusterFS daemons providing services like NFS, Proactive
+self-heal, Quota, User servicable snapshots etc. Following are some of the
+aspects that come under daemon management.
+
+Data members & functions of different management objects
+
+- **Connection Management**
+    - unix domain sockets based channel for internal communication
+    - rpc connection for the communication
+    - frame timeout value for UDS
+    - Methods - notify
+    - init, connect, termination, disconnect APIs can be invoked using the
+      connection management object
+
+- **Process Management**
+    - Name of the process
+    - pidfile to detect if the daemon is running
+    - loggging directory, log file, volfile, volfileserver & volfileid
+    - init, stop APIs can be invoked using the process management object
+
+- **Service Management**
+    - connection object
+    - process object
+    - online status
+    - Methods - manager, start, stop which can be abstracted as a common methods
+      or specific to service requirements
+    - init API can be invoked using the service management object
+
+ The above structures defines the skeleton of the daemon management framework.
+ Introduction of new daemons in GlusterFS needs to inherit these properties. Any
+ requirement specific to a daemon needs to be implemented in its own service
+ (for eg : snapd defines its own type glusterd_snapdsvc_t using glusterd_svc_t
+ and snapd specific data). New daemons will need to have its own service specific
+ code written in glusterd-<feature>-svc.h{c} and need to reuse the existing
+ framework.
diff --git a/doc/hacker-guide/en-US/markdown/inode.md b/doc/developer-guide/datastructure-inode.md
index a340ab9ca8e..45d7a941e5f 100644
--- a/doc/hacker-guide/en-US/markdown/inode.md
+++ b/doc/developer-guide/datastructure-inode.md
@@ -1,6 +1,6 @@
-#Inode and dentry management in GlusterFS:
+# Inode and dentry management in GlusterFS:
 
-##Background
+## Background
 Filesystems internally refer to files and directories via inodes. Inodes
 are unique identifiers of the entities stored in a filesystem. Whenever an
 application has to operate on a file/directory (read/modify), the filesystem
@@ -41,11 +41,10 @@ struct _inode_table {
 };
 ```
 
-#Life-cycle
+# Life-cycle
 ```
-
 inode_table_new (size_t lru_limit, xlator_t *xl)
-
+```
 This is a function which allocates a new inode table. Usually the top xlators in
 the graph such as protocol/server (for bricks), fuse and nfs (for fuse and nfs
 mounts) and libgfapi do inode managements. Hence they are the ones which will
@@ -59,11 +58,8 @@ new inode table.
 Thus an allocated inode table is destroyed only when the filesystem daemon is
 killed or unmounted.
 
-```
-
-#what it contains.
-```
 
+# what it contains.
 Inode table in glusterfs mainly contains a hash table for maintaining inodes.
 In general a file/directory is considered to be existing if there is a
 corresponding inode present in the inode table. If a inode for a file/directory
@@ -76,21 +72,21 @@ size of the hash table (as of now it is hard coded to 14057. The hash value of
 a inode is calculated using its gfid).
 
 Apart from the hash table, inode table also maintains 3 important list of inodes
-1) Active list:
+1. Active list:
 Active list contains all the active inodes (i.e inodes which are currently part
 of some fop).
-2) Lru list:
+2. Lru list:
 Least recently used inodes list. A limit can be set for the size of the lru
 list. For bricks it is 16384 and for clients it is infinity.
-3) Purge list:
+3. Purge list:
 List of all the inodes which have to be purged (i.e inodes which have to be
 deleted from the inode table due to unlink/rmdir/forget).
 
 And at last it also contains the mem-pool for allocating inodes, dentries so
 that frequent malloc/calloc and free of the data structures can be avoided.
-```
 
-#Data structure (inode)
+
+# Data structure (inode)
 ```
 struct _inode {
         inode_table_t       *table;         /* the table this inode belongs to */
@@ -108,7 +104,7 @@ struct _inode {
         struct _inode_ctx   *_ctx;          /* place holder for keeping the
         information about the inode by different xlators */
 };
-
+```
 As said above, inodes are internal way of identifying the files/directories. A
 inode uniquely represents a file/directory. A new inode is created whenever a
 create/mkdir/symlink/mknod operations are performed. Apart from that a new inode
@@ -128,9 +124,9 @@ inodes are those inodes whose refcount is greater than zero. Whenever some
 operation comes on a file/directory, and the resolver tries to find the inode
 for it, it increments the refcount of the inode before returning the inode. The
 refcount of an inode can be incremented by calling the below function
-
+```
 inode_ref (inode_t *inode)
-
+```
 Any xlator which wants to operate on a inode as part of some fop (or wants the
 inode in the callback), should hold a ref on the inode.
 Once the fop is completed before sending the reply of the fop to the above
@@ -139,18 +135,18 @@ zero, it is removed from the active inodes list and put into LRU list maintained
 by the inode table. Thus in short if some fop is happening on a file/directory,
 the corresponding inode will be in the active list or it will be in the LRU
 list.
-```
 
-#Life Cycle
+
+# Life Cycle
 
 A new inode is created whenever a new file/directory/symlink is created OR a
 successful lookup of an existing entry is done. The xlators which does inode
 management (as of now protocol/server, fuse, nfs, gfapi) will perform inode_link
 operation upon successful lookup or successful creation of a new entry.
-
+```
 inode_link (inode_t *inode, inode_t *parent, const char *name,
             struct iatt *buf);
-
+```
 inode_link actually adds the inode to the inode table (to be precise it adds
 the inode to the hash table maintained by the inode table. The hash value is
 calculated based on the gfid). Copies the gfid to the inode (the gfid is
@@ -160,7 +156,7 @@ A inode is removed from the inode table and eventually destroyed when unlink
 or rmdir operation is performed on a file/directory, or the the lru limit of
 the inode table has been exceeded.
 
-#Data structure (dentry)
+# Data structure (dentry)
 ```
 
 struct _dentry {
@@ -170,22 +166,22 @@ struct _dentry {
         char              *name;         /* name of the directory entry */
         inode_t           *parent;       /* directory of the entry */
 };
-
+```
 A dentry is the presence of an entry for a file/directory within its parent
 directory. A dentry usually points to the inode to which it belongs to. In
 glusterfs a dentry contains the following fields.
-1) a hook using which it can add itself to the list of
+1. a hook using which it can add itself to the list of
 the dentries maintained by the inode to which it points to.
-2) A hash table pointer.
-3) Pointer to the inode to which it belongs to.
-4) Name of the dentry
-5) Pointer to the inode of the parent directory in which the dentry is present
+2. A hash table pointer.
+3. Pointer to the inode to which it belongs to.
+4. Name of the dentry
+5. Pointer to the inode of the parent directory in which the dentry is present
 
 A new dentry is created when a new file/directory/symlink is created or a hard
 link to an existing file is created.
-
+```
 __dentry_create (inode_t *inode, inode_t *parent, const char *name);
-
+```
 A dentry holds a refcount on the parent
 directory so that the parent inode is never removed from the active inode's list
 and put to the lru list (If the lru limit of the lru list is exceeded, there is
@@ -212,15 +208,14 @@ deleted due to file removal or lru limit being exceeded  the inode is retired
 purge list maintained by the inode table), the nlookup count is set to 0 via
 inode_forget api. The inode table, then prunes all the inodes from the purge
 list by destroying the inode contexts maintained by each xlator.
-
+```
 unlinking of the dentry is done via inode_unlink;
 
 void
 inode_unlink (inode_t *inode, inode_t *parent, const char *name);
-
+```
 If the inode has multiple hard links, then the unlink operation performed by
 the application results just in the removal of the dentry with the name provided
 by the application. For the inode to be removed, all the dentries of the inode
 should be unlinked.
-```
 
diff --git a/doc/data-structures/iobuf.md b/doc/developer-guide/datastructure-iobuf.md
index 5f521f1485f..03604e3672c 100644
--- a/doc/data-structures/iobuf.md
+++ b/doc/developer-guide/datastructure-iobuf.md
@@ -1,6 +1,6 @@
-#Iobuf-pool
-##Datastructures
-###iobuf
+# Iobuf-pool
+## Datastructures
+### iobuf
 Short for IO Buffer. It is one allocatable unit for the consumers of the IOBUF
 API, each unit hosts @page_size(defined in arena structure) bytes of memory. As
 initial step of processing a fop, the IO buffer passed onto GlusterFS by the
@@ -28,7 +28,7 @@ struct iobuf {
 };
 ```
 
-###iobref
+### iobref
 There may be need of multiple iobufs for a single fop, like in vectored read/write.
 Hence multiple iobufs(default 16) are encapsulated under one iobref.
 ```
@@ -40,7 +40,7 @@ struct iobref {
         int                used;    /* number of iobufs added to this iobref */
 };
 ```
-###iobuf_arenas
+### iobuf_arenas
 One region of memory MMAPed from the operating system. Each region MMAPs
 @arena_size bytes of memory, and hosts @arena_size / @page_size IOBUFs.
 The same sized iobufs are grouped into one arena, for sanity of access.
@@ -77,7 +77,7 @@ struct iobuf_arena {
 };
 
 ```
-###iobuf_pool
+### iobuf_pool
 Pool of Iobufs. As there may be many Io buffers required by the filesystem,
 a pool of iobufs are preallocated and kept, if these preallocated ones are
 exhausted only then the standard malloc/free is called, thus improving the
@@ -139,8 +139,8 @@ arenas in the purge list are destroyed only if there is  atleast one arena in
 (e.g: If there is an arena (page_size=128KB, count=32) in purge list, this arena
 is destroyed(munmap) only if there is an arena in 'arenas' list with page_size=128KB).
 
-##APIs
-###iobuf_get
+## APIs
+### iobuf_get
 
 ```
 struct iobuf *iobuf_get (struct iobuf_pool *iobuf_pool);
@@ -149,7 +149,7 @@ Creates a new iobuf of the default page size(128KB hard coded as of yet).
 Also takes a reference(increments ref count), hence no need of doing it
 explicitly after getting iobuf.
 
-###iobuf_get2
+### iobuf_get2
 
 ```
 struct iobuf * iobuf_get2 (struct iobuf_pool *iobuf_pool, size_t page_size);
@@ -179,7 +179,7 @@ if (requested iobuf size > Max iobuf size in the pool(1MB as of yet))
 Also takes a reference(increments ref count), hence no need of doing it
 explicitly after getting iobuf.
 
-###iobuf_ref
+### iobuf_ref
 
 ```
 struct iobuf *iobuf_ref (struct iobuf *iobuf);
@@ -188,7 +188,7 @@ struct iobuf *iobuf_ref (struct iobuf *iobuf);
 xlator/function/, its a good practice to take a reference so that iobuf is not
 deleted by the allocator.
 
-###iobuf_unref
+### iobuf_unref
 ```
 void iobuf_unref (struct iobuf *iobuf);
 ```
@@ -203,33 +203,33 @@ Unreference the iobuf, if the ref count is zero iobuf is considered free.
 Every iobuf_ref should have a corresponding iobuf_unref, and also every
 iobuf_get/2 should have a correspondning iobuf_unref.
 
-###iobref_new
+### iobref_new
 ```
 struct iobref *iobref_new ();
 ```
 Creates a new iobref structure and returns its pointer.
 
-###iobref_ref
+### iobref_ref
 ```
 struct iobref *iobref_ref (struct iobref *iobref);
 ```
 Take a reference on the iobref.
 
-###iobref_unref
+### iobref_unref
 ```
 void iobref_unref (struct iobref *iobref);
 ```
 Decrements the reference count of the iobref. If the ref count is 0, then unref
 all the iobufs(iobuf_unref) in the iobref, and destroy the iobref.
 
-###iobref_add
+### iobref_add
 ```
 int iobref_add (struct iobref *iobref, struct iobuf *iobuf);
 ```
 Adds the given iobuf into the iobref, it takes a ref on the iobuf before adding
 it, hence explicit iobuf_ref is not required if adding to the iobref.
 
-###iobref_merge
+### iobref_merge
 ```
 int iobref_merge (struct iobref *to, struct iobref *from);
 ```
@@ -239,13 +239,13 @@ on all the iobufs added to the 'to' iobref. Hence iobref_unref should be
 performed both on 'from' and 'to' iobrefs (performing iobref_unref only on 'to'
 will not free the iobufs and may result in leak).
 
-###iobref_clear
+### iobref_clear
 ```
 void iobref_clear (struct iobref *iobref);
 ```
 Unreference all the iobufs in the iobref, and also unref the iobref.
 
-##Iobuf Leaks
+## Iobuf Leaks
 If all iobuf_refs/iobuf_new do not have correspondning iobuf_unref, then the
 iobufs are not freed and recurring execution of such code path may lead to huge
 memory leaks. The easiest way to identify if a memory leak is caused by iobufs
diff --git a/doc/data-structures/mem-pool.md b/doc/developer-guide/datastructure-mem-pool.md
index c71aa2a8ddd..225567cbf9f 100644
--- a/doc/data-structures/mem-pool.md
+++ b/doc/developer-guide/datastructure-mem-pool.md
@@ -1,5 +1,5 @@
-#Mem-pool
-##Background
+# Mem-pool
+## Background
 There was a time when every fop in glusterfs used to incur cost of allocations/de-allocations for every stack wind/unwind between xlators because stack/frame/*_localt_t in every wind/unwind was allocated and de-allocated. Because of all these system calls in the fop path there was lot of latency and the worst part is that most of the times the number of frames/stacks active at any time wouldn't cross a threshold. So it was decided that this threshold number of frames/stacks would be allocated in the beginning of the process only once. Get one of them from the pool of stacks/frames whenever `STACK_WIND` is performed and put it back into the pool in `STACK_UNWIND`/`STACK_DESTROY` without incurring any extra system calls. The data structures are allocated only when threshold number of such items are in active use i.e. pool is in complete use.% increase in the performance once this was added to all the common data structures (inode/fd/dict etc) in xlators throughout the stack was tremendous.
 
 ## Data structure
@@ -27,7 +27,7 @@ will be served from here until all the elements in the pool are in use i.e. cold
 };
 ```
 
-##Life-cycle
+## Life-cycle
 ```
 mem_pool_new (data_type, unsigned long count)
 
@@ -120,5 +120,5 @@ mem_pool_destroy (struct mem_pool *pool)
 Deletes this pool from the `global_list` maintained by `glusterfs-ctx` and frees all the memory allocated in `mem_pool_new`.
 
 
-###How to pick pool-size
+### How to pick pool-size
 This varies from work-load to work-load. Create the mem-pool with some random size and run the work-load. Take the statedump after the work-load is complete. In the statedump if `max_alloc` is always less than `cold_count` may be reduce the size of the pool closer to `max_alloc`. On the otherhand if there are lots of `pool-misses` then increase the `pool_size` by `max_stdalloc` to achieve better 'hit-rate' of the pool.
diff --git a/doc/developer-guide/dirops-transactions-in-dht.md b/doc/developer-guide/dirops-transactions-in-dht.md
new file mode 100644
index 00000000000..909a97001aa
--- /dev/null
+++ b/doc/developer-guide/dirops-transactions-in-dht.md
@@ -0,0 +1,273 @@
+# dirops transactions in dht
+Need for transactions during operations on directories arise from two
+basic design elements of DHT:
+
+ 1. A directory is created on all subvolumes of dht. Since glusterfs
+ associates each file-system object with an unique gfid, every
+ subvolume should have the same unique mapping of (path of directory,
+ gfid). To elaborate,
+    * Each subvolume should've same gfid associated with a path to
+ directory.
+    * A gfid should not be associated with more than one path in any
+ subvolume.
+
+ So, entire operations like mkdir, renamedir, rmdir and creation of
+ directories during self-heal need to be atomic in dht. In other words,
+ any of these operations shouldn't begin on an inode if one of them is
+ already in progress on the same inode, till it completes on all
+ subvolumes of dht. If not, more than one of these operations
+ happening in parallel can break any or all of the two requirements
+ listed above. This is referred in the rest of the document by the
+ name _Atomicity during namespace operations_.
+   
+ 2. Each directory has an independent layout persisted on
+ subvolumes. Each subvolume contains only part of the layout relevant
+ to it. For performance reasons _and_ since _only_ dht has aggregated
+ view, this layout is cached in memory of client. To make sure dht
+ reads or modifies a single complete layout while parallel modifications of    the layout are in progress, we need atomicity during layout modification and reading. This is referred in the rest of the document as  _Atomicity during layout modification and reading_.
+
+Rest of the document explains how atomicity is achieved for each of
+the case above.
+
+**Atomicity during layout modification and reading**
+File operations a.k.a fops can be classified into two categories based on how they consume layout.
+
+ - Layout writer. Setting of layout during selfheal of a directory is
+ layout writer of _that_ directory. 
+ - Layout reader. 
+     * Any entry fop like create, unlink, rename, link, symlink,
+ unlink, mknod, rename, mkdir, rmdir, renamedir which needs layout of the parent directory. Each of these fops are readers of layout on parent directory.
+     *  setting of layout during mkdir of a directory is considered as
+ a reader of the same directory's layout. The reason for this is that
+ only a parallel lookup on that directory can be a competing fop that modifies the layout (Other fops need gfid of the directory which can be got only after either lookup or mkdir completes). However, healing of layout is considered as a writer and a single writer blocks all readers.
+ 
+*Algorithm*
+Atomicity is achieved by locking on the inode of directory whose
+layout is being modified or read. The fop used is inodelk.
+ -  Writer acquires blocking inodelk (directory-inode, write-lock) on
+ all subvolumes serially. The order of subvols in which they are
+ locked by different clients remains constant for a directory. If locking fails on any subvolume, layout modification is abandoned.
+ - Reader acquires an inodelk (directory-inode, read-lock) on _any_
+ one subvolume. If locking fails on a subvolume (say with
+ ESTALE/ENOTCONN error), locking can be tried on other subvolumes till
+ we get one lock. If we cannot get lock on at least one subvolume,
+ consistency of layout is not guaranteed. Based on the consistency
+ requirements of fops, they can be failed or continued.
+
+Reasons why writer has to lock on _all_ subvols:
+
+ - DHT don't have a metadata server and locking is implemented by brick. So, there is no well-defined subvol/brick that can be used as an arbitrator by different clients while acquiring locks.
+ - readers should acquire as minimum number of locks as possible. In
+ other words, the algorithm aims to have less synchronization cost to
+ readers.
+ - The subvolume to which a directory hashes could be used as a
+ lock server. However, in the case of an entry fop like create
+ (/a/b/c) where we want to block modification of layout of b for the
+ duration of create, we would be required to acquire lock on the
+ subvol to which /a/b hashes. To find out the hashed-subvol of
+ /a/b, we would need layout of /a. Note that how there is a dependency
+ of locking the layouts of ancestors all the way to root. So this
+ locking is not preferred. Also, note that only the immediate parent
+ inode is available in arguments of a fop like create.
+
+**Atomicity during namespace operations**
+
+ -  We use locks on inode of parent directory in the namespace of
+ _"basename"_  during mkdir, rmdir, renamedir and directory
+ creation phase of self-heal. The exact fop we use is _entrylk
+ (parent-inode, "basename")_.
+ - refresh in-memory layout of parent-inode from values stored on backend
+ -  _entrylk (parent-inode, "basename")_ is done on subvolume to which
+ _"basename" hashes_. So, this operation is a _reader_ of the
+ layout on _parent-inode_. Which means an _inodelk (parent-inode,
+ read-lock)_ has to be completed before _entrylk (parent-inode,
+ "basename")_ is issued. Both the locks have to be held till the
+ operation is tried on all subvolumes. If acquiring of any/all of
+ these locks fail, the operation should be failed.
+
+With the above details, algorithms for mkdir, rmdir, renamedir,
+self-heal of directory are explicitly detailed below.
+
+**Self-heal of a directory**
+  
+ -  creation of directories on subvolumes is done only during
+ _named-lookup_ of a directory as we need < parent-inode,
+ "basename" >.
+ -  If a directory is missing on one or more subvolumes,
+    *  acquire _inodelk (parent-inode, read-lock)_ on _any one_ of the
+ subvolumes.
+    * refresh the in-memory layout of parent-inode from values stored on backend
+    *  acquire _entrylk (parent-inode, "basename")_ on the subvolume
+ to which _"basename"_ hashes.
+    *  If any/all of the locks fail, self-heal is aborted.
+    *  create directories on missing subvolumes.
+    *  release _entrylk (parent-inode, "basename")_.
+    *  release _inodelk (parent-inode, read-lock)_.
+    
+ -  If layout of a directory needs healing
+     * acquire _inodelk (directory-inode, write-lock)_ on _all_ the
+ subvolumes. If locking fails on any of the subvolumes,
+ self-heal is aborted. Blocking Locks are acquired serially across subvolumes in a _well-defined_ order which is _constant_ across all the healers of a directory. One order could be the order in which subvolumes are stored in the array _children_ of dht xlator.
+     *  heal the layout.
+     *  release _inodelk (directory-inode, write-lock)_ on _all_ the
+ subvolumes in parallel.
+     *  Note that healing of layout can be done in both _named_ and
+ _nameless_ lookups of a directory as _only directory-inode_ is needed
+ for healing and it is available during both.
+
+**mkdir (parent-inode, "basename")**
+
+* while creating directory across subvolumes,
+
+ - acquire _inodelk (parent-inode, read-lock)_ on _any one_ of the
+ subvolumes.
+ - refresh in-memory layout of parent-inode from values stored on backend
+ - acquire _entrylk (parent-inode, "basename")_ on the subvolume to
+ which _"basename"_ hashes.
+ -  If any/all of the above two locks fail, release the locks that
+ were acquired successfully and mkdir should be failed (as perceived by application).
+ - do _mkdir (parent-inode, "basename")_ on the subvolume to which
+ _"basename"_ hashes. If this mkdir fails, mkdir is failed.
+ - do _mkdir (parent-inode, "basename")_ on the remaining subvolumes.
+ - release _entrylk (parent-inode, "basename")_.
+ - release _inodelk (parent-inode, "read-lock")_.
+*  while setting the layout of a directory,
+ - acquire _inodelk (directory-inode, read-lock)_ on _any one_ of the
+ subvolumes.
+ -  If locking fails, cleanup the locks that were acquired
+ successfully and abort layout setting. Note that we'll have a
+ directory without a layout till a lookup happens on the
+ directory. This means entry operations within this directory fail
+ in this time window. We can also consider failing mkdir. The
+ problem of dealing with a directory without layout is out of the
+ scope of this document. 
+ -  set the layout on _directory-inode_.
+ -  release _inodelk (directory-inode, read-lock)_.
+* Note that during layout setting we consider mkdir as a  _reader_ not
+ _writer_, though it is setting the layout. Reasons are:
+ - Before any of other readers like create, link etc that operate on
+ this directory to happen, _gfid_ of this directory has to be
+ resolved. But _gfid_ is only available only if either of following
+ conditions are true: 
+      * after mkdir is complete.
+      * a lookup on the same path happens parallel to in-progress
+ mkdir.
+
+     But, on completion of any of the above two operations, layout
+     will be healed. So, none of the _readers_ will happen on a
+     directory with partial layout.
+
+* Note that since we've an _entrylk (parent-inode, "basename")_ for
+   the entire duration of (attempting) creating directories, parallel
+   mkdirs will no longer contend on _mkdir_ on subvolume _to which  "basename"  hashes_. But instead, contend on _entrylk (parent-inode, "basename")_ on the subvolume _to which "basename" hashes_. So, we can attempt the _mkdir_ in _parallel_ on all subvolumes instead of two stage mkdir on hashed first and the rest of them in parallel later. However, we need to make sure that mkdir is successful on the subvolume _to which "basename" hashes_ for mkdir to be successful (as perceived by application). In the case of failed mkdir (as perceived by application), a cleanup should be performed on all the subvolumes before _entrylk (parent-inode, "basename")_ is released.
+
+**rmdir (parent-inode, "basename", directory-inode)**
+
+ -  acquire _inodelk (parent-inode, read-lock)_ on _any one_
+ subvolume.
+ -  refresh in-memory layout of parent-inode from values stored on backend
+ -  acquire _entrylk (parent-inode, "basename")_ on the subvolume to
+ which _"basename" hashes_.
+ -  If any/all of the above locks fail, rmdir is failed after cleanup
+ of the locks that were acquired successfully.
+ -  do _rmdir (parent-inode, "basename")_ on the subvolumes to which
+ _"basename" doesn't hash to_.
+    * If successful, continue. 
+    * Else,
+     *  recreate directories on those subvolumes where rmdir
+  succeeded.
+     *  heal the layout of _directory-inode_. Note that this will have
+  same synchronization requirements as discussed during layout
+  healing part of the section "Directoy self-heal" above.
+     *  release _entrylk (parent-inode, "basename")_.
+     *  release _inodelk (parent-inode, read-lock)_.
+     *  fail _rmdir (parent-inode, "basename")_ to application.
+ - do _rmdir (parent-inode, "basename")_ on the subvolume to which
+ _"basename" hashes_. 
+ - If successful, continue.
+ -  Else, Go to the failure part of _rmdir (parent-inode, "basename")_
+ on subvolumes to which "basename" _doesn't hash to_. 
+ - release _entrylk (parent-inode, "basename")_.
+ - release _inodelk (parent-inode, read-lock)_.
+ - return success to application.
+ 
+**renamedir (src-parent-inode, "src-basename", src-directory-inode, dst-parent-inode, "dst-basename", dst-directory-inode)**
+
+ -  requirement is to prevent any operation in both _src-namespace_
+ and _dst-namespace_. So, we need to acquire locks on both
+ namespaces.We also need to have constant ordering while acquiring
+ locks during parallel renames of the form _rename (src, dst)_ and
+ _rename (dst, src)_ to prevent deadlocks. We can sort gfids of
+ _src-parent-inode_ and _dst-parent-inode_ and use that order to
+ acquire locks. For the sake of explanation lets say we ended up
+ with order of _src_ followed by _dst_.
+ -  acquire _inodelk (src-parent-inode, read-lock)_.
+ -  refresh in-memory layout of src-parent-inode from values stored on backend
+ -  acquire _entrylk (src-parent-inode, "src-basename")_.
+ -  acquire _inodelk (dst-parent-inode, read-lock)_.
+ -  refresh in-memory layout of dst-parent-inode from values stored on backend
+ -  acquire _entrylk (dst-parent-inode, "dst-basename")_.
+ - If acquiring any/all of the locks above fail,
+      * release the locks that were successfully acquired.
+      * fail the renamedir operation to application
+      *  done
+ - do _renamedir ("src", "dst")_ on the subvolume _to which "dst-basename" hashes_.
+    *  If failure, Goto point _If acquiring any/all of the locks above fail_.
+    *  else, continue.
+ - do _renamedir ("src", "dst")_ on rest of the subvolumes.
+    *  If there is any failure, 
+      * revert the successful renames.
+      *  Goto to point  _If acquiring any/all of the locks above fail_.
+    *  else,
+        -  release all the locks acquired.
+        -  return renamedir as success to application.
+
+**Some examples of races**
+This section gives concrete examples of races that can result in inconsistencies explained in the beginning of the document. 
+
+Some assumptions are:
+
+*  We consider an example distribute of three subvols s1, s2 and s3.
+*  For examples of renamedir ("src", "dst"), _src_ hashes to s1 and _dst_ hashes to s2. _src_ and _dst_ are associated with _gfid-src_ and _gfid-dst_ respectively
+*  For non renamedir examples, _dir_ is the name of directory and it hashes to s1.
+
+And the examples are:  
+
+ -  mkdir vs rmdir - inconsistency in namespace.
+   *  mkdir ("dir", gfid1) is complete on s1
+   *  rmdir is issued on same directory. Note that, since rmdir needs a gfid, a lookup should be complete before rmdir. lookup creates the directory on rest of the subvols as part of self-heal.
+   *  rmdir (gfid1) deletes directory from all subvols.
+   *  A new mkdir ("dir", gfid2) is issued. It is successful on s1 associating "dir" with gfid2.
+   *  mkdir ("dir", gfid1) resumes and creates directory on s2 and s3 associating "dir" with gfid1.
+   *  mkdir ("dir", gfid2) fails with EEXIST on s2 and s3. Since, EEXIST errors are ignored, mkdir is considered successful to application.
+   *  In this example we have multiple inconsitencies
+      *  "dir" is associated with gfid1 on s2, s3 and with gfid2 on s1
+      *  Even if mkdir ("dir", gfid2) was not issued, we would've a case of a directory magically reappearing after a successful rmdir.
+ -  lookup heal vs rmdir
+   * rmdir ("dir", gfid1) is issued. It is successful on s2 and s3 (non-hashed subvols for name "dir")
+   *  lookup ("dir") is issued. Since directory is present on s1 yet, it is created on s2 and s3 associating with gfid1 as part of self-heal
+   *  rmdir ("dir", gfid1) is complete on s1 and it is successful
+   *  Another lookup ("dir") creates the directory on s1 too
+   *  "dir" magically reappears after a successful rmdir
+ - lookup heal (src) vs renamedir ("src", "dst")
+   *  renamedir ("src", "dst") complete on s2
+   *  lookup ("src") recreates _src_ with _gfid-src_ on s2
+   *  renamedir ("src", "dst") completes on s1, s3. After rename is complete path _dst_ will be associated with gfid _gfid-src_
+   *  Another lookup ("src") recreates _src_ on subvols s1 and s3, associating it with gfid _gfid-src_
+   *  Inconsistencies are
+      * after a successful renamedir ("src", "dst"), both src and dst exist
+      *  Two directories - src and dst - are associated with same gfid. One common symptom is that some entries (of the earlier _src_ and current _dst_ directory) being missed out in readdir listing as the gfid handle might be pointing to the empty healed directory than the actual directory containing entries
+ - lookup heal (dst) vs renamedir ("src", "dst")
+   *  dst exists and empty when renamdir started
+   *  dst doesn't exist when renamedir started
+      -  renamedir ("src", "dst") complete on s2 and s3
+      -  lookup ("dst") creates _dst_ associating it with _gfid-src_ on s1
+      -  An entry is created in _dst_ on either s1
+      -  renamedir ("src", "dst") on s1 will result in a directory _dst/dst_ as _dst_ is no longer empty and _man 2 rename_ states that if _dst_ is not empty, _src_ is renamed _as a subdirectory of dst_
+      -  A lookup ( _dst/dst_) creates _dst/dst_ on s2 and s3 associating with _gfid-src_ as part of self-heal
+      -  Inconsistencies are:
+         * Two directories - _dst_ and _dst/dst_ - exist even though both of them didn't exist at the beginning of renamedir
+         *  Both _dst_ and _dst/dst_ have same gfid - _gfid-src_. As observed earlier, symptom might be directory listing being incomplete
+ - mkdir (dst) vs renamedir ("src", "dst")
+ - rmdir (src) vs renamedir ("src", "dst")
+ - rmdir (dst) vs renamedir ("src", "dst")
diff --git a/doc/developer-guide/ec-implementation.md b/doc/developer-guide/ec-implementation.md
new file mode 100644
index 00000000000..77e62583caa
--- /dev/null
+++ b/doc/developer-guide/ec-implementation.md
@@ -0,0 +1,588 @@
+Erasure coding implementation
+=============================
+
+This document provides information about how [erasure code][1] has
+been implemented into ec translator. It describes the algorithm used
+and the optimizations made, but it doesn't contain a full description
+of the mathematical background needed to understand erasure coding in
+general. It either describes the other parts of ec not directly
+related to the encoding/decoding procedure, like synchronization or
+fop management.
+
+
+Introduction
+------------
+
+EC is based on [Reed-Solomon][2] erasure code. It's a very old code.
+It's not considered the best one nowadays, but is good enough and it's
+one of the few codes that is not covered by any patent and can be
+freely used.
+
+To define the Reed-Solomon code we use 3 parameters:
+
+  * __Key fragments (K)__  
+    It represents the minimum number of healthy fragments that will be
+    needed to be able to recover the original data. Any subset of K
+    out of the total number of fragments will serve.
+
+  * __Redundancy fragments (R)__  
+    It represents the number of extra fragments to compute for each
+    original data block. This value determines how many fragments can
+    be lost before being unable to recover the original data.
+
+  * __Fragment size (S)__  
+    This determines the size of each fragment. The original data
+    block size is computed as S * K. Currently this values is fixed
+    to 512 bytes.
+
+  * __Total number of fragments (N = K + R)__  
+    This isn't a real parameter but it will be useful to simplify
+    the following descriptions.
+
+From the point of view of the implementation, it only consists on
+matrix multiplications. There are two kinds of matrices to use for
+Reed-Solomon:
+
+  * __[Systematic][3]__  
+    This kind of matrix has the particularity that K of the encoded
+    fragments are simply a copy of the original data, divided into K
+    pieces. Thus no real encoding needs to be done for them and only
+    the R redundancy fragments need to be computed.
+
+    This kind of matrices contain one KxK submatrix that is the
+    [identity matrix][4].
+
+  * __Non-systematic__  
+    This kind of matrix doesn't contain an identity submatrix. This
+    means that all of the N fragments need to be encoded, requiring
+    more computation. On the other hand, these matrices have some nice
+    properties that allow faster implementations of some algorithms,
+    like the matrix inversion used to decode the data.
+
+    Another advantage of non-systematic matrices is that the decoding
+    time is constant, independently of how many fragments are lost,
+    while systematic approach can suffer from performance degradation
+    when one fragment is lost.
+
+All non-systematic matrices can be converted to systematic ones, but
+then we lose the good properties of the non-systematic. We have to
+choose betwee best peek performance (systematic) and performance
+stability (non-systematic).
+
+
+Encoding procedure
+------------------
+
+To encode a block of data we need a KxN matrix where each subset of K
+rows is [linearly independent][5]. In other words, the determinant of
+each KxK submatrix is not 0.
+
+There are some known ways to obtain this kind of matrices. EC uses a
+small variation of a matrix known as [Vandermonde Matrix][6] where
+each element of the matrix is defined as:
+
+    a(i, j) = i ^ (K - j)
+
+    where i is the row from 1 to N, and j is the column from 1 to K.
+
+This is exactly the Vandermonde Matrix but with the elements of each
+row in reverse order. This change is made to be able to implement a
+small optimization in the matrix multiplication.
+
+Once we have the matrix, we only need to compute the multiplication
+of this matrix by a vector composed of K elements of data coming from
+the original data block.
+
+     /                 \             /                               \
+    |    1   1  1  1  1 |    / \    |     a +    b +   c +  d + e = t |
+    |   16   8  4  2  1 |   | a |   |   16a +   8b +  4c + 2d + e = u |
+    |   81  27  9  3  1 |   | b | = |   81a +  27b +  9c + 3d + e = v |
+    |  256  64 16  4  1 | * | c |   |  256a +  64b + 16c + 4d + e = w |
+    |  625 125 25  5  1 |   | d |   |  625a + 125b + 25c + 5d + e = x |
+    | 1296 216 36  6  1 |   | e |   | 1296a + 216b + 36c + 6d + e = y |
+    | 2401 343 49  7  1 |    \ /    | 2401a + 343b + 49c + 7d + e = z |
+     \                 /             \                               /
+
+The optimization that can be done here is this:
+
+    16a + 8b + 4c + 2d + e = 2(2(2(2a + b) + c) + d) + e
+
+So all the multiplications are always by the number of the row (2 in
+this case) and we don't need temporal storage for intermediate
+results:
+
+    a *= 2
+    a += b
+    a *= 2
+    a += c
+    a *= 2
+    a += d
+    a *= 2
+    a += e
+
+Once we have the result vector, each element is a fragment that needs
+to be stored in a separate place.
+
+
+Decoding procedure
+------------------
+
+To recover the data we need exactly K of the fragments. We need to
+know which K fragments we have (i.e. the original row number from
+which each fragment was calculated). Once we have this data we build
+a square KxK matrix composed by the rows corresponding to the given
+fragments and invert it.
+
+With the inverted matrix, we can recover the original data by
+multiplying it with the vector composed by the K fragments.
+
+In our previous example, if we consider that we have recovered
+fragments t, u, v, x and z, corresponding to rows 1, 2, 3, 5 and 7,
+we can build the following matrix:
+
+     /                 \
+    |    1   1  1  1  1 |
+    |   16   8  4  2  1 |
+    |   81  27  9  3  1 |
+    |  625 125 25  5  1 |
+    | 2401 343 49  7  1 |
+     \                 /
+
+And invert it:
+
+     /                                     \
+    |    1/48  -1/15    1/16  -1/48   1/240 |
+    |  -17/48  16/15  -15/16  13/48 -11/240 |
+    |  101/48 -86/15   73/16 -53/48  41/240 |
+    | -247/48 176/15 -129/16  83/48 -61/240 |
+    |   35/8   -7      35/8   -7/8    1/8   |
+     \                                     /
+
+Multiplying it by the vector (t, u, v, x, z) we recover the original
+data (a, b, c, d, e):
+
+     /                                     \     / \     / \
+    |    1/48  -1/15    1/16  -1/48   1/240 |   | t |   | a |
+    |  -17/48  16/15  -15/16  13/48 -11/240 |   | u |   | b |
+    |  101/48 -86/15   73/16 -53/48  41/240 | * | v | = | c |
+    | -247/48 176/15 -129/16  83/48 -61/240 |   | x |   | d |
+    |   35/8   -7      35/8   -7/8    1/8   |   | z |   | e |
+     \                                     /     \ /     \ /
+
+
+Galois Field
+------------
+
+This encoding/decoding procedure is quite complex to compute using
+regular mathematical operations and it's not well suited for what
+we want to do (note that matrix elements can grow unboundly).
+
+To solve this problem, exactly the same procedure is done inside a
+[Galois Field][7] of characteristic 2, which is a finite field with
+some interesting properties that make it specially useful for fast
+operations using computers.
+
+There are two main differences when we use this specific Galois Field:
+
+  * __All regular additions are replaced by bitwise xor's__  
+    For todays computers it's not really faster to execute an xor
+    compared to an addition, however replacing additions by xor's
+    inside a multiplication has many advantages (we will make use of
+    this to optimize the multiplication).
+
+    Another consequence of this change is that additions and
+    substractions are really the same xor operation.
+
+  * __The elements of the matrix are bounded__  
+    The field uses a modulus that keep all possible elements inside
+    a delimited region, avoiding really big numbers and fixing the
+    number of bits needed to represent each value.
+
+    In the current implementation EC uses 8 bits per field element.
+
+It's very important to understand how multiplications are computed
+inside a Galois Field to be able to understand how has it been
+optimized.
+
+We'll start with a simple 'old school' multiplication but in base 2.
+For example, if we want to multiply 7 * 5 (111b * 101b in binary), we
+do the following:
+
+          1 1 1 (= 7)
+    *     1 0 1 (= 5)
+    -----------
+          1 1 1 (= 7)
+    +   0 0 0   (= 0)
+    + 1 1 1     (= 7)
+    -----------
+    1 0 0 0 1 1 (= 35)
+
+This is quite simple. Note that the addition of the third column
+generates a carry that is propagated to all the other left columns.
+
+The next step is to define the modulus of the field. Suppose we use
+11 as the modulus. Then we convert the result into an element of the
+field by dividing by the modulus and taking the residue. We also use
+the 'old school' method in binary:
+
+
+      1 0 0 0 1 1 (= 35) | 1 0 1 1 (= 11)
+    - 0 0 0 0            ----------------
+    ---------              0 1 1 (= 3)
+      1 0 0 0 1
+    -   1 0 1 1
+    -----------
+      0 0 1 1 0 1
+    -     1 0 1 1
+    -------------
+          0 0 1 0 (= 2)
+
+So, 7 * 5 in a field with modulus 11 is 2. Note that the main
+objective in each iteration of the division is to make higher bits
+equal to 0 when possible (if it's not possible in one iteration, it
+will be zeroed on the next).
+
+If we do the same but changing additions with xors we get this:
+
+          1 1 1 (= 7)
+    *     1 0 1 (= 5)
+    -----------
+          1 1 1 (= 7)
+    x   0 0 0   (= 0)
+    x 1 1 1     (= 7)
+    -----------
+      1 1 0 1 1 (= 27)
+
+In this case, the xor of the third column doesn't generate any carry.
+
+Now we need to divide by the modulus. We can also use 11 as the
+modulus since it still satisfies the needed conditions to work on a
+Galois Field of characteristic 2 with 3 bits:
+
+      1 1 0 1 1 (= 27) | 1 0 1 1 (= 11)
+    x 1 0 1 1          ----------------
+    ---------            1 1 1 (= 7)
+      0 1 1 0 1
+    x   1 0 1 1
+    -----------
+        0 1 1 0 1
+    x     1 0 1 1
+    -------------
+          0 1 1 0 (= 6)
+
+Note that, in this case, to make zero the higher bit we need to
+consider the result of the xor operation, not the addition operation.
+
+So, 7 * 5 in a Galois Field of 3 bits with modulus 11 is 6.
+
+
+Optimization
+------------
+
+To compute all these operations in a fast way some methods have been
+traditionally used. Maybe the most common is the [lookup table][8].
+
+The problem with this method is that it requires 3 lookups for each
+byte multiplication, greatly amplifying the needed memory bandwidth
+and making it difficult to take advantage of any SIMD support on the
+processor.
+
+What EC does to improve the performance is based on the following
+property (using the 3 bits Galois Field of the last example):
+
+    A * B mod N = (A * b{2} * 4 mod N) x
+                  (A * b{1} * 2 mod N) x
+                  (A * b{0} mod N)
+
+This is basically a rewrite of the steps made in the previous example
+to multiply two numbers but moving the modulus calculation inside each
+intermediate result. What we can see here is that each term of the
+xor can be zeroed if the corresponding bit of B is 0, so we can ignore
+that factor. If the bit is 1, we need to compute A multiplied by a
+power of two and take the residue of the division by the modulus. We
+can precompute these values:
+
+    A0 = A (we don't need to compute the modulus here)
+    A1 = A0 * 2 mod N
+    A2 = A1 * 2 mod N
+
+Having these values we only need to add those corresponding to bits
+set to 1 in B. Using our previous example:
+
+    A = 1 1 1 (= 7)
+    B = 1 0 1 (= 5)
+
+    A0 = 1 1 1 (= 7)
+    A1 = 1 1 1 * 1 0 mod 1 0 1 1 = 1 0 1 (= 5)
+    A2 = 1 0 1 * 1 0 mod 1 0 1 1 = 0 0 1 (= 1)
+
+    Since only bits 0 and 2 are 1 in B, we add A0 and A2:
+
+    A0 + A2 = 1 1 1 x 0 0 1 = 1 1 0 (= 6)
+
+If we carefully look at what we are doing when computing each Ax, we
+see that we do two basic things:
+
+  - Shift the original value one bit to the left
+  - If the highest bit is 1, xor with the modulus
+
+Let's write this in a detailed way (representing each bit):
+
+    Original value:      a{2} a{1} a{0}
+    Shift 1 bit:    a{2} a{1} a{0}  0
+
+    If a{2} is 0 we already have the result:
+        a{1} a{0} 0
+
+    If a{2} is 1 we need to xor with the modulus:
+        1 a{1} a{0} 0 x 1 0 1 1 = a{1} (a{0} x 1) 1
+
+An important thing to see here is that if a{2} is 0, we can get the
+same result by xoring with all 0 instead of the modulus. For this
+reason we can rewrite the modulus as this:
+
+    Modulus: a{2} 0 a{2} a{2}
+
+This means that the modulus will be 0 0 0 0 is a{2} is 0, so the value
+won't change, and it will be 1 0 1 1 if a{2} is 1, giving the correct
+result. So, the computation is simply:
+
+    Original value:      a{2} a{1} a{0}
+    Shift 1 bit:    a{2} a{1} a{0}  0
+    Apply modulus:  a{1} (a{0} x a{2}) a{2}
+
+We can compute all Ax using this method. We'll get this:
+
+    A0 = a{2} a{1} a{0}
+    A1 = a{1} (a{0} x a{2}) a{2}
+    A2 = (a{0} x a{2}) (a{1} x a{2}) a{1}
+
+Once we have all terms, we xor the ones corresponding to the bits set
+to 1 in B. In out example this will be A0 and A2:
+
+   Result: (a{2} x a{0} x a{2}) (a{1} x a{1} x a{2}) (a{0} x a{1})
+
+We can easily see that we can remove some redundant factors:
+
+   Result: a{0} a{2} (a{0} x a{1})
+
+This way we have come up with a simply set of equations to compute the
+multiplication of any number by 5. If A is 1 1 1 (= 7), the result
+must be 1 1 0 (= 6) using the equations, as we expected. If we try
+another numbe for A, like 0 1 0 (= 2), the result must be 0 0 1 (= 1).
+
+This seems a really fast way to compute the multiplication without
+using any table lookup. The problem is that this is only valid for
+B = 5. For other values of B another set of equations will be found.
+To solve this problem we can pregenerate the equations for all
+possible values of B. Since the Galois Field we use is small, this is
+feasible.
+
+One thing to be aware of is that, in general, two equations for
+different bits of the same B can share common subexpressions. This
+gives space for further optimizations to reduce the total number of
+xors used in the final equations for a given B. However this is not
+easy to find, since finding the smallest number of xors that give the
+correct result is an NP-Problem. For EC an exhaustive search has been
+made to find the best combinations for each possible value.
+
+
+Implementation
+--------------
+
+All this seems great from the hardware point of view, but implementing
+this using normal processor instructions is not so easy because we
+would need a lot of shifts, ands, xors and ors to move the bits of
+each number to the correct position to compute the equation and then
+another shift to put each bit back to its final place.
+
+For example, to implement the functions to multiply by 5, we would
+need something like this:
+
+    Bit 2:  T2 = (A & 1) << 2
+    Bit 1:  T1 = (A & 4) >> 1
+    Bit 0:  T0 = ((A >> 1) x A) & 1
+    Result: T2 + T1 + T0
+
+This doesn't look good. So here we make a change in the way we get
+and process the data: instead of reading full numbers into variables
+and operate with them afterwards, we use a single independent variable
+for each bit of the number.
+
+Assume that we can read and write independent bits from memory (later
+we'll see how to solve this problem when this is not possible). In
+this case, the code would look something like this:
+
+    Bit 2:        T2 = Mem[2]
+    Bit 1:        T1 = Mem[1]
+    Bit 0:        T0 = Mem[0]
+    Computation:  T1 ^= T0
+    Store result: Mem[2] = T0
+                  Mem[1] = T2
+                  Mem[0] = T1
+
+Note that in this case we handle the final reordering of bits simply
+by storing the right variable to the right place, without any shifts,
+ands nor ors. In fact we only have memory loads, memory stores and
+xors. Note also that we can do all the computations directly using the
+variables themselves, without additional storage. This true for most
+of the values, but in some cases an additional one or two temporal
+variables will be needed to store intermediate results.
+
+The drawback of this approach is that additions, that are simply a
+xor of two numbers will need as many xors as bits are in each number.
+
+
+SIMD optimization
+-----------------
+
+So we have a good way to compute the multiplications, but even using
+this we'll need several operations for each byte of the original data.
+We can improve this by doing multiple multiplications using the same
+set of instructions.
+
+With the approach taken in the implementation section, we can see that
+in fact it's really easy to add SIMD support to this method. We only
+need to store in each variable one bit from multiple numbers. For
+example, when we load T2 from memory, instead of reading the bit 2 of
+the first number, we can read the bit 2 of the first, second, third,
+fourth, ... numbers. The same can be done when loading T1 and T0.
+
+Obviously this needs to have a special encoding of the numbers into
+memory to be able to do that in a single operation, but since we can
+choose whatever encoding we want for EC, we have chosen to have
+exactly that. We interpret the original data as a stream of bits, and
+we split it into subsequences of length L, each containing one bit of
+a number. Every S subsequences form a set of numbers of S bits that
+are encoded and decoded as a single group. This repeats for any
+remaining data.
+
+For example, in a simple case with L = 8 and S = 3, the original data
+would contain something like this (interpreted as a sequence of bits,
+offsets are also bit-based):
+
+    Offset  0: a{0} b{0} c{0} d{0} e{0} f{0} g{0} h{0}
+    Offset  8: a{1} b{1} c{1} d{1} e{1} f{1} g{1} h{1}
+    Offset 16: a{2} b{2} c{2} d{2} e{2} f{2} g{2} h{2}
+    Offset 24: i{0} j{0} k{0} l{0} m{0} n{0} o{0} p{0}
+    Offset 32: i{1} j{1} k{1} l{1} m{1} n{1} o{1} p{1}
+    Offset 40: i{2} j{2} k{2} l{2} m{2} n{2} o{2} p{2}
+
+Note: If the input file is not a multiple of S * L, 0-padding is done.
+
+Here we have 16 numbers encoded, from A to P. This way we can easily
+see that reading the first byte of the file will read all bits 0 of
+number A, B, C, D, E, F, G and H. The same happens with bits 1 and 2
+when we read the second and third bytes respectively. Using this
+encoding and the implementation described above, we can see that the
+same set of instructions will be computing the multiplication of 8
+numbers at the same time.
+
+This can be further improved if we use L = 64 with 64 bits variables
+on 64-bits processor. It's even faster if we use L = 128 using SSE
+registers or L = 256 using AVX registers on Intel processors.
+
+Currently EC uses L = 512 and S = 8. This means that numbers are
+packed in blocks of 512 bytes and gives space for even bigger
+processor registers up to 512 bits.
+
+
+Conclusions
+-----------
+
+This method requires a single variable/processor register for each
+bit. This can be challenging if we want to avoid additional memory
+accesses, even if we use modern processors that have many registers.
+However, the implementation we chose for the Vandermonde Matrix
+doesn't require temporary storage, so we don't need a full set of 8
+new registers (one for each bit) to store partial computations.
+Additionally, the computation of the multiplications requires, at
+most, 2 extra registers, but this is afordable.
+
+Xors are a really fast operation in modern processors. Intel CPU's
+can dispatch up to 3 xors per CPU cycle if there are no dependencies
+with ongoing previous instructions. Worst case is 1 xor per cycle. So,
+in some configurations, this method could be very near to the memory
+speed.
+
+Another interesting thing of this method is that all data it needs to
+operate is packed in small sequential blocks of memory, meaning that
+it can take advantage of the faster internal CPU caches.
+
+
+Results
+-------
+
+For the particular case of 8 bits, EC can compute each multiplication
+using 12.8 xors on average (without counting 0 and 1 that do not
+require any xor). Some numbers require less, like 2 that only requires
+3 xors.
+
+Having all this, we can check some numbers to see the performance of
+this method.
+
+Maybe the most interesting thing is the average number of xors needed
+to encode a single byte of data. To compute this we'll need to define
+some variables:
+
+  * K: Number of data fragments  
+  * R: Number of redundancy fragments  
+  * N: K + R  
+  * B: Number of bits per number  
+  * A: Average number of xors per number  
+  * Z: Bits per CPU register (can be up to 256 for AVX registers)  
+  * X: Average number of xors per CPU cycle  
+  * L: Average cycles per load  
+  * S: Average cycles per store  
+  * G: Core speed in Hz  
+
+_Total number of bytes processed for a single matrix multiplication_:
+
+  * __Read__:    K * B * Z / 8  
+  * __Written__: N * B * Z / 8  
+
+_Total number of memory accesses_:
+
+  * __Loads__:  K * B * N  
+  * __Stores__: B * N  
+
+>  We need to read the same K * B * Z bits, in registers of Z bits, N
+>  times, one for each row of the matrix. However the last N - 1 reads
+>  could be made from the internal CPU caches if conditions are good.
+
+_Total number of operations_:
+
+  * __Additions__:       (K - 1) * N  
+  * __Multiplications__: K * N  
+
+__Total number of xors__: B * (K - 1) * N + A * K * N =
+                          N * ((A + B) * K - B)
+
+__Xors per byte__: 8 * N * ((A + B) * K - B) / (K * B * Z)
+
+__CPU cycles per byte__: 8 * N * ((A + B) * K - B) / (K * B * Z * X) +
+                         8 * L * N / Z +      (loads)
+                         8 * S * N / (K * Z)  (stores)
+
+__Bytes per second__: G / {CPU cycles per byte}
+
+Some xors per byte numbers for specific configurations (B=8):
+
+              Z=64  Z=128  Z=256
+    K=2/R=1   0.79  0.39   0.20
+    K=4/R=2   1.76  0.88   0.44
+    K=4/R=3   2.06  1.03   0.51
+    K=8/R=3   3.40  1.70   0.85
+    K=8/R=4   3.71  1.86   0.93
+    K=16/R=4  6.34  3.17   1.59
+
+
+
+[1]: https://en.wikipedia.org/wiki/Erasure_code
+[2]: https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction
+[3]: https://en.wikipedia.org/wiki/Systematic_code
+[4]: https://en.wikipedia.org/wiki/Identity_matrix
+[5]: https://en.wikipedia.org/wiki/Linear_independence
+[6]: https://en.wikipedia.org/wiki/Vandermonde_matrix
+[7]: https://en.wikipedia.org/wiki/Finite_field
+[8]: https://en.wikipedia.org/wiki/Finite_field_arithmetic#Implementation_tricks
diff --git a/doc/developer-guide/fuse-interrupt.md b/doc/developer-guide/fuse-interrupt.md
new file mode 100644
index 00000000000..ec991b81ec5
--- /dev/null
+++ b/doc/developer-guide/fuse-interrupt.md
@@ -0,0 +1,211 @@
+# Fuse interrupt handling
+
+## Conventions followed
+
+- *FUSE* refers to the "wire protocol" between kernel and userspace and
+  related specifications.
+- *fuse* refers to the kernel subsystem and also to the GlusterFs translator.
+
+## FUSE interrupt handling spec
+
+The [Linux kernel FUSE documentation](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/filesystems/fuse.txt?h=v4.18#n148)
+desrcibes how interrupt handling happens in fuse.
+
+## Interrupt handling in the fuse translator
+
+### Declarations
+
+This document describes the internal API in the fuse translator with which
+interrupt can be handled.
+
+The API being internal (to be used only in fuse-bridge.c; the functions are
+not exported to a header file).
+
+```
+enum fuse_interrupt_state {
+    /* ... */
+    INTERRUPT_SQUELCHED,
+    INTERRUPT_HANDLED,
+    /* ... */
+};
+typedef enum fuse_interrupt_state fuse_interrupt_state_t;
+struct fuse_interrupt_record;
+typedef struct fuse_interrupt_record fuse_interrupt_record_t;
+typedef void (*fuse_interrupt_handler_t)(xlator_t *this,
+                                         fuse_interrupt_record_t *);
+struct fuse_interrupt_record {
+    fuse_in_header_t fuse_in_header;
+    void *data;
+    /*
+       ...
+     */
+};
+
+fuse_interrupt_record_t *
+fuse_interrupt_record_new(fuse_in_header_t *finh,
+                          fuse_interrupt_handler_t handler);
+
+void
+fuse_interrupt_record_insert(xlator_t *this, fuse_interrupt_record_t *fir);
+
+gf_boolean_t
+fuse_interrupt_finish_fop(call_frame_t *frame, xlator_t *this,
+                          gf_boolean_t sync, void **datap);
+
+void
+fuse_interrupt_finish_interrupt(xlator_t *this, fuse_interrupt_record_t *fir,
+                                fuse_interrupt_state_t intstat,
+                                gf_boolean_t sync, void **datap);
+```
+
+The code demonstrates the usage of the API through `fuse_flush()`. (It's a
+dummy implementation only for demonstration purposes.) Flush is chosen
+because a `FLUSH` interrupt is easy to trigger (see
+*tests/features/interrupt.t*). Interrupt handling for flush is switched on
+by `--fuse-flush-handle-interrupt` (a hidden glusterfs command line flag).
+The implementation of flush interrupt is contained in the
+`fuse_flush_interrupt_handler()` function and blocks guarded by the
+
+```
+if (priv->flush_handle_interrupt) { ...
+```
+
+conditional (where `priv` is a `*fuse_private_t`).
+
+### Overview
+
+"Regular" fuse fops and interrupt handlers interact via a list containing
+interrupt records.
+
+If a fop wishes to have its interrupts handled, it needs to set up an
+interrupt record and insert it into the list; also when it's to finish
+(ie. in its "cbk" stage) it needs to delete the record from the list.
+
+If no interrupt happens, basically that's all to it - a list insertion
+and deletion.
+
+However, if an interrupt comes for the fop, the interrupt FUSE request
+will carry the data identifying an ongoing fop (that is, its `unique`),
+and based on that, the interrupt record will be looked up in the list, and
+the specific interrupt handler (a member of the interrupt record) will be
+called.
+
+Usually the fop needs to share some data with the interrupt handler to
+enable it to perform its task (also shared via the interrupt record).
+The interrupt API offers two approaches to manage shared data:
+- _Async or reference-counting strategy_: from the point on when the interrupt
+  record is inserted to the list, it's owned jointly by the regular fop and
+  the prospective interrupt handler. Both of them need to check before they
+  return if the other is still holding a reference; if not, then they are
+  responsible for reclaiming the shared data.
+- _Sync or borrow strategy_: the interrupt handler is considered a borrower
+  of the shared data. The interrupt handler should not reclaim the shared
+  data. The fop will wait for the interrupt handler to finish (ie., the borrow
+  to be returned), then it has to reclaim the shared data.
+
+The user of the interrupt API need to call the following functions to
+instrument this control flow:
+- `fuse_interrupt_record_insert()` in the fop to insert the interrupt record to
+   the list;
+- `fuse_interrupt_finish_fop()`in the fop (cbk) and
+- `fuse_interrupt_finish_interrupt()`in the interrupt handler
+
+to perform needed synchronization at the end their tenure. The data management
+strategies are implemented by the `fuse_interrupt_finish_*()` functions (which
+have an argument to specify which strategy to use); these routines take care
+of freeing the interrupt record itself, while the reclamation of the shared data
+is left to the API user.
+
+### Usage
+
+A given FUSE fop can be enabled to handle interrupts via the following
+steps:
+
+- Define a handler function (of type `fuse_interrupt_handler_t`).
+  It should implement the interrupt handling logic and in the end
+  call (directly or as async callback) `fuse_interrupt_finish_interrupt()`.
+  The `intstat` argument to `fuse_interrupt_finish_interrupt` should be
+  either `INTERRUPT_SQUELCHED` or `INTERRUPT_HANDLED`.
+    - `INTERRUPT_SQUELCHED` means that the interrupt could not be delivered
+      and the fop is going on uninterrupted.
+    - `INTERRUPT_HANDLED` means that the interrupt was actually handled. In
+      this case the fop will be answered from interrupt context with errno
+      `EINTR` (that is, the fop should not send a response to the kernel).
+
+  (the enum `fuse_interrupt_state` includes further members, which are reserved
+  for internal use).
+
+  We return to the `sync` and `datap` arguments later.
+- In the `fuse_<FOP>` function create an interrupt record using
+  `fuse_interrupt_record_new()`, passing the incoming `fuse_in_header` and
+  the above handler function to it.
+    - Arbitrary further data can be referred to via the `data` member of the
+      interrupt record that is to be passed on from fop context to
+      interrupt context.
+- When it's set up, pass the interrupt record to
+  `fuse_interrupt_record_insert()`.
+- In `fuse_<FOP>_cbk` call `fuse_interrupt_finish_fop()`.
+    - `fuse_interrupt_finish_fop()` returns a Boolean according to whether the
+      interrupt was handled. If it was, then the FUSE request is already
+      answered and the stack gets destroyed in `fuse_interrupt_finish_fop` so
+      `fuse_<FOP>_cbk()` can just return (zero). Otherwise follow the standard
+      cbk logic (answer the FUSE request and destroy the stack -- these are
+      typically accomplished by `fuse_err_cbk()`).
+- The last two argument of `fuse_interrupt_finish_fop()` and
+  `fuse_interrupt_finish_interrupt()` are `gf_boolean_t sync` and
+  `void **datap`.
+    - `sync` represents the strategy for freeing the interrupt record. The
+      interrupt handler and the fop handler are in race to get at the interrupt
+      record first (interrupt handler for purposes of doing the interrupt
+      handling, fop handler for purposes of deactivating the interrupt record
+      upon completion of the fop handling).
+        - If `sync` is true, then the fop handler will wait for the interrupt
+          handler to finish and it takes care of freeing.
+        - If `sync` is false, the loser of the above race will perform freeing.
+
+      Freeing is done within the respective interrupt finish routines, except
+      for the `data` field of the interrupt record; with respect to that, see
+      the discussion of the `datap` parameter below. The strategy has to be
+      consensual, that is, `fuse_interrupt_finish_fop()` and
+      `fuse_interrupt_finish_interrupt()` must pass the same value for `sync`.
+      If dismantling the resources associated with the interrupt record is
+      simple, `sync = _gf_false` is the suggested choice; `sync = _gf_true` can
+      be useful in the opposite case, when dismantling those resources would
+      be inconvenient to implement in two places or to enact in non-fop context.
+    - If `datap` is `NULL`, the `data` member of the interrupt record will be
+      freed within the interrupt finish routine.  If it points to a valid
+      `void *` pointer, and if caller is doing the cleanup (see `sync` above),
+      then that pointer will be directed to the `data` member of the interrupt
+      record and it's up to the caller what it's doing with it.
+        - If `sync` is true, interrupt handler can use `datap = NULL`, and
+          fop handler will have `datap` point to a valid pointer.
+        - If `sync` is false, and handlers pass a pointer to a pointer for
+          `datap`, they should check if the pointed pointer is NULL before
+          attempting to deal with the data.
+
+### FUSE answer for the interrupted fop
+
+The kernel acknowledges a successful interruption for a given FUSE request
+if the filesystem daemon answers it with errno EINTR; upon that, the syscall
+which induced the request will be abruptly terminated with an interrupt, rather
+than returning a value.
+
+In glusterfs, this can be arranged in two ways.
+
+- If the interrupt handler wins the race for the interrupt record, ie.
+  `fuse_interrupt_finish_fop()` returns true to `fuse_<FOP>_cbk()`, then, as
+  said above, `fuse_<FOP>_cbk()` does not need to answer the FUSE request.
+  That's because then the interrupt handler will take care about answering
+  it (with errno EINTR).
+- If `fuse_interrupt_finish_fop()` returns false to `fuse_<FOP>_cbk()`, then
+  this return value does not inform the fop handler whether there was an interrupt
+  or not. This return value occurs both when fop handler won the race for the
+  interrupt record against the interrupt handler, and when there was no interrupt
+  at all.
+
+  However, the internal logic of the fop handler might detect from other
+  circumstances that an interrupt was delivered. For example, the fop handler
+  might be sleeping, waiting for some data to arrive, so that a premature
+  wakeup (with no data present) occurs if the interrupt handler intervenes. In
+  such cases it's the responsibility of the fop handler to reply the FUSE
+  request with errro EINTR.
diff --git a/doc/developer-guide/gfapi-symbol-versions.md b/doc/developer-guide/gfapi-symbol-versions.md
new file mode 100644
index 00000000000..e4f4fe9f052
--- /dev/null
+++ b/doc/developer-guide/gfapi-symbol-versions.md
@@ -0,0 +1,270 @@
+
+## Symbol Versions and SO_NAMEs
+
+  In general, adding new APIs to a shared library does not require that
+symbol versions be used or the the SO_NAME be "bumped." These actions
+are usually reserved for when a major change is introduced, e.g. many
+APIs change or a signficant change in the functionality occurs.
+
+  Over the normal lifetime of a When a new API is added, the library is
+recompiled, consumers of the new API are able to do so, and existing,
+legacy consumers of the original API continue as before. If by some
+chance an old copy of the library is installed on a system, it's unlikely
+that most applications will be affected. New applications that use the
+new API will incur a run-time error terminate.
+
+  Bumping the SO_NAME, i.e. changing the shared lib's file name, e.g.
+from libfoo.so.0 to libfoo.so.1, which also changes the ELF SO_NAME
+attribute inside the file, works a little differently. libfoo.so.0
+contains only the old APIs. libfoo.so.1 contains both the old and new
+APIs. Legacy software that was linked with libfoo.so.0 continues to work
+as libfoo.so.0 is usually left installed on the system. New software that
+uses the new APIs is linked with libfoo.so.1, and works as long as
+long as libfoo.so.1 is installed on the system. Accidentally (re)installing
+libfoo.so.0 doesn't break new software as long as reinstalling doesn't
+erase libfoo.so.1.
+
+  Using symbol versions is somewhere in the middle. The shared library
+file remains libfoo.so.0 forever. Legacy APIs may or may not have an
+associated symbol version. New APIs may or may not have an associated
+symbol version either. In general symbol versions are reserved for APIs
+that have changed. Either the function's signature has changed, i.e. the
+return time or the number of paramaters, and/or the parameter types have
+changed. Another reason for using symbol versions on an API is when the
+behaviour or functionality of the API changes dramatically. As with a
+library that doesn't use versioned symbols, old and new applications
+either find or don't find the versioned symbols they need. If the versioned
+symbol doesn't exist in the installed library, the application incurs a
+run-time error and terminates.
+
+  GlusterFS wanted to keep tight control over the APIs in libgfapi.
+Originally bumping the SO_NAME was considered, and GlusterFS-3.6.0 was
+released with libgfapi.so.7. Not only was "7" a mistake (it should have
+been "6"), but it was quickly pointed out that many dependent packages
+that use libgfapi would be forced to be recompiled/relinked. Thus no
+packages of 3.6.0 were ever released and 3.6.1 was quickly released with
+libgfapi.so.0, but with symbol versions. There's no strong technical
+reason for either; the APIs have not changed, only new APIs have been
+added. It's merely being done in anticipation that some APIs might change
+sometime in the future.
+
+  Enough about that now, let's get into the nitty gritty——
+
+## Adding new APIs
+
+### Adding a public API.
+
+  This is the default, and the easiest thing to do. Public APIs have
+declarations in either glfs.h, glfs-handles.h, or, at your discretion,
+in a new header file intended for consumption by other developers.
+
+Here's what you need to do to add a new public API:
+
++ Write the declaration, e.g. in glfs.h:
+
+```C
+    int glfs_dtrt (const char *volname, void *stuff) __THROW
+```
+
++ Write the definition, e.g. in glfs-dtrt.c:
+
+```C
+   int
+   pub_glfs_dtrt (const char *volname, void *stuff)
+   {
+       ...
+       return 0;
+   }
+```
+
++ Add the symbol version magic for ELF, gnu toolchain to the definition.
+
+  following the definition of your new function in glfs-dtrtops.c, add a
+  line like this:
+
+```C
+  GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_dtrt, 3.7.0)
+```
+
+  The whole thing should look like:
+
+```C
+      int
+      pub_glfs_dtrt (const char *volname, void *stuff)
+      {
+           ...
+      }
+      GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_dtrt, 3.7.0);
+```
+
+  In this example, 3.7.0 refers to the Version the symbol will first
+  appear in. There's nothing magic about it, it's just a string token.
+  The current versions we have are 3.4.0, 3.4.2, 3.5.0, 3.5.1, and 3.6.0.
+  They are to be considered locked or closed. You can not, must not add
+  any new APIs and use these versions. Most new APIs will use 3.7.0. If
+  you add a new API appearing in 3.6.2 (and mainline) then you would use
+  3.6.2.
+
++ Add the symbol version magic for OS X to the declaration.
+
+  following the declaration in glfs.h, add a line like this:
+
+```C
+  GFAPI_PUBLIC(glfs_dtrt, 3.7.0)
+```
+
+  The whole thing should look like:
+
+```C
+     int glfs_dtrt (const char *volname, void *stuff) __THROW
+              GFAPI_PUBLIC(glfs_dtrt, 3.7.0);
+```
+
+  The version here must match the version associated with the definition.
+
++ Add the new API to the ELF, gnu toolchain link map file, gfapi.map
+
+  Most new public APIs will probably be added to a new section that
+  looks like this:
+
+```
+  GFAPI_3.7.0 {
+          global:
+                  glfs_dtrt;
+  } GFAPI_PRIVATE_3.7.0;
+```
+
+  if you're adding your new API to, e.g. 3.6.2, it'll look like this:
+
+```
+  GFAPI_3.6.2 {
+          global:
+                  glfs_dtrt;
+  } GFAPI_3.6.0;
+```
+
+  and you must change the
+```
+    GFAPI_PRIVATE_3.7.0 { ...} GFAPI_3.6.0;
+```
+  section to:
+```
+    GFAPI_PRIVATE_3.7.0 { ...} GFAPI_3.6.2;
+```
+
++ Add the new API to the OS X alias list file, gfapi.aliases.
+
+  Most new APIs will use a line that looks like this:
+
+```C
+ _pub_glfs_dtrt _glfs_dtrt$GFAPI_3.7.0
+```
+
+  if you're adding your new API to, e.g. 3.6.2, it'll look like this:
+
+```C
+ _pub_glfs_dtrt _glfs_dtrt$GFAPI_3.6.2
+```
+
+And that's it.
+
+
+### Adding a private API.
+
+  If you're thinking about adding a private API that isn't declared in
+one of the header files, then you should seriously rethink what you're
+doing and figure out how to put it in libglusterfs instead.
+
+If that hasn't convinced you, follow the instructions above, but use the
+_PRIVATE versions of macros, symbol versions, and aliases. If you're 1337
+enough to ignore this advice, then you're 1337 enough to figure out how
+to do it.
+
+
+## Changing an API.
+
+### Changing a public API.
+
+  There are two ways an API might change, 1) its signature has changed, or
+2) its new functionality or behavior is substantially different than the
+old. An APIs signature consists of the function return type, and the number
+and/or type of its parameters. E.g. the original API:
+
+```C
+   int glfs_dtrt (const char *volname, void *stuff);
+```
+
+and the changed API:
+
+```C
+   void *glfs_dtrt (const char *volname, glfs_t *ctx, void *stuff);
+```
+
+  One way to avoid a change like this, and which is preferable in many
+ways, is to leave the legacy glfs_dtrt() function alone, document it as
+deprecated, and simply add a new API, e.g. glfs_dtrt2(). Practically
+speaking, that's effectively what we'll be doing anyway, the difference
+is only that we'll use a versioned symbol to do it.
+
+  On the assumption that adding a new API is undesirable for some reason,
+perhaps the use of glfs_gnu() is just so pervasive that we really don't
+want to add glfs_gnu2().
+
++ change the declaration in glfs.h:
+
+```C
+  glfs_t *glfs_gnu (const char *volname, void *stuff) __THROW
+          GFAPI_PUBLIC(glfs_gnu, 3.7.0);
+````
+
+Note that there is only the single, new declaration.
+
++ change the old definition of glfs_gnu() in glfs.c:
+
+```C
+    struct glfs *
+    pub_glfs_gnu340 (const char * volname)
+    {
+        ...
+    }
+    GFAPI_SYMVER_PUBLIC(glfs_gnu340, glfs_gnu, 3.4.0);
+```
+
++ create the new definition of glfs_gnu in glfs.c:
+
+```C
+    struct glfs *
+    pub_glfs_gnu (const char * volname, void *stuff)
+    {
+        ...
+    }
+    GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_gnu, 3.7.0);
+```
+
++ Add the new API to the ELF, gnu toolchain link map file, gfapi.map
+
+```
+  GFAPI_3.7.0 {
+          global:
+                  glfs_gnu;
+  } GFAPI_PRIVATE_3.7.0;
+```
+
++ Update the OS X alias list file, gfapi.aliases, for both versions:
+
+Change the old line:
+```C
+  _pub_glfs_gnu _glfs_gnu$GFAPI_3.4.0
+```
+to:
+```C
+  _pub_glfs_gnu340 _glfs_gnu$GFAPI_3.4.0
+```
+
+Add a new line:
+```C
+  _pub_glfs_gnu _glfs_gnu$GFAPI_3.7.0
+```
+
++ Lastly, change all gfapi internal calls glfs_gnu to the new API.
+
diff --git a/doc/developer-guide/identifying-resource-leaks.md b/doc/developer-guide/identifying-resource-leaks.md
new file mode 100644
index 00000000000..950cae79b0a
--- /dev/null
+++ b/doc/developer-guide/identifying-resource-leaks.md
@@ -0,0 +1,200 @@
+# Identifying Resource Leaks
+
+Like most other pieces of software, GlusterFS is not perfect in how it manages
+its resources like memory, threads and the like. Gluster developers try hard to
+prevent leaking resources but releasing and unallocating the used structures.
+Unfortunately every now and then some resource leaks are unintentionally added.
+
+This document tries to explain a few helpful tricks to identify resource leaks
+so that they can be addressed.
+
+
+## Debug Builds
+
+There are certain techniques used in GlusterFS that make it difficult to use
+tools like Valgrind for memory leak detection. There are some build options
+that make it more practical to use Valgrind and other tools. When running
+Valgrind, it is important to have GlusterFS builds that contain the
+debuginfo/symbols. Some distributions (try to) strip the debuginfo to get
+smaller executables. Fedora and RHEL based distributions have sub-packages
+called ...-debuginfo that need to be installed for symbol resolving.
+
+
+### Memory Pools
+
+By using memory pools, there are no allocation/freeing of single structures
+needed. This improves performance, but also makes it impossible to track the
+allocation and freeing of srtuctures.
+
+It is possible to disable the use of memory pools, and use standard `malloc()`
+and `free()` functions provided by the C library. Valgrind is then able to
+track the allocated areas and verify if they have been free'd. In order to
+disable memory pools, the Gluster sources needs to be configured with the
+`--enable-debug` option:
+
+```shell
+./configure --enable-debug
+```
+
+When building RPMs, the `.spec` handles the `--with=debug` option too:
+
+```shell
+make dist
+rpmbuild -ta --with=debug glusterfs-....tar.gz
+```
+
+### Dynamically Loaded xlators
+
+Valgrind tracks the call chain of functions that do memory allocations. The
+addresses of the functions are stored and before Valgrind exits the addresses
+are resolved into human readable function names and offsets (line numbers in
+source files). Because Gluster loads xlators dynamically, and unloads then
+before exiting, Valgrind is not able to resolve the function addresses into
+symbols anymore. Whenever this happend, Valgrind shows `???` in the output,
+like
+
+```
+  ==25170== 344 bytes in 1 blocks are definitely lost in loss record 233 of 324
+  ==25170==    at 0x4C29975: calloc (vg_replace_malloc.c:711)
+  ==25170==    by 0x52C7C0B: __gf_calloc (mem-pool.c:117)
+  ==25170==    by 0x12B0638A: ???
+  ==25170==    by 0x528FCE6: __xlator_init (xlator.c:472)
+  ==25170==    by 0x528FE16: xlator_init (xlator.c:498)
+  ...
+```
+
+These `???` can be prevented by not calling `dlclose()` for unloading the
+xlator. This will cause a small leak of the handle that was returned with
+`dlopen()`, but for improved debugging this can be acceptible. For this and
+other Valgrind features, a `--enable-valgrind` option is available to
+`./configure`. When GlusterFS is built with this option, Valgrind will be able
+to resolve the symbol names of the functions that do memory allocations inside
+xlators.
+
+```shell
+./configure --enable-valgrind
+```
+
+When building RPMs, the `.spec` handles the `--with=valgrind` option too:
+
+```shell
+make dist
+rpmbuild -ta --with=valgrind glusterfs-....tar.gz
+```
+
+## Running Valgrind against a single xlator
+
+Debugging a single xlator is not trivial. But there are some tools to make it
+easier. The `sink` xlator does not do any memory allocations itself, but
+contains just enough functionality to mount a volume with only the `sink`
+xlator. There is a little gfapi application under `tests/basic/gfapi/` in the
+GlusterFS sources that can be used to run only gfapi and the core GlusterFS
+infrastructure with the `sink` xlator. By extending the `.vol` file to load
+more xlators, each xlator can be debugged pretty much separately (as long as
+the xlators have no dependencies on each other). A basic Valgrind run with the
+suitable configure options looks like this:
+
+```shell
+./autogen.sh
+./configure --enable-debug --enable-valgrind
+make && make install
+cd tests/basic/gfapi/
+make gfapi-load-volfile
+valgrind ./gfapi-load-volfile sink.vol
+```
+
+Combined with other very useful options to Valgrind, the following execution
+shows many more useful details:
+
+```shell
+valgrind \
+        --fullpath-after= --leak-check=full --show-leak-kinds=all \
+        ./gfapi-load-volfile sink.vol
+```
+
+Note that the `--fullpath-after=` option is left empty, this makes Valgrind
+print the full path and filename that contains the functions:
+
+```
+==2450== 80 bytes in 1 blocks are definitely lost in loss record 8 of 60
+==2450==    at 0x4C29975: calloc (/builddir/build/BUILD/valgrind-3.11.0/coregrind/m_replacemalloc/vg_replace_malloc.c:711)
+==2450==    by 0x52C6F73: __gf_calloc (/usr/src/debug/glusterfs-3.11dev/libglusterfs/src/mem-pool.c:117)
+==2450==    by 0x12F10CDA: init (/usr/src/debug/glusterfs-3.11dev/xlators/meta/src/meta.c:231)
+==2450==    by 0x528EFD5: __xlator_init (/usr/src/debug/glusterfs-3.11dev/libglusterfs/src/xlator.c:472)
+==2450==    by 0x528F105: xlator_init (/usr/src/debug/glusterfs-3.11dev/libglusterfs/src/xlator.c:498)
+==2450==    by 0x52D9D8B: glusterfs_graph_init (/usr/src/debug/glusterfs-3.11dev/libglusterfs/src/graph.c:321)
+...
+```
+
+In the above example, the `init` function in `xlators/meta/src/meta.c` does a
+memory allocation on line 231. This memory is never free'd again, and hence
+Valgrind logs this call stack. When looking in the code, it seems that the
+allocation of `priv` is assigned to the `this->private` member of the
+`xlator_t` structure. Because the allocation is done in `init()`, free'ing is
+expected to happen in `fini()`. Both functions are shown below, with the
+inclusion of the empty `fini()`:
+
+
+```
+226 int
+227 init (xlator_t *this)
+228 {
+229         meta_priv_t *priv = NULL;
+230 
+231         priv = GF_CALLOC (sizeof(*priv), 1, gf_meta_mt_priv_t);
+232         if (!priv)
+233                 return -1;
+234 
+235         GF_OPTION_INIT ("meta-dir-name", priv->meta_dir_name, str, out);
+236 
+237         this->private = priv;
+238 out:
+239         return 0;
+240 }
+241 
+242 
+243 int
+244 fini (xlator_t *this)
+245 {
+246         return 0;
+247 }
+```
+
+In this case, the resource leak can be addressed by adding a single line to the
+`fini()` function:
+
+```
+243 int
+244 fini (xlator_t *this)
+245 {
+246         GF_FREE (this->private);
+247         return 0;
+248 }
+```
+
+Running the same Valgrind command and comparing the output will show that the
+memory leak in `xlators/meta/src/meta.c:init` is not reported anymore.
+
+### Running DRD, the Valgrind thread error detector
+
+When configuring GlusterFS with:
+
+```shell
+./configure --enable-valgrind
+```
+
+the default Valgrind tool (Memcheck) is enabled. But it's also possble to select
+one of Memcheck or DRD by using:
+
+```shell
+./configure --enable-valgrind=memcheck
+```
+
+or:
+
+```shell
+./configure --enable-valgrind=drd
+```
+
+respectively. When using DRD, it's recommended to consult
+https://valgrind.org/docs/manual/drd-manual.html before running.
diff --git a/doc/developer-guide/logging-guidelines.md b/doc/developer-guide/logging-guidelines.md
new file mode 100644
index 00000000000..0e6b2588535
--- /dev/null
+++ b/doc/developer-guide/logging-guidelines.md
@@ -0,0 +1,132 @@
+Guidelines on using the logging framework within a component
+============================================================
+Gluster library libglusterfs.so provides message logging abstractions that
+are intended to be used across all code/components within gluster.
+
+There could be potentially 2 major cases how the logging infrastructure is
+used,
+  - A new gluster service daemon or end point is created
+    - The service daemon infrastructure itself initlializes the logging
+    infrastructure (i.e calling gf_log_init and related set functions)
+      - See, glusterfsd.c:logging_init
+    - Alternatively there could be a case where an end point service (say
+    gfapi) may need to do the required initialization
+    - This document does not (yet?) cover guidelines for these cases. Best
+    bet would be to look at code in glusterfsd.c:logging_init (or equivalent)
+    in case a need arises and you reach this document.
+  - A new xlator or subcomponent is written as a part of the stack
+    - Primarily in this case, the consumer of the logging APIs would only
+    invoke an API to *log* a particular message at a certain severity
+    - This document elaborates on this use of the message logging framework
+    in this context
+
+There are 2 interfaces provided to log messages,
+1. The older gf_log* interface
+2. The newer gf_msg* interface
+
+1. Older _to_be_deprecated_ gf_log* interface
+  - Not much documentation is provided for this interface here, as these are
+  meant to be deprecated and all code should be moved to the newer gf_msg*
+  interface
+
+2. New gf_msg* interface
+  - The set of interfaces provided by gf_msg are,
+    - NOTE: It is best to consult logging.h for the latest interfaces, as
+    this document may get out of sync with the code
+
+    *gf_msg(dom, levl, errnum, msgid, fmt...)*
+      - Used predominantly to log a message above TRACE and DEBUG levels
+      - See further guidelines below
+
+    *gf_msg_debug(dom, errnum, fmt...)*
+    *gf_msg_trace(dom, errnum, fmt...)*
+      - Above are handy shortcuts for TRACE and DEBUG level messages
+      - See further guidelines below
+
+    *gf_msg_callingfn(dom, levl, errnum, msgid, fmt...)*
+      - Useful function when a backtrace to detect calling routines that
+      reached this execution point needs to be logged in addition to the
+      message
+      - This is very handy for framework libraries or code, as there could
+      be many callers to the same, and the real failure would need the call
+      stack to determine who the caller actually was
+      - A good example is the dict interfaces using this function to log who
+      called, when a particular error/warning needs to be displayed
+      - See further guidelines below
+
+    *gf_msg_plain(levl, fmt...)*
+    *gf_msg_plain_nomem(levl, msg)*
+    *gf_msg_vplain(levl, fmt, va)*
+    *gf_msg_backtrace_nomem*
+      - The above interfaces are provided to log messages without any typical
+      headers (like the time stamp, dom, errnum etc.). The primary users of
+      the above interfaces are, when printing the final graph, or printing
+      the configuration when a process is about dump core or abort, or
+      printing the backtrace when a process receives a critical signal
+      - These interfaces should not be used outside the scope of the users
+      above, unless you know what you are doing
+
+    *gf_msg_nomem(dom, levl, size)*
+      - Very crisp messages, throwing out file, function, line numbers, in
+      addition to the passed in size
+      - These are used in the memory allocation abstractions interfaces in
+      gluster, when the allocation fails (hence a no-mem log message, so that
+      further allocation is not attempted by the logging infrastructure)
+      - If you are contemplating using these, then you know why and how
+
+  - Guidelines for the various arguments to the interfaces
+    **dom**
+      - The domain from which this message appears, IOW for xlators it should
+      be the name of the xlator (as in this->name)
+
+      - The intention of this string is to distinguish from which
+      component/xlator the message is being printed
+
+      - This information can also be gleaned from the FILE:LINE:FUNCTION
+      information printed in the message anyway, but is retained to provide
+      backward compatability to the messages as seen by admins earlier
+
+    **levl**
+      - Consult logging.h:gf_loglevel_t for logging levels (they pretty much
+      map to syslog levels)
+
+    **errnum**
+      - errno should be passed in here, if available, 0 otherwise. This auto
+      enables the logging infrastructure to print (man 3) strerror form of the
+      same at the end of the message in a consistent format
+
+      - If any message is already adding the strerror as a parameter to the
+      message, it is suggested/encouraged to remove the same and pass it as
+      errnum
+
+      - The intention is that, if all messages did this using errnum and not
+      as a part of their own argument list, the output would look consistent
+      for the admin and cross component developers when reading logs
+
+    **msgid**
+      - This is a unique message ID per message (which now is more a message
+      ID per group of messages in the implementation)
+
+      - Rules for generating this ID can be found in dht-messages.h (it used
+      to be template-common-messages.h, but that template is not updated,
+      this comment should be changed once that is done) and glfs-message-id.h
+
+      - Every message that is *above* TRACE and DEBUG should get a message
+      ID, as this helps generating message catalogs that can help admins to
+      understand the context of messages better. Another intention is that
+      automated message parsers could detect a class of message IDs and send
+      out notifications on the same, rather than parse the message string
+      itself (which if it changes, the input to the parser has to change, etc.
+      and hence is better to retain message IDs)
+
+      - Ok so if intention is not yet clear, look at journald MESSAGE ID
+      motivation, as that coupled with ident/dom above is expected to provide
+      us with similar advantages
+
+      - Bottomline: Every message gets its own ID, in case a message is
+      *not* DEBUG or TRACE and still is developer centric, a generic message
+      ID per component *maybe* assigned to the same to provide ease of use
+      of the API
+
+    **fmt**
+      - As in format argument of (man 3) printf
diff --git a/doc/network_compression.md b/doc/developer-guide/network_compression.md
index 7327591ef63..1222a765276 100644
--- a/doc/network_compression.md
+++ b/doc/developer-guide/network_compression.md
@@ -1,9 +1,9 @@
-#On-Wire Compression + Decompression
+# On-Wire Compression + Decompression
 
 The 'compression translator' compresses and decompresses data in-flight
 between client and bricks.
 
-###Working
+### Working
 When a writev call occurs, the client compresses the data before sending it to
 brick. On the brick, compressed data is decompressed. Similarly, when a readv
 call occurs, the brick compresses the data before sending it to client. On the
@@ -19,7 +19,7 @@ During normal operation, this is the format of data sent over wire:
 The trailer contains the CRC32 checksum and length of original uncompressed
 data. This is used for validation.
 
-###Usage
+### Usage
 
 Turning on compression xlator:
 
@@ -27,7 +27,7 @@ Turning on compression xlator:
 gluster volume set <vol_name> network.compression on
 ~~~
 
-###Configurable parameters (optional)
+### Configurable parameters (optional)
 
 **Compression level**
 ~~~
@@ -35,10 +35,10 @@ gluster volume set <vol_name> network.compression.compression-level 8
 ~~~
 
 ~~~
-0  : no compression
-1  : best speed
-9  : best compression
--1 : default compression
+ 0  : no compression
+ 1  : best speed
+ 9  : best compression
+-1  : default compression
 ~~~
 
 **Minimum file size**
@@ -55,7 +55,7 @@ Other less frequently used parameters include `network.compression.mem-level`
 and `network.compression.window-size`. More details can about these options
 can be found by running `gluster volume set help` command.
 
-###Known Issues and Limitations
+### Known Issues and Limitations
 
 * Compression translator cannot work with striped volumes.
 * Mount point hangs when writing a file with write-behind xlator turned on. To
@@ -65,7 +65,7 @@ set`performance.strict-write-ordering` to on.
 distribute volumes. This limitation is caused by AFR not being able to
 propagate xdata. This issue has been fixed in glusterfs versions > 3.5
 
-###TODO
+### TODO
 Although zlib offers high compression ratio, it is very slow. We can make the
 translator pluggable to add support for other compression methods such as
 [lz4 compression](https://code.google.com/p/lz4/)
diff --git a/doc/developer-guide/options-to-contribute.md b/doc/developer-guide/options-to-contribute.md
new file mode 100644
index 00000000000..3f0d84e7645
--- /dev/null
+++ b/doc/developer-guide/options-to-contribute.md
@@ -0,0 +1,212 @@
+# A guide for contributors
+
+While you have gone through 'how to contribute' guides, if you are
+not sure what to work on, but really want to help the project, you
+have now landed on the right document :-)
+
+### Basic
+
+Instead of planning to fix **all** the below issues in one patch,
+we recommend you to have a a constant, continuous flow of improvements
+for the project. We recommend you to pick 1 file (or just few files) at
+a time to address below issues.
+Pick any `.c` (or `.h`) file, and you can send a patch which fixes **any**
+of the below themes. Ideally, fix all such occurrences in the file, even
+though, the reviewers would review even a single line change patch
+from you.
+
+1. Check for variable definitions, and if there is an array definition,
+which is very large at the top of the function, see if you can re-scope
+the variable to relevant sections (if it helps).
+
+Most of the time, some of these arrays may be used for 'error' handling,
+and it is possible to use them only in that scope.
+
+Reference: https://review.gluster.org/20846/
+
+
+2. Check for complete string initialization at the beginning of a function.
+Ideally, there is no reason to initialize a string. Fix it across the file.
+
+Example:
+
+`char new_path_name[PATH_MAX] = {0};` to `char new_path_name[PATH_MAX];`
+
+
+3. Change `calloc()` to `malloc()` wherever it makes sense.
+
+In a case of allocating a structures, where you expect certain (or most of)
+variables to be 0 (or NULL), it makes sense to use calloc(). But otherwise,
+there is an extra cost to `memset()` the whole object after allocating it.
+While it is not a significant improvement in performance, code which gets
+hit 1000s of times in a second, it would add some value.
+
+Reference: https://review.gluster.org/20878/
+
+
+4. You can consider using `snprintf()`, instead of `strncpy()` while dealing
+with strings.
+
+strncpy() won't null terminate if the dest buffer isn't big enough; snprintf()
+does. While most of the string operations in the code is on array, and larger
+size than required, strncpy() does an extra copy of 0s at the end of
+string till the size of the array. It makes sense to use `snprintf()`,
+which doesn't suffer from that behavior.
+
+Also check the return value from snprintf() for buffer overflow and handle
+accordingly
+
+Reference: https://review.gluster.org/20925/
+
+
+5. Now, pick a `.h` file, and see if a structure is very large, and see
+if re-aligning them as per [coding-standard](./coding-standard.md) gives any size benefit,
+if yes, go ahead and change it. Make sure you check all the structures
+in the file for similar pattern.
+
+Reference: [Check this section](https://github.com/gluster/glusterfs/blob/master/doc/developer-guide/coding-standard.md#structure-members-should-be-aligned-based-on-the-padding-requirements
+
+
+### If you are up for more :-)
+
+Good progress! Glad you are interested to know more. We are surely interested
+in next level of contributions from you!
+
+#### Coverity
+
+Visit [Coverity Dashboard](https://scan.coverity.com/projects/gluster-glusterfs?tab=overview).
+
+Now, if the number of defect is not 0, you have an opportunity to contribute.
+
+You get all the detail on why the particular defect is mentioned there, and
+most probable hint on how to fix it. Do it!
+
+Reference: https://review.gluster.org/21394/
+
+Use the same reference Id (789278) as the patch, so we can capture it is in
+single bugzilla.
+
+#### Clang-Scan
+
+Clang-Scan is a tool which scans the .c files and reports the possible issues,
+similar to coverity, but a different tool. Over the years we have seen, they
+both report very different set of issues, and hence there is a value in fixing it.
+
+GlusterFS project gets tested with clang-scan job every night, and the report is
+posted in the [job details page](https://build.gluster.org/job/clang-scan/lastCompletedBuild/clangScanBuildBugs/).
+As long as the number is not 0 in the report here, you have an opportunity to
+contribute! Similar to coverity dashboard, click on 'Details' to find out the
+reason behind that report, and send a patch.
+
+Reference: https://review.gluster.org/21025
+
+Again, you can use reference Id (1622665) for these patches!
+
+
+### I am good with programming, I would like to do more than above!
+
+#### Locked regions / Critical sections
+
+In the file you open, see if the lock is taken only to increment or decrement
+a flag, counter etc. If yes, then recommend you to convert it to ATOMIC locks.
+It is simple activity, but, if you know programing, you would know the benefit
+here.
+
+NOTE: There may not always a possibility to do this! You may have to check
+with developers first before going ahead.
+
+Reference: https://review.gluster.org/21221/
+
+
+#### ASan (address sanitizer)
+
+[The job](https://build.gluster.org/job/asan/) runs regression with asan builds,
+and you can also run glusterfs with asan on your workload to identify the leaks.
+If there are any leaks reported, feel free to check it, and send us patch.
+
+You can also run `valgrind` and let us know what it reports.
+
+Reference: https://review.gluster.org/21397
+
+
+#### Porting to different architecture
+
+This is something which we are not focusing right now, happy to collaborate!
+
+Reference: https://review.gluster.org/21276
+
+
+#### Fix 'TODO/FIXME' in codebase
+
+There are few cases of pending features, or pending validations, which are
+pending from sometime. You can pick them in the given file, and choose to
+fix it.
+
+
+### I don't know C, but I am interested to contribute in some way!
+
+You are most welcome! Our community is open for your contribution! First thing
+which comes to our mind is **documentation**. Next is, **testing** or validation.
+
+If you have some hardware, and want to run some performance comparisons with
+different version, or options, and help us to tune better is also a great help.
+
+
+#### Documentation
+
+1. We have some documentation in [glusterfs repo](../), go through these, and
+see if you can help us to keep up-to-date.
+
+2. The https://docs.gluster.org is powered by https://github.com/gluster/glusterdocs
+repo. You can check out the repo, and help in keeping that up-to-date.
+
+3. [Our website](https://gluster.org) is maintained by https://github.com/gluster/glusterweb
+repo. Help us to keep this up-to-date, and add content there.
+
+4. Write blogs about Gluster, and your experience, and make world know little
+more about Gluster, and your use-case, and how it helped to solve the problem.
+
+
+#### Testing
+
+1. There is a regression test suite in glusterfs, which runs with every patch, and is
+triggered by just running `./run-tests.sh` from the root of the project repo.
+
+You can add more test case to match your use-case, and send it as a patch, so you
+can make sure all future patches in glusterfs would keep your usecase intact.
+
+2. [Glusto-Tests](https://github.com/gluster/glusto-tests): This is another testing
+framework written for gluster, and makes use of clustered setup to test different
+use-cases, and helps to validate many bugs.
+
+
+#### Ansible
+
+Gluster Organization has rich set of ansible roles, which are actively maintained.
+Feel free to check them out here - https://github.com/gluster/gluster-ansible
+
+
+#### Monitoring
+
+We have prometheus repo, and are actively working on adding more metrics. Add what
+you need @ https://github.com/gluster/gluster-prometheus
+
+
+#### Health-Report
+
+This is a project, where at any given point in time, you want to run some set of
+commands locally, and get an output to analyze the status, it can be added.
+Contribute @ https://github.com/gluster/gluster-health-report
+
+
+### All these C/bash/python is old-school, I want something in containers.
+
+We have something for you too :-)
+
+Please visit our https://github.com/gluster/gcs repo for checking how you can help,
+and how gluster can help you in container world.
+
+
+### Note
+
+For any queries, best way is to contact us through mailing-list, <mailto:gluster-devel@gluster.org>
diff --git a/doc/hacker-guide/en-US/markdown/posix.md b/doc/developer-guide/posix.md
index 84c813e55a2..84c813e55a2 100644
--- a/doc/hacker-guide/en-US/markdown/posix.md
+++ b/doc/developer-guide/posix.md
diff --git a/doc/rpc-for-glusterfs.changes-done.txt b/doc/developer-guide/rpc-for-glusterfs.changes-done.txt
index 6bbbca78826..6bbbca78826 100644
--- a/doc/rpc-for-glusterfs.changes-done.txt
+++ b/doc/developer-guide/rpc-for-glusterfs.changes-done.txt
diff --git a/doc/developer-guide/rpc-for-glusterfs.new-versions.md b/doc/developer-guide/rpc-for-glusterfs.new-versions.md
new file mode 100644
index 00000000000..e3da5efa4a2
--- /dev/null
+++ b/doc/developer-guide/rpc-for-glusterfs.new-versions.md
@@ -0,0 +1,32 @@
+# GlusterFS RPC program versions
+
+## Compatibility
+
+RPC layer of glusterfs is implemented with possible changes over the protocol layers in mind. If there are any changes in the FOPs from what is assumed to be client side, and whats in serverside, they are to be added as a separate program table.
+
+### Program tables and Versions
+
+A given RPC program has a specific Task, and Version along with actors belonging to the program. In any of the programs, if a new actor is added, it is very important to define one more program with different version, and then keep both, if both are supported. Or else, it is important to handle the 'handshake' properly.
+
+#### Server details
+
+More info on RPC program is at `rpc/rpc-lib/src/rpcsvc.h` and check for structure `rpcsvc_actor_t` and `struct rpcsvc_program`. For usage, check `xlators/protocol/server/src/server-rpc-fops.c`
+
+#### Client details
+
+For details on client structures check `rpc/rpc-lib/src/rpc-clnt.h` for `rpc_clnt_procedure_t` and `rpc_clnt_program_t`. For usage, check `xlators/protocol/client/src/client-rpc-fops.c`
+
+## Protocol
+
+A protocol is what is agreed between two parties. In glusterfs, a RPC protocol is defined as .x file, which then gets converted to .c/.h file using `rpcgen`. There are different protocols defined for communication between `xlators/mgmt/glusterd <==> glusterfsd`, `gluster CLI <==> glusterd`, and `client-protocol <==> server-protocol`
+
+Once a protocol is defined and a release is made with that protocol, make sure no one changes it. Any edits to a given structure there should be a new version of the structure, and also it should get used in new actor, and thus new program version.
+
+## Server and Client Handshake
+
+When a client succeeds to establish a connect (it can be any transport, socket, ib-verbs or unix), client sends a dump (GF_DUMP_DUMP) request to server, which will respond back with all the supported versions of the server RPC (the supported programs which are registered with `rpcsvc_program_register()`).
+
+A client which expects certain programs to be present in server, it should be taking care of looking for it in the handshake methods, and take appropriate action depending on what to do next. In general a compatibility issue should be handled at handshake level itself, thus we can clearly let user/admin know of any 'in-compatibilities'. 
+As a developer of GlusterFS protocol layer, one just has to make sure *never to make changes to existing program structures*, but they have to add new programs if required. New programs can have the same actors as present in existing, and also little more. Or it can even have same actor behave differently, take different parameter.
+
+If this is followed properly, there would be smooth upgrade / downgrade of versions. If not, technically, it is 100% guarantee of getting compatibility related issues.
diff --git a/doc/developer-guide/syncop.md b/doc/developer-guide/syncop.md
new file mode 100644
index 00000000000..bcc8bd08e01
--- /dev/null
+++ b/doc/developer-guide/syncop.md
@@ -0,0 +1,72 @@
+# syncop framework
+A coroutines-based, cooperative multi-tasking framework.
+
+## Topics
+
+- Glossary
+- Lifecycle of a synctask
+- Existing usage
+
+
+## Glossary
+
+### syncenv
+
+syncenv is an object that provides access to a pool of worker threads.
+synctasks execute in a syncenv.
+
+### synctask
+
+synctask can be informally defined as a pair of function pointers, namely _the
+call_ and _the callback_ (see syncop.h for more details).
+
+        synctask_fn_t - 'the call'
+        synctask_cbk_t - 'the callback'
+
+synctask has two modes of operation,
+
+1. The calling thread waits for the synctask to complete.
+2. The calling thread schedules the synctask and continues.
+
+synctask guarantees that the callback is called _after_ the call completes.
+
+### Lifecycle of a synctask
+
+A synctask could go into the following stages while in execution.
+
+- CREATED  - On calling synctask_create/synctask_new.
+
+- RUNNABLE - synctask is queued in env->runq.
+
+- RUNNING  - When one of syncenv's worker threads calls synctask_switch_to.
+
+- WAITING  - When a synctask calls synctask_yield.
+
+- DONE     - When a synctask has run to completion.
+
+
+                                +-------------------------------+
+                                |            CREATED            |
+                                +-------------------------------+
+                                  |
+                                  | synctask_new/synctask_create
+                                  v
+                                +-------------------------------+
+                                |    RUNNABLE (in env->runq)    | <+
+                                +-------------------------------+  |
+                                  |                                |
+                                  | synctask_switch_to             |
+                                  v                                |
+ +------+  on task completion   +-------------------------------+  |
+ | DONE | <-------------------- |            RUNNING            |  | synctask_wake/wake
+ +------+                       +-------------------------------+  |
+                                  |                                |
+                                  | synctask_yield/yield           |
+                                  v                                |
+                                +-------------------------------+  |
+                                |    WAITING (in env->waitq)    | -+
+                                +-------------------------------+
+
+Note: A synctask is not guaranteed to run on the same thread throughout its
+lifetime. Every time a synctask yields, it is possible for it to run on a
+different thread.
diff --git a/doc/developer-guide/thread-naming.md b/doc/developer-guide/thread-naming.md
new file mode 100644
index 00000000000..513140d4437
--- /dev/null
+++ b/doc/developer-guide/thread-naming.md
@@ -0,0 +1,104 @@
+Thread Naming
+================
+Gluster processes spawn many threads; some threads are created by libglusterfs
+library, while others are created by xlators. When gfapi library is used in an
+application, some threads belong to the application and some are spawned by
+gluster libraries. We also have features where n number of threads are spawned
+to act as worker threads for same operation.
+
+In all the above cases, it is useful to be able to determine the list of threads
+that exist in runtime. Naming threads when you create them is the easiest way to
+provide that information to kernel so that it can then be queried by any means.
+
+How to name threads
+-------------------
+We have two wrapper functions in libglusterfs for creating threads. They take
+name as an argument and set thread name after its creation.
+
+```C
+gf_thread_create (pthread_t *thread, const pthread_attr_t *attr,
+                  void *(*start_routine)(void *), void *arg, const char *name)
+```
+
+```C
+gf_thread_create_detached (pthread_t *thread,
+                           void *(*start_routine)(void *), void *arg,
+                           const char *name)
+```
+
+As max name length for a thread in POSIX is only 16 characters including the
+'\0' character, you have to be a little creative with naming. Also, it is
+important that all Gluster threads have common prefix. Considering these
+conditions, we have "glfs_" as prefix for all the threads created by these
+wrapper functions. It is responsibility of the owner of thread to provide the
+suffix part of the name. It does not have to be a descriptive name, as it has
+only 10 letters to work with. However, it should be unique enough such that it
+can be matched with a table which describes it.
+
+If n number of threads are spwaned to perform same function, it is must that the
+threads are numbered.
+
+Table of thread names
+---------------------
+Thread names don't have to be a descriptive; however, it should be unique enough
+such that it can be matched with a table below without ambiguity.
+
+- bdaio    - block device aio
+- brfsscan - bit rot fs scanner
+- brhevent - bit rot event handler
+- brmon    - bit rot monitor
+- brosign  - bit rot one shot signer
+- brpobj   - bit rot object processor
+- brsproc  - bit rot scrubber
+- brssign  - bit rot stub signer
+- brswrker - bit rot worker
+- clogc    - changelog consumer
+- clogcbki - changelog callback invoker
+- clogd    - changelog dispatcher
+- clogecon - changelog reverse connection
+- clogfsyn - changelog fsync
+- cloghcon - changelog history consumer
+- clogjan  - changelog janitor
+- clogpoll - changelog poller
+- clogproc - changelog process
+- clogro   - changelog rollover
+- ctrcomp  - change time recorder compaction
+- dhtdf    - dht defrag task
+- dhtdg    - dht defrag start
+- dhtfcnt  - dht rebalance file counter
+- ecshd    - ec heal daemon
+- epollN   -  epoll thread
+- fdlwrker - fdl worker
+- fusenoti - fuse notify
+- fuseproc - fuse main thread
+- gdhooks  - glusterd hooks
+- glfspoll -  gfapi poller thread
+- idxwrker - index worker
+- iosdump  - io stats dump
+- iotwr    - io thread worker
+- jbrflush - jbr flush
+- leasercl - lease recall
+- memsweep - sweeper thread for mem pools
+- nfsauth  - nfs auth
+- nfsnsm   - nfs nsm
+- nfsudp   - nfs udp mount
+- nlmmon   - nfs nlm/nsm mon
+- posixaio - posix aio
+- posixfsy - posix fsync
+- posixhc  - posix heal
+- posixjan - posix janitor
+- posixrsv - posix reserve
+- quiesce  - quiesce dequeue
+- rdmaAsyn - rdma async event handler
+- rdmaehan - rdma completion handler
+- rdmarcom - rdma receive completion handler
+- rdmascom - rdma send completion handler
+- rpcsvcrh - rpcsvc request handler
+- scleanup - socket cleanup
+- shdheal  - self heal daemon
+- sigwait  -  glusterfsd sigwaiter
+- spoller  - socket poller
+- sprocN   - syncop worker thread
+- tbfclock - token bucket filter token generator thread
+- timer    - timer thread
+- upreaper - upcall reaper
diff --git a/doc/hacker-guide/en-US/markdown/translator-development.md b/doc/developer-guide/translator-development.md
index edadd5150dc..f75935519f6 100644
--- a/doc/hacker-guide/en-US/markdown/translator-development.md
+++ b/doc/developer-guide/translator-development.md
@@ -51,7 +51,7 @@ if (!(xl->fini = dlsym (handle, "fini"))) {
 In this example, `xl` is a pointer to the in-memory object for the translator
 we're loading. As you can see, it's looking up various symbols *by name* in the
  shared object it just loaded, and storing pointers to those symbols. Some of
-them (e.g. init are functions, while others e.g. fops are dispatch tables
+them (e.g. init) are functions, while others (e.g. fops) are dispatch tables
 containing pointers to many functions. Together, these make up the translator's
  public interface.
 
@@ -102,7 +102,7 @@ various structures in logs. I've never used it myself, though I probably
 should. What's noteworthy here is that we don't even define dumpops. That's
 because all of the functions that might use these dispatch functions will check
  for `xl->dumpops` being `NULL` before calling through it. This is in sharp
-contrast to the behavior for `fops` and `cbks1`, which *must* be present. If
+contrast to the behavior for `fops` and `cbks`, which *must* be present. If
 they're not, translator loading will fail because these pointers are not
 checked every time and if they're `NULL` then we'll segfault. That's why we
 provide an empty definition for cbks; it's OK for the individual function
@@ -472,7 +472,7 @@ hello
 Now let's interrupt the process and see where we are.
 
 ```
-^C
+
 Program received signal SIGINT, Interrupt.
 0x0000003a0060b3dc in pthread_cond_wait@@GLIBC_2.3.2 ()
                    from /lib64/libpthread.so.0
@@ -664,3 +664,20 @@ started writing my first for-real translators, and probably even a bit more. I
 hope you've enjoyed this series, or at least found it useful, and if you have
 any suggestions for other topics I should cover please let me know (via
 comments or email, IRC or Twitter).
+
+Other versions
+--------------
+
+Original author's site:
+
+ * [Translator 101 - Setting the Stage](http://pl.atyp.us/hekafs.org/index.php/2011/11/translator-101-class-1-setting-the-stage/)
+
+ * [Translator 101 - Init, Fini and Private Context](http://pl.atyp.us/hekafs.org/index.php/2011/11/translator-101-lesson-2-init-fini-and-private-context/)
+
+ * [Translator 101 - This Time for Real](http://pl.atyp.us/hekafs.org/index.php/2011/11/translator-101-lesson-3-this-time-for-real/)
+
+ * [Translator 101 - Debugging a Translator](http://pl.atyp.us/hekafs.org/index.php/2011/11/translator-101-lesson-4-debugging-a-translator/)
+
+Gluster community site:
+
+ * [Translators](https://docs.gluster.org/en/latest/Quick-Start-Guide/Architecture/#translators)
diff --git a/doc/hacker-guide/en-US/markdown/unittest.md b/doc/developer-guide/unittest.md
index 42dc210959e..5c6c0a8a039 100644
--- a/doc/hacker-guide/en-US/markdown/unittest.md
+++ b/doc/developer-guide/unittest.md
@@ -14,10 +14,13 @@
 * Maintainable
 * Trustworthy (when you see its result, you don’t need to debug the code just to be sure)
 
-## Cmockery2
-GlusterFS unit test framework is based on [Cmockery2][].  Cmockery provides developers with methods to isolate and test modules written in C language.  It also provides integration with Jenkins by providing JUnit XML compliant unit test results.
+## cmocka
+GlusterFS unit test framework is based on [cmocka][].  cmocka provides
+developers with methods to isolate and test modules written in C language.  It
+also provides integration with Jenkins by providing JUnit XML compliant unit
+test results.
 
-Before continuing, you may want to familiarize yourself with Cmockery2 by reading the [usage guide][cmockery2usage].
+cmocka
 
 ## Running Unit Tests
 To execute the unit tests, all you need is to type `make check`.  Here is a step-by-step example assuming you just cloned a GlusterFS tree:
@@ -90,7 +93,7 @@ Expected assertion ((void *)0) != ptr occurred
 Add the following to your C file:
 
 ```c
-#include <cmockery/pbc.h>
+#include <cmocka_pbc.h>
 ```
 
 ```c
@@ -171,7 +174,7 @@ int divide (int n, int d)
 Cmockery2 provides its own memory allocation functions which check for buffer overrun and memory leaks.  The following header file must be included **last** to be able to override any of the memory allocation functions:
 
 ```c
-#include <cmockery/cmockery_override.h>
+#include <cmocka.h>
 ```
 
 This file will only take effect with the `UNIT_TESTING` CFLAG is set.
@@ -204,7 +207,7 @@ TESTS += xxx_unittest
 
 Where `xxx` is the name of your C file. For example, look at `libglusterfs/src/Makefile.am`.
 
-Copy the simple unit test from `cmockery2/src/example/run_tests.c` to `unittest/xxx_unittest.c`.  If you would like to see an example of a unit test, please refer to `libglusterfs/src/unittest/mem_pool_unittest.c`.
+Copy the simple unit test from the [cmocka API][cmockaapi] to `unittest/xxx_unittest.c`.  If you would like to see an example of a unit test, please refer to `libglusterfs/src/unittest/mem_pool_unittest.c`.
 
 #### Mocking
 You may see that the linker will complain about missing functions needed by the C file you would like to test.  Identify the required functions, then place their stubs in a file called `unittest/xxx_mock.c`, then include this file in `Makefile.am` in `xxx_unittest_SOURCES`.  This will allow you to you Cmockery2's mocking functions.
@@ -220,6 +223,6 @@ $ gdb libglusterfs/src/mem_pool_unittest
 ```
 
 
-[Cmockery2]: https://github.com/lpabon/cmockery2
+[cmocka]: https://cmocka.org
 [definitionofunittest]: http://artofunittesting.com/definition-of-a-unit-test/
-[cmockery2usage]: https://github.com/lpabon/cmockery2/blob/master/doc/usage.md
+[cmockapi]: https://api.cmocka.org
diff --git a/doc/versioning.md b/doc/developer-guide/versioning.md
index 10c1511b79b..10c1511b79b 100644
--- a/doc/versioning.md
+++ b/doc/developer-guide/versioning.md
diff --git a/doc/hacker-guide/en-US/markdown/write-behind.md b/doc/developer-guide/write-behind.md
index 0d78964fa20..0d78964fa20 100644
--- a/doc/hacker-guide/en-US/markdown/write-behind.md
+++ b/doc/developer-guide/write-behind.md
diff --git a/doc/developer-guide/writing-a-cloudsync-plugin.md b/doc/developer-guide/writing-a-cloudsync-plugin.md
new file mode 100644
index 00000000000..907860aaed8
--- /dev/null
+++ b/doc/developer-guide/writing-a-cloudsync-plugin.md
@@ -0,0 +1,164 @@
+## How to write your Cloudsync Plugin
+
+### Background
+
+Cloudsync translator is part of the archival feature in Gluster. This translator
+does the retrieval/download part. Each cold  file will be archived to a remote
+storage (public or private cloud). On future access to the file, it will be
+retrieved from the remote storage by Cloudsync translator. Each remote storage
+would need a unique plugin. Cloudsync translator will load this plugin and
+call the necessary plugin functions.
+
+Upload can be done by a script or program. There are some basic mandatory steps
+for uploading the data. There is a sample script for crawl and upload given at
+the end of this guide.
+
+### Necessary changes to create a plugin
+
+1. Define store_methods:
+
+* This structure is the container of basic functions that will be called by
+  cloudsync xlator.
+
+        typedef struct store_methodds {
+                int (*fop_download) (call_frame_t *frame, void *config);
+                /* return type should be the store config */
+                void *(*fop_init) (xlator_t *this);
+                int (*fop_reconfigure) (xlator_t *this, dict_t *options);
+                void (*fop_fini) (void *config);
+        } store_methods_t;
+
+
+        Member details:
+        fop_download:
+                This is the download function pointer.
+
+                frame: This will have the fd to write data downloaded from
+                       cloud to GlusterFS.(frame->local->fd)
+
+                config: This is the plugin configuration variable.
+
+                Note: Structure cs_local_t has member dlfd and dloffset which
+                      can be used to manage the writes to Glusterfs.
+                      Include cloudsync-common.h to access these structures.
+
+        fop_init:
+                This is similar to xlator init. But here the return value is
+                the plugin configuration pointer. This pointer will be stored
+                in the cloudsync private object (priv->stores->config). And
+                the cloudsync private object can be accessed by "this->private"
+                where "this" is of type xlator_t.
+
+        fop_reconfigure:
+                This is similar to xlator_reconfigure.
+
+        fop_fini:
+                Free plugin resources.
+
+        Note: Store_methods_t is part of cs_private_t which in turn part of
+              xlator_t. Create a store_methods_t object named "store_ops" in
+              your plugin. For example
+
+              store_methods_t store_ops = {
+                .fop_download = aws_download_s3,
+                .fop_init     = aws_init,
+                .fop_reconfigure = aws_reconfigure,
+                .fop_fini     = aws_fini,
+              };
+
+
+2 - Making Cloudsync xlator aware of the plugin:
+
+        Add an entry in to the cs_plugin structure. For example
+        struct cs_plugin plugins[] = {
+                {
+                  .name = "amazons3",
+                  .library = "libamazons3.so",
+                  .description = "amazon s3 store."
+                },
+
+                {.name = NULL},
+        };
+
+        Description about individual members:
+                name: name of the plugin
+                library: This is the shared object created. Cloudsync will load
+                         this library during init.
+                description: Describe about the plugin.
+
+3- Makefile Changes in Cloudsync:
+
+		Add <plugin.la> to cloudsync_la_LIBADD variable.
+
+4 - Configure.ac changes:
+
+                In cloudsync section add the necessary dependency checks for
+                the plugin.
+
+5 - Export symbols:
+
+        Cloudsync needs "store_ops" to resolve all plugin functions.
+        Create a file <plugin>.sym and add write "store_ops" to it.
+
+
+### Sample script for upload
+This script assumes amazon s3 is the target cloud and bucket name is
+gluster-bucket. User can do necessary aws configuration using command
+"aws configure". Currently for amazons3 there are four gluster settings
+available.
+1- features.s3plugin-seckey -> s3 secret key
+2- features.s3plugin-keyid -> s3 key id
+3- features.s3plugin-bucketid -> bucketid
+4- features.s3plugin-hostname -> hostname e.g. s3.amazonaws.com
+
+Additionally set cloudsync storetype to amazons3.
+
+gluster v set <VOLNAME> cloudsync-storetype amazons3
+
+Now create a mount dedicated for this upload task.
+
+That covers necessary configurations needed.
+
+Below is the sample script for upload. The script will crawl directly on the
+brick and will upload those files which are not modified for last one month.
+It needs two arguments.
+1st arguement - Gluster Brick path
+2nd arguement - coldness that is how many days since the file was modified.
+3rd argument - dedicated gluster mount point created for uploading.
+
+Once the cloud setup is done, run the following script on individual bricks.
+Note: For an AFR volume, pick only the fully synchronized brick among the
+replica bricks.
+
+```
+target_folder=$1
+coldness=$2
+mnt=$3
+
+cd $target_folder
+for i in `find . -type f  | grep -v "glusterfs" | sed 's/..//'`
+do
+        echo "processing $mnt/$i"
+
+        #check whether the file is already archived
+        getfattr -n trusted.glusterfs.cs.remote $i &> /dev/null
+        if [ $? -eq 0 ]
+        then
+                echo "file $mnt/$i is already archived"
+        else
+                #upload to cloud
+                aws s3 cp $mnt/$i s3://gluster-bucket/
+                mtime=`stat -c "%Y" $mnt/$i`
+
+                #post processing of upload
+                setfattr -n trusted.glusterfs.csou.complete -v $mtime $mnt/$i
+                if [ $? -ne 0 ]
+                then
+                        echo "archiving of file $mnt/$i failed"
+                else
+                        echo "archiving of file $mnt/$i succeeded"
+                fi
+
+        fi
+done
+```
diff --git a/doc/developer-guide/xlator-classification.md b/doc/developer-guide/xlator-classification.md
new file mode 100644
index 00000000000..6073df9375f
--- /dev/null
+++ b/doc/developer-guide/xlator-classification.md
@@ -0,0 +1,221 @@
+# xlator categories and expectations
+
+The purpose of the document is to define a category for various xlators
+and expectations around what each category means from a perspective of
+health and maintenance of a xlator.
+
+The need to do this is to ensure certain categories are kept in good
+health, and helps the community and contributors focus their efforts around the
+same.
+
+This document also provides implementation details for xlator developers to
+declare a category for any xlator.
+
+## Table of contents
+1. Audience
+2. Categories (and expectations of each category)
+3. Implementation and usage details
+
+## Audience
+
+This document is intended for the following community participants,
+- New xlator contributors
+- Existing xlator maintainers
+- Packaging and gluster management stack maintainers
+
+For a more user facing understanding it is recommended to read section (TBD)
+in the gluster documentation.
+
+## Categories
+1. Experimental (E)
+2. TechPreview  (TP)
+3. Maintained (M)
+4. Deprecated (D)
+5. Obsolete (O)
+
+### Experimental (E)
+
+Developed in the experimental branch, for exploring new features. These xlators
+are NEVER packaged as a part of releases, interested users and contributors can
+build and work with these from sources. In the future, these maybe available as
+an package based on a weekly build of the same.
+
+#### Quality expectations
+- Compiles or passes smoke tests
+- Does not break nightly experimental regressions
+  - NOTE: If a nightly is broken, then all patches that were merged are reverted
+  till the errant patch is found and subsequently fixed
+
+### TechPreview (TP)
+
+Xlators in master or release branches that are not deemed fit to be in
+production deployments, but are feature complete to invite feedback and host
+user data.
+
+These xlators will be worked upon with priority by maintainers/authors who are
+involved in making them more stable than xlators in the Experimental/Deprecated/
+Obsolete categories.
+
+There is no guarantee that these xlators will move to the Maintained state, and
+may just get Obsoleted based on feedback, or other project goals or technical
+alternatives.
+
+#### Quality expectations
+- Same as Maintained, minus
+  - Performance, Scale, other(?)
+  - *TBD* *NOTE* Need inputs, Intention is all quality goals as in Maintained,
+  other than the list above (which for now has scale and performance)
+
+### Maintained (M)
+
+These xltors are part of the core Gluster functionality and are maintained
+actively. These are part of master and release branches and are higher in
+priority of maintainers and other interested contributors.
+
+#### Quality expectations
+
+NOTE: A short note on what each of these mean are added here, details to follow.
+
+NOTE: Out of the gate all of the following are not mandated, consider the
+following a desirable state to reach as we progress on each
+
+- Bug backlog: Actively address bug backlog
+- Enhancement backlog: Actively maintain outstanding enhancement backlog (need
+        not be acted on, but should be visible to all)
+- Review backlog: Actively keep this below desired counts and states
+- Static code health:  Actively meet near-zero issues in this regard
+  - Coverity, spellcheck and other checks
+- Runtime code health: Actively meet defined coverage levels in this regard
+  - Coverage, others?
+  - Per-patch regressions
+  - Glusto runs
+  - Performance
+  - Scalability
+- Technical specifications: Implementation details should be documented and
+        updated at regular cadence (even per patch that change assumptions in
+        here)
+- User documentation: User facing details should be maintained to current
+        status in the documentation
+- Debuggability: Steps, tools, procedures should be documented and maintained
+        each release/patch as applicable
+- Troubleshooting: Steps, tools, procedures should be documented and maintained
+        each release/patch as applicable
+  - Steps/guides for self service
+  - Knowledge base for problems
+- Other common criteria that will apply: Required metrics/desired states to be
+        defined per criteria
+  - Monitoring, usability, statedump, and other such xlator expectations
+
+### Deprecated (D)
+
+Xlators on master or release branches that would be obsoleted and/or replaced
+with similar or other functionality in the next major release.
+
+#### Quality expectations
+- Retain status-quo when moved to this state, till it is moved to obsoleted
+- Provide migration steps if feature provided by the xlator is replaced with
+other xlators
+
+### Obsolete (O)
+
+Xlator/code still in tree, but not packaged or shipped or maintained in any
+form. This is noted as a category till the code is removed from the tree.
+
+These xlators and their corresponding code and test health will not be executed.
+
+#### Quality expectations
+- None
+
+## Implementation and usage details
+
+### How to specify an xlators category
+
+While defining 'xlator_api_t' structure for the corresponding xlator, add a
+flag like below:
+
+```
+diff --git a/xlators/performance/nl-cache/src/nl-cache.c b/xlators/performance/nl-cache/src/nl-cache.c
+index 0f0e53bac2..8267d6897c 100644
+--- a/xlators/performance/nl-cache/src/nl-cache.c
++++ b/xlators/performance/nl-cache/src/nl-cache.c
+@@ -869,4 +869,5 @@ xlator_api_t xlator_api = {
+         .cbks          = &nlc_cbks,
+         .options       = nlc_options,
+         .identifier    = "nl-cache",
++        .category      = GF_TECH_PREVIEW,
+ };
+diff --git a/xlators/performance/quick-read/src/quick-read.c b/xlators/performance/quick-read/src/quick-read.c
+index 8d39720e7f..235de27c19 100644
+--- a/xlators/performance/quick-read/src/quick-read.c
++++ b/xlators/performance/quick-read/src/quick-read.c
+@@ -1702,4 +1702,5 @@ xlator_api_t xlator_api = {
+         .cbks          = &qr_cbks,
+         .options       = qr_options,
+         .identifier    = "quick-read",
++        .category      = GF_MAINTAINED,
+ };
+```
+
+Similarly, if a particular option is in different state other than
+the xlator state, one can add the same flag in options structure too.
+
+```
+diff --git a/xlators/cluster/afr/src/afr.c b/xlators/cluster/afr/src/afr.c
+index 0e86e33d03..81996743d1 100644
+--- a/xlators/cluster/afr/src/afr.c
++++ b/xlators/cluster/afr/src/afr.c
+@@ -772,6 +772,7 @@ struct volume_options options[] = {
+           .description = "Maximum latency for shd halo replication in msec."
+         },
+         { .key   = {"halo-enabled"},
++          .category = GF_TECH_PREVIEW,
+           .type  = GF_OPTION_TYPE_BOOL,
+           .default_value = "False",
+
+```
+
+
+### User experience using the categories
+
+#### Ability to use a category
+
+This section details which category of xlators can be used when and specifics
+around when each category is enabled.
+
+1. Maintained category xlators can be used by default, this implies, volumes
+created with these xlators enabled will throw no warnings, or need no user
+intervention to use the xlator.
+
+2. Tech Preview category xlators needs cluster configuration changes to allow
+these xlatorss to be used in volumes, further, logs will contain a message
+stating TP xlators are in use. Without the cluster configured to allow TP
+xlators, volumes created or edited to use such xlators would result in errors.
+  - (TBD) Cluster configuration option
+  - (TBD) Warning message
+  - (TBD) Code mechanics on how this is achieved
+
+3. Deprecated category xlators can be used by default, but will throw a warning
+in the logs that such are in use and will be deprecated in the future.
+  - (TBD) Warning message
+
+4. Obsolete category xlators will not be packaged and hence cannot be used from
+release builds.
+
+5. Experimental category xlators will not be packaged and hence cannot be used
+from release builds, if running experimental (weekly or other such) builds,
+these will throw a warning in the logs stating experimental xlators are in use.
+  - (TBD) Warning message
+
+#### Ability to query xlator category
+
+(TBD) Need to provide the ability to query xlator categories, or list xlators
+and their respective categories.
+
+#### User facing changes
+
+User facing changes that are expected due to this change include the following,
+- Cluster wide option to enable TP xlators, or more generically a category
+level of xlators
+- Errors in commands that fail due to invalid categories
+- Warning messages in logs to denote certain categories of xlators are in use
+- (TBD) Ability to query xlators and their respective categories
diff --git a/doc/features/afr-statistics.md b/doc/features/afr-statistics.md
deleted file mode 100644
index d0705845aa4..00000000000
--- a/doc/features/afr-statistics.md
+++ /dev/null
@@ -1,142 +0,0 @@
-##gluster volume heal <volume-name> statistics
-
-##Description
-In case of index self-heal, self-heal daemon reads the entries from the
-local bricks, from /brick-path/.glusterfs/indices/xattrop/ directory.
-So based on the entries read by self heal daemon, it will attempt self-heal.
-Executing this command will list the  crawl statistics of self heal done for
-each brick.
-
-For each brick, it will list:
-1. Starting time of crawl done for that brick.
-2. Ending time of crawl done for that brick.
-3. No of entries for which self-heal is successfully attempted.
-4. No of split-brain entries found while self-healing.
-5. No of entries for which heal failed.
-
-
-
-Example:
-a) Create a gluster volume with replica count 2.
-b) Create 10 files.
-c) kill brick_1 of this replica.
-d) Overwrite all 10 files.
-e) Kill the other brick (brick_2), and bring back (brick_1) up.
-f) Overwrite all 10 files.
-
-Now we have 10 files, which are in split brain. Self heal daemon will crawl for
-both the bricks, and will count 10 files from each brick.
-It will report 10 files under split-brain with respect to given brick.
-
-Gathering crawl statistics on volume volume1 has been successful
-------------------------------------------------
-
-Crawl statistics for brick no 0
-Hostname of brick 192.168.122.1
-
-Starting time of crawl: Tue May 20 19:13:11 2014
-
-Ending time of crawl: Tue May 20 19:13:12 2014
-
-Type of crawl: INDEX
-No. of entries healed: 0
-No. of entries in split-brain: 10
-No. of heal failed entries: 0
-------------------------------------------------
-
-Crawl statistics for brick no 1
-Hostname of brick 192.168.122.1
-
-Starting time of crawl: Tue May 20 19:13:12 2014
-
-Ending time of crawl: Tue May 20 19:13:12 2014
-
-Type of crawl: INDEX
-No. of entries healed: 0
-No. of entries in split-brain: 10
-No. of heal failed entries: 0
-
-------------------------------------------------
-
-
-As the output shows, self-heal daemon detects 10 files in split-brain with
-resept to given brick.
-
-
-
-
-##gluster volume heal <volume-name> statistics heal-count
-It lists the number of entries present in
-/<brick>/.glusterfs/indices/xattrop from each-brick.
-
-
-1. Create a replicate volume.
-2. Kill one brick of a replicate volume1.
-3. Create 10 files.
-4. Execute above command.
-
---------------------------------------------------------------------------------
-Gathering count of entries to be healed on volume volume1 has been successful
-
-Brick 192.168.122.1:/brick_1
-Number of entries: 10
-
-Brick 192.168.122.1:/brick_2
-No gathered input for this brick
--------------------------------------------------------------------------------
-
-
-
-
-
-
-##gluster volume heal <volume-name> statistics heal-count replica \
-  ip_addr:/brick_location
-
-To list the number of entries to be healed from a particular replicate
-subvolume, listing any one child of that replicate subvolume in the command,
-will list the entries for all the childrens of that replicate subvolume.
-
-Example:           dht
-             /              \
-            /                \
-         replica-1         replica-2
-         /    \               /    \
-     child-1  child-2   child-3   child-4
-    /brick1   /brick2   /brick3   /brick4
-
-gluster volume heal <vol-name> statistics heal-count ip:/brick1
-will list  count only for child-1 and child-2.
-
-gluster volume heal <vol-name> statistics heal-count ip:/brick3
-will list count only for child-3 and child-4.
-
-
-
-1. Create a volume same as mentioned in the above graph.
-2. Kill Brick-2.
-3. Create some files.
-4. If we are interested in knowing the number of files to be healed from each
-   brick of replica-1 only, mention any one child of that replica.
-
-gluster volume heal volume1 statistics heal-count replica 192.168.122.1:/brick2
-
-output:
--------
-Gathering count of entries to be healed per replica on volume volume1 has \
-been successful
-
-Brick 192.168.122.1:/brick_1
-Number of entries: 10                                <--10 files
-
-Brick 192.168.122.1:/brick_2
-No gathered input for this brick                     <-Brick is down
-
-Brick 192.168.122.1:/brick_3
-No gathered input for this brick                     <--No result, as we are not
-                                                        interested.
-
-Brick 192.168.122.1:/brick_4                         <--No result, as we are not
-No gathered input for this brick                        interested.
-
-
diff --git a/doc/features/afr-v1.md b/doc/features/afr-v1.md
deleted file mode 100644
index 8df66658613..00000000000
--- a/doc/features/afr-v1.md
+++ /dev/null
@@ -1,340 +0,0 @@
-#Automatic File Replication
-Afr xlator in glusterfs is responsible for replicating the data across the bricks.
-
-###Responsibilities of AFR
-Its responsibilities include the following:
-
-1. Maintain replication consistency (i.e. Data on both the bricks should be same, even in the cases where there are operations happening on same file/directory in parallel from multiple applications/mount points as long as all the bricks in replica set are up)
-
-2. Provide a way of recovering data in case of failures as long as there is
-  at least one brick which has the correct data.
-
-3. Serve fresh data for read/stat/readdir etc
-
-###Transaction framework
-For 1, 2 above afr uses transaction framework which consists of the following 5
-phases which happen on all the bricks in replica set(Bricks which are in replication):
-
-####1.Lock Phase
-####2. Pre-op Phase
-####3. Op Phase
-####4. Post-op Phase
-####5. Unlock Phase
-
-*Op Phase* is the actual operation sent by application (`write/create/unlink` etc). For every operation which afr receives that modifies data it sends that same operation in parallel to all the bricks in its replica set. This is how it achieves replication.
-
-*Lock, Unlock Phases* take necessary locks so that *Op phase* can provide **replication consistency** in normal work flow.
-
-#####For example:
-If an application performs `touch a` and the other one does `mkdir a`, afr makes sure that either file with name `a` or directory with name `a` is created on both the bricks.
-
-*Pre-op, Post-op Phases* provide changelogging which enables afr to figure out which copy is fresh.
-Once afr knows how to figure out fresh copy in the replica set it can **recover data** from fresh copy to stale copy. Also it can **serve fresh** data for `read/stat/readdir` etc.
-
-##Internal Operations
-Brief introduction to internal operations in Glusterfs which make *Locking, Unlocking, Pre/Post ops* possible:
-
-###Internal Locking Operations
-Glusterfs has **locks** translator which provides the following internal locking operations called `inodelk`, `entrylk` which are used by afr to achieve synchronization of operations on files or directories that conflict with each other.
-
-`Inodelk` gives the facility for translators in Glusterfs to obtain range (denoted by tuple with **offset**, **length**) locks in a given domain for an inode.
-Full file lock is denoted by the tuple (offset: `0`, length: `0`) i.e. length `0` is considered as infinity.
-
-`Entrylk` enables translators of Glusterfs to obtain locks on `name` in a given domain for an inode, typically a directory.
-
-**Locks** translator provides both *blocking* and *nonblocking* variants and of these operations.
-
-###Xattrop
-For pre/post ops posix translator provides an operation called xattrop.
-xattrop is a way of *incrementing*/*decrementing* a number present in the extended attribute of the inode *atomically*.
-
-##Transaction Types
-There are 3 types of transactions in AFR.
-1. Data transactions
-  - Operations that add/modify/truncate the file contents.
-  - `Write`/`Truncate`/`Ftruncate` etc
-
-2. Metadata transactions
-  - Operations that modify the data kept in inode.
-  - `Chmod`/`Chown` etc
-
-3) Entry transactions
-  - Operations that add/remove/rename entries in a directory
-  - `Touch`/`Mkdir`/`Mknod` etc
-
-###Data transactions:
-
-*write* (`offset`, `size`) - writes data from `offset` of `size`
-
-*ftruncate*/*truncate* (`offset`) - truncates data from `offset` till the end of file.
-
-Afr internal locking needs to make sure that two conflicting data operations happen in order, one after the other so that it does not result in replication inconsistency. Afr data operations take inodelks in same domain (lets call it `data` domain).
-
-*Write* operation with offset `O` and size `S` takes an inode lock in data domain with range `(O, S)`.
-
-*Ftruncate*/*Truncate* operations with offset `O` take inode locks in `data` domain with range `(O, 0)`. Please note that size `0` means size infinity.
-
-These ranges make sure that overlapping write/truncate/ftruncate operations are done one after the other.
-
-Now that we know the ranges the operations take locks on, we will see how locking happens in afr.
-
-####Lock:
-Afr initially attempts **non-blocking** locks on **all** the bricks of the replica set in **parallel**. If all the locks are successful then it goes on to perform pre-op. But in case **non-blocking** locks **fail** because there is *at least one conflicting operation* which already has a **granted lock** then it **unlocks** the **non-blocking** locks it already acquired in this previous step and proceeds to perform **blocking** locks **one after the other** on each of the subvolumes in the order of subvolumes specified in the volfile.
-
-Chances of **conflicting operations** is **very low** and time elapsed in **non-blocking** locks phase is `Max(latencies of the bricks for responding to inodelk)`, where as time elapsed in **blocking locks** phase is `Sum(latencies of the bricks for responding to inodelk)`. That is why afr always tries for non-blocking locks first and only then it moves to blocking locks.
-
-####Pre-op:
-Each file/dir in a brick maintains the changelog(roughly pending operation count) of itself and that of the files
-present in all the other bricks in it's replica set as seen by that brick.
-
-Lets consider an example replica volume with 2 bricks brick-a and brick-b.
-all files in brick-a will have 2 entries
-one for itself and the other for the file present in it's replica set, i.e.brick-b:
-One can inspect changelogs using getfattr command.
-
-    # getfattr -d -e hex -m. brick-a/file.txt
-    trusted.afr.vol-client-0=0x000000000000000000000000 -->changelog for itself (brick-a)
-    trusted.afr.vol-client-1=0x000000000000000000000000 -->changelog for brick-b as seen by brick-a
-
-Likewise, all files in brick-b will have:
-
-    # getfattr -d -e hex -m. brick-b/file.txt
-    trusted.afr.vol-client-0=0x000000000000000000000000 -->changelog for brick-a as seen by brick-b
-    trusted.afr.vol-client-1=0x000000000000000000000000 -->changelog for itself (brick-b)
-
-#####Interpreting Changelog Value:
-Each extended attribute has a value which is `24` hexa decimal digits. i.e. `12` bytes
-First `8` digits (`4` bytes) represent changelog of `data`. Second `8` digits represent changelog
-of `metadata`. Last 8 digits represent Changelog of `directory entries`.
-
-Pictorially representing the same, we have:
-
-    0x 00000000 00000000 00000000
-          |        |        |
-          |        |        \_ changelog of directory entries
-          |        \_ changelog of metadata
-          \ _ changelog of data
-
-Before write operation is performed on the brick, afr marks the file saying there is a pending operation.
-
-As part of this pre-op afr sends xattrop operation with increment 1 for data operation to make the extended attributes the following:
-    # getfattr -d -e hex -m. brick-a/file.txt
-    trusted.afr.vol-client-0=0x000000010000000000000000 -->changelog for itself (brick-a)
-    trusted.afr.vol-client-1=0x000000010000000000000000 -->changelog for brick-b as seen by brick-a
-
-Likewise, all files in brick-b will have:
-
-    # getfattr -d -e hex -m. brick-b/file.txt
-    trusted.afr.vol-client-0=0x000000010000000000000000 -->changelog for brick-a as seen by brick-b
-    trusted.afr.vol-client-1=0x000000010000000000000000 -->changelog for itself (brick-b)
-
-As the operation is in progress on files on both the bricks all the extended attributes show the same value.
-
-####Op:
-Now it sends the actual write operation to both the bricks. Afr remembers whether the operation is successful or not on all the subvolumes.
-
-####Post-Op:
-If the operation succeeds on all the bricks then there is no pending operations on any of the bricks so as part of POST-OP afr sends xattrop operation with increment -1 i.e. decrement by 1 for data operation to make the extended attributes back to all zeros again.
-
-In case there is a failure on brick-b then there is still a pending operation on brick-b where as no pending operations are there on brick-a. So xattrop operation for both of these extended attributes differs now. For extended attribute corresponding to brick-a i.e. trusted.afr.vol-client-0 decrement by 1 is sent where as for extended attribute corresponding to brick-b increment by '0' is sent to retain the pending operation count.
-
-    # getfattr -d -e hex -m. brick-a/file.txt
-    trusted.afr.vol-client-0=0x000000000000000000000000 -->changelog for itself (brick-a)
-    trusted.afr.vol-client-1=0x000000010000000000000000 -->changelog for brick-b as seen by brick-a
-
-    # getfattr -d -e hex -m. brick-b/file.txt
-    trusted.afr.vol-client-0=0x000000000000000000000000 -->changelog for brick-a as seen by brick-b
-    trusted.afr.vol-client-1=0x000000010000000000000000 -->changelog for itself (brick-b)
-
-####Unlock:
-Once the transaction is completed unlock is sent on all the bricks where lock is acquired.
-
-
-###Meta Data transactions:
-
-setattr, setxattr, removexattr
-All metadata operations take same inode lock with same range in metadata domain.
-
-####Lock:
-Metadata locking also starts initially with non-blocking locks then move on to blocking locks on any failures because of conflicting operations.
-
-####Pre-op:
-Before metadata operation is performed on the brick, afr marks the file saying there is a pending operation.
-As part of this pre-op afr sends xattrop operation with increment 1 for metadata operation to make the extended attributes the following:
-    # getfattr -d -e hex -m. brick-a/file.txt
-    trusted.afr.vol-client-0=0x000000000000000100000000 -->changelog for itself (brick-a)
-    trusted.afr.vol-client-1=0x000000000000000100000000 -->changelog for brick-b as seen by brick-a
-
-Likewise, all files in brick-b will have:
-    # getfattr -d -e hex -m. brick-b/file.txt
-    trusted.afr.vol-client-0=0x000000000000000100000000 -->changelog for brick-a as seen by brick-b
-    trusted.afr.vol-client-1=0x000000000000000100000000 -->changelog for itself (brick-b)
-
-As the operation is in progress on files on both the bricks all the extended attributes show the same value.
-
-####Op:
-Now it sends the actual metadata operation to both the bricks. Afr remembers whether the operation is successful or not on all the subvolumes.
-
-Post-Op:
-If the operation succeeds on all the bricks then there is no pending operations on any of the bricks so as part of POST-OP afr sends xattrop operation with increment -1 i.e. decrement by 1 for metadata operation to make the extended attributes back to all zeros again.
-
-In case there is a failure on brick-b then there is still a pending operation on brick-b where as no pending operations are there on brick-a. So xattrop operation for both of these extended attributes differs now. For extended attribute corresponding to brick-a i.e. trusted.afr.vol-client-0 decrement by 1 is sent where as for extended attribute corresponding to brick-b increment by '0' is sent to retain the pending operation count.
-
-    # getfattr -d -e hex -m. brick-a/file.txt
-    trusted.afr.vol-client-0=0x000000000000000000000000 -->changelog for itself (brick-a)
-    trusted.afr.vol-client-1=0x000000000000000100000000 -->changelog for brick-b as seen by brick-a
-
-    # getfattr -d -e hex -m. brick-b/file.txt
-    trusted.afr.vol-client-0=0x000000000000000000000000 -->changelog for brick-a as seen by brick-b
-    trusted.afr.vol-client-1=0x000000000000000100000000 -->changelog for itself (brick-b)
-
-####Unlock:
-Once the transaction is completed unlock is sent on all the bricks where lock is acquired.
-
-
-###Entry transactions:
-
-create, mknod, mkdir, link, symlink, rename, unlink, rmdir
-Pre-op/Post-op (done using xattrop) always happens on the parent directory.
-
-Entry Locks taken by these entry operations:
-
-**Create** (file `dir/a`): Lock on name `a` in inode of `dir`
-
-**mknod** (file `dir/a`): Lock on name `a` in inode of `dir`
-
-**mkdir** (dir `dir/a`): Lock on name `a` in inode of `dir`
-
-**link** (file `oldfile`, file `dir/newfile`): Lock on name `newfile` in inode of `dir`
-
-**Symlink** (file `oldfile`, file `dir`/`symlinkfile`): Lock on name `symlinkfile` in inode of `dir`
-
-**rename** of (file `dir1`/`file1`, file `dir2`/`file2`): Lock on name `file1` in inode of `dir1`, Lock on name `file2` in inode of `dir2`
-
-**rename** of (dir `dir1`/`dir2`, dir `dir3`/`dir4`): Lock on name `dir2` in inode of `dir1`, Lock on name `dir4` in inode of `dir3`, Lock on `NULL` in inode of `dir4`
-
-**unlink** (file `dir`/`a`): Lock on name `a` in inode of `dir`
-
-**rmdir** (dir dir/a): Lock on name `a` in inode of `dir`, Lock on `NULL` in inode of `a`
-
-####Lock:
-Even entry locking starts initially with non-blocking locks then move on to blocking locks on any failures because of conflicting operations.
-
-####Pre-op:
-Before entry operation is performed on the brick, afr marks the directory saying there is a pending operation.
-
-As part of this pre-op afr sends xattrop operation with increment 1 for entry operation to make the extended attributes the following:
-
-    # getfattr -d -e hex -m. brick-a/
-    trusted.afr.vol-client-0=0x000000000000000000000001 -->changelog for itself (brick-a)
-    trusted.afr.vol-client-1=0x000000000000000000000001 -->changelog for brick-b as seen by brick-a
-
-Likewise, all files in brick-b will have:
-    # getfattr -d -e hex -m. brick-b/
-    trusted.afr.vol-client-0=0x000000000000000000000001 -->changelog for brick-a as seen by brick-b
-    trusted.afr.vol-client-1=0x000000000000000000000001 -->changelog for itself (brick-b)
-
-As the operation is in progress on files on both the bricks all the extended attributes show the same value.
-
-####Op:
-Now it sends the actual entry operation to both the bricks. Afr remembers whether the operation is successful or not on all the subvolumes.
-
-####Post-Op:
-If the operation succeeds on all the bricks then there is no pending operations on any of the bricks so as part of POST-OP afr sends xattrop operation with increment -1 i.e. decrement by 1 for entry operation to make the extended attributes back to all zeros again.
-
-In case there is a failure on brick-b then there is still a pending operation on brick-b where as no pending operations are there on brick-a. So xattrop operation for both of these extended attributes differs now. For extended attribute corresponding to brick-a i.e. trusted.afr.vol-client-0 decrement by 1 is sent where as for extended attribute corresponding to brick-b increment by '0' is sent to retain the pending operation count.
-
-    # getfattr -d -e hex -m. brick-a/file.txt
-    trusted.afr.vol-client-0=0x000000000000000000000000 -->changelog for itself (brick-a)
-    trusted.afr.vol-client-1=0x000000000000000000000001 -->changelog for brick-b as seen by brick-a
-
-    # getfattr -d -e hex -m. brick-b/file.txt
-    trusted.afr.vol-client-0=0x000000000000000000000000 -->changelog for brick-a as seen by brick-b
-    trusted.afr.vol-client-1=0x000000000000000000000001 -->changelog for itself (brick-b)
-
-####Unlock:
-Once the transaction is completed unlock is sent on all the bricks where lock is acquired.
-
-The parts above cover how replication consistency is achieved in afr.
-
-Now let us look at how afr can figure out how to recover from failures given the changelog extended attributes
-
-###Recovering from failures (Self-heal)
-For recovering from failures afr tries to determine which copy is the fresh copy based on the extended attributes.
-
-There are 3 possibilities:
-1. All the extended attributes are zero on all the bricks. This means there are no pending operations on any of the bricks so there is nothing to recover.
-2. According to the extended attributes there is a brick(brick-a) which noticed that there are operations pending on the other brick(brick-b).
- - There are 4 possibilities for brick-b
-
-    - It did not even participate in transaction (all extended attributes on brick-b are zeros). Choose brick-a as source and perform recovery to brick-b.
-
-    - It participated in the transaction but died even before post-op. (All extended attributes on brick-b have a pending-count). Choose brick-a as source and perform recovery to brick-b.
-
-    - It participated in the transaction and after the post-op extended attributes on brick-b show that there are pending operations on itself. Choose brick-a as source and perform recovery to brick-b.
-
-    - It participated in the transaction and after the post-op extended attributes on brick-b show that there are pending operations on brick-a. This situation is called Split-brain and there is no way to recover. This situation can happen in cases of network partition.
-
-3. The only possibility now is where both brick-a, brick-b have pending operations. In this case changelogs extended attributes are all non-zeros on all the bricks. Basically what could have happened is the operations started on the file but either the whole replica set went down or the mount process itself dies before post-op is performed. In this case there is a possibility that data on the bricks is different. In this case afr chooses file with bigger size as source, if both files have same size then it choses the subvolume which has witnessed large number of pending operations on the other brick as source. If both have same number of pending operations then it chooses the file with newest ctime as source. If this is also same then it just picks one of the two bricks as source and syncs data on to the other to make sure that the files are replicas to each other.
-
-###Self-healing:
-Afr does 3 types of self-heals for data recovery.
-
-1. Data self-heal
-
-2. Metadata self-heal
-
-3. Data self-heal
-
-As we have seen earlier, afr depends on changelog extended attributes to figure out which copy is source and which copy is sink. General algorithm for performing this recovery (self-heal) is same for all of these different self-heals.
-
-1. Take appropriate full locks on the file/directory to make sure no other transaction is in progress while inspecting changelog extended attributes.
-In this step, for
- - Data self-heal afr takes inode lock with `offset: 0` and `size: 0`(infinity) in data domain.
- - Entry self-heal takes entry lock on directory with `NULL` name i.e. full directory lock.
- - Metadata self-heal it takes pre-defined range in metadata domain on which all the metadata operations on that inode take locks on. To prevent duplicate data self-heal an inode lock is taken in self-heal domain as well.
-
-2. Perform Sync from fresh copy to stale copy.
-In this step,
- - Metadata self-heal gets the inode attributes, extended attributes from source copy and sets them on the stale copy.
-
- - Entry self-heal reads entries on stale directories and see if they are present on source directory, if they are not present it deletes them. Then it reads entries on fresh directory and creates the missing entries on stale directories.
-
- - Data self-heal does things a bit differently to make sure no other writes on the file are blocked for the duration of self-heal because files sizes could be as big as 100G(VM files) and we don't want to block all the transactions until the self-heal is over. Locks translator allows two overlapping locks to be granted if they are from same lock owner. Using this what data self-heal does is it takes a small 128k size range lock and unlock previous acquired lock, heals just that 128k chunk and takes next 128k chunk lock and unlock previous lock and moves to the next one. It always makes sure that at least one lock is present on the file by selfheal throughout the duration of self-heal so that two self-heals don't happen in parallel.
-
- - Data self-heal has two algorithms, where the file can be copied only when there is data mismatch for that chunk called as 'diff' self-heal. The otherone is blind copy of each chunk called 'full' self-heal
-
-3. Change extended attributes to mark new sources after the sync.
-
-4. Unlock the locks acquired to perform self-heal.
-
-### Transaction Optimizations:
-As we saw earlier afr transaction for all the operations that modify data happens in 5 phases, i.e. it sends 5 operations on the network for every operation. In the following sections we will see optimizations already implemented in afr which reduce the number of operations on the network to just 1 per transaction in best case.
-
-####Changelog-piggybacking
-This optimization comes into picture when on same file descriptor, before write1's post op is complete write2's pre-op starts and the operations are succeeding. When writes come in that manner we can piggyback on the pre-op of write1 for write2 and somehow tell write1 that write2 will do the post-op that was supposed to be done by write1. So write1's post-op does not happen over network, write2's pre-op does not happen over network. This optimization does not hold if there are any failures in write1's phases.
-
-####Delayed Post-op
-This optimization just delays post-op of the write transaction(write1) by a pre-configured amount time to increase the probability of next write piggybacking on the pre-op done by write1.
-
-With the combination of these two optimizations for operations like full file copy which are write intensive operations, what will essentially happen is for the first write a pre-op will happen. Then for the last write on the file post-op happens. So for all the write transactions between first write and last write afr reduced network operations from 5 to 3.
-
-####Eager-locking:
-This optimization comes into picture when only one file descriptor is open on the file and performing writes just like in the previous optimization. What this optimization does is it takes a full file lock on the file irrespective of the offset, size of the write, so that lock acquired by write1 can be piggybacked by write2 and write2 takes the responsibility of unlocking it. both write1, write2 will have same lock owner and afr takes the responsibility of serializing overlapping writes so that replication consistency is maintained.
-
-With the combination of these optimizations for operations like full file copy which are write intensive operations, what will essentially happen is for the first write a pre-op, full-file lock will happen. Then for the last write on the file post-op, unlock happens. So for all the write transactions between first write and last write afr reduced network operations from 5 to 1.
-
-###Quorum in afr:
-To avoid split-brains, afr employs the following quorum policies.
- - In replica set with odd number of bricks, replica set is said to be in quorum if more than half of the bricks are up.
- - In replica set with even number of bricks, if more than half of the bricks are up then it is said to be in quorum but if number of bricks that are up is equal to number of bricks that are down then, it is said to be in quorum if the first brick is also up in the set of bricks that are up.
-
-When quorum is not met in the replica set then modify operations on the mount are not allowed by afr.
-
-###Self-heal daemon and Index translator usage by afr:
-
-####Index xlator:
-On each brick index xlator is loaded. This xlator keeps track of what is happening in afr's pre-op and post-op. If there is an ongoing I/O or a pending self-heal, changelog xattrs would have non-zero values. Whenever xattrop/fxattrop fop (pre-op, post-ops are done using these fops) comes to index xlator a link (with gfid as name of the file on which the fop is performed) is added in <brick>/.glusterfs/indices/xattrop directory. If the value returned by the fop is zero the link is removed from the index otherwise it is kept until zero is returned in the subsequent xattrop/fxattrop fops.
-
-####Self-heal-daemon:
-self-heal-daemon process keeps running on each machine of the trusted storage pool. This process has afr xlators of all the volumes which are started. Its job is to crawl indices on bricks that are local to that machine. If any of the files represented by the gfid of the link name need healing and automatically heal them. This operation is performed every 10 minutes for each replica set. Additionally when a brick comes online also this operation is performed.
diff --git a/doc/features/brick-failure-detection.md b/doc/features/brick-failure-detection.md
deleted file mode 100644
index 24f2a18f39f..00000000000
--- a/doc/features/brick-failure-detection.md
+++ /dev/null
@@ -1,67 +0,0 @@
-# Brick Failure Detection
-
-This feature attempts to identify storage/file system failures and disable the failed brick without disrupting the remainder of the node's operation.
-
-## Description
-
-Detecting failures on the filesystem that a brick uses makes it possible to handle errors that are caused from outside of the Gluster environment.
-
-There have been hanging brick processes when the underlying storage of a brick went unavailable. A hanging brick process can still use the network and repond to clients, but actual I/O to the storage is impossible and can cause noticible delays on the client side.
-
-Provide better detection of storage subsytem failures and prevent bricks from hanging. It should prevent hanging brick processes when storage-hardware or the filesystem fails.
-
-A health-checker (thread) has been added to the posix xlator. This thread periodically checks the status of the filesystem (implies checking of functional storage-hardware).
-
-`glusterd` can detect that the brick process has exited, `gluster volume status` will show that the brick process is not running anymore. System administrators checking the logs should be able to triage the cause.
-
-## Usage and Configuration
-
-The health-checker is enabled by default and runs a check every 30 seconds. This interval can be changed per volume with:
-
-    # gluster volume set <VOLNAME> storage.health-check-interval <SECONDS>
-
-If `SECONDS` is set to 0, the health-checker will be disabled.
-
-## Failure Detection
-
-Error are logged to the standard syslog (mostly `/var/log/messages`):
-
-    Jun 24 11:31:49 vm130-32 kernel: XFS (dm-2): metadata I/O error: block 0x0 ("xfs_buf_iodone_callbacks") error 5 buf count 512
-    Jun 24 11:31:49 vm130-32 kernel: XFS (dm-2): I/O Error Detected. Shutting down filesystem
-    Jun 24 11:31:49 vm130-32 kernel: XFS (dm-2): Please umount the filesystem and rectify the problem(s)
-    Jun 24 11:31:49 vm130-32 kernel: VFS:Filesystem freeze failed
-    Jun 24 11:31:50 vm130-32 GlusterFS[1969]: [2013-06-24 10:31:50.500674] M [posix-helpers.c:1114:posix_health_check_thread_proc] 0-failing_xfs-posix: health-check failed, going down
-    Jun 24 11:32:09 vm130-32 kernel: XFS (dm-2): xfs_log_force: error 5 returned.
-    Jun 24 11:32:20 vm130-32 GlusterFS[1969]: [2013-06-24 10:32:20.508690] M [posix-helpers.c:1119:posix_health_check_thread_proc] 0-failing_xfs-posix: still alive! -> SIGTERM
-
-The messages labelled with `GlusterFS` in the above output are also written to the logs of the brick process.
-
-## Recovery after a failure
-
-When a brick process detects that the underlaying storage is not responding anymore, the process will exit. There is no automated way that the brick process gets restarted, the sysadmin will need to fix the problem with the storage first.
-
-After correcting the storage (hardware or filesystem) issue, the following command will start the brick process again:
-
-    # gluster volume start <VOLNAME> force
-
-## How To Test
-
-The health-checker thread that is part of each brick process will get started automatically when a volume has been started. Verifying its functionality can be done in different ways.
-
-On virtual hardware:
-
-* disconnect the disk from the VM that holds the brick
-
-On real hardware:
-
-* simulate a RAID-card failure by unplugging the card or cables
-
-On a system that uses LVM for the bricks:
-
-* use device-mapper to load an error-table for the disk, see [this description](http://review.gluster.org/5176).
-
-On any system (writing to random offsets of the block device, more difficult to trigger):
-
-1. cause corruption on the filesystem that holds the brick
-2. read contents from the brick, hoping to hit the corrupted area
-3. the filsystem should abort after hitting a bad spot, the health-checker should notice that shortly afterwards
diff --git a/doc/features/ctime.md b/doc/features/ctime.md
new file mode 100644
index 00000000000..74a77abed4b
--- /dev/null
+++ b/doc/features/ctime.md
@@ -0,0 +1,68 @@
+# Consistent time attributes in gluster across replica/distribute
+
+
+#### Problem:
+Traditionally gluster has been using time attributes (ctime, atime, mtime) of files/dirs from bricks. The problem with this approach is that, it is not consisteant  across replica and distribute bricks. And applications which depend on it breaks as replica  might not  always return time attributes from same brick.
+
+Tar especially gives "file changed as we read it" whenever it detects ctime differences when stat is served from  different bricks. The way we have been trying to solve it is to serve  the stat structures from same brick in afr, max-time in dht. But it doesn't avoid the problem completely. Because there is no way to change ctime at the moment(lutimes() only allows mtime, atime), there is little we can do to make sure ctimes match after self-heals/xattr updates/rebalance.
+
+#### Solution Proposed:
+Store time attribues (ctime, mtime, atime) as an xattr of the file. The xattr is updated based
+on the fop. If a filesystem fop changes only mtime and ctime, update only those in xattr for
+that file.
+
+#### Design Overview:
+1) As part of each fop, top layer will generate a time stamp and pass it to the down along
+ with other information
+  - This will bring a dependency for NTP synced clients along with servers
+  - There can be a diff in time if the fop stuck in the xlator for various reason,
+for ex: because of locks.
+
+ 2)  On the server, posix layer stores the value in the memory (inode ctx) and will sync the data periodically to the disk as an extended attr
+  -  Of course sync call also will force it. And fop comes for an inode which is not linked, we do the sync immediately.
+
+ 3)  Each time when inodes are created or initialized it read the data from disk and store in inode ctx.
+
+ 4)  Before setting to inode_ctx we compare the timestamp stored and the timestamp received, and only store if the stored value is lesser than the current value.
+
+ 5)  So in best case data will be stored and retrieved from the memory. We replace the values in iatt with the values in inode_ctx.
+
+ 6)  File ops that changes the parent directory attr time need to be consistent across all the distributed directories across the subvolumes. (for eg: a create call will change ctime and mtime of parent dir)
+
+  - This has to handle separately because we only send the fop to the hashed subvolume.
+  - We can asynchronously send the timeupdate setattr fop to the other subvoumes and change the values for parent directory if the file fops is successful on hashed subvolume.
+  -  This will have a window where the times are inconsistent across dht subvolume (Please provide your suggestions)
+
+7)  Currently we have couple of mount options for time attributes like noatime, relatime , nodiratime etc. But we are not explicitly handled those options even if it is given as mount option when gluster mount.
+
+
+#### Implementation Overview:
+This features involves changes in following xlators.
+ - utime xlator
+ - posix xlator
+
+##### utime xlator:
+This is a new client side xlator which does following tasks.
+
+1. It will generate a time stamp and passes it down in frame->root->ctime  and over the network.
+2.  Based on fop, it also decides the time attributes to be updated and this passed using "frame->root->flags"
+
+    Patches:
+    1. https://review.gluster.org/#/c/19857/
+
+##### posix xlator:
+Following tasks are done in posix xlator:
+
+1. Provides APIs to set and get the xattr from backend. It also caches the xattr in inode context. During get, it updates time attributes stored in xattr into iatt structure.
+2. Based on the flags from utime xlator, relevant fops update the time attributes in the xattr.
+
+    Patches:
+    1. https://review.gluster.org/#/c/19267/
+    2. https://review.gluster.org/#/c/19795/
+    3. https://review.gluster.org/#/c/19796/
+
+#### Pending Work:
+1. Handling of time related mount options (noatime, realatime,etc)
+2. flag based create (depending on flags in open, create behaviour might change)
+3. Changes in dht for direcotory sync acrosss multiple subvolumes
+4. readdirp stat need to be worked on.
diff --git a/doc/features/file-snapshot.md b/doc/features/file-snapshot.md
deleted file mode 100644
index 7f7c419fc7f..00000000000
--- a/doc/features/file-snapshot.md
+++ /dev/null
@@ -1,91 +0,0 @@
-#File Snapshot
-This feature gives the ability to take snapshot of files.
-
-##Descritpion
-This feature adds file snapshotting support to glusterfs. Snapshots can be created , deleted and reverted.
-
-To take a snapshot of a file, file should be in QCOW2 format as the code for the block layer snapshot has been taken from Qemu and put into gluster as a translator.
-
-With this feature, glusterfs will have better integration with Openstack Cinder, and in general ability to take snapshots of files (typically VM images).
-
-New extended attribute (xattr) will be added to identify files which are 'snapshot managed' vs raw files.
-
-##Volume Options
-Following volume option needs to be set on the volume for taking file snapshot.
-
-    # features.file-snapshot on
-##CLI parameters
-Following cli parameters needs to be passed with setfattr command to create, delete and revert file snapshot.
-
-    # trusted.glusterfs.block-format
-    # trusted.glusterfs.block-snapshot-create
-    # trusted.glusterfs.block-snapshot-goto
-##Fully loaded Example
-Download glusterfs3.5 rpms from download.gluster.org
-Install these rpms.
-
-start glusterd by using the command
-
-    # service glusterd start
-Now create a volume by using the command
-
-    # gluster volume create <vol_name> <brick_path>
-Run the command below to make sure that volume is created.
-
-    # gluster volume info
-Now turn on the snapshot feature on the volume by using the command
-
-    # gluster volume set <vol_name> features.file-snapshot on
-Verify that the option is set by using the command
-
-    # gluster volume info
-User should be able to see another option in the volume info
-
-    # features.file-snapshot: on
-Now mount the volume using fuse mount
-
-    # mount -t glusterfs <vol_name> <mount point>
-cd into the mount point
-    # cd <mount_point>
-    # touch <file_name>
-Size of the file can be set and format of the file can be changed to QCOW2 by running the command below. File size can be in KB/MB/GB
-
-    # setfattr -n trusted.glusterfs.block-format -v qcow2:<file_size> <file_name>
-Now create another file and send data to that file by running the command
-
-    # echo 'ABCDEFGHIJ' > <data_file1>
-copy the data to the one file to another by running the command
-
-    # dd if=data-file1 of=big-file conv=notrunc
-Now take the `snapshot of the file` by running the command
-
-    # setfattr -n trusted.glusterfs.block-snapshot-create -v <image1> <file_name>
-Add some more contents to the file and take another file snaphot by doing the following steps
-
-    # echo '1234567890' > <data_file2>
-    # dd if=<data_file2> of=<file_name> conv=notrunc
-    # setfattr -n trusted.glusterfs.block-snapshot-create -v <image2> <file_name>
-Now `revert` both the file snapshots and write data to some files so that data can be compared.
-
-    # setfattr -n trusted.glusterfs.block-snapshot-goto -v <image1> <file_name>
-    # dd if=<file_name> of=<out-file1> bs=11 count=1
-    # setfattr -n trusted.glusterfs.block-snapshot-goto -v <image2> <file_name>
-    # dd if=<file_name> of=<out-file2> bs=11 count=1
-Now read the contents of the files and compare as below:
-
-    # cat <data_file1>, <out_file1>  and compare contents.
-    # cat <data_file2>, <out_file2>  and compare contents.
-##one line description for the variables used
-file_name = File which will be creating in the mount point intially.
-
-data_file1 = File which contains data 'ABCDEFGHIJ'
-
-image1 = First file snapshot which has 'ABCDEFGHIJ' + some null values.
-
-data_file2 = File which contains data '1234567890'
-
-image2 = second file snapshot which has '1234567890' + some null values.
-
-out_file1 = After reverting image1 this contains 'ABCDEFGHIJ'
-
-out_file2 = After reverting image2 this contians '1234567890'
diff --git a/doc/features/ganesha-ha.md b/doc/features/ganesha-ha.md
new file mode 100644
index 00000000000..4b226a22ccf
--- /dev/null
+++ b/doc/features/ganesha-ha.md
@@ -0,0 +1,43 @@
+# Overview of Ganesha HA Resource Agents in GlusterFS 3.7
+
+The ganesha_mon RA monitors its ganesha.nfsd daemon. While the
+daemon is running, it creates two attributes: ganesha-active and
+grace-active. When the daemon stops for any reason, the attributes
+are deleted. Deleting the ganesha-active attribute triggers the
+failover of the virtual IP (the IPaddr RA) to another node —
+according to constraint location rules — where ganesha.nfsd is
+still running.
+
+The ganesha_grace RA monitors the grace-active attribute. When
+the grace-active attibute is deleted, the ganesha_grace RA stops,
+and will not restart. This triggers pacemaker to invoke the notify
+action in the ganesha_grace RAs on the other nodes in the cluster;
+which send a DBUS message to their respective ganesha.nfsd.
+
+(N.B. grace-active is a bit of a misnomer. while the grace-active
+attribute exists, everything is normal and healthy. Deleting the
+attribute triggers putting the surviving ganesha.nfsds into GRACE.)
+
+To ensure that the remaining/surviving ganesha.nfsds are put into
+ NFS-GRACE before the IPaddr (virtual IP) fails over there is a
+short delay (sleep) between deleting the grace-active attribute
+and the ganesha-active attribute. To summarize, e.g. in a four
+node cluster:
+
+1. on node 2 ganesha_mon::monitor notices that ganesha.nfsd has died
+
+2. on node 2 ganesha_mon::monitor deletes its grace-active attribute
+
+3. on node 2 ganesha_grace::monitor notices that grace-active is gone
+and returns OCF_ERR_GENERIC, a.k.a. new error. When pacemaker tries
+to (re)start ganesha_grace, its start action will return
+OCF_NOT_RUNNING, a.k.a. known error, don't attempt further restarts.
+
+4. on nodes 1, 3, and 4, ganesha_grace::notify receives a post-stop
+notification indicating that node 2 is gone, and sends a DBUS message
+to its ganesha.nfsd, putting it into NFS-GRACE.
+
+5. on node 2 ganesha_mon::monitor waits a short period, then deletes
+its ganesha-active attribute. This triggers the IPaddr (virt IP)
+failover according to constraint location rules.
+
diff --git a/doc/features/geo-replication/distributed-geo-rep.md b/doc/features/geo-replication/distributed-geo-rep.md
deleted file mode 100644
index 0a3183d6269..00000000000
--- a/doc/features/geo-replication/distributed-geo-rep.md
+++ /dev/null
@@ -1,71 +0,0 @@
-Introduction
-============
-
-This document goes through the new design of distributed geo-replication, it's features and the nature of changes involved. First we list down some of the important features.
-
-  - Distributed asynchronous replication
-  - Fast and versatile change detection
-  - Replica failover
-  - Hardlink synchronization
-  - Effective handling of deletes and renames
-  - Configurable sync engine (rsync, tar+ssh)
-  - Adaptive to a wide variety of workloads
-  - GFID synchronization
-
-Geo-replication makes use of the all new *journaling* infrastructure (a.k.a. changelog) to achieve great performance and feature improvements as mentioned above. To understand more about changelogging and the helper library (*libgfchangelog*) refer to document: doc/features/geo-replication/libgfchangelog.md
-
-Data Replication
-----------------
-
-Geo-replication is responsible to incrementally replicate data from the master node to the slave. But isn't that similar to what AFR does? Yes, but here the slave is located geographically distant from the master. Geo-replication follows the eventually consistent replication model, which implies, at any point of time, the slave would be lagging w.r.t. master, but would eventually catch up. Replication performance is dependent on two crucial factors:
-  - Network latency
-  - Change detection
-
-Network latency is something that is not in direct control for many reasons, but still there is always a best effort. Therefore, geo-replication offloads the data replicaiton part to common UNIX file transfer utilities. We choose the grand daddy of file transfers [rsync(1)] [1] as the default synchronization engine, as it's best known for it's diff transfer algorithm for effcient usage of network and lightning fast transfers (leave alone the flexibiliy). But what about small files performance? Due to it's checksumming algorithm, rsync has more overhead for small files -- the overhead of checksumming outweighs the bytes to be transferred for small files. Therefore, geo-replication can also use combination of tar piped over ssh to transfer large number of small files. Tests have shown a great improvement over standard rsync. However, sync engine is not yet dynamic to the file type and needs to be chosen manually by a configuration option.
-
-OTOH, change detection is something that is in full control of the application. Earlier (< release 3.5), geo-replicaiton would perform a file system crawl to indentify changes in the file system. This was not an unintelligent *check-every-single-inode* in the file system, but crawl logic based on *xtime*. xtime is an extended attribute maintained by the *marker* translator for each inode on the master and follows an upward-recursive marking pattern. Geo-replication would traverse a directory based on this simple condition:
-
-> xtime(master) > xtime(slave)
-
-E.g.:
-
->                    MASTER                          SLAVE
->
->                      /\                             /\
->                    d0  dir0                        d0 dir0
->                   /      \                        /     \
->                 d1       dir1                   d1      dir1
->                /                               /
->              d2                              d2
->             /                               /
->          file0                           file0
-
-Consider the directory tree above. Assume that master and slave were in sync and the following operation happens on master:
-```
-touch /d0/d1/d2/file0
-```
-This would trigger a xtime marking (xtime being the current timestamp) from the leaf (*file0*) upto the root (*/*), i.e. an *xattr* of *file0*, *d2*, *d1*, *d0* and finally */*. Geo-replication daemon would crawl the file system based the condition mentioned before and hence would only crawl the **left** part of the directory tree (as the **right** part would hve equal xtimes).
-
-Although the above crawling algorithm is fast, it still has to crawl a good part of the file system. Also, to decide whether to crawl a particular subdirectory, geo-rep need to compare xtime -- which is basically a **getxattr()** call on the master and slave (remember, *slave* is over a WAN).
-
-Therefore, in 3.5 the need arised to take crawling to the next level. Geo-replication now uses the changelogging infrastructure to idenitify changes in the filesystem. Actually, there is absolutely no crawl involved. Changelogging based detection is notification based. Geo-replication daemon registers itself with the changelog consumer library (*libgfchangelog*) and basically invokes a set of APIs to get the list of changes in the filesystem and replays them onto the slave. There is absolutely no crawl or any kind of extended attribute gets involved.
-
-Distributed Geo-Replication
----------------------------
-Geo-replication (also known as gsyncd or geo-rep) used to be non-distributed before release 3.5. The node on which geo-rep start command was executed was responsible for replication data to the slave. If this node goes offline due to some reason (reboot, crash, etc..), replication would thereby be ceased. So one of the main development efforts for release 3.5 was to *distributify* geo-replication. Geo-rep daemon running on each node (per brick) is responsible for replicating data **local** to each brick. This results in full parallelism and effective use of cluster/network resource.
-
-With release 3.5, geo-rep start command would spawn a geo-replication daemon on each node in the master cluster (one per brick). Geo-rep *status* command shown geo-rep session status from each master node. Similary, *stop* would gracefully tear down the session from all nodes.
-
-What else is synced?
---------------------
-  - GFID: Synchronizing the inode number (GFID) between master and the slave helps in synchronizing hardlinks.
-  - Purges are also handled effectively as there is no entry comparison between master and slave. With changelog replay, geo-rep perform unlink operation without having to resort to expensive **readdir()** over the WAN.
-  - Renames: With earlier geo-replication, because of the path based nature of crawling, renames were actually a delete and a create on the slave, followed by data transfer (not to mention the inode number change). Now, with changelogging, it's actually a **rename()** call on the slave.
-
-Replica Failover
-----------------
-One of the basic volume configuration is a replicated volume (synchronous replication). Having geo-replication sync data from all replicas would mean wastage of network bandwidth and possibly data corruption on the slave (though that's unlikely). Therefore, geo-rep on such volume configurations works in an **ACTIVE** and **PASSIVE** mode. Geo-rep daemon on one of the replicas is responsible for replicating data (**ACTIVE**), while the other geo-rep daemon is basically doing nothing (**PASSIVE**).
-
-On the event of the *ACTIVE* node going offline, the *PASSIVE* node identifies this event (there's a lag of max 60 seconds for this identification) and switches to *ACTIVE*; thereby taking over the role of replicating data from where the earlier *ACTIVE* node left off. This guarantees uninterrupted data replication even on node reboot/failures.
-
-[1]:http://rsync.samba.org
diff --git a/doc/features/geo-replication/libgfchangelog.md b/doc/features/geo-replication/libgfchangelog.md
deleted file mode 100644
index 1dd0d24253a..00000000000
--- a/doc/features/geo-replication/libgfchangelog.md
+++ /dev/null
@@ -1,119 +0,0 @@
-libgfchangelog: "GlusterFS changelog" consumer library
-======================================================
-
-This document puts forward the intended need for GlusterFS changelog consumer library (a.k.a. libgfchangelog) for consuming changlogs produced by the Changelog translator. Further, it mentions the proposed design and the API exposed by it. A brief explanation of changelog translator can also be found as a commit message in the upstream source tree and the review link can be [accessed here] [1].
-
-Initial consumer of changelogs would be Geo-Replication (release 3.5). Possible consumers in the future could be backup utilities, GlusterFS self-heal, bit-rot detection, AV scanners. All these utilities have one thing in common - to get a list of changed entities (created/modified/deleted) in the file system. Therefore, the need arises to provide such functionality in the form of a shared library that applications can link against and query for changes (See API section). There is no plan as of now to provide language bindings as such, but for shell script friendliness: 'gfind' command line utility (which would be dynamically linked with libgfchangelog) would be helpful. As of now, development for this utility is still not commenced.
-
-The next section gives a brief introduction about how changelogs are organized and managed. Then we propose couple of designs for libgfchangelog. API set is not covered in this document (maybe later).
-
-Changelogs
-==========
-
-Changelogs can be thought as a running history for an entity in the file system from the time the entity came into existance. The goal is to capture all possible transitions the entity underwent till the time it got purged. The transition namespace is broken up into three categories with each category represented by a specific changelog format. Changes are recorded in a flat file in the filesystem and are rolled over after a specific time interval. All three types of categories are recorded in a single changelog file (sequentially) with a type for each entry. Having a single file reduces disk seeks and fragmentation and less number of files to deal with. Stratergy for pruning of old logs is still undecided.
-
-
-Changelog Transition Namespace
-------------------------------
-
-As mentioned before the transition namespace is categorized into three types:
-  - TYPE-I   : Data operation
-  - TYPE-II  : Metadata operation
-  - TYPE-III : Entry operation
-
-One could visualize the transition of an file system entity as a state machine transitioning from one type to another. For TYPE-I and TYPE-II operations there is no state transition as such, but TYPE-III operation involves a state change from the file systems perspective. We can now classify file operations (fops) into one of the three types:
-  - Data operation: write(), writev(), truncate(), ftruncate()
-  - Metadata operation: setattr(), fsetattr(), setxattr(), fsetxattr(), removexattr(), fremovexattr()
-  - Entry operation: create(), mkdir(), mknod(), symlink(), link(), rename(), unlink(), rmdir()
-
-Changelog Entry Format
-----------------------
-
-In order to record the type of operation and entity underwent, a type identifier is used. Normally, the entity on which the operation is performed would be identified by the pathname, which is the most common way of addressing in a file system, but we choose to use GlusterFS internal file identifier (GFID) instead (as GlusterFS supports GFID based backend and the pathname field may not always be valid and other reasons which are out of scope of this this document). Therefore, the format of the record for the three types of operation can be summarized as follows:
-
-  - TYPE-I   : GFID of the file
-  - TYPE-II  : GFID of the file
-  - TYPE-III : GFID + FOP + MODE + UID + GID + PARGFID/BNAME [PARGFID/BNAME]
-
-GFID's are analogous to inodes. TYPE-I and TYPE-II fops record the GFID of the entity on which the operation was performed: thereby recording that there was an data/metadata change on the inode. TYPE-III fops record at the minimum a set of six or seven records (depending on the type of operation), that is sufficient to identify what type of operation the entity underwent. Normally this record inculdes the GFID of the entity, the type of file operation (which is an integer [an enumerated value which is used in GluterFS]) and the parent GFID and the basename (analogous to parent inode and basename).
-
-Changelogs can be either in ascii or binary format, the difference being the format of the records that is persisted. In a binary changelog the gfids are recorded in it's native format ie. 16 byte record and the fop number as a 4 byte integer. In an ascii changelog, the gfids are stored in their canonical form and the fop number is stringified and persisted. Null charater is used as the record serarator and changelogs. This makes it hard to read changelogs from the command line, but the packed format is needed to support file names with spaces and special characters. Below is a snippet of a changelog along side it's hexdump.
-
-```
-00000000  47 6c 75 73 74 65 72 46  53 20 43 68 61 6e 67 65  |GlusterFS Change|
-00000010  6c 6f 67 20 7c 20 76 65  72 73 69 6f 6e 3a 20 76  |log | version: v|
-00000020  31 2e 31 20 7c 20 65 6e  63 6f 64 69 6e 67 20 3a  |1.1 | encoding :|
-00000030  20 32 0a 45 61 36 39 33  63 30 34 65 2d 61 66 39  | 2.Ea693c04e-af9|
-00000040  65 2d 34 62 61 35 2d 39  63 61 37 2d 31 63 34 61  |e-4ba5-9ca7-1c4a|
-00000050  34 37 30 31 30 64 36 32  00 32 33 00 33 33 32 36  |47010d62.23.3326|
-00000060  31 00 30 00 30 00 66 36  35 34 32 33 32 65 2d 61  |1.0.0.f654232e-a|
-00000070  34 32 62 2d 34 31 62 33  2d 62 35 61 61 2d 38 30  |42b-41b3-b5aa-80|
-00000080  33 62 33 64 61 34 35 39  33 37 2f 6c 69 62 76 69  |3b3da45937/libvi|
-00000090  72 74 5f 64 72 69 76 65  72 5f 6e 65 74 77 6f 72  |rt_driver_networ|
-000000a0  6b 2e 73 6f 00 44 61 36  39 33 63 30 34 65 2d 61  |k.so.Da693c04e-a|
-000000b0  66 39 65 2d 34 62 61 35  2d 39 63 61 37 2d 31 63  |f9e-4ba5-9ca7-1c|
-000000c0  34 61 34 37 30 31 30 64  36 32 00 45 36 65 39 37  |4a47010d62.E6e97|
-```
-
-As you can see, there is an *entry* operation (journal record starting with an "E"). Records for this operation are:
-  - GFID  : a693c04e-af9e-4ba5-9ca7-1c4a-47010d62
-  - FOP   : 23  (create)
-  - Mode  : 33261
-  - UID   : 0
-  - GID   : 0
-  - PARGFID/BNAME: f654232e-a42b-41b3-b5aa-803b3da45937
-
-**NOTE**: In case of a rename operation, there would be an additional record (for the target PARGFID/BNAME).
-
-libgfchangelog
---------------
-
-NOTE: changelogs generated by the changelog translator are rolled over [with the timestamp as the suffix] after a specific interval, after which a new change is started. The current changelog [changelog file without the timestamp as the suffix] should never be processed unless it's rolled over. The rolled over logs should be treated read-only.
-
-Capturing changes performed on a file system is useful for applications that rely on file system scan (crawl) to figure out such information. Backup utilities, automatic file healing in a replicated environment, bit-rot detection and the likes are some of the end user applications that require a set of changed entities in a file system to act on. Goal of libgfchangelog is to provide the application (consumer) a fast and easy to use common query interface (API). The consumer need not worry about the changelog format, nomenclature of the changelog files etc.
-
-Now we list functionality and some of the features.
-
-Functionality
--------------
-
-Changelog Processing: Processing involes reading changelog file(s), converting the entries into human-readable (or application understandable) format (in case of binary log format).
-Book-keeping: Keeping track of how much the application has consumed the changelog (ie. changes during the time slice start-time -> end-time).
-Serve API request: Update the consumer by providing the set of changes.
-
-Processing could be done in two ways:
-
-* Pre-processing (pre-processing from the library POV):
-Once a changelog file is rolled over (by the changelog translator), a set of post processing operations are performed. These operations could include conversion of a binary log file to an understandable format, collate a bunch of logs into a larger sampling period or just keep a private copy of the changelog (in ascii format). Extra disk space is consumed to store this private copy. The library would then be free to consume these logs and serve application requests.
-
-* On-demand:
-The processing of the changelogs is trigerred when an application requests for changes. Downside of this being additional time spent on decoding the logs and data accumulation during application request time (but no additional disk space is used over the time period).
-
-After processing, the changelog is ready to be consumed by the application. The function of processing is to convert the logs into human/application readable format (an example is shown below):
-
-```
-E a7264fe2-dd6b-43e1-8786-a03b42cc2489 CREATE 33188 0 0 00000000-0000-0000-0000-000000000001%2Fservices1
-M a7264fe2-dd6b-43e1-8786-a03b42cc2489 NULL
-M 00000000-0000-0000-0000-000000000001 NULL
-D a7264fe2-dd6b-43e1-8786-a03b42cc2489
-```
-
-Features
---------
-
-The following points mention some of the features that the library could provide.
-
-  - Consumer could choose the update type when it registers with the library. 'types' could be:
-    - Streaming: The consumer is updated via stream of changes, ie. the library would just replay the logs
-    - Consolidated: The consumer is provided with a consolidated view of the changelog, eg. if <gfid> had an DATA and a METADATA operation, it would be presented as a single update. Similarly for ENTRY operations.
-    - Raw: This mode provides the consumer with the pathnames of the changelog files itself (after processing). The changelogs should be strictly treated as read-only. This gives the flexibility to the consumer to extract updates using thier own preferred way (eg. using command line tools like sed, awk, sort | uniq etc.).
-  - Application may choose to adopt a synchronous (blocking) or an asynchronous (callback) notification mechanism.
-  - Provide a unified view of changelogs from multiple peers (replication scenario) or a global changelog view of the entire cluster.
-
-
-** The first cut of the library supports:**
-  - Raw access mode
-  - Synchronous programming model
-  - Per brick changelog consumption ie. no unified/globally aggregated changelog
-
-[1]:http://review.gluster.org/5127
diff --git a/doc/features/gfid-access.md b/doc/features/gfid-access.md
deleted file mode 100644
index 2d324a18bdb..00000000000
--- a/doc/features/gfid-access.md
+++ /dev/null
@@ -1,73 +0,0 @@
-#Gfid-access Translator
-The 'gfid-access' translator provides access to data in glusterfs using a
-virtual path. This particular translator is designed to provide direct access to
-files in glusterfs using its gfid. 'GFID' is glusterfs's inode number for a file
-to identify it uniquely. As of now, Geo-replication is the only consumer of this
-translator. The changelog translator logs the 'gfid' with corresponding file
-operation in journals which are consumed by Geo-Replication to replicate the
-files using gfid-access translator very efficiently.
-
-###Implications and Usage
-A new virtual directory called '.gfid' is exposed in the aux-gfid mount
-point when gluster volume is mounted with 'aux-gfid-mount' option.
-All the gfids of files are exposed in one level under the '.gfid' directory.
-No matter at what level the file resides, it is accessed using its
-gfid under this virutal directory as shown in example below. All access
-protocols work seemlessly, as the complexities are handled internally.
-
-###Testing
-1. Mount glusterfs client with '-o aux-gfid-mount' as follows.
-
-      mount -t glusterfs -o aux-gfid-mount <node-ip>:<volname> <mountpoint>
-
-      Example:
-
-        #mount -t glusterfs -o aux-gfid-mount rhs1:master /master-aux-mnt
-
-2. Get the 'gfid' of a file using normal mount or aux-gfid-mount and do some
-   operations as follows.
-
-      getfattr -n glusterfs.gfid.string <file>
-
-      Example:
-
-        #getfattr -n glusterfs.gfid.string  /master-aux-mnt/file
-        # file: file
-        glusterfs.gfid.string="796d3170-0910-4853-9ff3-3ee6b1132080"
-
-        #cat /master-aux-mnt/file
-        sample data
-
-        #stat /master-aux-mnt/file
-           File: `file'
-           Size: 12             Blocks: 1          IO Block: 131072 regular file
-         Device: 13h/19d        Inode: 11525625031905452160  Links: 1
-         Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/   root)
-         Access: 2014-05-23 20:43:33.239999863 +0530
-         Modify: 2014-05-23 17:36:48.224999989 +0530
-         Change: 2014-05-23 20:44:10.081999938 +0530
-
-
-3. Access files using virtual path as follows.
-
-      /mountpoint/.gfid/<actual-canonical-gfid-of-the-file\>'
-
-      Example:
-
-        #cat /master-aux-mnt/.gfid/796d3170-0910-4853-9ff3-3ee6b1132080
-        sample data
-        #stat /master-aux-mnt/.gfid/796d3170-0910-4853-9ff3-3ee6b1132080
-           File: `.gfid/796d3170-0910-4853-9ff3-3ee6b1132080'
-           Size: 12             Blocks: 1          IO Block: 131072 regular file
-         Device: 13h/19d        Inode: 11525625031905452160  Links: 1
-         Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/   root)
-         Access: 2014-05-23 20:43:33.239999863 +0530
-         Modify: 2014-05-23 17:36:48.224999989 +0530
-         Change: 2014-05-23 20:44:10.081999938 +0530
-
-   We can notice that 'cat' command on the 'file' using path and using virtual
-   path displays the same data. Similarly 'stat' command on the 'file' and using
-   virtual path with gfid gives same Inode Number confirming that its same file.
-
-###Nature of changes
-This feature is introduced with 'gfid-access' translator.
diff --git a/doc/features/libgfapi.md b/doc/features/libgfapi.md
deleted file mode 100644
index dfc8cfe6527..00000000000
--- a/doc/features/libgfapi.md
+++ /dev/null
@@ -1,381 +0,0 @@
-One of the known methods to access glusterfs is via fuse module. However, it has some overhead or performance issues because of the number of context switches which need to be performed to complete one i/o transaction[1].
-
-
-To over come this limitation, a new method called ‘libgfapi’ is introduced. libgfapi support is available from GlusterFS-3.4 release.
-
-libgfapi is a userspace library for accessing data in glusterfs. libgfapi library perform IO on gluster volumes directly without FUSE mount. It is a filesystem like api and runs/sits in application process context. libgfapi eliminates the fuse and the kernel vfs layer from the glusterfs volume access. The speed and latency have improved with libgfapi access. [1]
-
-
-Using libgfapi, various user-space filesystems (like NFS-Ganesha or Samba) or the virtualizer (like QEMU) can interact with GlusterFS which serves as back-end filesystem. Currently below projects integrate with glusterfs using libgfapi interfaces.
-
-
-* qemu storage layer
-* Samba VFS plugin
-* NFS-Ganesha
-
-All the APIs in libgfapi make use of `struct glfs` object. This object
-contains information about volume name, glusterfs context associated,
-subvols in the graph etc which makes it unique for each volume.
-
-
-For any application to make use of libgfapi, it should typically start
-with the below APIs in the following order -
-
-* To create a new glfs object :
-
-    glfs_t *glfs_new (const char *volname) ;
-
-    glfs_new() returns glfs_t object.
-
-
-* On this newly created glfs_t, you need to be either set a volfile path
-   (glfs_set_volfile) or a volfile server (glfs_set_volfile_server).
-    Incase of failures, the corresponding cleanup routine is
-    "glfs_unset_volfile_server"
-
-    int glfs_set_volfile (glfs_t *fs, const char *volfile);
-
-    int glfs_set_volfile_server (glfs_t *fs, const char *transport,const char *host, int port) ;
-
-    int glfs_unset_volfile_server (glfs_t *fs, const char *transport,const char *host, int port) ;
-
-*  Specify logging parameters using glfs_set_logging():
-
-    int glfs_set_logging (glfs_t *fs, const char *logfile, int loglevel) ;
-
-* Initializes the glfs_t object using glfs_init()
-    int glfs_init (glfs_t *fs) ;
-
-#### FOPs APIs available with libgfapi :
-
-
-
-    int glfs_get_volumeid (struct glfs *fs, char *volid, size_t size);
-
-    int glfs_setfsuid (uid_t fsuid) ;
-
-    int glfs_setfsgid (gid_t fsgid) ;
-
-    int glfs_setfsgroups (size_t size, const gid_t *list) ;
-
-    glfs_fd_t *glfs_open (glfs_t *fs, const char *path, int flags) ;
-
-    glfs_fd_t *glfs_creat (glfs_t *fs, const char *path, int flags,mode_t mode) ;
-
-    int glfs_close (glfs_fd_t *fd) ;
-
-    glfs_t *glfs_from_glfd (glfs_fd_t *fd) ;
-
-    int glfs_set_xlator_option (glfs_t *fs, const char *xlator, const char *key,const char *value) ;
-
-    typedef void (*glfs_io_cbk) (glfs_fd_t *fd, ssize_t ret, void *data);
-
-    ssize_t glfs_read (glfs_fd_t *fd, void *buf,size_t count, int flags) ;
-
-    ssize_t glfs_write (glfs_fd_t *fd, const void *buf,size_t count, int flags) ;
-
-    int glfs_read_async (glfs_fd_t *fd, void *buf, size_t count, int flags, glfs_io_cbk fn, void *data) ;
-
-    int glfs_write_async (glfs_fd_t *fd, const void *buf, size_t count, int flags, glfs_io_cbk fn, void *data) ;
-
-    ssize_t glfs_readv (glfs_fd_t *fd, const struct iovec *iov, int iovcnt,int flags) ;
-
-    ssize_t glfs_writev (glfs_fd_t *fd, const struct iovec *iov, int iovcnt,int flags) ;
-
-    int glfs_readv_async (glfs_fd_t *fd, const struct iovec *iov, int count, int flags, glfs_io_cbk fn, void *data) ;
-
-    int glfs_writev_async (glfs_fd_t *fd, const struct iovec *iov, int count, int flags, glfs_io_cbk fn, void *data) ;
-
-    ssize_t glfs_pread (glfs_fd_t *fd, void *buf, size_t count, off_t offset,int flags) ;
-
-    ssize_t glfs_pwrite (glfs_fd_t *fd, const void *buf, size_t count, off_t offset, int flags) ;
-
-    int glfs_pread_async (glfs_fd_t *fd, void *buf, size_t count, off_t offset,int flags, glfs_io_cbk fn, void *data) ;
-
-    int glfs_pwrite_async (glfs_fd_t *fd, const void *buf, int count, off_t offset,int flags, glfs_io_cbk fn, void *data) ;
-
-    ssize_t glfs_preadv (glfs_fd_t *fd, const struct iovec *iov, int iovcnt, int count, off_t offset, int flags,glfs_io_cbk fn, void *data) ;
-
-    ssize_t glfs_pwritev (glfs_fd_t *fd, const struct iovec *iov, int iovcnt,int count, off_t offset, int flags, glfs_io_cbk fn, void *data) ;
-
-    int glfs_preadv_async (glfs_fd_t *fd, const struct iovec *iov, glfs_io_cbk fn, void *data) ;
-
-    int glfs_pwritev_async (glfs_fd_t *fd, const struct iovec *iov, glfs_io_cbk fn, void *data) ;
-
-    off_t glfs_lseek (glfs_fd_t *fd, off_t offset, int whence) ;
-
-    int glfs_truncate (glfs_t *fs, const char *path, off_t length) ;
-
-    int glfs_ftruncate (glfs_fd_t *fd, off_t length) ;
-
-    int glfs_ftruncate_async (glfs_fd_t *fd, off_t length, glfs_io_cbk fn,void *data) ;
-
-    int glfs_lstat (glfs_t *fs, const char *path, struct stat *buf) ;
-
-    int glfs_stat (glfs_t *fs, const char *path, struct stat *buf) ;
-
-    int glfs_fstat (glfs_fd_t *fd, struct stat *buf) ;
-
-    int glfs_fsync (glfs_fd_t *fd) ;
-
-    int glfs_fsync_async (glfs_fd_t *fd, glfs_io_cbk fn, void *data) ;
-
-    int glfs_fdatasync (glfs_fd_t *fd) ;
-
-    int glfs_fdatasync_async (glfs_fd_t *fd, glfs_io_cbk fn, void *data) ;
-
-    int glfs_access (glfs_t *fs, const char *path, int mode) ;
-
-    int glfs_symlink (glfs_t *fs, const char *oldpath, const char *newpath) ;
-
-    int glfs_readlink (glfs_t *fs, const char *path,char *buf, size_t bufsiz) ;
-
-    int glfs_mknod (glfs_t *fs, const char *path, mode_t mode, dev_t dev) ;
-
-    int glfs_mkdir (glfs_t *fs, const char *path, mode_t mode) ;
-
-    int glfs_unlink (glfs_t *fs, const char *path) ;
-
-    int glfs_rmdir (glfs_t *fs, const char *path) ;
-
-    int glfs_rename (glfs_t *fs, const char *oldpath, const char *newpath) ;
-
-    int glfs_link (glfs_t *fs, const char *oldpath, const char *newpath) ;
-
-    glfs_fd_t *glfs_opendir (glfs_t *fs, const char *path) ;
-
-    int glfs_readdir_r (glfs_fd_t *fd, struct dirent *dirent,struct dirent **result) ;
-
-    int glfs_readdirplus_r (glfs_fd_t *fd, struct stat *stat, struct dirent *dirent, struct dirent **result) ;
-
-    struct dirent *glfs_readdir (glfs_fd_t *fd) ;
-
-    struct dirent *glfs_readdirplus (glfs_fd_t *fd, struct stat *stat) ;
-
-    long glfs_telldir (glfs_fd_t *fd) ;
-
-    void glfs_seekdir (glfs_fd_t *fd, long offset) ;
-
-    int glfs_closedir (glfs_fd_t *fd) ;
-
-    int glfs_statvfs (glfs_t *fs, const char *path, struct statvfs *buf) ;
-
-    int glfs_chmod (glfs_t *fs, const char *path, mode_t mode) ;
-
-    int glfs_fchmod (glfs_fd_t *fd, mode_t mode) ;
-
-    int glfs_chown (glfs_t *fs, const char *path, uid_t uid, gid_t gid) ;
-
-    int glfs_lchown (glfs_t *fs, const char *path, uid_t uid, gid_t gid) ;
-
-    int glfs_fchown (glfs_fd_t *fd, uid_t uid, gid_t gid) ;
-
-    int glfs_utimens (glfs_t *fs, const char *path,struct timespec times[2]) ;
-
-    int glfs_lutimens (glfs_t *fs, const char *path,struct timespec times[2]) ;
-
-    int glfs_futimens (glfs_fd_t *fd, struct timespec times[2]) ;
-
-    ssize_t glfs_getxattr (glfs_t *fs, const char *path, const char *name,void *value, size_t size) ;
-
-    ssize_t glfs_lgetxattr (glfs_t *fs, const char *path, const char *name,void *value, size_t size) ;
-
-    ssize_t glfs_fgetxattr (glfs_fd_t *fd, const char *name,void *value, size_t size) ;
-
-    ssize_t glfs_listxattr (glfs_t *fs, const char *path,void *value, size_t size) ;
-
-    ssize_t glfs_llistxattr (glfs_t *fs, const char *path, void *value,size_t size) ;
-
-    ssize_t glfs_flistxattr (glfs_fd_t *fd, void *value, size_t size) ;
-
-    int glfs_setxattr (glfs_t *fs, const char *path, const char *name,const void *value, size_t size, int flags) ;
-
-    int glfs_lsetxattr (glfs_t *fs, const char *path, const char *name,const void *value, size_t size, int flags) ;
-
-    int glfs_fsetxattr (glfs_fd_t *fd, const char *name,const void *value, size_t size, int flags) ;
-
-    int glfs_removexattr (glfs_t *fs, const char *path, const char *name) ;
-
-    int glfs_lremovexattr (glfs_t *fs, const char *path, const char *name) ;
-
-    int glfs_fremovexattr (glfs_fd_t *fd, const char *name) ;
-
-    int glfs_fallocate(glfs_fd_t *fd, int keep_size, off_t offset, size_t len) ;
-
-    int glfs_discard(glfs_fd_t *fd, off_t offset, size_t len) ;
-
-    int glfs_discard_async (glfs_fd_t *fd, off_t length, size_t lent, glfs_io_cbk fn, void *data) ;
-
-    int glfs_zerofill(glfs_fd_t *fd, off_t offset, off_t len) ;
-
-    int glfs_zerofill_async (glfs_fd_t *fd, off_t length, off_t len, glfs_io_cbk fn, void *data) ;
-
-    char *glfs_getcwd (glfs_t *fs, char *buf, size_t size) ;
-
-    int glfs_chdir (glfs_t *fs, const char *path) ;
-
-    int glfs_fchdir (glfs_fd_t *fd) ;
-
-    char *glfs_realpath (glfs_t *fs, const char *path, char *resolved_path) ;
-
-    int glfs_posix_lock (glfs_fd_t *fd, int cmd, struct flock *flock) ;
-
-    glfs_fd_t *glfs_dup (glfs_fd_t *fd) ;
-
-
-    struct glfs_object *glfs_h_lookupat (struct glfs *fs,struct glfs_object *parent,
-				     const char *path,
-                                     struct stat *stat) ;
-
-    struct glfs_object *glfs_h_creat (struct glfs *fs, struct glfs_object *parent,
-				  const char *path, int flags, mode_t mode,
-				  struct stat *sb) ;
-
-    struct glfs_object *glfs_h_mkdir (struct glfs *fs, struct glfs_object *parent,
-				  const char *path, mode_t flags,
-				  struct stat *sb) ;
-
-    struct glfs_object *glfs_h_mknod (struct glfs *fs, struct glfs_object *parent,
-				  const char *path, mode_t mode, dev_t dev,
-				  struct stat *sb) ;
-
-    struct glfs_object *glfs_h_symlink (struct glfs *fs, struct glfs_object *parent,
-				    const char *name, const char *data,
-				    struct stat *stat) ;
-
-
-    int glfs_h_unlink (struct glfs *fs, struct glfs_object *parent,
-		   const char *path) ;
-
-    int glfs_h_close (struct glfs_object *object) ;
-
-    int glfs_caller_specific_init (void *uid_caller_key, void *gid_caller_key,
-			       void *future) ;
-
-    int glfs_h_truncate (struct glfs *fs, struct glfs_object *object,
-                     off_t offset) ;
-
-    int glfs_h_stat(struct glfs *fs, struct glfs_object *object,
-                struct stat *stat) ;
-
-    int glfs_h_getattrs (struct glfs *fs, struct glfs_object *object,
-		     struct stat *stat) ;
-
-    int glfs_h_getxattrs (struct glfs *fs, struct glfs_object *object,
-		      const char *name, void *value,
-		      size_t size) ;
-
-    int glfs_h_setattrs (struct glfs *fs, struct glfs_object *object,
-		     struct stat *sb, int valid) ;
-
-    int glfs_h_setxattrs (struct glfs *fs, struct glfs_object *object,
-		      const char *name, const void *value,
-		      size_t size, int flags) ;
-
-    int glfs_h_readlink (struct glfs *fs, struct glfs_object *object, char *buf,
-		     size_t bufsiz) ;
-
-    int glfs_h_link (struct glfs *fs, struct glfs_object *linktgt,
-		 struct glfs_object *parent, const char *name) ;
-
-    int glfs_h_rename (struct glfs *fs, struct glfs_object *olddir,
-		   const char *oldname, struct glfs_object *newdir,
-		   const char *newname) ;
-
-    int glfs_h_removexattrs (struct glfs *fs, struct glfs_object *object,
-			 const char *name) ;
-
-    ssize_t glfs_h_extract_handle (struct glfs_object *object,
-			       unsigned char *handle, int len) ;
-
-    struct glfs_object *glfs_h_create_from_handle (struct glfs *fs,
-					       unsigned char *handle, int len,
-					       struct stat *stat) ;
-
-
-    struct glfs_fd *glfs_h_opendir (struct glfs *fs,
-                                struct glfs_object *object) ;
-
-    struct glfs_fd *glfs_h_open (struct glfs *fs, struct glfs_object *object,
-			     int flags) ;
-
-For more details on these apis please refer glfs.h and glfs-handles.h  in the source tree (api/src/) of glusterfs:
-
-* Incase of failures or to close the connection and destroy glfs_t
-object, use glfs_fini.
-
-    int glfs_fini (glfs_t *fs) ;
-
-
-All the fileops are typically divided into below categories
-
-* a) Handle based Operations -
-
-These APIs create/make use of a glfs_object (referred as handles) unique
-to each file within a volume.
-The structure glfs_object contains inode pointer and gfid.
-
-For example: Since NFS protocol uses file handles to access files, these APIs are
-mainly used by NFS-Ganesha server.
-
-Eg:
-
-    struct glfs_object *glfs_h_lookupat (struct glfs *fs,
-                                      struct glfs_object *parent,
-                                      const char *path,
-                                      struct stat *stat);
-
-    struct glfs_object *glfs_h_creat (struct glfs *fs,
-                                   struct glfs_object *parent,
-                                   const char *path,
-                                   int flags, mode_t mode,
-                                   struct stat *sb);
-
-    struct glfs_object *glfs_h_mkdir (struct glfs *fs,
-                                struct glfs_object *parent,
-                                const char *path, mode_t flags,
-                                struct stat *sb);
-
-
-
-* b) File path/descriptor based Operations -
-
-These APIs make use of file path/descriptor to determine the file on
-which it needs to operate on.
-
-For example: Samba uses these APIs for file operations.
-
-Examples of the APIs using file path -
-
-    int glfs_chdir (glfs_t *fs, const char *path) ;
-
-    char *glfs_realpath (glfs_t *fs, const char *path, char *resolved_path) ;
-
-Once the file is opened, the file-descriptor generated is used for
-further operations.
-
-Eg:
-
-    int glfs_posix_lock (glfs_fd_t *fd, int cmd, struct flock *flock) ;
-    glfs_fd_t *glfs_dup (glfs_fd_t *fd) ;
-
-
-
-#### libgfapi bindings :
-
-libgfapi bindings are available for below languages:
-
-    - Go
-    - Java
-    - python [2]
-    - Ruby
-
-For more details on these bindings,please refer :
-
-    #http://www.gluster.org/community/documentation/index.php/Language_Bindings
-
-References:
-
-[1] http://humblec.com/libgfapi-interface-glusterfs/
-[2] http://www.gluster.org/2014/04/play-with-libgfapi-and-its-python-bindings/
-
diff --git a/doc/features/nufa.md b/doc/features/nufa.md
deleted file mode 100644
index 03b8194b4c0..00000000000
--- a/doc/features/nufa.md
+++ /dev/null
@@ -1,20 +0,0 @@
-# NUFA Translator
-
-The NUFA ("Non Uniform File Access") is a variant of the DHT ("Distributed Hash
-Table") translator, intended for use with workloads that have a high locality
-of reference.  Instead of placing new files pseudo-randomly, it places them on
-the same nodes where they are created so that future accesses can be made
-locally.  For replicated volumes, this means that one copy will be local and
-others will be remote; the read-replica selection mechanisms will then favor
-the local copy for reads.  For non-replicated volumes, the only copy will be
-local.
-
-## Interface
-
-Use of NUFA is controlled by a volume option, as follows.
-
-	gluster volume set myvolume cluster.nufa on
-
-This will cause the NUFA translator to be used wherever the DHT translator
-otherwise would be.  The rest is all automatic.
-
diff --git a/doc/features/ovirt-integration.md b/doc/features/ovirt-integration.md
deleted file mode 100644
index 46dbeabbbaa..00000000000
--- a/doc/features/ovirt-integration.md
+++ /dev/null
@@ -1,106 +0,0 @@
-##Ovirt Integration with glusterfs
-
-oVirt is an opensource virtualization management platform. You can use oVirt to manage
-hardware nodes, storage and network resources, and to deploy and monitor virtual machines
-running in your data center.  oVirt serves as the bedrock for Red Hat''s Enterprise Virtualization product,
-and is the "upstream" project where new features are developed in advance of their inclusion
-in that supported product offering.
-
-To know more about ovirt please visit http://www.ovirt.org/ and to configure
-#http://www.ovirt.org/Quick_Start_Guide#Install_oVirt_Engine_.28Fedora.29%60
-
-For the installation step of ovirt, please refer
-#http://www.ovirt.org/Quick_Start_Guide#Install_oVirt_Engine_.28Fedora.29%60
-
-When oVirt integrated with gluster, glusterfs can be used in below forms:
-
-* As a storage domain to host VM disks.
-
-There are mainly two ways to exploit glusterfs as a storage domain.
-	- POSIXFS_DOMAIN ( >=oVirt 3.1 )
-	- GLUSTERFS_DOMAIN ( >=oVirt 3.3)
-
-The former one has performance overhead and is not an ideal way to consume images hosted in glusterfs volumes.
-When used by this method, qemu uses glusterfs `mount point` to access VM images and invite FUSE overhead.
-The libvirt treats this as a file type disk in its xml schema.
-
-The latter is the recommended way of using glusterfs with ovirt as a storage domain. This provides better
-and efficient way to access images hosted under glusterfs volumes.When qemu accessing glusterfs volume using this method,
-it make use of `libgfapi` implementation of glusterfs and this method is called native integration.
-Here the glusterfs is added as a block backend to qemu and libvirt treat this as a `network` type disk.
-
-For more details on this, please refer # http://www.ovirt.org/Features/GlusterFS_Storage_Domain
-However there are 2 bugs which block usage of this feature.
-
-https://bugzilla.redhat.com/show_bug.cgi?id=1022961
-https://bugzilla.redhat.com/show_bug.cgi?id=1017289
-
-Please check above bugs for latest status.
-
-* To manage gluster trusted pools.
-
-oVirt web admin console can be used to -
-	- add new / import existing gluster cluster
-	- add/delete volumes
-	- add/delete bricks
-	- set/reset volume options
-	- optimize volume for virt store
-	- Rebalance and Remove bricks
-	- Monitor gluster deployment - node, brick, volume status,
-	  Enhanced service monitoring (Physical node resources as well Quota, geo-rep and self-heal status) through Nagios integration(>=oVirt 3.4)
-
-
-
-When configuing ovirt to manage only gluster cluster/trusted pool, you need to select `gluster` as an input for
-`Application mode` in OVIRT ENGINE CONFIGURATION option of `engine-setup` command.
-Refer # http://www.ovirt.org/Quick_Start_Guide#Install_oVirt_Engine_.28Fedora.29%60
-
-If you want to use gluster as both ( as a storage domain to host VM disks and to manage gluster trusted pools)
-you need to input `both` as a value for `Application mode` in engine-setup command.
-
-Once you have successfully installed oVirt Engine as mentioned above, you will be provided with instructions
-to access oVirt''s web console.
-
-Below example shows how to configure gluster nodes in fedora.
-
-
-#Configuring gluster nodes.
-
-On the machine designated as your host, install any supported distribution( ex:Fedora/CentOS/RHEL...etc).
-A minimal installation is sufficient.
-
-Refer # http://www.ovirt.org/Quick_Start_Guide#Install_Hosts
-
-
-##Connect to Ovirt Engine
-
-Log In to Administration Console
-
-Ensure that you have the administrator password configured during installation of  oVirt engine.
-
-- To connect to oVirt webadmin console
-
-
-Open a browser and navigate to https://domain.example.com/webadmin. Substitute domain.example.com with the URL provided during installation
-
-If this is your first time connecting to the administration console, oVirt Engine will issue
-security certificates for your browser. Click the link labelled this certificate to trust the
-ca.cer certificate. A pop-up displays, click Open to launch the Certificate dialog.
-Click `Install Certificate` and select to place the certificate in Trusted Root Certification Authorities store.
-
-
-The console login screen displays. Enter admin as your User Name, and enter the Password that
-you provided during installation. Ensure that your domain is set to Internal. Click Login.
-
-
-You have now successfully logged in to the oVirt web administration console. Here, you can configure and manage all your gluster resources.
-
-To manage gluster trusted pool:
-
-- Create a cluster with "Enable gluster service" - turned on. (Turn on "Enable virt service" if the same nodes are used as hypervisor as well)
-- Add hosts which have already been set up as in step Configuring gluster nodes.
-- Create a volume, and click on "Optimize for virt store",This sets the volume tunables optimize volume to be used as an image store
-
-To use this volume as a storage domain:
-
-Please refer `User interface` section of www.ovirt.org/Features/GlusterFS_Storage_Domain
diff --git a/doc/features/qemu-integration.md b/doc/features/qemu-integration.md
deleted file mode 100644
index b44dc06bb43..00000000000
--- a/doc/features/qemu-integration.md
+++ /dev/null
@@ -1,231 +0,0 @@
-Using GlusterFS volumes to host VM images and data was sub-optimal due to the FUSE overhead involved in accessing gluster volumes via GlusterFS native client. However this has changed now with two specific enhancements:
-
-- A new library called libgfapi is now available as part of GlusterFS that  provides POSIX-like C APIs for accessing gluster volumes. libgfapi support is available from GlusterFS-3.4 release.
-- QEMU (starting from QEMU-1.3) will have GlusterFS block driver that uses libgfapi and hence there is no FUSE overhead any longer when QEMU works with VM images on gluster volumes.
-
-GlusterFS with its pluggable translator model can serve as a flexible storage backend for QEMU. QEMU has to just talk to GlusterFS and GlusterFS will hide different file systems and storage types underneath. Various GlusterFS storage features like replication and striping will automatically be available for QEMU. Efforts are also on to add block device backend in Gluster via Block Device (BD) translator that will expose underlying block devices as files to QEMU. This allows GlusterFS to be a single storage backend for both file and block based storage types.
-
-###GlusterFS specifcation in QEMU
-
-VM image residing on gluster volume can be specified on QEMU command line using URI format
-
-    gluster[+transport]://[server[:port]]/volname/image[?socket=...]
-
-
-
-* `gluster` is the protocol.
-
-* `transport` specifies the transport type used to connect to gluster management daemon (glusterd). Valid transport types are `tcp, unix and rdma.` If a transport type isn’t specified, then tcp type is assumed.
-
-* `server` specifies the server where the volume file specification for the given volume resides. This can be either hostname, ipv4 address or ipv6 address. ipv6 address needs to be within square brackets [ ]. If transport type is unix, then server field should not be specified. Instead the socket field needs to be populated with the path to unix domain socket.
-
-* `port` is the port number on which glusterd is listening. This is optional and if not specified, QEMU will send 0 which will make gluster to use the default port. If the transport type is unix, then port should not be specified.
-
-* `volname` is the name of the gluster volume which contains the VM image.
-
-* `image` is the path to the actual VM image that resides on gluster volume.
-
-
-###Examples:
-
-    gluster://1.2.3.4/testvol/a.img
-    gluster+tcp://1.2.3.4/testvol/a.img
-    gluster+tcp://1.2.3.4:24007/testvol/dir/a.img
-    gluster+tcp://[1:2:3:4:5:6:7:8]/testvol/dir/a.img
-    gluster+tcp://[1:2:3:4:5:6:7:8]:24007/testvol/dir/a.img
-    gluster+tcp://server.domain.com:24007/testvol/dir/a.img
-    gluster+unix:///testvol/dir/a.img?socket=/tmp/glusterd.socket
-    gluster+rdma://1.2.3.4:24007/testvol/a.img
-
-
-
-NOTE: (GlusterFS URI description and above examples are taken from QEMU documentation)
-
-###Configuring QEMU with GlusterFS backend
-
-While building QEMU from source, in addition to the normal configuration options, ensure that  –enable-glusterfs options are  specified explicitly with ./configure script to get glusterfs support in qemu.
-
-Starting with QEMU-1.6, pkg-config is used to configure the GlusterFS backend in QEMU. If you are using GlusterFS compiled and installed from sources, then the GlusterFS package config file (glusterfs-api.pc) might not be present at the standard path and you will have to explicitly add the path by executing this command before running the QEMU configure script:
-
-    export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/
-
-Without this, GlusterFS driver will not be compiled into QEMU even when GlusterFS is present in the system.
-
-* Creating a VM image on GlusterFS backend
-
-qemu-img command can be used to create VM images on gluster backend. The general syntax for image creation looks like this:
-
-For ex:
-
-    qemu-img create gluster://server/volname/path/to/image size
-
-## How to setup the environment:
-
-This usecase ( using glusterfs backend for VM disk store), is known as 'Virt-Store' usecase. Steps for the entire procedure could be split to:
-
-*    Steps to be done on gluster volume side
-*    Steps to be done on Hypervisor side
-
-
-##Steps to be done on gluster side
-
-These are the steps that needs to be done on the gluster side. Precisely this involves
-
-    Creating "Trusted Storage Pool"
-    Creating a volume
-    Tuning the volume for virt-store
-    Tuning glusterd to accept requests from QEMU
-    Tuning glusterfsd to accept requests from QEMU
-    Setting ownership on the volume
-    Starting the volume
-
-* Creating "Trusted Storage Pool"
-
-Install glusterfs rpms on the NODE. You can create a volume with a single node. You can also scale up the cluster, as we call as `Trusted Storage Pool`, by adding more nodes to the cluster
-
-    gluster peer probe <hostname>
-
-* Creating a volume
-
-It is highly recommended to have replicate volume or distribute-replicate volume for virt-store usecase, as it would add high availability and fault-tolerance. Remember the plain distribute works equally well
-
-    gluster volume create replica 2 <brick1> .. <brickN>
-
-where,  `<brick1> is <hostname>:/<path-of-dir>  `
-
-
-Note: It is recommended to create sub-directories inside brick and that could be used to create a volume.For example, say, /home/brick1 is the mountpoint of XFS, then you can create a sub-directory inside it /home/brick1/b1 and use it while creating a volume.You can also use space available in root filesystem for bricks. Gluster cli, by default, throws warning in that case. You can override it by using force option
-
-    gluster volume create replica 2 <brick1> .. <brickN> force
-
-If you are new to GlusterFS, you can take a look at QuickStart (http://www.gluster.org/community/documentation/index.php/QuickStart) guide.
-
-* Tuning the volume for virt-store
-
-There are recommended settings available for virt-store. This provide good performance characteristics when enabled on the volume that was used for virt-store
-
-Refer to  http://www.gluster.org/community/documentation/index.php/Virt-store-usecase#Tunables for recommended tunables and for applying them on the volume, http://www.gluster.org/community/documentation/index.php/Virt-store-usecase#Applying_the_Tunables_on_the_volume
-
-
-* Tuning glusterd to accept requests from QEMU
-
-glusterd receives the request only from the applications that run with port number less than 1024 and it blocks otherwise. QEMU uses port number greater than 1024 and to make glusterd accept requests from QEMU, edit the glusterd vol file, /etc/glusterfs/glusterd.vol and add the following,
-
-    option rpc-auth-allow-insecure on
-
-Note: If you have installed glusterfs from source, you can find glusterd vol file at `/usr/local/etc/glusterfs/glusterd.vol`
-
-Restart glusterd after adding that option to glusterd vol file
-
-    service glusterd restart
-
-* Tuning glusterfsd to accept requests from QEMU
-
-Enable the option `allow-insecure` on the particular volume
-
-    gluster volume set <volname> server.allow-insecure on
-
-IMPORTANT : As of now(april 2,2014)there is a bug, as allow-insecure is not dynamically set on a volume.You need to restart the volume for the change to take effect
-
-
-* Setting ownership on the volume
-
-Set the ownership of qemu:qemu on to the volume
-
-    gluster volume set <vol-name> storage.owner-uid 107
-    gluster volume set <vol-name> storage.owner-gid 107
-
-* Starting the volume
-
-Start the volume
-
-    gluster volume start <vol-name>
-
-## Steps to be done on Hypervisor Side:
-
-To create a raw image,
-
-    qemu-img create gluster://1.2.3.4/testvol/dir/a.img 5G
-
-To create a qcow2 image,
-
-    qemu-img create -f qcow2 gluster://server.domain.com:24007/testvol/a.img 5G
-
-
-
-
-
-## Booting VM image from GlusterFS backend
-
-A VM image 'a.img' residing on gluster volume testvol can be booted using QEMU like this:
-
-
-    qemu-system-x86_64 -drive file=gluster://1.2.3.4/testvol/a.img,if=virtio
-
-In addition to VM images, gluster drives can also be used as data drives:
-
-    qemu-system-x86_64 -drive file=gluster://1.2.3.4/testvol/a.img,if=virtio -drive file=gluster://1.2.3.4/datavol/a-data.img,if=virtio
-
-Here 'a-data.img' from datavol gluster volume appears as a 2nd drive for the guest.
-
-It is also possible to make use of libvirt to define a disk and use it with qemu:
-
-
-### Create libvirt XML to define Virtual Machine
-
-virt-install is python wrapper which is mostly used to create VM using set of params. How-ever virt-install doesn't support any network filesystem [ https://bugzilla.redhat.com/show_bug.cgi?id=1017308 ]
-
-Create a libvirt VM xml - http://libvirt.org/formatdomain.html where  the disk section is formatted in such a way, qemu driver for glusterfs is being used. This can be seen in the following example xml description
-
-
-    <disk type='network' device='disk'>
-        <driver name='qemu' type='raw' cache='none'/>
-        <source protocol='gluster' name='distrepvol/vm3.img'>
-        <host name='10.70.37.106' port='24007'/>
-        </source>
-    <target dev='vda' bus='virtio'/>
-    <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
-    </disk>
-
-
-
-
-
-* Define the VM from the XML file that was created earlier
-
-
-    virsh define <xml-file-description>
-
-* Verify that the VM is created successfully
-
-
-    virsh list --all
-
-* Start the VM
-
-
-    virsh start <VM>
-
-* Verification
-
-You can verify the disk image file that is being used by VM
-
-    virsh domblklist <VM-Domain-Name/ID>
-
-The above should show the volume name and image name. Here is the example,
-
-
-    [root@test ~]# virsh domblklist vm-test2
-    Target     Source
-    ------------------------------------------------
-    vda        distrepvol/test.img
-    hdc        -
-
-
-Reference:
-
-For more details on this feature implementation and its advantages, please refer:
-
-http://raobharata.wordpress.com/2012/10/29/qemu-glusterfs-native-integration/
-
-http://www.gluster.org/community/documentation/index.php/Libgfapi_with_qemu_libvirt
diff --git a/doc/features/quota-scalability.md b/doc/features/quota-scalability.md
deleted file mode 100644
index e47c898dd2a..00000000000
--- a/doc/features/quota-scalability.md
+++ /dev/null
@@ -1,52 +0,0 @@
-Issues with older implemetation:
------------------------------------
-* >#### Enforcement of quota was done on client side. This had following two issues :
-  > >* All clients are not trusted and hence enforcement is not secure.
-  > >* Quota enforcer caches directory size for a certain time out period to reduce network calls to fetch size. On time out, this cache is validated by querying server. With more clients, the traffic caused due to this
-validation increases.
-
-* >#### Relying on lookup calls on a file/directory (inode) to update its contribution [time consuming]
-
-* >####Hardlimits were stored in a comma separated list.
-  > >* Hence, changing hard limit of one directory is not an independent operation and would invalidate hard limits of other directories. We need to parse the string once for each of these directories just to identify whether its hard limit is changed. This limits the number of hard limits we can configure.
-
-* >####Cli used to fetch the list of directories on which quota-limit is set, from glusterd.
-  > >* With more number of limits, the network overhead incurred to fetch this list limits the scalability of number of directories on which we can set quota.
-
-* >#### Problem with NFS mount
-  > >*  Quota, for its enforcement and accounting requires all the ancestors of a file/directory till root. However, with NFS relying heavily on nameless lookups (through which there is no guarantee that ancestry can be
-accessed) this ancestry is not always present. Hence accounting and enforcement was not correct.
-
-
-New Design Implementation:
---------------------------------
-
-* Quota enforcement is moved to server side. This addresses issues that arose because of client side enforcement.
-
-* Two levels of quota limits, soft and hard quota is introduced.
-  This will result in a message being logged on reaching soft quota and    writes will fail with EDQUOT after hard limit is reached.
-
-Work Flow
------------------
-
-* Accounting
-      # This is done using the marker translator loaded on each brick of the volume. Accounting happens in the background. Ie, it doesn't happen in-flight with the file operation. The file operations latency is not
-directly affected by the time taken to perform accounting. This update is sent recursively upwards up to the root of the volume.
-
-* Enforcement
-      # The enforcer updates its 'view' (cached) of directory's disk usage on the incidence of a file operation after the expiry of hard/soft timeout, depending on the current usage. Enforcer uses quotad to get the
-aggregated disk usage of a directory from the accounting information present on each brick (viz, provided by marker).
-
-* Aggregator (quotad)
-      # Quotad is a daemon that serves volume-wide disk usage of a directory, on which quota is configured. It is present on all nodes in the cluster (trusted storage pool) as bricks don't have a global view of cluster.
-Quotad queries the disk usage information from all the bricks in that volume and aggregates. It manages all the volumes on which quota is enabled.
-
-
-Benefit to GlusterFS
----------------------------------
-
-* Support upto 65536 quota configurations per volume.
-* More quotas can be configured in a single volume thereby leading to support GlusterFS for use cases like home directory.
-
-###For more information on quota usability refer the following link :
-> https://access.redhat.com/site/documentation/en-US/Red_Hat_Storage/2.1/html-single/Administration_Guide/index.html#chap-User_Guide-Dir_Quota-Enable
diff --git a/doc/features/rdma-cm-in-3.4.0.txt b/doc/features/rdma-cm-in-3.4.0.txt
deleted file mode 100644
index fd953e56b3f..00000000000
--- a/doc/features/rdma-cm-in-3.4.0.txt
+++ /dev/null
@@ -1,9 +0,0 @@
-Following is the impact of http://review.gluster.org/#change,149.
-
-New userspace packages needed:
-librdmacm
-librdmacm-devel
-
-rdmacm needs an IPoIB address for connection establishment. This requirement results in following issues:
-* Because of bug #890502, we've to probe the peer on an IPoIB address. This imposes a restriction that all volumes created in the future have to communicate over IPoIB address (irrespective of whether they use gluster's tcp or rdma transport).
-* Currently client has an independence to choose b/w tcp and rdma transports while communicating with the server (by creating volumes with transport-type tcp,rdma). This independence was a byproduct of our ability use the normal channel used with transport-type tcp for rdma connectiion establishment handshake too. However, with new requirement of IPoIB address for connection establishment, we loose this independence (till we bring in multi-network support - where a brick can be identified by a set of ip-addresses and we can choose different pairs of ip-addresses for communication based on our requirements - in glusterd).
diff --git a/doc/features/rdmacm.md b/doc/features/rdmacm.md
deleted file mode 100644
index caacab40452..00000000000
--- a/doc/features/rdmacm.md
+++ /dev/null
@@ -1,17 +0,0 @@
-## Rdma Connection manager ##
-
-### What? ###
-Infiniband requires addresses of end points to be exchanged using an out-of-band channel (like tcp/ip). Glusterfs used a custom protocol over a tcp/ip channel to exchange this address. However, librdmacm provides the same functionality with the advantage of being a standard protocol. This helps if we want to communicate with a non-glusterfs entity (say nfs client with gluster nfs server) over infiniband.
-
-### Dependencies ###
-* [IP over Infiniband](http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-5.html) - The value to *option* **remote-host** in glusterfs transport configuration  should be an IPoIB address
-* [rdma cm kernel module](http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-4.html#ss4.4)
-* [user space rdmacm library - librdmacm](https://www.openfabrics.org/downloads/rdmacm)
-
-### Limitations ###
-* Because of bug [890502](https://bugzilla.redhat.com/show_bug.cgi?id=890502), we've to probe the peer on an IPoIB address. This imposes a restriction that all volumes created in the future have to communicate over IPoIB address (irrespective of whether they use gluster's tcp or rdma transport).
-
-* Currently client has independence to choose b/w tcp and rdma transports while communicating with the server (by creating volumes with **transport-type tcp,rdma**). This independence was a by-product of our ability to use the tcp/ip channel - transports with *option transport-type tcp* - for rdma connection establishment handshake too. However, with new requirement of IPoIB address for connection establishment, we loose this independence (till we bring in [multi-network support](https://bugzilla.redhat.com/show_bug.cgi?id=765437) - where a brick can be identified by a set of ip-addresses and we can choose different pairs of ip-addresses for communication based on our requirements - in glusterd).
-
-### External links ###
-* [Infiniband Howto](http://pkg-ofed.alioth.debian.org/howto/infiniband-howto.html)
diff --git a/doc/features/readdir-ahead.md b/doc/features/readdir-ahead.md
deleted file mode 100644
index 5302a021202..00000000000
--- a/doc/features/readdir-ahead.md
+++ /dev/null
@@ -1,14 +0,0 @@
-## Readdir-ahead ##
-
-### Summary ###
-Provide read-ahead support for directories to improve sequential directory read performance.
-
-### Owners ###
-Brian Foster
-
-### Detailed Description ###
-The read-ahead feature for directories is analogous to read-ahead for files. The objective is to detect sequential directory read operations and establish a pipeline for directory content. When a readdir request is received and fulfilled, preemptively issue subsequent readdir requests to the server in anticipation of those requests from the user. If sequential readdir requests are received, the directory content is already immediately available in the client. If subsequent requests are not sequential or not received, said data is simply dropped and the optimization is bypassed.
-
-readdir-ahead is currently disabled by default. It can be enabled with the following command:
-
-    gluster volume set <volname> readdir-ahead on
diff --git a/doc/features/rebalance.md b/doc/features/rebalance.md
deleted file mode 100644
index 29b993008d2..00000000000
--- a/doc/features/rebalance.md
+++ /dev/null
@@ -1,74 +0,0 @@
-## Background
-
-
-For a more detailed description, view Jeff Darcy's blog post [here] 
-(http://hekafs.org/index.php/2012/03/glusterfs-algorithms-distribution/) 
-
-GlusterFS uses the distribute translator (DHT) to aggregate space of multiple servers. DHT distributes files among its subvolumes using a consistent hashing method providing 32-bit hashes. Each DHT subvolume is given a range in the 32-bit hash space. A hash value is calculated for every file using a combination of its name. The file is then placed in the subvolume with the hash range that contains the hash value. 
-
-## What is rebalance?
-
-The rebalance process migrates files between the DHT subvolumes when necessary.
-
-## When is rebalance required?
-
-Rebalancing is required for two main cases.
-
-1. Addition/Removal of bricks
-
-2. Renaming of a file 
-
-## Addition/Removal of  bricks
-
-Whenever the number or order of DHT subvolumes change, the hash range given to each subvolume is recalculated. When this happens, already existing files on the volume will need to be moved to the correct subvolume based on their hash. Rebalance does this activity.
-
-Addition of bricks which increase the size of a volume will increase the number of DHT subvolumes and lead to recalculation of hash ranges (This doesn't happen when bricks are added to a volume to increase redundancy, i.e. increase replica count of a volume). This will require an explicit rebalance command to be issued to migrate the files.
-
-Removal of bricks which decrease the size of a volumes also causes the hash ranges of DHT to be recalculated. But we don't need to issue an explicit rebalance command in this case, as rebalance is done automatically by the remove-brick process if needed.
-
-## Renaming of a file
-
-Renaming of file will cause its hash to change. The file now needs to be moved to the correct subvolume based on its new hash. Rebalance does this.
-
-## How does rebalance work?
-
-At a high level, the rebalance process consists of the following 3 steps:
-
-1. Crawl the volume to access all files
-2. Calculate the hash for the file
-3. If needed move the migrate the file to the correct subvolume.
-
-
-The rebalance process has been optimized by making it distributed across the trusted storage pool. With distributed rebalance, a rebalance process is launched on each peer in the cluster. Each rebalance process will crawl files on only those bricks of the volume which are present on it, and migrate the files which need migration to the correct brick. This speeds up the rebalance process considerably.
-
-## What will happen if rebalance is not run?
-
-### Addition of bricks
-
-With the current implementation of add-brick, when the size of a volume is augmented by adding new bricks, the new bricks are not put into use immediately i.e., the hash ranges there not recalculated immediately. This means that the files will still be placed only onto the existing bricks, leaving the newly added storage space unused. Starting a rebalance process on the volume will cause the hash ranges to be recalculated with the new bricks included, which allows the newly added storage space to be used.
-
-### Renaming a file
-
-When a file rename causes the file to be hashed to a new subvolume, DHT writes a link file on the new subvolume leaving the actual file on the original subvolume. A link file is an empty file, which has an extended attribute set that points to the subvolume on which the actual file exists. So, when a client accesses the renamed file, DHT first looks for the file in the hashed subvolume and gets the link file. DHT understands the link file, and gets the actual file from the subvolume pointed to by the link file. This leads to a slight reduction in performance. A rebalance will move the actual file to the hashed subvolume, allowing clients to access the file directly once again.
-
-## Are clients affected during a rebalance process?
-
-The rebalance process is transparent to applications on the clients. Applications which have open files on the volume will not be affected by the rebalance process, even if the open file requires migration. The DHT translator on the client will hide the migration from the applications.
-
-##How are open files migrated?
-
-(A more technical description of the algorithm used can be seen in the commit message of commit a07bb18c8adeb8597f62095c5d1361c5bad01f09.)
-
-To achieve migration of open files, two things need to be assured of,
-a) any writes or changes happening to the file during migration are correctly synced to destination subvolume after the migration is complete.
-b) any further changes should be made to the destination subvolume
-
-Both of these requirements require sending notificatoins to clients. Clients are notified by overloading an attribute used in every callback functions. DHT understands these attributes in the callbacks and can be notified if a file is being migrated or not.
-
-During rebalance, a file will be in two phases
-
-1. Migration in process - In this phase the file is being migrated by the rebalance process from the source subvolume to the destination subvolume. The rebalance process will set a 'in-migration' attribute on the file, which will notify the clients' DHT translator. The clients' DHT translator will then take care to send any further changes to the destination subvolume as well. This way we satisfy the first requirement
-
-2. Migration completed - Once the file has been migrated, the rebalance process will set a 'migration-complete' attribute on the file. The clients will be notified of the completion and all further operations on the file will happen on the destination subvolume.
-
-The DHT translator handles the above and allows the applications on the clients to continue working on a file under migration.
diff --git a/doc/features/server-quorum.md b/doc/features/server-quorum.md
deleted file mode 100644
index 7b20084cea8..00000000000
--- a/doc/features/server-quorum.md
+++ /dev/null
@@ -1,44 +0,0 @@
-# Server Quorum
-
-Server quorum is a feature intended to reduce the occurrence of "split brain"
-after a brick failure or network partition.  Split brain happens when different
-sets of servers are allowed to process different sets of writes, leaving data
-in a state that can not be reconciled automatically.  The key to avoiding split
-brain is to ensure that there can be only one set of servers - a quorum - that
-can continue handling writes.  Server quorum does this by the brutal but
-effective means of forcing down all brick daemons on cluster nodes that can no
-longer reach enough of their peers to form a majority.  Because there can only
-be one majority, there can be only one set of bricks remaining, and thus split
-brain can not occur.
-
-## Options
-
-Server quorum is controlled by two parameters:
-
- * **cluster.server-quorum-type**
- 
-   This value may be "server" to indicate that server quorum is enabled, or
-   "none" to mean it's disabled.
-	
- * **cluster.server-quorum-ratio**
-
-   This is the percentage of cluster nodes that must be up to maintain quorum.
-   More precisely, this percentage of nodes *plus one* must be up.
-
-Note that these are cluster-wide flags.  All volumes served by the cluster will
-be affected.  Once these values are set, quorum actions - starting or stopping
-brick daemons in response to node or network events - will be automatic.
-
-## Best Practices
-
-If a cluster with an even number of nodes is split exactly down the middle,
-neither half can have quorum (which requires **more than** half of the total).
-This is particularly important when N=2, in which case the loss of either node
-leads to loss of quorum.  Therefore, it is highly advisable to ensure that the
-cluster size is three or greater.  The "extra" node in this case need not have
-any bricks or serve any data.  It need only be present to preserve the notion
-of a quorum majority less than the entire cluster membership, allowing the
-cluster to survive the loss of a single node without losing quorum.
-
-
-
diff --git a/doc/features/worm.md b/doc/features/worm.md
deleted file mode 100644
index dba99777da5..00000000000
--- a/doc/features/worm.md
+++ /dev/null
@@ -1,75 +0,0 @@
-#WORM (Write Once Read Many)
-This features enables you to create a `WORM volume` using gluster CLI.
-##Description
-WORM (write once,read many) is a desired feature for users who want to store data such as `log files` and where data is not allowed to get modified.
-
-GlusterFS provides a new key `features.worm` which takes boolean values(enable/disable) for volume set.
-
-Internally, the volume set command with 'feature.worm' key will add 'features/worm' translator in the brick's volume file.
-
-`This change would be reflected on a subsequent restart of the volume`, i.e gluster volume stop, followed by a gluster volume start.
-
-With a volume converted to WORM, the changes are as follows:
-
-* Reads are handled normally
-* Only files with O_APPEND flag will be supported.
-* Truncation,deletion wont be supported.
-
-##Volume Options
-Use the volume set command on a volume and see if the volume is actually turned into WORM type.
-
-    # features.worm enable
-##Fully loaded Example
-WORM feature is being supported from glusterfs version 3.4
-start glusterd by using the command
-
-    # service glusterd start
-Now create a volume by using the command
-
-    # gluster volume create <vol_name> <brick_path>
-start the volume created by running the command below.
-
-    # gluster vol start <vol_name>
-Run the command below to make sure that volume is created.
-
-    # gluster volume info
-Now turn on the WORM feature on the volume by using the command
-
-    # gluster vol set <vol_name> worm enable
-Verify that the option is set by using the command
-
-    # gluster volume info
-User should be able to see another option in the volume info
-
-    # features.worm: enable
-Now restart the volume for the changes to reflect, by performing volume stop and start.
-
-    # gluster volume <vol_name> stop
-    # gluster volume <vol_name> start
-Now mount the volume using fuse mount
-
-    # mount -t glusterfs <vol_name> <mnt_point>
-create a file inside the mount point by running the command below
-
-    # touch <file_name>
-Verify that user is able to create a file by running the command below
-
-    # ls <file_name>
-
-##How To Test
-Now try deleting the above file which is been created
-
-    # rm <file_name>
-Since WORM is enabled on the volume, it gives the following error message `rm: cannot remove '/<mnt_point>/<file_name>': Read-only file system`
-
-put some content into the file which is created above.
-
-    # echo "at the end of the file" >> <file_name>
-Now try editing the file by running the commnad below and verify that the following error message is displayed `rm: cannot remove '/<mnt_point>/<file_name>': Read-only file system`
-
-    # sed -i "1iAt the beginning of the file" <file_name>
-Now read the contents of the file and verify that file can be read.
-
-    cat <file_name>
-
-`Note: If WORM option is set on the volume before it is started, then volume need not be restarted for the changes to get reflected`.
diff --git a/doc/features/zerofill.md b/doc/features/zerofill.md
deleted file mode 100644
index c0f1fc5014c..00000000000
--- a/doc/features/zerofill.md
+++ /dev/null
@@ -1,26 +0,0 @@
-#zerofill API for GlusterFS
-zerofill() API would allow creation of pre-allocated and zeroed-out files on GlusterFS volumes by offloading the zeroing part to server and/or storage (storage offloads use SCSI WRITESAME).
-## Description
-
-Zerofill writes zeroes to a file in the specified range. This fop will be useful when a whole file needs to be initialized with zero (could be useful for zero filled VM disk image provisioning or during scrubbing of VM disk images).
-
-Client/application can issue this FOP for zeroing out. Gluster server will zero out required range of bytes ie server offloaded zeroing. In the absence of this fop, client/application has to repetitively issue write (zero) fop to the server, which is very inefficient method because of the overheads involved in RPC calls and acknowledgements.
-
-WRITESAME is a SCSI T10 command that takes a block of data as input and writes the same data to other blocks and this write is handled completely within the storage and hence is known as offload . Linux ,now has support for SCSI WRITESAME command which is exposed to the user in the form of BLKZEROOUT ioctl. BD Xlator can exploit BLKZEROOUT ioctl to implement this fop. Thus zeroing out operations can be completely offloaded to the storage device,
-making it highly efficient.
-
-The fop takes two arguments offset and size. It zeroes out 'size' number of bytes in an opened file starting from 'offset' position.
-This feature adds zerofill support to the following areas:
-> - libglusterfs
-- io-stats
-- performance/md-cache,open-behind
-- quota
-- cluster/afr,dht,stripe
-- rpc/xdr
-- protocol/client,server
-- io-threads
-- marker
-- storage/posix
-- libgfapi
-
-Client applications can exploit this fop by using glfs_zerofill introduced in libgfapi.FUSE support to this fop has not been added as there is no system call for this fop.
diff --git a/doc/gluster.8 b/doc/gluster.8
index d13cb6f6a17..ba595edca15 100644
--- a/doc/gluster.8
+++ b/doc/gluster.8
@@ -16,15 +16,14 @@ gluster - Gluster Console Manager (command line utility)
 .PP
 To run the program and display gluster prompt:
 .PP
-.B gluster
+.B gluster [--remote-host=<gluster_node>] [--mode=script] [--xml]
 .PP
 (or)
 .PP
 To specify a command directly:
 .PP
 .B gluster
-.I [commands] [options]
-
+.I  [commands] [options] [--remote-host=<gluster_node>] [--mode=script] [--xml]
 .SH DESCRIPTION
 The Gluster Console Manager is a command line utility for elastic volume management. You can run the gluster command on any export server. The command enables administrators to perform cloud operations, such as creating, expanding, shrinking, rebalancing, and migrating volumes without needing to schedule server downtime.
 .SH COMMANDS
@@ -36,7 +35,13 @@ The Gluster Console Manager is a command line utility for elastic volume managem
 \fB\ volume info [all|<VOLNAME>] \fR
 Display information about all volumes, or the specified volume.
 .TP
-\fB\ volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT>] [disperse [<COUNT>]] [redundancy <COUNT>] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK> ... \fR
+\fB\ volume list \fR
+List all volumes in cluster
+.TP
+\fB\ volume status [all | <VOLNAME> [nfs|shd|<BRICK>|quotad]] [detail|clients|mem|inode|fd|callpool|tasks|client-list] \fR
+Display status of all or specified volume(s)/brick
+.TP
+\fB\ volume create <NEW-VOLNAME> [stripe <COUNT>] [[replica <COUNT> [arbiter <COUNT>]]|[replica 2 thin-arbiter 1]] [disperse [<COUNT>]] [disperse-data <COUNT>] [redundancy <COUNT>] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK> ... <TA-BRICK> \fR
 Create a new volume of the specified type using the specified bricks and transport type (the default transport type is tcp).
 To create a volume with both transports (tcp and rdma), give 'transport tcp,rdma' as an option.
 .TP
@@ -52,8 +57,17 @@ Stop the specified volume.
 \fB\ volume set <VOLNAME> <OPTION> <PARAMETER> [<OPTION> <PARAMETER>] ... \fR
 Set the volume options.
 .TP
-\fB\ volume get <VOLNAME> <OPTION/all>\fR
-Get the volume options.
+\fB\ volume get <VOLNAME/all> <OPTION/all> \fR
+Get the value of the all options or given option for volume <VOLNAME> or all option. gluster volume get all all is to get all global options
+.TP
+\fB\ volume reset <VOLNAME> [option] [force] \fR
+Reset all the reconfigured options
+.TP
+\fB\ volume barrier <VOLNAME> {enable|disable} \fR
+Barrier/unbarrier file operations on a volume
+.TP
+\fB\ volume clear-locks <VOLNAME> <path> kind {blocked|granted|all}{inode [range]|entry [basename]|posix [range]} \fR
+Clear locks held on path
 .TP
 \fB\ volume help \fR
 Display help for the volume command.
@@ -71,8 +85,11 @@ If you remove the brick, the data stored in that brick will not be available. Yo
 .B replace-brick
 option.
 .TP
-\fB\ volume replace-brick <VOLNAME> (<BRICK> <NEW-BRICK>) start|pause|abort|status|commit \fR
-Replace the specified brick.
+\fB\ volume reset-brick <VOLNAME> <SOURCE-BRICK> {{start} | {<NEW-BRICK> commit}} \fR
+Brings down or replaces the specified source brick with the new brick.
+.TP
+\fB\ volume replace-brick <VOLNAME> <SOURCE-BRICK> <NEW-BRICK> commit force \fR
+Replace the specified source brick with a new brick.
 .TP
 \fB\ volume rebalance <VOLNAME> start \fR
 Start rebalancing the specified volume.
@@ -92,6 +109,18 @@ Locate the log file for corresponding volume/brick.
 .TP
 \fB\ volume log rotate <VOLNAME> [BRICK] \fB
 Rotate the log file for corresponding volume/brick.
+.TP
+\fB\ volume profile <VOLNAME> {start|info [peek|incremental [peek]|cumulative|clear]|stop} [nfs] \fR
+Profile operations on the volume. Once started, volume profile <volname> info provides cumulative statistics of the FOPs performed.
+.TP
+\fB\ volume top <VOLNAME> {open|read|write|opendir|readdir|clear} [nfs|brick <brick>] [list-cnt <value>] | {read-perf|write-perf} [bs <size> count <count>] [brick <brick>] [list-cnt <value>] \fR
+Generates a profile of a volume representing the performance and bottlenecks/hotspots of each brick.
+.TP
+\fB\ volume statedump <VOLNAME> [[nfs|quotad] [all|mem|iobuf|callpool|priv|fd|inode|history]... | [client <hostname:process-id>]] \fR
+Dumps the in memory state of the specified process or the bricks of the volume.
+.TP
+\fB\ volume sync <HOSTNAME> [all|<VOLNAME>] \fR
+Sync the volume information from a peer
 .SS "Peer Commands"
 .TP
 \fB\ peer probe <HOSTNAME> \fR
@@ -103,17 +132,121 @@ Detach the specified peer.
 \fB\ peer status \fR
 Display the status of peers.
 .TP
+\fB\ pool list \fR
+List all the nodes in the pool (including localhost)
+.TP
 \fB\ peer help \fR
 Display help for the peer command.
+.SS "Quota Commands"
+.TP
+\fB\ volume quota <VOLNAME> enable \fR
+Enable quota on the specified volume. This will cause all the directories in the filesystem hierarchy to be accounted and updated thereafter on each operation in the the filesystem. To kick start this accounting, a crawl is done over the hierarchy with an auxiliary client.
+.TP
+\fB\ volume quota <VOLNAME> disable \fR
+Disable quota on the volume. This will disable enforcement and accounting in the filesystem. Any configured limits will be lost.
+.TP
+\fB\ volume quota <VOLNAME> limit-usage <PATH> <SIZE> [<PERCENT>] \fR
+Set a usage  limit on the given path. Any previously set limit is overridden to the new value. The soft limit can optionally be specified (as a percentage of hard limit). If soft limit percentage is not provided the default soft limit value for the volume is used to decide the soft limit.
+.TP
+\fB\ volume quota <VOLNAME> limit-objects <PATH> <SIZE> [<PERCENT>] \fR
+Set an inode limit on the given path. Any previously set limit is overridden to the new value. The soft limit can optionally be specified (as a percentage of hard limit). If soft limit percentage is not provided the default soft limit value for the volume is used to decide the soft limit.
+.TP
+NOTE: valid units of SIZE are : B, KB, MB, GB, TB, PB. If no unit is specified, the unit defaults to bytes.
+.TP
+\fB\ volume quota <VOLNAME> remove <PATH> \fR
+Remove any usage limit configured on the specified directory. Note that if any limit is configured on the ancestors of this directory (previous directories along the path), they will still be honored and enforced.
+.TP
+\fB\ volume quota <VOLNAME> remove-objects <PATH> \fR
+Remove any inode limit configured on the specified directory. Note that if any limit is configured on the ancestors of this directory (previous directories along the path), they will still be honored and enforced.
+.TP
+\fB\ volume quota <VOLNAME> list <PATH> \fR
+Lists the  usage and limits configured on directory(s). If a path is given only the limit that has been configured on the directory(if any) is displayed along with the directory's usage. If no path is given, usage and limits are displayed for all directories that has limits configured.
+.TP
+\fB\ volume quota <VOLNAME> list-objects <PATH> \fR
+Lists the inode usage and inode limits configured on directory(s). If a path is given only the limit that has been configured on the directory(if any) is displayed along with the directory's inode usage. If no path is given, usage and limits are displayed for all directories that has limits configured.
+.TP
+\fB\ volume quota <VOLNAME> default-soft-limit <PERCENT> \fR
+Set the percentage value for default soft limit for the volume.
+.TP
+\fB\ volume quota <VOLNAME> soft-timeout <TIME> \fR
+Set the soft timeout for the volume. The interval in which limits are retested before the soft limit is breached.
+.TP
+\fB\ volume quota <VOLNAME> hard-timeout <TIME> \fR
+Set the hard timeout for the volume. The interval in which limits are retested after the soft limit is breached.
+.TP
+\fB\ volume quota <VOLNAME> alert-time <TIME> \fR
+Set the frequency in which warning messages need to be logged (in the brick logs) once soft limit is breached.
+.TP
+\fB\ volume inode-quota <VOLNAME> enable/disable \fR
+Enable/disable inode-quota for <VOLNAME>
+.TP
+\fB\ volume quota help \fR
+Display help for volume quota commands
+.TP
+NOTE: valid units of time and their symbols are : hours(h/hr), minutes(m/min), seconds(s/sec), weeks(w/wk), Days(d/days).
+.SS "Geo-replication Commands"
+.TP
+\fI\ Note\fR: password-less ssh, from the master node (where these commands are executed) to the slave node <SLAVE_HOST>, is a prerequisite for the geo-replication commands.
+.TP
+\fB\ system:: execute gsec_create\fR
+Generates pem keys which are required for push-pem
+.TP
+\fB\ volume geo-replication <MASTER_VOL> <SLAVE_HOST>::<SLAVE_VOL> create [[ssh-port n][[no-verify]|[push-pem]]] [force]\fR
+Create a new geo-replication session from <MASTER_VOL> to <SLAVE_HOST> host machine having <SLAVE_VOL>.
+Use ssh-port n if custom SSH port is configured in slave nodes.
+Use no-verify if the rsa-keys of nodes in master volume is distributed to slave nodes through an external agent.
+Use push-pem to push the keys automatically.
+.TP
+\fB\ volume geo-replication <MASTER_VOL> <SLAVE_HOST>::<SLAVE_VOL> {start|stop} [force] \fR
+Start/stop the geo-replication session from <MASTER_VOL> to <SLAVE_HOST> host machine having <SLAVE_VOL>.
+.TP
+\fB\ volume geo-replication [<MASTER_VOL> [<SLAVE_HOST>::<SLAVE_VOL>]] status [detail] \fR
+Query status of the geo-replication session from <MASTER_VOL> to <SLAVE_HOST> host machine having <SLAVE_VOL>.
+.TP
+\fB\ volume geo-replication <MASTER_VOL> <SLAVE_HOST>::<SLAVE_VOL> {pause|resume} [force] \fR
+Pause/resume the geo-replication session from <MASTER_VOL> to <SLAVE_HOST> host machine having <SLAVE_VOL>.
+.TP
+\fB\ volume geo-replication <MASTER_VOL> <SLAVE_HOST>::<SLAVE_VOL> delete [reset-sync-time]\fR
+Delete the geo-replication session from <MASTER_VOL> to <SLAVE_HOST> host machine having <SLAVE_VOL>.
+Optionally you can also reset the sync time in case you need to resync the entire volume on session recreate.
+.TP
+\fB\ volume geo-replication <MASTER_VOL> <SLAVE_HOST>::<SLAVE_VOL> config [[!]<options> [<value>]] \fR
+View (when no option provided) or set configuration for this geo-replication session.
+Use "!<OPTION>" to reset option <OPTION> to default value.
+.SS "Bitrot Commands"
+.TP
+\fB\ volume bitrot <VOLNAME> {enable|disable} \fR
+Enable/disable bitrot for volume <VOLNAME>
+.TP
+\fB\ volume bitrot <VOLNAME> signing-time <time-in-secs> \fR
+Waiting time for an object after last fd is closed to start signing process.
+.TP
+\fB\ volume bitrot <VOLNAME> signer-threads <count> \fR
+Number of signing process threads. Usually set to number of available cores.
+.TP
+\fB\ volume bitrot <VOLNAME> scrub-throttle {lazy|normal|aggressive} \fR
+Scrub-throttle value is a measure of how fast or slow the scrubber scrubs the filesystem for volume <VOLNAME>
+.TP
+\fB\ volume bitrot <VOLNAME> scrub-frequency {hourly|daily|weekly|biweekly|monthly} \fR
+Scrub frequency for volume <VOLNAME>
+.TP
+\fB\ volume bitrot <VOLNAME> scrub {pause|resume|status|ondemand} \fR
+Pause/Resume scrub. Upon resume, scrubber continues where it left off. status option shows the statistics of scrubber. ondemand option starts the scrubbing immediately if the scrubber is not paused or already running.
+.TP
+\fB\ volume bitrot help \fR
+Display help for volume bitrot commands
+.TP
 .SS "Snapshot Commands"
-.PP
 .TP
-\fB\ snapshot create <snapname> <volname> [description <description>] [force] \fR
-Creates a snapshot of a GlusterFS volume. User can provide a snap-name and a description to identify the snap. The description cannot be more than 1024 characters. To be able to take a snapshot, volume should be present and it should be in started state.
+\fB\ snapshot create <snapname> <volname> [no-timestamp] [description <description>] [force] \fR
+Creates a snapshot of a GlusterFS volume. User can provide a snap-name and a description to identify the snap. Snap will be created by appending timestamp in GMT. User can override this behaviour using "no-timestamp" option. The description cannot be more than 1024 characters. To be able to take a snapshot, volume should be present and it should be in started state.
 .TP
 \fB\ snapshot restore <snapname> \fR
 Restores an already taken snapshot of a GlusterFS volume. Snapshot restore is an offline activity therefore if the volume is online (in started state) then the restore operation will fail. Once the snapshot is restored it will not be available in the list of snapshots.
 .TP
+\fB\ snapshot clone <clonename> <snapname> \fR
+Create a clone of a snapshot volume, the resulting volume will be GlusterFS volume. User can provide a clone-name. To be able to take a clone, snapshot should be present and it should be in activated state.
+.TP
 \fB\ snapshot delete ( all | <snapname> | volume <volname> ) \fR
 If snapname is specified then mentioned snapshot is deleted. If volname is specified then all snapshots belonging to that particular volume is deleted. If keyword *all* is used then all snapshots belonging to the system is deleted.
 .TP
@@ -121,7 +254,7 @@ If snapname is specified then mentioned snapshot is deleted. If volname is speci
 Lists all snapshots taken. If volname is provided, then only the snapshots belonging to that particular volume is listed.
 .TP
 \fB\ snapshot info [snapname | (volume <volname>)] \fR
-This command gives information such as snapshot name, snapshot UUID, time at which snapshot was created, and it lists down the snap-volume-name, number of snapshots already taken and number of snapshots still available for that particular volume, and the state of the snapshot.
+This command gives information such as snapshot name, snapshot UUID, time at which snapshot was created, and it lists down the snap-volume-name, number of snapshots already taken and number of snapshots still available for that particular volume, and the state of the snapshot. If snapname is specified then info of the  mentioned  snapshot is  displayed.  If volname is specified then info of all snapshots belonging to that volume is displayed.  If  both  snapname and  volname  is  not specified then info of all the snapshots present in the system are displayed.
 .TP
 \fB\ snapshot status [snapname | (volume <volname>)] \fR
 This command gives status of the snapshot. The details included are snapshot brick path, volume group(LVM details), status of the snapshot bricks, PID of the bricks, data percentage filled for that particular volume group to which the snapshots belong to, and total size of the logical volume.
@@ -137,13 +270,16 @@ Snapshot config command along with keywords can be used to change the existing c
 
 snap-max-soft-limit and auto-delete are global options, that will be inherited by all volumes in the system and cannot be set to individual volumes.
 
-The system limit takes precedence over the volume specific limit.
+snap-max-hard-limit can be set globally, as well as per volume. The lowest limit between the global system limit and the volume specific limit, becomes the
+"Effective snap-max-hard-limit" for a volume.
+
+snap-max-soft-limit is a percentage value, which is applied on the "Effective snap-max-hard-limit" to get the "Effective snap-max-soft-limit".
 
-When auto-delete feature is enabled, then upon reaching the soft-limit, with every successful snapshot creation, the oldest snapshot will be deleted.
+When auto-delete feature is enabled, then upon reaching the "Effective snap-max-soft-limit", with every successful snapshot creation, the oldest snapshot will be deleted.
 
-When auto-delete feature is disabled, then upon reaching the soft-limit, the user gets a warning with every successful snapshot creation.
+When auto-delete feature is disabled, then upon reaching the "Effective snap-max-soft-limit", the user gets a warning with every successful snapshot creation.
 
-Upon reaching the hard-limit, further snapshot creations will not be allowed.
+When auto-delete feature is disabled, then upon reaching the "Effective snap-max-hard-limit", further  snapshot  creations  will not be allowed.
 
 activate-on-create is disabled by default. If you enable activate-on-create, then further snapshot will be activated during the time of snapshot creation.
 .TP
@@ -154,8 +290,58 @@ Note : By default the snapshot is activated during snapshot creation.
 .TP
 \fB\ snapshot deactivate <snapname> \fR
 Deactivates the mentioned snapshot.
+.TP
+\fB\ snapshot help \fR
+Display help for the snapshot commands.
+.SS "Self-heal Commands"
+.TP
+\fB\ volume heal <VOLNAME>\fR
+Triggers index self heal for the files that need healing.
+
+.TP
+\fB\ volume heal  <VOLNAME> [enable | disable]\fR
+Enable/disable self-heal-daemon for volume <VOLNAME>.
+
+.TP
+\fB\ volume heal <VOLNAME> full\fR
+Triggers self heal on all the files.
+
+.TP
+\fB\ volume heal <VOLNAME> info \fR
+Lists the files that need healing.
+
+.TP
+\fB\ volume heal <VOLNAME> info split-brain \fR
+Lists the files which are in split-brain state.
+
+.TP
+\fB\ volume heal <VOLNAME> statistics \fR
+Lists the crawl statistics.
+
+.TP
+\fB\ volume heal <VOLNAME> statistics heal-count \fR
+Displays the count of files to be healed.
+
+.TP
+\fB\ volume heal <VOLNAME> statistics heal-count replica <HOSTNAME:BRICKNAME> \fR
+Displays the number of files to be healed from a particular replica subvolume to which the brick <HOSTNAME:BRICKNAME> belongs.
+
+.TP
+\fB\ volume heal <VOLNAME> split-brain bigger-file <FILE> \fR
+Performs healing of <FILE> which is in split-brain by choosing the bigger file in the replica as source.
+
+.TP
+\fB\ volume heal <VOLNAME> split-brain source-brick <HOSTNAME:BRICKNAME> \fR
+Selects <HOSTNAME:BRICKNAME> as the source for all the files that are in split-brain in that replica and heals them.
+
+.TP
+\fB\ volume heal <VOLNAME> split-brain source-brick <HOSTNAME:BRICKNAME> <FILE> \fR
+Selects the split-brained <FILE> present in <HOSTNAME:BRICKNAME> as source and completes heal.
 .SS "Other Commands"
 .TP
+\fB\ get-state [<daemon>] [[odir </path/to/output/dir/>] [file <filename>]] [detail|volumeoptions] \fR
+Get local state representation of mentioned daemon and store data in provided path information
+.TP
 \fB\ help \fR
 Display the command options.
 .TP
diff --git a/doc/glusterd.8 b/doc/glusterd.8
index 04a43481eec..e3768c78761 100644
--- a/doc/glusterd.8
+++ b/doc/glusterd.8
@@ -30,6 +30,9 @@ File to use for logging.
 \fB\-L <LOGLEVEL>, \fB\-\-log\-level=<LOGLEVEL>\fR
 Logging severity.  Valid options are TRACE, DEBUG, INFO, WARNING, ERROR and CRITICAL (the default is INFO).
 .TP
+\fB\-\-localtime\-logging\fR
+Enable localtime log timestamps.
+.TP
 \fB\-\-debug\fR
 Run the program in debug mode. This option sets \fB\-\-no\-daemon\fR, \fB\-\-log\-level\fR to DEBUG
 and \fB\-\-log\-file\fR to console.
diff --git a/doc/glusterfs.8 b/doc/glusterfs.8
index fc28ef68be6..3d359ea85e4 100644
--- a/doc/glusterfs.8
+++ b/doc/glusterfs.8
@@ -53,6 +53,9 @@ Maximum number of connect attempts to server. This option should be provided wit
 \fB\-\-acl\fR
 Mount the filesystem with POSIX ACL support.
 .TP
+\fB\-\-localtime\-logging\fR
+Enable localtime log timestamps.
+.TP
 \fB\-\-debug\fR
 Run in debug mode.  This option sets \fB\-\-no\-daemon\fR, \fB\-\-log\-level\fR to DEBUG,
 and \fB\-\-log\-file\fR to console.
@@ -60,8 +63,8 @@ and \fB\-\-log\-file\fR to console.
 \fB\-\-enable\-ino32=BOOL\fR
 Use 32-bit inodes when mounting to workaround application that doesn't support 64-bit inodes.
 .TP
-\fB\-\-fopen\-keep\-cache\fR
-Do not purge the cache on file open.
+\fB\-\-fopen\-keep\-cache[=BOOL]\fR
+Do not purge the cache on file open (default: false).
 .TP
 \fB\-\-mac\-compat=BOOL\fR
 Provide stubs for attributes needed for seamless operation on Macs (the default is off).
@@ -98,11 +101,17 @@ Mount the filesystem in 'worm' mode.
 .TP
 \fB\-\-xlator\-option=VOLUME\-NAME.OPTION=VALUE\fR
 Add/Override a translator option for a volume with the specified value.
+.TP
+\fB\-\-subdir\-mount=SUBDIR\-MOUNT\-PATH\fR
+Mount subdirectory instead of the '/' of volume.
 
 .SS "Fuse options"
 .PP
 .TP
 
+\fB\-\-attr\-times\-granularity=NANOSECONDS\fR
+Declare supported granularity of file attribute times (default is 0 which kernel handles as unspecified; valid real values are between 1 and 1000000000).
+.TP
 \fB\-\-attribute\-timeout=SECONDS\fR
 Set attribute timeout to SECONDS for inodes in fuse kernel module (the default is 1).
 .TP
@@ -112,8 +121,8 @@ Set fuse module's background queue length to N (the default is 64).
 \fB\-\-congestion\-threshold=N\fR
 Set fuse module's congestion threshold to N (the default is 48).
 .TP
-\fB\-\-direct\-io\-mode=BOOL\fR
-Enable/Disable the direct-I/O mode in fuse module (the default is enable).
+\fB\-\-direct\-io\-mode=BOOL|auto\fR
+Specify fuse direct I/O strategy (the default is auto).
 .TP
 \fB\-\-dump-fuse=PATH\f\R
 Dump fuse traffic to PATH
@@ -124,9 +133,17 @@ Set entry timeout to SECONDS in fuse kernel module (the default is 1).
 \fB\-\-gid\-timeout=SECONDS\fR
 Set auxiliary group list timeout to SECONDS for fuse translator (the default is 0).
 .TP
+\fB\-\-kernel-writeback-cache=BOOL\fR
+Enable fuse in-kernel writeback cache.
+.TP
 \fB\-\-negative\-timeout=SECONDS\fR
 Set negative timeout to SECONDS in fuse kernel module (the default is 0).
 .TP
+\fB\-\-auto\-invalidation=BOOL\fR
+controls whether fuse-kernel can auto-invalidate attribute, dentry and
+page-cache. Disable this only if same files/directories are not
+accessed across two different mounts concurrently [default: on].
+.TP
 \fB\-\-volfile-check\fR
 Enable strict volume file checking.
 
diff --git a/doc/glusterfsd.8 b/doc/glusterfsd.8
index 176d042367c..bc1de2a8c80 100644
--- a/doc/glusterfsd.8
+++ b/doc/glusterfsd.8
@@ -51,6 +51,9 @@ Server to get the volume from.  This option overrides \fB\-\-volfile option
 .PP
 .TP
 
+\fB\-\-localtime\-logging\fR
+Enable localtime log timestamps.
+.TP
 \fB\-\-debug\fR
 Run in debug mode.  This option sets \fB\-\-no\-daemon\fR, \fB\-\-log\-level\fR to DEBUG
 and \fB\-\-log\-file\fR to console
@@ -101,6 +104,14 @@ Set entry timeout to SECONDS in fuse kernel module [default: 1]
 .TP
 \fB\-\-direct\-io\-mode=BOOL\fR
 Enable/Disable direct-io mode in fuse module [default: enable]
+.TP
+\fB\-\-resolve-gids\fR
+Resolve all auxiliary groups in fuse translator (max 32 otherwise)
+.TP
+\fB\-\-auto\-invalidation=BOOL\fR
+controls whether fuse-kernel can auto-invalidate attribute, dentry and
+page-cache. Disable this only if same files/directories are not
+accessed across two different mounts concurrently [default: on]
 
 .SS "Miscellaneous Options"
 .PP
diff --git a/doc/hacker-guide/en-US/markdown/coding-standard.md b/doc/hacker-guide/en-US/markdown/coding-standard.md
deleted file mode 100644
index 368c5553464..00000000000
--- a/doc/hacker-guide/en-US/markdown/coding-standard.md
+++ /dev/null
@@ -1,402 +0,0 @@
-GlusterFS Coding Standards
-==========================
-
-Structure definitions should have a comment per member
-------------------------------------------------------
-
-Every member in a structure definition must have a comment about its
-purpose. The comment should be descriptive without being overly verbose.
-
-*Bad:*
-
-```
-gf_lock_t   lock;           /* lock */
-```
-
-*Good:*
-
-```
-DBTYPE      access_mode;    /* access mode for accessing
-                             * the databases, can be
-                             * DB_HASH, DB_BTREE
-                             * (option access-mode <mode>)
-                             */
-```
-
-Declare all variables at the beginning of the function
-------------------------------------------------------
-
-All local variables in a function must be declared immediately after the
-opening brace. This makes it easy to keep track of memory that needs to be freed
-during exit. It also helps debugging, since gdb cannot handle variables
-declared inside loops or other such blocks.
-
-Always initialize local variables
----------------------------------
-
-Every local variable should be initialized to a sensible default value
-at the point of its declaration. All pointers should be initialized to NULL,
-and all integers should be zero or (if it makes sense) an error value.
-
-
-*Good:*
-
-```
-int ret       = 0;
-char *databuf = NULL;
-int _fd       = -1;
-```
-
-Initialization should always be done with a constant value
-----------------------------------------------------------
-
-Never use a non-constant expression as the initialization value for a variable.
-
-
-*Bad:*
-
-```
-pid_t pid     = frame->root->pid;
-char *databuf = malloc (1024);
-```
-
-Validate all arguments to a function
-------------------------------------
-
-All pointer arguments to a function must be checked for `NULL`.
-A macro named `VALIDATE` (in `common-utils.h`)
-takes one argument, and if it is `NULL`, writes a log message and
-jumps to a label called `err` after setting op_ret and op_errno
-appropriately. It is recommended to use this template.
-
-
-*Good:*
-
-```
-VALIDATE(frame);
-VALIDATE(this);
-VALIDATE(inode);
-```
-
-Never rely on precedence of operators
--------------------------------------
-
-Never write code that relies on the precedence of operators to execute
-correctly.  Such code can be hard to read and someone else might not
-know the precedence of operators as accurately as you do.
-
-*Bad:*
-
-```
-if (op_ret == -1 && errno != ENOENT)
-```
-
-*Good:*
-
-```
-if ((op_ret == -1) && (errno != ENOENT))
-```
-
-Use exactly matching types
---------------------------
-
-Use a variable of the exact type declared in the manual to hold the
-return value of a function. Do not use an ``equivalent'' type.
-
-
-*Bad:*
-
-```
-int len = strlen (path);
-```
-
-*Good:*
-
-```
-size_t len = strlen (path);
-```
-
-Never write code such as `foo->bar->baz`; check every pointer
--------------------------------------------------------------
-
-Do not write code that blindly follows a chain of pointer
-references. Any pointer in the chain may be `NULL` and thus
-cause a crash. Verify that each pointer is non-null before following
-it.
-
-Check return value of all functions and system calls
-----------------------------------------------------
-
-The return value of all system calls and API functions must be checked
-for success or failure.
-
-*Bad:*
-
-```
-close (fd);
-```
-
-*Good:*
-
-```
-op_ret = close (_fd);
-if (op_ret == -1) {
-        gf_log (this->name, GF_LOG_ERROR,
-                "close on file %s failed (%s)", real_path,
-                strerror (errno));
-        op_errno = errno;
-        goto out;
-}
-```
-
-
-Gracefully handle failure of malloc
------------------------------------
-
-GlusterFS should never crash or exit due to lack of memory. If a
-memory allocation fails, the call should be unwound and an error
-returned to the user.
-
-*Use result args and reserve the return value to indicate success or failure:*
-
-The return value of every functions must indicate success or failure (unless
-it is impossible for the function to fail --- e.g., boolean functions). If
-the function needs to return additional data, it must be returned using a
-result (pointer) argument.
-
-*Bad:*
-
-```
-int32_t dict_get_int32 (dict_t *this, char *key);
-```
-
-*Good:*
-
-```
-int dict_get_int32 (dict_t *this, char *key, int32_t *val);
-```
-
-Always use the `n' versions of string functions
------------------------------------------------
-
-Unless impossible, use the length-limited versions of the string functions.
-
-*Bad:*
-
-```
-strcpy (entry_path, real_path);
-```
-
-*Good:*
-
-```
-strncpy (entry_path, real_path, entry_path_len);
-```
-
-No dead or commented code
--------------------------
-
-There must be no dead code (code to which control can never be passed) or
-commented out code in the codebase.
-
-Only one unwind and return per function
----------------------------------------
-
-There must be only one exit out of a function. `UNWIND` and return
-should happen at only point in the function.
-
-Function length or Keep functions small
----------------------------------------
-
-We live in the UNIX-world where modules do one thing and do it well.
-This rule should apply to our functions also. If a function is very long, try splitting it
-into many little helper functions. The question is, in a coding
-spree, how do we know a function is long and unreadable. One rule of
-thumb given by Linus Torvalds is that, a function should be broken-up
-if you have 4 or more levels of indentation going on for more than 3-4
-lines.
-
-*Example for a helper function:*
-```
-static int
-same_owner (posix_lock_t *l1, posix_lock_t *l2)
-{
-        return ((l1->client_pid == l2->client_pid) &&
-               (l1->transport  == l2->transport));
-}
-```
-
-Defining functions as static
-----------------------------
-
-Define internal functions as static only if you're
-very sure that there will not be a crash(..of any kind..) emanating in
-that function. If there is even a remote possibility, perhaps due to
-pointer derefering, etc, declare the function as non-static. This
-ensures that when a crash does happen, the function name shows up the
-in the back-trace generated by libc. However, doing so has potential
-for polluting the function namespace, so to avoid conflicts with other
-components in other parts, ensure that the function names are
-prepended with a prefix that identify the component to which it
-belongs. For eg. non-static functions in io-threads translator start
-with iot_.
-
-Ensure function calls wrap around after 80-columns
---------------------------------------------------
-
-Place remaining arguments on the next line if needed.
-
-Functions arguments and function definition
--------------------------------------------
-
-Place all the arguments of a function definition on the same line
-until the line goes beyond 80-cols. Arguments that extend beyind
-80-cols should be placed on the next line.
-
-Style issues
-------------
-
-### Brace placement
-
-Use K&R/Linux style of brace placement for blocks.
-
-*Good:*
-
-```
-int some_function (...)
-{
-        if (...) {
-                /* ... */
-        } else if (...) {
-                /* ... */
-        } else {
-                /* ... */
-        }
-
-        do {
-                /* ... */
-        } while (cond);
-}
-```
-
-### Indentation
-
-Use *eight* spaces for indenting blocks. Ensure that your
-file contains only spaces and not tab characters. You can do this
-in Emacs by selecting the entire file (`C-x h`) and
-running `M-x untabify`.
-
-To make Emacs indent lines automatically by eight spaces, add this
-line to your `.emacs`:
-
-```
-(add-hook 'c-mode-hook (lambda () (c-set-style "linux")))
-```
-
-### Comments
-
-Write a comment before every function describing its purpose (one-line),
-its arguments, and its return value. Mention whether it is an internal
-function or an exported function.
-
-Write a comment before every structure describing its purpose, and
-write comments about each of its members.
-
-Follow the style shown below for comments, since such comments
-can then be automatically extracted by doxygen to generate
-documentation.
-
-*Good:*
-
-```
-/**
-* hash_name -hash function for filenames
-* @par:  parent inode number
-* @name: basename of inode
-* @mod:  number of buckets in the hashtable
-*
-* @return: success: bucket number
-*          failure: -1
-*
-* Not for external use.
-*/
-```
-
-### Indicating critical sections
-
-To clearly show regions of code which execute with locks held, use
-the following format:
-
-```
-pthread_mutex_lock (&mutex);
-{
-        /* code */
-}
-pthread_mutex_unlock (&mutex);
-```
-
-*A skeleton fop function:*
-
-This is the recommended template for any fop. In the beginning come
-the initializations. After that, the `success' control flow should be
-linear.  Any error conditions should cause a `goto` to a single
-point, `out`.  At that point, the code should detect the error
-that has occurred and do appropriate cleanup.
-
-```
-int32_t
-sample_fop (call_frame_t *frame, xlator_t *this, ...)
-{
-        char *            var1     = NULL;
-        int32_t           op_ret   = -1;
-        int32_t           op_errno = 0;
-        DIR *             dir      = NULL;
-        struct posix_fd * pfd      = NULL;
-
-        VALIDATE_OR_GOTO (frame, out);
-        VALIDATE_OR_GOTO (this, out);
-
-        /* other validations */
-
-        dir = opendir (...);
-
-        if (dir == NULL) {
-                op_errno = errno;
-                gf_log (this->name, GF_LOG_ERROR,
-                        "opendir failed on %s (%s)", loc->path,
-                        strerror (op_errno));
-                goto out;
-        }
-
-        /* another system call */
-        if (...) {
-                op_errno = ENOMEM;
-                gf_log (this->name, GF_LOG_ERROR,
-                        "out of memory :(");
-                goto out;
-        }
-
-        /* ... */
-
- out:
-        if (op_ret == -1) {
-
-          /* check for all the cleanup that needs to be
-             done */
-
-                if (dir) {
-                        closedir (dir);
-                        dir = NULL;
-                }
-
-                if (pfd) {
-                        FREE (pfd->path);
-                        FREE (pfd);
-                        pfd = NULL;
-                }
-        }
-
-        STACK_UNWIND (frame, op_ret, op_errno, fd);
-        return 0;
-}
-```
diff --git a/doc/legacy/Makefile.am b/doc/legacy/Makefile.am
deleted file mode 100644
index b2caabaa2f3..00000000000
--- a/doc/legacy/Makefile.am
+++ /dev/null
@@ -1,3 +0,0 @@
-info_TEXINFOS = user-guide.texi
-CLEANFILES = *~
-DISTCLEANFILES = .deps/*.P *.info *vti
diff --git a/doc/legacy/advanced-stripe.odg b/doc/legacy/advanced-stripe.odg
deleted file mode 100644
index 7686d7091b2..00000000000
--- a/doc/legacy/advanced-stripe.odg
+++ /dev/null
diff --git a/doc/legacy/advanced-stripe.pdf b/doc/legacy/advanced-stripe.pdf
deleted file mode 100644
index ec8b03dcfbb..00000000000
--- a/doc/legacy/advanced-stripe.pdf
+++ /dev/null
diff --git a/doc/legacy/booster.txt b/doc/legacy/booster.txt
deleted file mode 100644
index 051401a28fc..00000000000
--- a/doc/legacy/booster.txt
+++ /dev/null
@@ -1,54 +0,0 @@
-Introduction
-============
-* booster is a LD_PRELOADable library which boosts read/write performance by bypassing fuse for
-  read() and write() calls.
-
-Requirements
-============
-* fetch volfile from glusterfs.
-* identify whether multiple files are from the same mount point. If so, use only one context.
-
-Design
-======
-* for a getxattr, along with other attributes, fuse returns following attributes.
-  * contents of client volume-file.
-  * mount point.
-
-* LD_PRELOADed booster.so maintains an hash table storing mount-points and libglusterfsclient handles
-  so that handles are reused for files from same mount point.
-
-* it also maintains a fdtable. fdtable maps the fd (integer) returned to application to fd (pointer to fd struct)
-  used by libglusterfsclient. application is returned the same fd as the one returned from libc apis.
-
-* During fork, these tables are overwritten to enable creation of fresh glusterfs context in child.
-
-Working
-=======
-* application willing to use booster LD_PRELOADs booster.so which is a wrapper library implementing
-  open, read and write.
-
-* application should specify the path to logfile through the environment variable GLFS_BOOSTER_LOGFILE. If
-  not specified, logging is done to /dev/stderr.
-
-* open call does,
-  * real_open on the file.
-  * fgetxattr(fd).
-  * store the volume-file content got in the dictionary to a temporary file.
-  * look in the hashtable for the mount-point, if already present get the libglusterfsclient handle from the
-    hashtable. Otherwise get a new handle from libglusterfsclient (be careful about mount point not present in
-    the hashtable and multiple glusterfs_inits running simultaneously for the same mount-point there by using
-    multiple handles for the same mount point).
-  * real_close (fd).
-  * delete temporary volume-volfile.
-  * glusterfs_open (handle, path, mode).
-  * store the fd returned by glusterfs_open in the fdtable at the same index as the fd returned by real_open.
-  * return the index as fd.
-
-* read/write calls do,
-  * get the libglusterfsclient fd from fdtable.
-  * if found use glusterfs_read/glusterfs_write, else use real_read/real_write.
-
-* close call does,
-  * remove the fd from the fdtable.
-
-* other calls use real_calls.
diff --git a/doc/legacy/colonO-icon.jpg b/doc/legacy/colonO-icon.jpg
deleted file mode 100644
index 3e66f7a2775..00000000000
--- a/doc/legacy/colonO-icon.jpg
+++ /dev/null
diff --git a/doc/legacy/errno.list.bsd.txt b/doc/legacy/errno.list.bsd.txt
deleted file mode 100644
index 350af25e4ab..00000000000
--- a/doc/legacy/errno.list.bsd.txt
+++ /dev/null
@@ -1,376 +0,0 @@
-/*-
- * Copyright (c) 1982, 1986, 1989, 1993
- *	The Regents of the University of California.  All rights reserved.
- * (c) UNIX System Laboratories, Inc.
- * All or some portions of this file are derived from material licensed
- * to the University of California by American Telephone and Telegraph
- * Co. or Unix System Laboratories, Inc. and are reproduced herein with
- * the permission of UNIX System Laboratories, Inc.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- * 4. Neither the name of the University nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- *	@(#)errno.h	8.5 (Berkeley) 1/21/94
- * $FreeBSD: src/sys/sys/errno.h,v 1.28 2005/04/02 12:33:28 das Exp $
- */
-
-#ifndef _SYS_ERRNO_H_
-#define _SYS_ERRNO_H_
-
-#ifndef _KERNEL
-#include <sys/cdefs.h>
-__BEGIN_DECLS
-int *	__error(void);
-__END_DECLS
-#define	errno		(* __error())
-#endif
-
-#define	EPERM		1		/* Operation not permitted */
-#define	ENOENT		2		/* No such file or directory */
-#define	ESRCH		3		/* No such process */
-#define	EINTR		4		/* Interrupted system call */
-#define	EIO		5		/* Input/output error */
-#define	ENXIO		6		/* Device not configured */
-#define	E2BIG		7		/* Argument list too long */
-#define	ENOEXEC		8		/* Exec format error */
-#define	EBADF		9		/* Bad file descriptor */
-#define	ECHILD		10		/* No child processes */
-#define	EDEADLK		11		/* Resource deadlock avoided */
-					/* 11 was EAGAIN */
-#define	ENOMEM		12		/* Cannot allocate memory */
-#define	EACCES		13		/* Permission denied */
-#define	EFAULT		14		/* Bad address */
-#ifndef _POSIX_SOURCE
-#define	ENOTBLK		15		/* Block device required */
-#endif
-#define	EBUSY		16		/* Device busy */
-#define	EEXIST		17		/* File exists */
-#define	EXDEV		18		/* Cross-device link */
-#define	ENODEV		19		/* Operation not supported by device */
-#define	ENOTDIR		20		/* Not a directory */
-#define	EISDIR		21		/* Is a directory */
-#define	EINVAL		22		/* Invalid argument */
-#define	ENFILE		23		/* Too many open files in system */
-#define	EMFILE		24		/* Too many open files */
-#define	ENOTTY		25		/* Inappropriate ioctl for device */
-#ifndef _POSIX_SOURCE
-#define	ETXTBSY		26		/* Text file busy */
-#endif
-#define	EFBIG		27		/* File too large */
-#define	ENOSPC		28		/* No space left on device */
-#define	ESPIPE		29		/* Illegal seek */
-#define	EROFS		30		/* Read-only filesystem */
-#define	EMLINK		31		/* Too many links */
-#define	EPIPE		32		/* Broken pipe */
-
-/* math software */
-#define	EDOM		33		/* Numerical argument out of domain */
-#define	ERANGE		34		/* Result too large */
-
-/* non-blocking and interrupt i/o */
-#define	EAGAIN		35		/* Resource temporarily unavailable */
-#ifndef _POSIX_SOURCE
-#define	EWOULDBLOCK	EAGAIN		/* Operation would block */
-#define	EINPROGRESS	36		/* Operation now in progress */
-#define	EALREADY	37		/* Operation already in progress */
-
-/* ipc/network software -- argument errors */
-#define	ENOTSOCK	38		/* Socket operation on non-socket */
-#define	EDESTADDRREQ	39		/* Destination address required */
-#define	EMSGSIZE	40		/* Message too long */
-#define	EPROTOTYPE	41		/* Protocol wrong type for socket */
-#define	ENOPROTOOPT	42		/* Protocol not available */
-#define	EPROTONOSUPPORT	43		/* Protocol not supported */
-#define	ESOCKTNOSUPPORT	44		/* Socket type not supported */
-#define	EOPNOTSUPP	45		/* Operation not supported */
-#define	ENOTSUP		EOPNOTSUPP	/* Operation not supported */
-#define	EPFNOSUPPORT	46		/* Protocol family not supported */
-#define	EAFNOSUPPORT	47		/* Address family not supported by protocol family */
-#define	EADDRINUSE	48		/* Address already in use */
-#define	EADDRNOTAVAIL	49		/* Can't assign requested address */
-
-/* ipc/network software -- operational errors */
-#define	ENETDOWN	50		/* Network is down */
-#define	ENETUNREACH	51		/* Network is unreachable */
-#define	ENETRESET	52		/* Network dropped connection on reset */
-#define	ECONNABORTED	53		/* Software caused connection abort */
-#define	ECONNRESET	54		/* Connection reset by peer */
-#define	ENOBUFS		55		/* No buffer space available */
-#define	EISCONN		56		/* Socket is already connected */
-#define	ENOTCONN	57		/* Socket is not connected */
-#define	ESHUTDOWN	58		/* Can't send after socket shutdown */
-#define	ETOOMANYREFS	59		/* Too many references: can't splice */
-#define	ETIMEDOUT	60		/* Operation timed out */
-#define	ECONNREFUSED	61		/* Connection refused */
-
-#define	ELOOP		62		/* Too many levels of symbolic links */
-#endif /* _POSIX_SOURCE */
-#define	ENAMETOOLONG	63		/* File name too long */
-
-/* should be rearranged */
-#ifndef _POSIX_SOURCE
-#define	EHOSTDOWN	64		/* Host is down */
-#define	EHOSTUNREACH	65		/* No route to host */
-#endif /* _POSIX_SOURCE */
-#define	ENOTEMPTY	66		/* Directory not empty */
-
-/* quotas & mush */
-#ifndef _POSIX_SOURCE
-#define	EPROCLIM	67		/* Too many processes */
-#define	EUSERS		68		/* Too many users */
-#define	EDQUOT		69		/* Disc quota exceeded */
-
-/* Network File System */
-#define	ESTALE		70		/* Stale NFS file handle */
-#define	EREMOTE		71		/* Too many levels of remote in path */
-#define	EBADRPC		72		/* RPC struct is bad */
-#define	ERPCMISMATCH	73		/* RPC version wrong */
-#define	EPROGUNAVAIL	74		/* RPC prog. not avail */
-#define	EPROGMISMATCH	75		/* Program version wrong */
-#define	EPROCUNAVAIL	76		/* Bad procedure for program */
-#endif /* _POSIX_SOURCE */
-
-#define	ENOLCK		77		/* No locks available */
-#define	ENOSYS		78		/* Function not implemented */
-
-#ifndef _POSIX_SOURCE
-#define	EFTYPE		79		/* Inappropriate file type or format */
-#define	EAUTH		80		/* Authentication error */
-#define	ENEEDAUTH	81		/* Need authenticator */
-#define	EIDRM		82		/* Identifier removed */
-#define	ENOMSG		83		/* No message of desired type */
-#define	EOVERFLOW	84		/* Value too large to be stored in data type */
-#define	ECANCELED	85		/* Operation canceled */
-#define	EILSEQ		86		/* Illegal byte sequence */
-#define	ENOATTR		87		/* Attribute not found */
-
-#define EDOOFUS		88		/* Programming error */
-#endif /* _POSIX_SOURCE */
-
-#define	EBADMSG		89		/* Bad message */
-#define	EMULTIHOP	90		/* Multihop attempted */
-#define	ENOLINK		91		/* Link has been severed */
-#define	EPROTO		92		/* Protocol error */
-
-#ifndef _POSIX_SOURCE
-#define	ELAST		92		/* Must be equal largest errno */
-#endif /* _POSIX_SOURCE */
-
-#ifdef _KERNEL
-/* pseudo-errors returned inside kernel to modify return to process */
-#define	ERESTART	(-1)		/* restart syscall */
-#define	EJUSTRETURN	(-2)		/* don't modify regs, just return */
-#define	ENOIOCTL	(-3)		/* ioctl not handled by this layer */
-#define	EDIRIOCTL	(-4)		/* do direct ioctl in GEOM */
-#endif
-
-#endif
-/*-
- * Copyright (c) 1982, 1986, 1989, 1993
- *	The Regents of the University of California.  All rights reserved.
- * (c) UNIX System Laboratories, Inc.
- * All or some portions of this file are derived from material licensed
- * to the University of California by American Telephone and Telegraph
- * Co. or Unix System Laboratories, Inc. and are reproduced herein with
- * the permission of UNIX System Laboratories, Inc.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- * 4. Neither the name of the University nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- *	@(#)errno.h	8.5 (Berkeley) 1/21/94
- * $FreeBSD: src/sys/sys/errno.h,v 1.28 2005/04/02 12:33:28 das Exp $
- */
-
-#ifndef _SYS_ERRNO_H_
-#define _SYS_ERRNO_H_
-
-#ifndef _KERNEL
-#include <sys/cdefs.h>
-__BEGIN_DECLS
-int *	__error(void);
-__END_DECLS
-#define	errno		(* __error())
-#endif
-
-#define	EPERM		1		/* Operation not permitted */
-#define	ENOENT		2		/* No such file or directory */
-#define	ESRCH		3		/* No such process */
-#define	EINTR		4		/* Interrupted system call */
-#define	EIO		5		/* Input/output error */
-#define	ENXIO		6		/* Device not configured */
-#define	E2BIG		7		/* Argument list too long */
-#define	ENOEXEC		8		/* Exec format error */
-#define	EBADF		9		/* Bad file descriptor */
-#define	ECHILD		10		/* No child processes */
-#define	EDEADLK		11		/* Resource deadlock avoided */
-					/* 11 was EAGAIN */
-#define	ENOMEM		12		/* Cannot allocate memory */
-#define	EACCES		13		/* Permission denied */
-#define	EFAULT		14		/* Bad address */
-#ifndef _POSIX_SOURCE
-#define	ENOTBLK		15		/* Block device required */
-#endif
-#define	EBUSY		16		/* Device busy */
-#define	EEXIST		17		/* File exists */
-#define	EXDEV		18		/* Cross-device link */
-#define	ENODEV		19		/* Operation not supported by device */
-#define	ENOTDIR		20		/* Not a directory */
-#define	EISDIR		21		/* Is a directory */
-#define	EINVAL		22		/* Invalid argument */
-#define	ENFILE		23		/* Too many open files in system */
-#define	EMFILE		24		/* Too many open files */
-#define	ENOTTY		25		/* Inappropriate ioctl for device */
-#ifndef _POSIX_SOURCE
-#define	ETXTBSY		26		/* Text file busy */
-#endif
-#define	EFBIG		27		/* File too large */
-#define	ENOSPC		28		/* No space left on device */
-#define	ESPIPE		29		/* Illegal seek */
-#define	EROFS		30		/* Read-only filesystem */
-#define	EMLINK		31		/* Too many links */
-#define	EPIPE		32		/* Broken pipe */
-
-/* math software */
-#define	EDOM		33		/* Numerical argument out of domain */
-#define	ERANGE		34		/* Result too large */
-
-/* non-blocking and interrupt i/o */
-#define	EAGAIN		35		/* Resource temporarily unavailable */
-#ifndef _POSIX_SOURCE
-#define	EWOULDBLOCK	EAGAIN		/* Operation would block */
-#define	EINPROGRESS	36		/* Operation now in progress */
-#define	EALREADY	37		/* Operation already in progress */
-
-/* ipc/network software -- argument errors */
-#define	ENOTSOCK	38		/* Socket operation on non-socket */
-#define	EDESTADDRREQ	39		/* Destination address required */
-#define	EMSGSIZE	40		/* Message too long */
-#define	EPROTOTYPE	41		/* Protocol wrong type for socket */
-#define	ENOPROTOOPT	42		/* Protocol not available */
-#define	EPROTONOSUPPORT	43		/* Protocol not supported */
-#define	ESOCKTNOSUPPORT	44		/* Socket type not supported */
-#define	EOPNOTSUPP	45		/* Operation not supported */
-#define	ENOTSUP		EOPNOTSUPP	/* Operation not supported */
-#define	EPFNOSUPPORT	46		/* Protocol family not supported */
-#define	EAFNOSUPPORT	47		/* Address family not supported by protocol family */
-#define	EADDRINUSE	48		/* Address already in use */
-#define	EADDRNOTAVAIL	49		/* Can't assign requested address */
-
-/* ipc/network software -- operational errors */
-#define	ENETDOWN	50		/* Network is down */
-#define	ENETUNREACH	51		/* Network is unreachable */
-#define	ENETRESET	52		/* Network dropped connection on reset */
-#define	ECONNABORTED	53		/* Software caused connection abort */
-#define	ECONNRESET	54		/* Connection reset by peer */
-#define	ENOBUFS		55		/* No buffer space available */
-#define	EISCONN		56		/* Socket is already connected */
-#define	ENOTCONN	57		/* Socket is not connected */
-#define	ESHUTDOWN	58		/* Can't send after socket shutdown */
-#define	ETOOMANYREFS	59		/* Too many references: can't splice */
-#define	ETIMEDOUT	60		/* Operation timed out */
-#define	ECONNREFUSED	61		/* Connection refused */
-
-#define	ELOOP		62		/* Too many levels of symbolic links */
-#endif /* _POSIX_SOURCE */
-#define	ENAMETOOLONG	63		/* File name too long */
-
-/* should be rearranged */
-#ifndef _POSIX_SOURCE
-#define	EHOSTDOWN	64		/* Host is down */
-#define	EHOSTUNREACH	65		/* No route to host */
-#endif /* _POSIX_SOURCE */
-#define	ENOTEMPTY	66		/* Directory not empty */
-
-/* quotas & mush */
-#ifndef _POSIX_SOURCE
-#define	EPROCLIM	67		/* Too many processes */
-#define	EUSERS		68		/* Too many users */
-#define	EDQUOT		69		/* Disc quota exceeded */
-
-/* Network File System */
-#define	ESTALE		70		/* Stale NFS file handle */
-#define	EREMOTE		71		/* Too many levels of remote in path */
-#define	EBADRPC		72		/* RPC struct is bad */
-#define	ERPCMISMATCH	73		/* RPC version wrong */
-#define	EPROGUNAVAIL	74		/* RPC prog. not avail */
-#define	EPROGMISMATCH	75		/* Program version wrong */
-#define	EPROCUNAVAIL	76		/* Bad procedure for program */
-#endif /* _POSIX_SOURCE */
-
-#define	ENOLCK		77		/* No locks available */
-#define	ENOSYS		78		/* Function not implemented */
-
-#ifndef _POSIX_SOURCE
-#define	EFTYPE		79		/* Inappropriate file type or format */
-#define	EAUTH		80		/* Authentication error */
-#define	ENEEDAUTH	81		/* Need authenticator */
-#define	EIDRM		82		/* Identifier removed */
-#define	ENOMSG		83		/* No message of desired type */
-#define	EOVERFLOW	84		/* Value too large to be stored in data type */
-#define	ECANCELED	85		/* Operation canceled */
-#define	EILSEQ		86		/* Illegal byte sequence */
-#define	ENOATTR		87		/* Attribute not found */
-
-#define EDOOFUS		88		/* Programming error */
-#endif /* _POSIX_SOURCE */
-
-#define	EBADMSG		89		/* Bad message */
-#define	EMULTIHOP	90		/* Multihop attempted */
-#define	ENOLINK		91		/* Link has been severed */
-#define	EPROTO		92		/* Protocol error */
-
-#ifndef _POSIX_SOURCE
-#define	ELAST		92		/* Must be equal largest errno */
-#endif /* _POSIX_SOURCE */
-
-#ifdef _KERNEL
-/* pseudo-errors returned inside kernel to modify return to process */
-#define	ERESTART	(-1)		/* restart syscall */
-#define	EJUSTRETURN	(-2)		/* don't modify regs, just return */
-#define	ENOIOCTL	(-3)		/* ioctl not handled by this layer */
-#define	EDIRIOCTL	(-4)		/* do direct ioctl in GEOM */
-#endif
-
-#endif
diff --git a/doc/legacy/errno.list.linux.txt b/doc/legacy/errno.list.linux.txt
deleted file mode 100644
index 3f3b18c46ec..00000000000
--- a/doc/legacy/errno.list.linux.txt
+++ /dev/null
@@ -1,1586 +0,0 @@
-#define ICONV_SUPPORTS_ERRNO 1
-#include <errno.h>
-/* Error constants.  Linux specific version.
-   Copyright (C) 1996, 1997, 1998, 1999, 2005 Free Software Foundation, Inc.
-   This file is part of the GNU C Library.
-
-   The GNU C Library is free software; you can redistribute it and/or
-   modify it under the terms of the GNU Lesser General Public
-   License as published by the Free Software Foundation; either
-   version 2.1 of the License, or (at your option) any later version.
-
-   The GNU C Library is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-   Lesser General Public License for more details.
-
-   You should have received a copy of the GNU Lesser General Public
-   License along with the GNU C Library; if not, write to the Free
-   Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
-   02111-1307 USA.  */
-
-#ifdef _ERRNO_H
-
-# undef EDOM
-# undef EILSEQ
-# undef ERANGE
-# include <linux/errno.h>
-
-/* Linux has no ENOTSUP error code.  */
-# define ENOTSUP EOPNOTSUPP
-
-/* Older Linux versions also had no ECANCELED error code.  */
-# ifndef ECANCELED
-#  define ECANCELED	125
-# endif
-
-/* Support for error codes to support robust mutexes was added later, too.  */
-# ifndef EOWNERDEAD
-#  define EOWNERDEAD		130
-#  define ENOTRECOVERABLE	131
-# endif
-
-# ifndef __ASSEMBLER__
-/* Function to get address of global `errno' variable.  */
-extern int *__errno_location (void) __THROW __attribute__ ((__const__));
-
-#  if !defined _LIBC || defined _LIBC_REENTRANT
-/* When using threads, errno is a per-thread value.  */
-#   define errno (*__errno_location ())
-#  endif
-# endif /* !__ASSEMBLER__ */
-#endif /* _ERRNO_H */
-
-#if !defined _ERRNO_H && defined __need_Emath
-/* This is ugly but the kernel header is not clean enough.  We must
-   define only the values EDOM, EILSEQ and ERANGE in case __need_Emath is
-   defined.  */
-# define EDOM	33	/* Math argument out of domain of function.  */
-# define EILSEQ	84	/* Illegal byte sequence.  */
-# define ERANGE	34	/* Math result not representable.  */
-#endif /* !_ERRNO_H && __need_Emath */
-/* Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements.  See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-#ifndef APR_ERRNO_H
-#define APR_ERRNO_H
-
-/**
- * @file apr_errno.h
- * @brief APR Error Codes
- */
-
-#include "apr.h"
-
-#if APR_HAVE_ERRNO_H
-#include <errno.h>
-#endif
-
-#ifdef __cplusplus
-extern "C" {
-#endif /* __cplusplus */
-
-/**
- * @defgroup apr_errno Error Codes
- * @ingroup APR
- * @{
- */
-
-/**
- * Type for specifying an error or status code.
- */
-typedef int apr_status_t;
-
-/**
- * Return a human readable string describing the specified error.
- * @param statcode The error code the get a string for.
- * @param buf A buffer to hold the error string.
- * @param bufsize Size of the buffer to hold the string.
- */
-APR_DECLARE(char *) apr_strerror(apr_status_t statcode, char *buf,
-                                 apr_size_t bufsize);
-
-#if defined(DOXYGEN)
-/**
- * @def APR_FROM_OS_ERROR(os_err_type syserr)
- * Fold a platform specific error into an apr_status_t code.
- * @return apr_status_t
- * @param e The platform os error code.
- * @warning  macro implementation; the syserr argument may be evaluated
- *      multiple times.
- */
-#define APR_FROM_OS_ERROR(e) (e == 0 ? APR_SUCCESS : e + APR_OS_START_SYSERR)
-
-/**
- * @def APR_TO_OS_ERROR(apr_status_t statcode)
- * @return os_err_type
- * Fold an apr_status_t code back to the native platform defined error.
- * @param e The apr_status_t folded platform os error code.
- * @warning  macro implementation; the statcode argument may be evaluated
- *      multiple times.  If the statcode was not created by apr_get_os_error
- *      or APR_FROM_OS_ERROR, the results are undefined.
- */
-#define APR_TO_OS_ERROR(e) (e == 0 ? APR_SUCCESS : e - APR_OS_START_SYSERR)
-
-/** @def apr_get_os_error()
- * @return apr_status_t the last platform error, folded into apr_status_t, on most platforms
- * @remark This retrieves errno, or calls a GetLastError() style function, and
- *      folds it with APR_FROM_OS_ERROR.  Some platforms (such as OS2) have no
- *      such mechanism, so this call may be unsupported.  Do NOT use this
- *      call for socket errors from socket, send, recv etc!
- */
-
-/** @def apr_set_os_error(e)
- * Reset the last platform error, unfolded from an apr_status_t, on some platforms
- * @param e The OS error folded in a prior call to APR_FROM_OS_ERROR()
- * @warning This is a macro implementation; the statcode argument may be evaluated
- *      multiple times.  If the statcode was not created by apr_get_os_error
- *      or APR_FROM_OS_ERROR, the results are undefined.  This macro sets
- *      errno, or calls a SetLastError() style function, unfolding statcode
- *      with APR_TO_OS_ERROR.  Some platforms (such as OS2) have no such
- *      mechanism, so this call may be unsupported.
- */
-
-/** @def apr_get_netos_error()
- * Return the last socket error, folded into apr_status_t, on all platforms
- * @remark This retrieves errno or calls a GetLastSocketError() style function,
- *      and folds it with APR_FROM_OS_ERROR.
- */
-
-/** @def apr_set_netos_error(e)
- * Reset the last socket error, unfolded from an apr_status_t
- * @param e The socket error folded in a prior call to APR_FROM_OS_ERROR()
- * @warning This is a macro implementation; the statcode argument may be evaluated
- *      multiple times.  If the statcode was not created by apr_get_os_error
- *      or APR_FROM_OS_ERROR, the results are undefined.  This macro sets
- *      errno, or calls a WSASetLastError() style function, unfolding
- *      socketcode with APR_TO_OS_ERROR.
- */
-
-#endif /* defined(DOXYGEN) */
-
-/**
- * APR_OS_START_ERROR is where the APR specific error values start.
- */
-#define APR_OS_START_ERROR     20000
-/**
- * APR_OS_ERRSPACE_SIZE is the maximum number of errors you can fit
- *    into one of the error/status ranges below -- except for
- *    APR_OS_START_USERERR, which see.
- */
-#define APR_OS_ERRSPACE_SIZE 50000
-/**
- * APR_OS_START_STATUS is where the APR specific status codes start.
- */
-#define APR_OS_START_STATUS    (APR_OS_START_ERROR + APR_OS_ERRSPACE_SIZE)
-/**
- * APR_OS_START_USERERR are reserved for applications that use APR that
- *     layer their own error codes along with APR's.  Note that the
- *     error immediately following this one is set ten times farther
- *     away than usual, so that users of apr have a lot of room in
- *     which to declare custom error codes.
- */
-#define APR_OS_START_USERERR    (APR_OS_START_STATUS + APR_OS_ERRSPACE_SIZE)
-/**
- * APR_OS_START_USEERR is obsolete, defined for compatibility only.
- * Use APR_OS_START_USERERR instead.
- */
-#define APR_OS_START_USEERR     APR_OS_START_USERERR
-/**
- * APR_OS_START_CANONERR is where APR versions of errno values are defined
- *     on systems which don't have the corresponding errno.
- */
-#define APR_OS_START_CANONERR  (APR_OS_START_USERERR \
-                                 + (APR_OS_ERRSPACE_SIZE * 10))
-/**
- * APR_OS_START_EAIERR folds EAI_ error codes from getaddrinfo() into
- *     apr_status_t values.
- */
-#define APR_OS_START_EAIERR    (APR_OS_START_CANONERR + APR_OS_ERRSPACE_SIZE)
-/**
- * APR_OS_START_SYSERR folds platform-specific system error values into
- *     apr_status_t values.
- */
-#define APR_OS_START_SYSERR    (APR_OS_START_EAIERR + APR_OS_ERRSPACE_SIZE)
-
-/** no error. */
-#define APR_SUCCESS 0
-
-/**
- * @defgroup APR_Error APR Error Values
- * <PRE>
- * <b>APR ERROR VALUES</b>
- * APR_ENOSTAT      APR was unable to perform a stat on the file
- * APR_ENOPOOL      APR was not provided a pool with which to allocate memory
- * APR_EBADDATE     APR was given an invalid date
- * APR_EINVALSOCK   APR was given an invalid socket
- * APR_ENOPROC      APR was not given a process structure
- * APR_ENOTIME      APR was not given a time structure
- * APR_ENODIR       APR was not given a directory structure
- * APR_ENOLOCK      APR was not given a lock structure
- * APR_ENOPOLL      APR was not given a poll structure
- * APR_ENOSOCKET    APR was not given a socket
- * APR_ENOTHREAD    APR was not given a thread structure
- * APR_ENOTHDKEY    APR was not given a thread key structure
- * APR_ENOSHMAVAIL  There is no more shared memory available
- * APR_EDSOOPEN     APR was unable to open the dso object.  For more
- *                  information call apr_dso_error().
- * APR_EGENERAL     General failure (specific information not available)
- * APR_EBADIP       The specified IP address is invalid
- * APR_EBADMASK     The specified netmask is invalid
- * APR_ESYMNOTFOUND Could not find the requested symbol
- * </PRE>
- *
- * <PRE>
- * <b>APR STATUS VALUES</b>
- * APR_INCHILD        Program is currently executing in the child
- * APR_INPARENT       Program is currently executing in the parent
- * APR_DETACH         The thread is detached
- * APR_NOTDETACH      The thread is not detached
- * APR_CHILD_DONE     The child has finished executing
- * APR_CHILD_NOTDONE  The child has not finished executing
- * APR_TIMEUP         The operation did not finish before the timeout
- * APR_INCOMPLETE     The operation was incomplete although some processing
- *                    was performed and the results are partially valid
- * APR_BADCH          Getopt found an option not in the option string
- * APR_BADARG         Getopt found an option that is missing an argument
- *                    and an argument was specified in the option string
- * APR_EOF            APR has encountered the end of the file
- * APR_NOTFOUND       APR was unable to find the socket in the poll structure
- * APR_ANONYMOUS      APR is using anonymous shared memory
- * APR_FILEBASED      APR is using a file name as the key to the shared memory
- * APR_KEYBASED       APR is using a shared key as the key to the shared memory
- * APR_EINIT          Ininitalizer value.  If no option has been found, but
- *                    the status variable requires a value, this should be used
- * APR_ENOTIMPL       The APR function has not been implemented on this
- *                    platform, either because nobody has gotten to it yet,
- *                    or the function is impossible on this platform.
- * APR_EMISMATCH      Two passwords do not match.
- * APR_EABSOLUTE      The given path was absolute.
- * APR_ERELATIVE      The given path was relative.
- * APR_EINCOMPLETE    The given path was neither relative nor absolute.
- * APR_EABOVEROOT     The given path was above the root path.
- * APR_EBUSY          The given lock was busy.
- * APR_EPROC_UNKNOWN  The given process wasn't recognized by APR
- * </PRE>
- * @{
- */
-/** @see APR_STATUS_IS_ENOSTAT */
-#define APR_ENOSTAT        (APR_OS_START_ERROR + 1)
-/** @see APR_STATUS_IS_ENOPOOL */
-#define APR_ENOPOOL        (APR_OS_START_ERROR + 2)
-/* empty slot: +3 */
-/** @see APR_STATUS_IS_EBADDATE */
-#define APR_EBADDATE       (APR_OS_START_ERROR + 4)
-/** @see APR_STATUS_IS_EINVALSOCK */
-#define APR_EINVALSOCK     (APR_OS_START_ERROR + 5)
-/** @see APR_STATUS_IS_ENOPROC */
-#define APR_ENOPROC        (APR_OS_START_ERROR + 6)
-/** @see APR_STATUS_IS_ENOTIME */
-#define APR_ENOTIME        (APR_OS_START_ERROR + 7)
-/** @see APR_STATUS_IS_ENODIR */
-#define APR_ENODIR         (APR_OS_START_ERROR + 8)
-/** @see APR_STATUS_IS_ENOLOCK */
-#define APR_ENOLOCK        (APR_OS_START_ERROR + 9)
-/** @see APR_STATUS_IS_ENOPOLL */
-#define APR_ENOPOLL        (APR_OS_START_ERROR + 10)
-/** @see APR_STATUS_IS_ENOSOCKET */
-#define APR_ENOSOCKET      (APR_OS_START_ERROR + 11)
-/** @see APR_STATUS_IS_ENOTHREAD */
-#define APR_ENOTHREAD      (APR_OS_START_ERROR + 12)
-/** @see APR_STATUS_IS_ENOTHDKEY */
-#define APR_ENOTHDKEY      (APR_OS_START_ERROR + 13)
-/** @see APR_STATUS_IS_EGENERAL */
-#define APR_EGENERAL       (APR_OS_START_ERROR + 14)
-/** @see APR_STATUS_IS_ENOSHMAVAIL */
-#define APR_ENOSHMAVAIL    (APR_OS_START_ERROR + 15)
-/** @see APR_STATUS_IS_EBADIP */
-#define APR_EBADIP         (APR_OS_START_ERROR + 16)
-/** @see APR_STATUS_IS_EBADMASK */
-#define APR_EBADMASK       (APR_OS_START_ERROR + 17)
-/* empty slot: +18 */
-/** @see APR_STATUS_IS_EDSOPEN */
-#define APR_EDSOOPEN       (APR_OS_START_ERROR + 19)
-/** @see APR_STATUS_IS_EABSOLUTE */
-#define APR_EABSOLUTE      (APR_OS_START_ERROR + 20)
-/** @see APR_STATUS_IS_ERELATIVE */
-#define APR_ERELATIVE      (APR_OS_START_ERROR + 21)
-/** @see APR_STATUS_IS_EINCOMPLETE */
-#define APR_EINCOMPLETE    (APR_OS_START_ERROR + 22)
-/** @see APR_STATUS_IS_EABOVEROOT */
-#define APR_EABOVEROOT     (APR_OS_START_ERROR + 23)
-/** @see APR_STATUS_IS_EBADPATH */
-#define APR_EBADPATH       (APR_OS_START_ERROR + 24)
-/** @see APR_STATUS_IS_EPATHWILD */
-#define APR_EPATHWILD      (APR_OS_START_ERROR + 25)
-/** @see APR_STATUS_IS_ESYMNOTFOUND */
-#define APR_ESYMNOTFOUND   (APR_OS_START_ERROR + 26)
-/** @see APR_STATUS_IS_EPROC_UNKNOWN */
-#define APR_EPROC_UNKNOWN  (APR_OS_START_ERROR + 27)
-/** @see APR_STATUS_IS_ENOTENOUGHENTROPY */
-#define APR_ENOTENOUGHENTROPY (APR_OS_START_ERROR + 28)
-/** @} */
-
-/**
- * @defgroup APR_STATUS_IS Status Value Tests
- * @warning For any particular error condition, more than one of these tests
- *      may match. This is because platform-specific error codes may not
- *      always match the semantics of the POSIX codes these tests (and the
- *      corresponding APR error codes) are named after. A notable example
- *      are the APR_STATUS_IS_ENOENT and APR_STATUS_IS_ENOTDIR tests on
- *      Win32 platforms. The programmer should always be aware of this and
- *      adjust the order of the tests accordingly.
- * @{
- */
-/**
- * APR was unable to perform a stat on the file
- * @warning always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_ENOSTAT(s)        ((s) == APR_ENOSTAT)
-/**
- * APR was not provided a pool with which to allocate memory
- * @warning always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_ENOPOOL(s)        ((s) == APR_ENOPOOL)
-/** APR was given an invalid date  */
-#define APR_STATUS_IS_EBADDATE(s)       ((s) == APR_EBADDATE)
-/** APR was given an invalid socket */
-#define APR_STATUS_IS_EINVALSOCK(s)     ((s) == APR_EINVALSOCK)
-/** APR was not given a process structure */
-#define APR_STATUS_IS_ENOPROC(s)        ((s) == APR_ENOPROC)
-/** APR was not given a time structure */
-#define APR_STATUS_IS_ENOTIME(s)        ((s) == APR_ENOTIME)
-/** APR was not given a directory structure */
-#define APR_STATUS_IS_ENODIR(s)         ((s) == APR_ENODIR)
-/** APR was not given a lock structure */
-#define APR_STATUS_IS_ENOLOCK(s)        ((s) == APR_ENOLOCK)
-/** APR was not given a poll structure */
-#define APR_STATUS_IS_ENOPOLL(s)        ((s) == APR_ENOPOLL)
-/** APR was not given a socket */
-#define APR_STATUS_IS_ENOSOCKET(s)      ((s) == APR_ENOSOCKET)
-/** APR was not given a thread structure */
-#define APR_STATUS_IS_ENOTHREAD(s)      ((s) == APR_ENOTHREAD)
-/** APR was not given a thread key structure */
-#define APR_STATUS_IS_ENOTHDKEY(s)      ((s) == APR_ENOTHDKEY)
-/** Generic Error which can not be put into another spot */
-#define APR_STATUS_IS_EGENERAL(s)       ((s) == APR_EGENERAL)
-/** There is no more shared memory available */
-#define APR_STATUS_IS_ENOSHMAVAIL(s)    ((s) == APR_ENOSHMAVAIL)
-/** The specified IP address is invalid */
-#define APR_STATUS_IS_EBADIP(s)         ((s) == APR_EBADIP)
-/** The specified netmask is invalid */
-#define APR_STATUS_IS_EBADMASK(s)       ((s) == APR_EBADMASK)
-/* empty slot: +18 */
-/**
- * APR was unable to open the dso object.
- * For more information call apr_dso_error().
- */
-#if defined(WIN32)
-#define APR_STATUS_IS_EDSOOPEN(s)       ((s) == APR_EDSOOPEN \
-                       || APR_TO_OS_ERROR(s) == ERROR_MOD_NOT_FOUND)
-#else
-#define APR_STATUS_IS_EDSOOPEN(s)       ((s) == APR_EDSOOPEN)
-#endif
-/** The given path was absolute. */
-#define APR_STATUS_IS_EABSOLUTE(s)      ((s) == APR_EABSOLUTE)
-/** The given path was relative. */
-#define APR_STATUS_IS_ERELATIVE(s)      ((s) == APR_ERELATIVE)
-/** The given path was neither relative nor absolute. */
-#define APR_STATUS_IS_EINCOMPLETE(s)    ((s) == APR_EINCOMPLETE)
-/** The given path was above the root path. */
-#define APR_STATUS_IS_EABOVEROOT(s)     ((s) == APR_EABOVEROOT)
-/** The given path was bad. */
-#define APR_STATUS_IS_EBADPATH(s)       ((s) == APR_EBADPATH)
-/** The given path contained wildcards. */
-#define APR_STATUS_IS_EPATHWILD(s)      ((s) == APR_EPATHWILD)
-/** Could not find the requested symbol.
- * For more information call apr_dso_error().
- */
-#if defined(WIN32)
-#define APR_STATUS_IS_ESYMNOTFOUND(s)   ((s) == APR_ESYMNOTFOUND \
-                       || APR_TO_OS_ERROR(s) == ERROR_PROC_NOT_FOUND)
-#else
-#define APR_STATUS_IS_ESYMNOTFOUND(s)   ((s) == APR_ESYMNOTFOUND)
-#endif
-/** The given process was not recognized by APR. */
-#define APR_STATUS_IS_EPROC_UNKNOWN(s)  ((s) == APR_EPROC_UNKNOWN)
-
-/** APR could not gather enough entropy to continue. */
-#define APR_STATUS_IS_ENOTENOUGHENTROPY(s) ((s) == APR_ENOTENOUGHENTROPY)
-
-/** @} */
-
-/**
- * @addtogroup APR_Error
- * @{
- */
-/** @see APR_STATUS_IS_INCHILD */
-#define APR_INCHILD        (APR_OS_START_STATUS + 1)
-/** @see APR_STATUS_IS_INPARENT */
-#define APR_INPARENT       (APR_OS_START_STATUS + 2)
-/** @see APR_STATUS_IS_DETACH */
-#define APR_DETACH         (APR_OS_START_STATUS + 3)
-/** @see APR_STATUS_IS_NOTDETACH */
-#define APR_NOTDETACH      (APR_OS_START_STATUS + 4)
-/** @see APR_STATUS_IS_CHILD_DONE */
-#define APR_CHILD_DONE     (APR_OS_START_STATUS + 5)
-/** @see APR_STATUS_IS_CHILD_NOTDONE */
-#define APR_CHILD_NOTDONE  (APR_OS_START_STATUS + 6)
-/** @see APR_STATUS_IS_TIMEUP */
-#define APR_TIMEUP         (APR_OS_START_STATUS + 7)
-/** @see APR_STATUS_IS_INCOMPLETE */
-#define APR_INCOMPLETE     (APR_OS_START_STATUS + 8)
-/* empty slot: +9 */
-/* empty slot: +10 */
-/* empty slot: +11 */
-/** @see APR_STATUS_IS_BADCH */
-#define APR_BADCH          (APR_OS_START_STATUS + 12)
-/** @see APR_STATUS_IS_BADARG */
-#define APR_BADARG         (APR_OS_START_STATUS + 13)
-/** @see APR_STATUS_IS_EOF */
-#define APR_EOF            (APR_OS_START_STATUS + 14)
-/** @see APR_STATUS_IS_NOTFOUND */
-#define APR_NOTFOUND       (APR_OS_START_STATUS + 15)
-/* empty slot: +16 */
-/* empty slot: +17 */
-/* empty slot: +18 */
-/** @see APR_STATUS_IS_ANONYMOUS */
-#define APR_ANONYMOUS      (APR_OS_START_STATUS + 19)
-/** @see APR_STATUS_IS_FILEBASED */
-#define APR_FILEBASED      (APR_OS_START_STATUS + 20)
-/** @see APR_STATUS_IS_KEYBASED */
-#define APR_KEYBASED       (APR_OS_START_STATUS + 21)
-/** @see APR_STATUS_IS_EINIT */
-#define APR_EINIT          (APR_OS_START_STATUS + 22)
-/** @see APR_STATUS_IS_ENOTIMPL */
-#define APR_ENOTIMPL       (APR_OS_START_STATUS + 23)
-/** @see APR_STATUS_IS_EMISMATCH */
-#define APR_EMISMATCH      (APR_OS_START_STATUS + 24)
-/** @see APR_STATUS_IS_EBUSY */
-#define APR_EBUSY          (APR_OS_START_STATUS + 25)
-/** @} */
-
-/**
- * @addtogroup APR_STATUS_IS
- * @{
- */
-/**
- * Program is currently executing in the child
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code */
-#define APR_STATUS_IS_INCHILD(s)        ((s) == APR_INCHILD)
-/**
- * Program is currently executing in the parent
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_INPARENT(s)       ((s) == APR_INPARENT)
-/**
- * The thread is detached
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_DETACH(s)         ((s) == APR_DETACH)
-/**
- * The thread is not detached
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_NOTDETACH(s)      ((s) == APR_NOTDETACH)
-/**
- * The child has finished executing
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_CHILD_DONE(s)     ((s) == APR_CHILD_DONE)
-/**
- * The child has not finished executing
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_CHILD_NOTDONE(s)  ((s) == APR_CHILD_NOTDONE)
-/**
- * The operation did not finish before the timeout
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_TIMEUP(s)         ((s) == APR_TIMEUP)
-/**
- * The operation was incomplete although some processing was performed
- * and the results are partially valid.
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_INCOMPLETE(s)     ((s) == APR_INCOMPLETE)
-/* empty slot: +9 */
-/* empty slot: +10 */
-/* empty slot: +11 */
-/**
- * Getopt found an option not in the option string
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_BADCH(s)          ((s) == APR_BADCH)
-/**
- * Getopt found an option not in the option string and an argument was
- * specified in the option string
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_BADARG(s)         ((s) == APR_BADARG)
-/**
- * APR has encountered the end of the file
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_EOF(s)            ((s) == APR_EOF)
-/**
- * APR was unable to find the socket in the poll structure
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_NOTFOUND(s)       ((s) == APR_NOTFOUND)
-/* empty slot: +16 */
-/* empty slot: +17 */
-/* empty slot: +18 */
-/**
- * APR is using anonymous shared memory
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_ANONYMOUS(s)      ((s) == APR_ANONYMOUS)
-/**
- * APR is using a file name as the key to the shared memory
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_FILEBASED(s)      ((s) == APR_FILEBASED)
-/**
- * APR is using a shared key as the key to the shared memory
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_KEYBASED(s)       ((s) == APR_KEYBASED)
-/**
- * Ininitalizer value.  If no option has been found, but
- * the status variable requires a value, this should be used
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_EINIT(s)          ((s) == APR_EINIT)
-/**
- * The APR function has not been implemented on this
- * platform, either because nobody has gotten to it yet,
- * or the function is impossible on this platform.
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_ENOTIMPL(s)       ((s) == APR_ENOTIMPL)
-/**
- * Two passwords do not match.
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_EMISMATCH(s)      ((s) == APR_EMISMATCH)
-/**
- * The given lock was busy
- * @warning always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_EBUSY(s)          ((s) == APR_EBUSY)
-
-/** @} */
-
-/**
- * @addtogroup APR_Error APR Error Values
- * @{
- */
-/* APR CANONICAL ERROR VALUES */
-/** @see APR_STATUS_IS_EACCES */
-#ifdef EACCES
-#define APR_EACCES EACCES
-#else
-#define APR_EACCES         (APR_OS_START_CANONERR + 1)
-#endif
-
-/** @see APR_STATUS_IS_EXIST */
-#ifdef EEXIST
-#define APR_EEXIST EEXIST
-#else
-#define APR_EEXIST         (APR_OS_START_CANONERR + 2)
-#endif
-
-/** @see APR_STATUS_IS_ENAMETOOLONG */
-#ifdef ENAMETOOLONG
-#define APR_ENAMETOOLONG ENAMETOOLONG
-#else
-#define APR_ENAMETOOLONG   (APR_OS_START_CANONERR + 3)
-#endif
-
-/** @see APR_STATUS_IS_ENOENT */
-#ifdef ENOENT
-#define APR_ENOENT ENOENT
-#else
-#define APR_ENOENT         (APR_OS_START_CANONERR + 4)
-#endif
-
-/** @see APR_STATUS_IS_ENOTDIR */
-#ifdef ENOTDIR
-#define APR_ENOTDIR ENOTDIR
-#else
-#define APR_ENOTDIR        (APR_OS_START_CANONERR + 5)
-#endif
-
-/** @see APR_STATUS_IS_ENOSPC */
-#ifdef ENOSPC
-#define APR_ENOSPC ENOSPC
-#else
-#define APR_ENOSPC         (APR_OS_START_CANONERR + 6)
-#endif
-
-/** @see APR_STATUS_IS_ENOMEM */
-#ifdef ENOMEM
-#define APR_ENOMEM ENOMEM
-#else
-#define APR_ENOMEM         (APR_OS_START_CANONERR + 7)
-#endif
-
-/** @see APR_STATUS_IS_EMFILE */
-#ifdef EMFILE
-#define APR_EMFILE EMFILE
-#else
-#define APR_EMFILE         (APR_OS_START_CANONERR + 8)
-#endif
-
-/** @see APR_STATUS_IS_ENFILE */
-#ifdef ENFILE
-#define APR_ENFILE ENFILE
-#else
-#define APR_ENFILE         (APR_OS_START_CANONERR + 9)
-#endif
-
-/** @see APR_STATUS_IS_EBADF */
-#ifdef EBADF
-#define APR_EBADF EBADF
-#else
-#define APR_EBADF          (APR_OS_START_CANONERR + 10)
-#endif
-
-/** @see APR_STATUS_IS_EINVAL */
-#ifdef EINVAL
-#define APR_EINVAL EINVAL
-#else
-#define APR_EINVAL         (APR_OS_START_CANONERR + 11)
-#endif
-
-/** @see APR_STATUS_IS_ESPIPE */
-#ifdef ESPIPE
-#define APR_ESPIPE ESPIPE
-#else
-#define APR_ESPIPE         (APR_OS_START_CANONERR + 12)
-#endif
-
-/**
- * @see APR_STATUS_IS_EAGAIN
- * @warning use APR_STATUS_IS_EAGAIN instead of just testing this value
- */
-#ifdef EAGAIN
-#define APR_EAGAIN EAGAIN
-#elif defined(EWOULDBLOCK)
-#define APR_EAGAIN EWOULDBLOCK
-#else
-#define APR_EAGAIN         (APR_OS_START_CANONERR + 13)
-#endif
-
-/** @see APR_STATUS_IS_EINTR */
-#ifdef EINTR
-#define APR_EINTR EINTR
-#else
-#define APR_EINTR          (APR_OS_START_CANONERR + 14)
-#endif
-
-/** @see APR_STATUS_IS_ENOTSOCK */
-#ifdef ENOTSOCK
-#define APR_ENOTSOCK ENOTSOCK
-#else
-#define APR_ENOTSOCK       (APR_OS_START_CANONERR + 15)
-#endif
-
-/** @see APR_STATUS_IS_ECONNREFUSED */
-#ifdef ECONNREFUSED
-#define APR_ECONNREFUSED ECONNREFUSED
-#else
-#define APR_ECONNREFUSED   (APR_OS_START_CANONERR + 16)
-#endif
-
-/** @see APR_STATUS_IS_EINPROGRESS */
-#ifdef EINPROGRESS
-#define APR_EINPROGRESS EINPROGRESS
-#else
-#define APR_EINPROGRESS    (APR_OS_START_CANONERR + 17)
-#endif
-
-/**
- * @see APR_STATUS_IS_ECONNABORTED
- * @warning use APR_STATUS_IS_ECONNABORTED instead of just testing this value
- */
-
-#ifdef ECONNABORTED
-#define APR_ECONNABORTED ECONNABORTED
-#else
-#define APR_ECONNABORTED   (APR_OS_START_CANONERR + 18)
-#endif
-
-/** @see APR_STATUS_IS_ECONNRESET */
-#ifdef ECONNRESET
-#define APR_ECONNRESET ECONNRESET
-#else
-#define APR_ECONNRESET     (APR_OS_START_CANONERR + 19)
-#endif
-
-/** @see APR_STATUS_IS_ETIMEDOUT
- *  @deprecated */
-#ifdef ETIMEDOUT
-#define APR_ETIMEDOUT ETIMEDOUT
-#else
-#define APR_ETIMEDOUT      (APR_OS_START_CANONERR + 20)
-#endif
-
-/** @see APR_STATUS_IS_EHOSTUNREACH */
-#ifdef EHOSTUNREACH
-#define APR_EHOSTUNREACH EHOSTUNREACH
-#else
-#define APR_EHOSTUNREACH   (APR_OS_START_CANONERR + 21)
-#endif
-
-/** @see APR_STATUS_IS_ENETUNREACH */
-#ifdef ENETUNREACH
-#define APR_ENETUNREACH ENETUNREACH
-#else
-#define APR_ENETUNREACH    (APR_OS_START_CANONERR + 22)
-#endif
-
-/** @see APR_STATUS_IS_EFTYPE */
-#ifdef EFTYPE
-#define APR_EFTYPE EFTYPE
-#else
-#define APR_EFTYPE        (APR_OS_START_CANONERR + 23)
-#endif
-
-/** @see APR_STATUS_IS_EPIPE */
-#ifdef EPIPE
-#define APR_EPIPE EPIPE
-#else
-#define APR_EPIPE         (APR_OS_START_CANONERR + 24)
-#endif
-
-/** @see APR_STATUS_IS_EXDEV */
-#ifdef EXDEV
-#define APR_EXDEV EXDEV
-#else
-#define APR_EXDEV         (APR_OS_START_CANONERR + 25)
-#endif
-
-/** @see APR_STATUS_IS_ENOTEMPTY */
-#ifdef ENOTEMPTY
-#define APR_ENOTEMPTY ENOTEMPTY
-#else
-#define APR_ENOTEMPTY     (APR_OS_START_CANONERR + 26)
-#endif
-
-/** @} */
-
-#if defined(OS2) && !defined(DOXYGEN)
-
-#define APR_FROM_OS_ERROR(e) (e == 0 ? APR_SUCCESS : e + APR_OS_START_SYSERR)
-#define APR_TO_OS_ERROR(e)   (e == 0 ? APR_SUCCESS : e - APR_OS_START_SYSERR)
-
-#define INCL_DOSERRORS
-#define INCL_DOS
-
-/* Leave these undefined.
- * OS2 doesn't rely on the errno concept.
- * The API calls always return a result codes which
- * should be filtered through APR_FROM_OS_ERROR().
- *
- * #define apr_get_os_error()   (APR_FROM_OS_ERROR(GetLastError()))
- * #define apr_set_os_error(e)  (SetLastError(APR_TO_OS_ERROR(e)))
- */
-
-/* A special case, only socket calls require this;
- */
-#define apr_get_netos_error()   (APR_FROM_OS_ERROR(errno))
-#define apr_set_netos_error(e)  (errno = APR_TO_OS_ERROR(e))
-
-/* And this needs to be greped away for good:
- */
-#define APR_OS2_STATUS(e) (APR_FROM_OS_ERROR(e))
-
-/* These can't sit in a private header, so in spite of the extra size,
- * they need to be made available here.
- */
-#define SOCBASEERR              10000
-#define SOCEPERM                (SOCBASEERR+1)             /* Not owner */
-#define SOCESRCH                (SOCBASEERR+3)             /* No such process */
-#define SOCEINTR                (SOCBASEERR+4)             /* Interrupted system call */
-#define SOCENXIO                (SOCBASEERR+6)             /* No such device or address */
-#define SOCEBADF                (SOCBASEERR+9)             /* Bad file number */
-#define SOCEACCES               (SOCBASEERR+13)            /* Permission denied */
-#define SOCEFAULT               (SOCBASEERR+14)            /* Bad address */
-#define SOCEINVAL               (SOCBASEERR+22)            /* Invalid argument */
-#define SOCEMFILE               (SOCBASEERR+24)            /* Too many open files */
-#define SOCEPIPE                (SOCBASEERR+32)            /* Broken pipe */
-#define SOCEOS2ERR              (SOCBASEERR+100)           /* OS/2 Error */
-#define SOCEWOULDBLOCK          (SOCBASEERR+35)            /* Operation would block */
-#define SOCEINPROGRESS          (SOCBASEERR+36)            /* Operation now in progress */
-#define SOCEALREADY             (SOCBASEERR+37)            /* Operation already in progress */
-#define SOCENOTSOCK             (SOCBASEERR+38)            /* Socket operation on non-socket */
-#define SOCEDESTADDRREQ         (SOCBASEERR+39)            /* Destination address required */
-#define SOCEMSGSIZE             (SOCBASEERR+40)            /* Message too long */
-#define SOCEPROTOTYPE           (SOCBASEERR+41)            /* Protocol wrong type for socket */
-#define SOCENOPROTOOPT          (SOCBASEERR+42)            /* Protocol not available */
-#define SOCEPROTONOSUPPORT      (SOCBASEERR+43)            /* Protocol not supported */
-#define SOCESOCKTNOSUPPORT      (SOCBASEERR+44)            /* Socket type not supported */
-#define SOCEOPNOTSUPP           (SOCBASEERR+45)            /* Operation not supported on socket */
-#define SOCEPFNOSUPPORT         (SOCBASEERR+46)            /* Protocol family not supported */
-#define SOCEAFNOSUPPORT         (SOCBASEERR+47)            /* Address family not supported by protocol family */
-#define SOCEADDRINUSE           (SOCBASEERR+48)            /* Address already in use */
-#define SOCEADDRNOTAVAIL        (SOCBASEERR+49)            /* Can't assign requested address */
-#define SOCENETDOWN             (SOCBASEERR+50)            /* Network is down */
-#define SOCENETUNREACH          (SOCBASEERR+51)            /* Network is unreachable */
-#define SOCENETRESET            (SOCBASEERR+52)            /* Network dropped connection on reset */
-#define SOCECONNABORTED         (SOCBASEERR+53)            /* Software caused connection abort */
-#define SOCECONNRESET           (SOCBASEERR+54)            /* Connection reset by peer */
-#define SOCENOBUFS              (SOCBASEERR+55)            /* No buffer space available */
-#define SOCEISCONN              (SOCBASEERR+56)            /* Socket is already connected */
-#define SOCENOTCONN             (SOCBASEERR+57)            /* Socket is not connected */
-#define SOCESHUTDOWN            (SOCBASEERR+58)            /* Can't send after socket shutdown */
-#define SOCETOOMANYREFS         (SOCBASEERR+59)            /* Too many references: can't splice */
-#define SOCETIMEDOUT            (SOCBASEERR+60)            /* Connection timed out */
-#define SOCECONNREFUSED         (SOCBASEERR+61)            /* Connection refused */
-#define SOCELOOP                (SOCBASEERR+62)            /* Too many levels of symbolic links */
-#define SOCENAMETOOLONG         (SOCBASEERR+63)            /* File name too long */
-#define SOCEHOSTDOWN            (SOCBASEERR+64)            /* Host is down */
-#define SOCEHOSTUNREACH         (SOCBASEERR+65)            /* No route to host */
-#define SOCENOTEMPTY            (SOCBASEERR+66)            /* Directory not empty */
-
-/* APR CANONICAL ERROR TESTS */
-#define APR_STATUS_IS_EACCES(s)         ((s) == APR_EACCES \
-                || (s) == APR_OS_START_SYSERR + ERROR_ACCESS_DENIED \
-                || (s) == APR_OS_START_SYSERR + ERROR_SHARING_VIOLATION)
-#define APR_STATUS_IS_EEXIST(s)         ((s) == APR_EEXIST \
-                || (s) == APR_OS_START_SYSERR + ERROR_OPEN_FAILED \
-                || (s) == APR_OS_START_SYSERR + ERROR_FILE_EXISTS \
-                || (s) == APR_OS_START_SYSERR + ERROR_ALREADY_EXISTS \
-                || (s) == APR_OS_START_SYSERR + ERROR_ACCESS_DENIED)
-#define APR_STATUS_IS_ENAMETOOLONG(s)   ((s) == APR_ENAMETOOLONG \
-                || (s) == APR_OS_START_SYSERR + ERROR_FILENAME_EXCED_RANGE \
-                || (s) == APR_OS_START_SYSERR + SOCENAMETOOLONG)
-#define APR_STATUS_IS_ENOENT(s)         ((s) == APR_ENOENT \
-                || (s) == APR_OS_START_SYSERR + ERROR_FILE_NOT_FOUND \
-                || (s) == APR_OS_START_SYSERR + ERROR_PATH_NOT_FOUND \
-                || (s) == APR_OS_START_SYSERR + ERROR_NO_MORE_FILES \
-                || (s) == APR_OS_START_SYSERR + ERROR_OPEN_FAILED)
-#define APR_STATUS_IS_ENOTDIR(s)        ((s) == APR_ENOTDIR)
-#define APR_STATUS_IS_ENOSPC(s)         ((s) == APR_ENOSPC \
-                || (s) == APR_OS_START_SYSERR + ERROR_DISK_FULL)
-#define APR_STATUS_IS_ENOMEM(s)         ((s) == APR_ENOMEM)
-#define APR_STATUS_IS_EMFILE(s)         ((s) == APR_EMFILE \
-                || (s) == APR_OS_START_SYSERR + ERROR_TOO_MANY_OPEN_FILES)
-#define APR_STATUS_IS_ENFILE(s)         ((s) == APR_ENFILE)
-#define APR_STATUS_IS_EBADF(s)          ((s) == APR_EBADF \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_HANDLE)
-#define APR_STATUS_IS_EINVAL(s)         ((s) == APR_EINVAL \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_PARAMETER \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_FUNCTION)
-#define APR_STATUS_IS_ESPIPE(s)         ((s) == APR_ESPIPE \
-                || (s) == APR_OS_START_SYSERR + ERROR_NEGATIVE_SEEK)
-#define APR_STATUS_IS_EAGAIN(s)         ((s) == APR_EAGAIN \
-                || (s) == APR_OS_START_SYSERR + ERROR_NO_DATA \
-                || (s) == APR_OS_START_SYSERR + SOCEWOULDBLOCK \
-                || (s) == APR_OS_START_SYSERR + ERROR_LOCK_VIOLATION)
-#define APR_STATUS_IS_EINTR(s)          ((s) == APR_EINTR \
-                || (s) == APR_OS_START_SYSERR + SOCEINTR)
-#define APR_STATUS_IS_ENOTSOCK(s)       ((s) == APR_ENOTSOCK \
-                || (s) == APR_OS_START_SYSERR + SOCENOTSOCK)
-#define APR_STATUS_IS_ECONNREFUSED(s)   ((s) == APR_ECONNREFUSED \
-                || (s) == APR_OS_START_SYSERR + SOCECONNREFUSED)
-#define APR_STATUS_IS_EINPROGRESS(s)    ((s) == APR_EINPROGRESS \
-                || (s) == APR_OS_START_SYSERR + SOCEINPROGRESS)
-#define APR_STATUS_IS_ECONNABORTED(s)   ((s) == APR_ECONNABORTED \
-                || (s) == APR_OS_START_SYSERR + SOCECONNABORTED)
-#define APR_STATUS_IS_ECONNRESET(s)     ((s) == APR_ECONNRESET \
-                || (s) == APR_OS_START_SYSERR + SOCECONNRESET)
-/* XXX deprecated */
-#define APR_STATUS_IS_ETIMEDOUT(s)         ((s) == APR_ETIMEDOUT \
-                || (s) == APR_OS_START_SYSERR + SOCETIMEDOUT)
-#undef APR_STATUS_IS_TIMEUP
-#define APR_STATUS_IS_TIMEUP(s)         ((s) == APR_TIMEUP \
-                || (s) == APR_OS_START_SYSERR + SOCETIMEDOUT)
-#define APR_STATUS_IS_EHOSTUNREACH(s)   ((s) == APR_EHOSTUNREACH \
-                || (s) == APR_OS_START_SYSERR + SOCEHOSTUNREACH)
-#define APR_STATUS_IS_ENETUNREACH(s)    ((s) == APR_ENETUNREACH \
-                || (s) == APR_OS_START_SYSERR + SOCENETUNREACH)
-#define APR_STATUS_IS_EFTYPE(s)         ((s) == APR_EFTYPE)
-#define APR_STATUS_IS_EPIPE(s)          ((s) == APR_EPIPE \
-                || (s) == APR_OS_START_SYSERR + ERROR_BROKEN_PIPE \
-                || (s) == APR_OS_START_SYSERR + SOCEPIPE)
-#define APR_STATUS_IS_EXDEV(s)          ((s) == APR_EXDEV \
-                || (s) == APR_OS_START_SYSERR + ERROR_NOT_SAME_DEVICE)
-#define APR_STATUS_IS_ENOTEMPTY(s)      ((s) == APR_ENOTEMPTY \
-                || (s) == APR_OS_START_SYSERR + ERROR_DIR_NOT_EMPTY \
-                || (s) == APR_OS_START_SYSERR + ERROR_ACCESS_DENIED)
-
-/*
-    Sorry, too tired to wrap this up for OS2... feel free to
-    fit the following into their best matches.
-
-    { ERROR_NO_SIGNAL_SENT,     ESRCH           },
-    { SOCEALREADY,              EALREADY        },
-    { SOCEDESTADDRREQ,          EDESTADDRREQ    },
-    { SOCEMSGSIZE,              EMSGSIZE        },
-    { SOCEPROTOTYPE,            EPROTOTYPE      },
-    { SOCENOPROTOOPT,           ENOPROTOOPT     },
-    { SOCEPROTONOSUPPORT,       EPROTONOSUPPORT },
-    { SOCESOCKTNOSUPPORT,       ESOCKTNOSUPPORT },
-    { SOCEOPNOTSUPP,            EOPNOTSUPP      },
-    { SOCEPFNOSUPPORT,          EPFNOSUPPORT    },
-    { SOCEAFNOSUPPORT,          EAFNOSUPPORT    },
-    { SOCEADDRINUSE,            EADDRINUSE      },
-    { SOCEADDRNOTAVAIL,         EADDRNOTAVAIL   },
-    { SOCENETDOWN,              ENETDOWN        },
-    { SOCENETRESET,             ENETRESET       },
-    { SOCENOBUFS,               ENOBUFS         },
-    { SOCEISCONN,               EISCONN         },
-    { SOCENOTCONN,              ENOTCONN        },
-    { SOCESHUTDOWN,             ESHUTDOWN       },
-    { SOCETOOMANYREFS,          ETOOMANYREFS    },
-    { SOCELOOP,                 ELOOP           },
-    { SOCEHOSTDOWN,             EHOSTDOWN       },
-    { SOCENOTEMPTY,             ENOTEMPTY       },
-    { SOCEPIPE,                 EPIPE           }
-*/
-
-#elif defined(WIN32) && !defined(DOXYGEN) /* !defined(OS2) */
-
-#define APR_FROM_OS_ERROR(e) (e == 0 ? APR_SUCCESS : e + APR_OS_START_SYSERR)
-#define APR_TO_OS_ERROR(e)   (e == 0 ? APR_SUCCESS : e - APR_OS_START_SYSERR)
-
-#define apr_get_os_error()   (APR_FROM_OS_ERROR(GetLastError()))
-#define apr_set_os_error(e)  (SetLastError(APR_TO_OS_ERROR(e)))
-
-/* A special case, only socket calls require this:
- */
-#define apr_get_netos_error()   (APR_FROM_OS_ERROR(WSAGetLastError()))
-#define apr_set_netos_error(e)   (WSASetLastError(APR_TO_OS_ERROR(e)))
-
-/* APR CANONICAL ERROR TESTS */
-#define APR_STATUS_IS_EACCES(s)         ((s) == APR_EACCES \
-                || (s) == APR_OS_START_SYSERR + ERROR_ACCESS_DENIED \
-                || (s) == APR_OS_START_SYSERR + ERROR_CANNOT_MAKE \
-                || (s) == APR_OS_START_SYSERR + ERROR_CURRENT_DIRECTORY \
-                || (s) == APR_OS_START_SYSERR + ERROR_DRIVE_LOCKED \
-                || (s) == APR_OS_START_SYSERR + ERROR_FAIL_I24 \
-                || (s) == APR_OS_START_SYSERR + ERROR_LOCK_VIOLATION \
-                || (s) == APR_OS_START_SYSERR + ERROR_LOCK_FAILED \
-                || (s) == APR_OS_START_SYSERR + ERROR_NOT_LOCKED \
-                || (s) == APR_OS_START_SYSERR + ERROR_NETWORK_ACCESS_DENIED \
-                || (s) == APR_OS_START_SYSERR + ERROR_SHARING_VIOLATION)
-#define APR_STATUS_IS_EEXIST(s)         ((s) == APR_EEXIST \
-                || (s) == APR_OS_START_SYSERR + ERROR_FILE_EXISTS \
-                || (s) == APR_OS_START_SYSERR + ERROR_ALREADY_EXISTS)
-#define APR_STATUS_IS_ENAMETOOLONG(s)   ((s) == APR_ENAMETOOLONG \
-                || (s) == APR_OS_START_SYSERR + ERROR_FILENAME_EXCED_RANGE \
-                || (s) == APR_OS_START_SYSERR + WSAENAMETOOLONG)
-#define APR_STATUS_IS_ENOENT(s)         ((s) == APR_ENOENT \
-                || (s) == APR_OS_START_SYSERR + ERROR_FILE_NOT_FOUND \
-                || (s) == APR_OS_START_SYSERR + ERROR_PATH_NOT_FOUND \
-                || (s) == APR_OS_START_SYSERR + ERROR_OPEN_FAILED \
-                || (s) == APR_OS_START_SYSERR + ERROR_NO_MORE_FILES)
-#define APR_STATUS_IS_ENOTDIR(s)        ((s) == APR_ENOTDIR \
-                || (s) == APR_OS_START_SYSERR + ERROR_PATH_NOT_FOUND \
-                || (s) == APR_OS_START_SYSERR + ERROR_BAD_NETPATH \
-                || (s) == APR_OS_START_SYSERR + ERROR_BAD_NET_NAME \
-                || (s) == APR_OS_START_SYSERR + ERROR_BAD_PATHNAME \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_DRIVE)
-#define APR_STATUS_IS_ENOSPC(s)         ((s) == APR_ENOSPC \
-                || (s) == APR_OS_START_SYSERR + ERROR_DISK_FULL)
-#define APR_STATUS_IS_ENOMEM(s)         ((s) == APR_ENOMEM \
-                || (s) == APR_OS_START_SYSERR + ERROR_ARENA_TRASHED \
-                || (s) == APR_OS_START_SYSERR + ERROR_NOT_ENOUGH_MEMORY \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_BLOCK \
-                || (s) == APR_OS_START_SYSERR + ERROR_NOT_ENOUGH_QUOTA \
-                || (s) == APR_OS_START_SYSERR + ERROR_OUTOFMEMORY)
-#define APR_STATUS_IS_EMFILE(s)         ((s) == APR_EMFILE \
-                || (s) == APR_OS_START_SYSERR + ERROR_TOO_MANY_OPEN_FILES)
-#define APR_STATUS_IS_ENFILE(s)         ((s) == APR_ENFILE)
-#define APR_STATUS_IS_EBADF(s)          ((s) == APR_EBADF \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_HANDLE \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_TARGET_HANDLE)
-#define APR_STATUS_IS_EINVAL(s)         ((s) == APR_EINVAL \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_ACCESS \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_DATA \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_FUNCTION \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_HANDLE \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_PARAMETER \
-                || (s) == APR_OS_START_SYSERR + ERROR_NEGATIVE_SEEK)
-#define APR_STATUS_IS_ESPIPE(s)         ((s) == APR_ESPIPE \
-                || (s) == APR_OS_START_SYSERR + ERROR_SEEK_ON_DEVICE \
-                || (s) == APR_OS_START_SYSERR + ERROR_NEGATIVE_SEEK)
-#define APR_STATUS_IS_EAGAIN(s)         ((s) == APR_EAGAIN \
-                || (s) == APR_OS_START_SYSERR + ERROR_NO_DATA \
-                || (s) == APR_OS_START_SYSERR + ERROR_NO_PROC_SLOTS \
-                || (s) == APR_OS_START_SYSERR + ERROR_NESTING_NOT_ALLOWED \
-                || (s) == APR_OS_START_SYSERR + ERROR_MAX_THRDS_REACHED \
-                || (s) == APR_OS_START_SYSERR + ERROR_LOCK_VIOLATION \
-                || (s) == APR_OS_START_SYSERR + WSAEWOULDBLOCK)
-#define APR_STATUS_IS_EINTR(s)          ((s) == APR_EINTR \
-                || (s) == APR_OS_START_SYSERR + WSAEINTR)
-#define APR_STATUS_IS_ENOTSOCK(s)       ((s) == APR_ENOTSOCK \
-                || (s) == APR_OS_START_SYSERR + WSAENOTSOCK)
-#define APR_STATUS_IS_ECONNREFUSED(s)   ((s) == APR_ECONNREFUSED \
-                || (s) == APR_OS_START_SYSERR + WSAECONNREFUSED)
-#define APR_STATUS_IS_EINPROGRESS(s)    ((s) == APR_EINPROGRESS \
-                || (s) == APR_OS_START_SYSERR + WSAEINPROGRESS)
-#define APR_STATUS_IS_ECONNABORTED(s)   ((s) == APR_ECONNABORTED \
-                || (s) == APR_OS_START_SYSERR + WSAECONNABORTED)
-#define APR_STATUS_IS_ECONNRESET(s)     ((s) == APR_ECONNRESET \
-                || (s) == APR_OS_START_SYSERR + ERROR_NETNAME_DELETED \
-                || (s) == APR_OS_START_SYSERR + WSAECONNRESET)
-/* XXX deprecated */
-#define APR_STATUS_IS_ETIMEDOUT(s)         ((s) == APR_ETIMEDOUT \
-                || (s) == APR_OS_START_SYSERR + WSAETIMEDOUT \
-                || (s) == APR_OS_START_SYSERR + WAIT_TIMEOUT)
-#undef APR_STATUS_IS_TIMEUP
-#define APR_STATUS_IS_TIMEUP(s)         ((s) == APR_TIMEUP \
-                || (s) == APR_OS_START_SYSERR + WSAETIMEDOUT \
-                || (s) == APR_OS_START_SYSERR + WAIT_TIMEOUT)
-#define APR_STATUS_IS_EHOSTUNREACH(s)   ((s) == APR_EHOSTUNREACH \
-                || (s) == APR_OS_START_SYSERR + WSAEHOSTUNREACH)
-#define APR_STATUS_IS_ENETUNREACH(s)    ((s) == APR_ENETUNREACH \
-                || (s) == APR_OS_START_SYSERR + WSAENETUNREACH)
-#define APR_STATUS_IS_EFTYPE(s)         ((s) == APR_EFTYPE \
-                || (s) == APR_OS_START_SYSERR + ERROR_EXE_MACHINE_TYPE_MISMATCH \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_DLL \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_MODULETYPE \
-                || (s) == APR_OS_START_SYSERR + ERROR_BAD_EXE_FORMAT \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_EXE_SIGNATURE \
-                || (s) == APR_OS_START_SYSERR + ERROR_FILE_CORRUPT \
-                || (s) == APR_OS_START_SYSERR + ERROR_BAD_FORMAT)
-#define APR_STATUS_IS_EPIPE(s)          ((s) == APR_EPIPE \
-                || (s) == APR_OS_START_SYSERR + ERROR_BROKEN_PIPE)
-#define APR_STATUS_IS_EXDEV(s)          ((s) == APR_EXDEV \
-                || (s) == APR_OS_START_SYSERR + ERROR_NOT_SAME_DEVICE)
-#define APR_STATUS_IS_ENOTEMPTY(s)      ((s) == APR_ENOTEMPTY \
-                || (s) == APR_OS_START_SYSERR + ERROR_DIR_NOT_EMPTY)
-
-#elif defined(NETWARE) && defined(USE_WINSOCK) && !defined(DOXYGEN) /* !defined(OS2) && !defined(WIN32) */
-
-#define APR_FROM_OS_ERROR(e) (e == 0 ? APR_SUCCESS : e + APR_OS_START_SYSERR)
-#define APR_TO_OS_ERROR(e)   (e == 0 ? APR_SUCCESS : e - APR_OS_START_SYSERR)
-
-#define apr_get_os_error()    (errno)
-#define apr_set_os_error(e)   (errno = (e))
-
-/* A special case, only socket calls require this: */
-#define apr_get_netos_error()   (APR_FROM_OS_ERROR(WSAGetLastError()))
-#define apr_set_netos_error(e)  (WSASetLastError(APR_TO_OS_ERROR(e)))
-
-/* APR CANONICAL ERROR TESTS */
-#define APR_STATUS_IS_EACCES(s)         ((s) == APR_EACCES)
-#define APR_STATUS_IS_EEXIST(s)         ((s) == APR_EEXIST)
-#define APR_STATUS_IS_ENAMETOOLONG(s)   ((s) == APR_ENAMETOOLONG)
-#define APR_STATUS_IS_ENOENT(s)         ((s) == APR_ENOENT)
-#define APR_STATUS_IS_ENOTDIR(s)        ((s) == APR_ENOTDIR)
-#define APR_STATUS_IS_ENOSPC(s)         ((s) == APR_ENOSPC)
-#define APR_STATUS_IS_ENOMEM(s)         ((s) == APR_ENOMEM)
-#define APR_STATUS_IS_EMFILE(s)         ((s) == APR_EMFILE)
-#define APR_STATUS_IS_ENFILE(s)         ((s) == APR_ENFILE)
-#define APR_STATUS_IS_EBADF(s)          ((s) == APR_EBADF)
-#define APR_STATUS_IS_EINVAL(s)         ((s) == APR_EINVAL)
-#define APR_STATUS_IS_ESPIPE(s)         ((s) == APR_ESPIPE)
-
-#define APR_STATUS_IS_EAGAIN(s)         ((s) == APR_EAGAIN \
-                || (s) ==                       EWOULDBLOCK \
-                || (s) == APR_OS_START_SYSERR + WSAEWOULDBLOCK)
-#define APR_STATUS_IS_EINTR(s)          ((s) == APR_EINTR \
-                || (s) == APR_OS_START_SYSERR + WSAEINTR)
-#define APR_STATUS_IS_ENOTSOCK(s)       ((s) == APR_ENOTSOCK \
-                || (s) == APR_OS_START_SYSERR + WSAENOTSOCK)
-#define APR_STATUS_IS_ECONNREFUSED(s)   ((s) == APR_ECONNREFUSED \
-                || (s) == APR_OS_START_SYSERR + WSAECONNREFUSED)
-#define APR_STATUS_IS_EINPROGRESS(s)    ((s) == APR_EINPROGRESS \
-                || (s) == APR_OS_START_SYSERR + WSAEINPROGRESS)
-#define APR_STATUS_IS_ECONNABORTED(s)   ((s) == APR_ECONNABORTED \
-                || (s) == APR_OS_START_SYSERR + WSAECONNABORTED)
-#define APR_STATUS_IS_ECONNRESET(s)     ((s) == APR_ECONNRESET \
-                || (s) == APR_OS_START_SYSERR + WSAECONNRESET)
-/* XXX deprecated */
-#define APR_STATUS_IS_ETIMEDOUT(s)       ((s) == APR_ETIMEDOUT \
-                || (s) == APR_OS_START_SYSERR + WSAETIMEDOUT \
-                || (s) == APR_OS_START_SYSERR + WAIT_TIMEOUT)
-#undef APR_STATUS_IS_TIMEUP
-#define APR_STATUS_IS_TIMEUP(s)         ((s) == APR_TIMEUP \
-                || (s) == APR_OS_START_SYSERR + WSAETIMEDOUT \
-                || (s) == APR_OS_START_SYSERR + WAIT_TIMEOUT)
-#define APR_STATUS_IS_EHOSTUNREACH(s)   ((s) == APR_EHOSTUNREACH \
-                || (s) == APR_OS_START_SYSERR + WSAEHOSTUNREACH)
-#define APR_STATUS_IS_ENETUNREACH(s)    ((s) == APR_ENETUNREACH \
-                || (s) == APR_OS_START_SYSERR + WSAENETUNREACH)
-#define APR_STATUS_IS_ENETDOWN(s)       ((s) == APR_OS_START_SYSERR + WSAENETDOWN)
-#define APR_STATUS_IS_EFTYPE(s)         ((s) == APR_EFTYPE)
-#define APR_STATUS_IS_EPIPE(s)          ((s) == APR_EPIPE)
-#define APR_STATUS_IS_EXDEV(s)          ((s) == APR_EXDEV)
-#define APR_STATUS_IS_ENOTEMPTY(s)      ((s) == APR_ENOTEMPTY)
-
-#else /* !defined(NETWARE) && !defined(OS2) && !defined(WIN32) */
-
-/*
- *  os error codes are clib error codes
- */
-#define APR_FROM_OS_ERROR(e)  (e)
-#define APR_TO_OS_ERROR(e)    (e)
-
-#define apr_get_os_error()    (errno)
-#define apr_set_os_error(e)   (errno = (e))
-
-/* A special case, only socket calls require this:
- */
-#define apr_get_netos_error() (errno)
-#define apr_set_netos_error(e) (errno = (e))
-
-/**
- * @addtogroup APR_STATUS_IS
- * @{
- */
-
-/** permission denied */
-#define APR_STATUS_IS_EACCES(s)         ((s) == APR_EACCES)
-/** file exists */
-#define APR_STATUS_IS_EEXIST(s)         ((s) == APR_EEXIST)
-/** path name is too long */
-#define APR_STATUS_IS_ENAMETOOLONG(s)   ((s) == APR_ENAMETOOLONG)
-/**
- * no such file or directory
- * @remark
- * EMVSCATLG can be returned by the automounter on z/OS for
- * paths which do not exist.
- */
-#ifdef EMVSCATLG
-#define APR_STATUS_IS_ENOENT(s)         ((s) == APR_ENOENT \
-                                      || (s) == EMVSCATLG)
-#else
-#define APR_STATUS_IS_ENOENT(s)         ((s) == APR_ENOENT)
-#endif
-/** not a directory */
-#define APR_STATUS_IS_ENOTDIR(s)        ((s) == APR_ENOTDIR)
-/** no space left on device */
-#ifdef EDQUOT
-#define APR_STATUS_IS_ENOSPC(s)         ((s) == APR_ENOSPC \
-                                      || (s) == EDQUOT)
-#else
-#define APR_STATUS_IS_ENOSPC(s)         ((s) == APR_ENOSPC)
-#endif
-/** not enough memory */
-#define APR_STATUS_IS_ENOMEM(s)         ((s) == APR_ENOMEM)
-/** too many open files */
-#define APR_STATUS_IS_EMFILE(s)         ((s) == APR_EMFILE)
-/** file table overflow */
-#define APR_STATUS_IS_ENFILE(s)         ((s) == APR_ENFILE)
-/** bad file # */
-#define APR_STATUS_IS_EBADF(s)          ((s) == APR_EBADF)
-/** invalid argument */
-#define APR_STATUS_IS_EINVAL(s)         ((s) == APR_EINVAL)
-/** illegal seek */
-#define APR_STATUS_IS_ESPIPE(s)         ((s) == APR_ESPIPE)
-
-/** operation would block */
-#if !defined(EWOULDBLOCK) || !defined(EAGAIN)
-#define APR_STATUS_IS_EAGAIN(s)         ((s) == APR_EAGAIN)
-#elif (EWOULDBLOCK == EAGAIN)
-#define APR_STATUS_IS_EAGAIN(s)         ((s) == APR_EAGAIN)
-#else
-#define APR_STATUS_IS_EAGAIN(s)         ((s) == APR_EAGAIN \
-                                      || (s) == EWOULDBLOCK)
-#endif
-
-/** interrupted system call */
-#define APR_STATUS_IS_EINTR(s)          ((s) == APR_EINTR)
-/** socket operation on a non-socket */
-#define APR_STATUS_IS_ENOTSOCK(s)       ((s) == APR_ENOTSOCK)
-/** Connection Refused */
-#define APR_STATUS_IS_ECONNREFUSED(s)   ((s) == APR_ECONNREFUSED)
-/** operation now in progress */
-#define APR_STATUS_IS_EINPROGRESS(s)    ((s) == APR_EINPROGRESS)
-
-/**
- * Software caused connection abort
- * @remark
- * EPROTO on certain older kernels really means ECONNABORTED, so we need to
- * ignore it for them.  See discussion in new-httpd archives nh.9701 & nh.9603
- *
- * There is potentially a bug in Solaris 2.x x<6, and other boxes that
- * implement tcp sockets in userland (i.e. on top of STREAMS).  On these
- * systems, EPROTO can actually result in a fatal loop.  See PR#981 for
- * example.  It's hard to handle both uses of EPROTO.
- */
-#ifdef EPROTO
-#define APR_STATUS_IS_ECONNABORTED(s)    ((s) == APR_ECONNABORTED \
-                                       || (s) == EPROTO)
-#else
-#define APR_STATUS_IS_ECONNABORTED(s)    ((s) == APR_ECONNABORTED)
-#endif
-
-/** Connection Reset by peer */
-#define APR_STATUS_IS_ECONNRESET(s)      ((s) == APR_ECONNRESET)
-/** Operation timed out
- *  @deprecated */
-#define APR_STATUS_IS_ETIMEDOUT(s)      ((s) == APR_ETIMEDOUT)
-/** no route to host */
-#define APR_STATUS_IS_EHOSTUNREACH(s)    ((s) == APR_EHOSTUNREACH)
-/** network is unreachable */
-#define APR_STATUS_IS_ENETUNREACH(s)     ((s) == APR_ENETUNREACH)
-/** inappropriate file type or format */
-#define APR_STATUS_IS_EFTYPE(s)          ((s) == APR_EFTYPE)
-/** broken pipe */
-#define APR_STATUS_IS_EPIPE(s)           ((s) == APR_EPIPE)
-/** cross device link */
-#define APR_STATUS_IS_EXDEV(s)           ((s) == APR_EXDEV)
-/** Directory Not Empty */
-#define APR_STATUS_IS_ENOTEMPTY(s)       ((s) == APR_ENOTEMPTY || \
-                                          (s) == APR_EEXIST)
-/** @} */
-
-#endif /* !defined(NETWARE) && !defined(OS2) && !defined(WIN32) */
-
-/** @} */
-
-#ifdef __cplusplus
-}
-#endif
-
-#endif  /* ! APR_ERRNO_H */
-#ifndef _LINUX_ERRNO_H
-#define _LINUX_ERRNO_H
-
-#include <asm/errno.h>
-
-#ifdef __KERNEL__
-
-/* Should never be seen by user programs */
-#define ERESTARTSYS	512
-#define ERESTARTNOINTR	513
-#define ERESTARTNOHAND	514	/* restart if no handler.. */
-#define ENOIOCTLCMD	515	/* No ioctl command */
-#define ERESTART_RESTARTBLOCK 516 /* restart by calling sys_restart_syscall */
-
-/* Defined for the NFSv3 protocol */
-#define EBADHANDLE	521	/* Illegal NFS file handle */
-#define ENOTSYNC	522	/* Update synchronization mismatch */
-#define EBADCOOKIE	523	/* Cookie is stale */
-#define ENOTSUPP	524	/* Operation is not supported */
-#define ETOOSMALL	525	/* Buffer or request is too small */
-#define ESERVERFAULT	526	/* An untranslatable error occurred */
-#define EBADTYPE	527	/* Type not supported by server */
-#define EJUKEBOX	528	/* Request initiated, but will not complete before timeout */
-#define EIOCBQUEUED	529	/* iocb queued, will get completion event */
-#define EIOCBRETRY	530	/* iocb queued, will trigger a retry */
-
-#endif
-
-#endif
-// Copyright (c) 1994 James Clark
-// See the file COPYING for copying permission.
-
-#ifndef ErrnoMessageArg_INCLUDED
-#define ErrnoMessageArg_INCLUDED 1
-
-#include "MessageArg.h"
-#include "rtti.h"
-
-#ifdef SP_NAMESPACE
-namespace SP_NAMESPACE {
-#endif
-
-class SP_API ErrnoMessageArg : public OtherMessageArg {
-  RTTI_CLASS
-public:
-  ErrnoMessageArg(int errnum) : errno_(errnum) { }
-  MessageArg *copy() const;
-  // errno might be a macro so we must use a different name
-  int errnum() const;
-private:
-  int errno_;
-};
-
-inline
-int ErrnoMessageArg::errnum() const
-{
-  return errno_;
-}
-
-#ifdef SP_NAMESPACE
-}
-#endif
-
-#endif /* not ErrnoMessageArg_INCLUDED */
-/* Copyright (C) 1991,92,93,94,95,96,97,2002 Free Software Foundation, Inc.
-   This file is part of the GNU C Library.
-
-   The GNU C Library is free software; you can redistribute it and/or
-   modify it under the terms of the GNU Lesser General Public
-   License as published by the Free Software Foundation; either
-   version 2.1 of the License, or (at your option) any later version.
-
-   The GNU C Library is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-   Lesser General Public License for more details.
-
-   You should have received a copy of the GNU Lesser General Public
-   License along with the GNU C Library; if not, write to the Free
-   Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
-   02111-1307 USA.  */
-
-/*
- *	ISO C99 Standard: 7.5 Errors	<errno.h>
- */
-
-#ifndef	_ERRNO_H
-
-/* The includer defined __need_Emath if he wants only the definitions
-   of EDOM and ERANGE, and not everything else.  */
-#ifndef	__need_Emath
-# define _ERRNO_H	1
-# include <features.h>
-#endif
-
-__BEGIN_DECLS
-
-/* Get the error number constants from the system-specific file.
-   This file will test __need_Emath and _ERRNO_H.  */
-#include <bits/errno.h>
-#undef	__need_Emath
-
-#ifdef	_ERRNO_H
-
-/* Declare the `errno' variable, unless it's defined as a macro by
-   bits/errno.h.  This is the case in GNU, where it is a per-thread
-   variable.  This redeclaration using the macro still works, but it
-   will be a function declaration without a prototype and may trigger
-   a -Wstrict-prototypes warning.  */
-#ifndef	errno
-extern int errno;
-#endif
-
-#ifdef __USE_GNU
-
-/* The full and simple forms of the name with which the program was
-   invoked.  These variables are set up automatically at startup based on
-   the value of ARGV[0] (this works only if you use GNU ld).  */
-extern char *program_invocation_name, *program_invocation_short_name;
-#endif /* __USE_GNU */
-#endif /* _ERRNO_H */
-
-__END_DECLS
-
-#endif /* _ERRNO_H */
-
-/* The Hurd <bits/errno.h> defines `error_t' as an enumerated type so
-   that printing `error_t' values in the debugger shows the names.  We
-   might need this definition sometimes even if this file was included
-   before.  */
-#if defined __USE_GNU || defined __need_error_t
-# ifndef __error_t_defined
-typedef int error_t;
-#  define __error_t_defined	1
-# endif
-# undef __need_error_t
-#endif
-#ifndef _I386_ERRNO_H
-#define _I386_ERRNO_H
-
-#include <asm-generic/errno.h>
-
-#endif
-#ifndef _ASM_GENERIC_ERRNO_BASE_H
-#define _ASM_GENERIC_ERRNO_BASE_H
-
-#define	EPERM		 1	/* Operation not permitted */
-#define	ENOENT		 2	/* No such file or directory */
-#define	ESRCH		 3	/* No such process */
-#define	EINTR		 4	/* Interrupted system call */
-#define	EIO		 5	/* I/O error */
-#define	ENXIO		 6	/* No such device or address */
-#define	E2BIG		 7	/* Argument list too long */
-#define	ENOEXEC		 8	/* Exec format error */
-#define	EBADF		 9	/* Bad file number */
-#define	ECHILD		10	/* No child processes */
-#define	EAGAIN		11	/* Try again */
-#define	ENOMEM		12	/* Out of memory */
-#define	EACCES		13	/* Permission denied */
-#define	EFAULT		14	/* Bad address */
-#define	ENOTBLK		15	/* Block device required */
-#define	EBUSY		16	/* Device or resource busy */
-#define	EEXIST		17	/* File exists */
-#define	EXDEV		18	/* Cross-device link */
-#define	ENODEV		19	/* No such device */
-#define	ENOTDIR		20	/* Not a directory */
-#define	EISDIR		21	/* Is a directory */
-#define	EINVAL		22	/* Invalid argument */
-#define	ENFILE		23	/* File table overflow */
-#define	EMFILE		24	/* Too many open files */
-#define	ENOTTY		25	/* Not a typewriter */
-#define	ETXTBSY		26	/* Text file busy */
-#define	EFBIG		27	/* File too large */
-#define	ENOSPC		28	/* No space left on device */
-#define	ESPIPE		29	/* Illegal seek */
-#define	EROFS		30	/* Read-only file system */
-#define	EMLINK		31	/* Too many links */
-#define	EPIPE		32	/* Broken pipe */
-#define	EDOM		33	/* Math argument out of domain of func */
-#define	ERANGE		34	/* Math result not representable */
-
-#endif
-#ifndef _ASM_GENERIC_ERRNO_H
-#define _ASM_GENERIC_ERRNO_H
-
-#include <asm-generic/errno-base.h>
-
-#define	EDEADLK		35	/* Resource deadlock would occur */
-#define	ENAMETOOLONG	36	/* File name too long */
-#define	ENOLCK		37	/* No record locks available */
-#define	ENOSYS		38	/* Function not implemented */
-#define	ENOTEMPTY	39	/* Directory not empty */
-#define	ELOOP		40	/* Too many symbolic links encountered */
-#define	EWOULDBLOCK	EAGAIN	/* Operation would block */
-#define	ENOMSG		42	/* No message of desired type */
-#define	EIDRM		43	/* Identifier removed */
-#define	ECHRNG		44	/* Channel number out of range */
-#define	EL2NSYNC	45	/* Level 2 not synchronized */
-#define	EL3HLT		46	/* Level 3 halted */
-#define	EL3RST		47	/* Level 3 reset */
-#define	ELNRNG		48	/* Link number out of range */
-#define	EUNATCH		49	/* Protocol driver not attached */
-#define	ENOCSI		50	/* No CSI structure available */
-#define	EL2HLT		51	/* Level 2 halted */
-#define	EBADE		52	/* Invalid exchange */
-#define	EBADR		53	/* Invalid request descriptor */
-#define	EXFULL		54	/* Exchange full */
-#define	ENOANO		55	/* No anode */
-#define	EBADRQC		56	/* Invalid request code */
-#define	EBADSLT		57	/* Invalid slot */
-
-#define	EDEADLOCK	EDEADLK
-
-#define	EBFONT		59	/* Bad font file format */
-#define	ENOSTR		60	/* Device not a stream */
-#define	ENODATA		61	/* No data available */
-#define	ETIME		62	/* Timer expired */
-#define	ENOSR		63	/* Out of streams resources */
-#define	ENONET		64	/* Machine is not on the network */
-#define	ENOPKG		65	/* Package not installed */
-#define	EREMOTE		66	/* Object is remote */
-#define	ENOLINK		67	/* Link has been severed */
-#define	EADV		68	/* Advertise error */
-#define	ESRMNT		69	/* Srmount error */
-#define	ECOMM		70	/* Communication error on send */
-#define	EPROTO		71	/* Protocol error */
-#define	EMULTIHOP	72	/* Multihop attempted */
-#define	EDOTDOT		73	/* RFS specific error */
-#define	EBADMSG		74	/* Not a data message */
-#define	EOVERFLOW	75	/* Value too large for defined data type */
-#define	ENOTUNIQ	76	/* Name not unique on network */
-#define	EBADFD		77	/* File descriptor in bad state */
-#define	EREMCHG		78	/* Remote address changed */
-#define	ELIBACC		79	/* Can not access a needed shared library */
-#define	ELIBBAD		80	/* Accessing a corrupted shared library */
-#define	ELIBSCN		81	/* .lib section in a.out corrupted */
-#define	ELIBMAX		82	/* Attempting to link in too many shared libraries */
-#define	ELIBEXEC	83	/* Cannot exec a shared library directly */
-#define	EILSEQ		84	/* Illegal byte sequence */
-#define	ERESTART	85	/* Interrupted system call should be restarted */
-#define	ESTRPIPE	86	/* Streams pipe error */
-#define	EUSERS		87	/* Too many users */
-#define	ENOTSOCK	88	/* Socket operation on non-socket */
-#define	EDESTADDRREQ	89	/* Destination address required */
-#define	EMSGSIZE	90	/* Message too long */
-#define	EPROTOTYPE	91	/* Protocol wrong type for socket */
-#define	ENOPROTOOPT	92	/* Protocol not available */
-#define	EPROTONOSUPPORT	93	/* Protocol not supported */
-#define	ESOCKTNOSUPPORT	94	/* Socket type not supported */
-#define	EOPNOTSUPP	95	/* Operation not supported on transport endpoint */
-#define	EPFNOSUPPORT	96	/* Protocol family not supported */
-#define	EAFNOSUPPORT	97	/* Address family not supported by protocol */
-#define	EADDRINUSE	98	/* Address already in use */
-#define	EADDRNOTAVAIL	99	/* Cannot assign requested address */
-#define	ENETDOWN	100	/* Network is down */
-#define	ENETUNREACH	101	/* Network is unreachable */
-#define	ENETRESET	102	/* Network dropped connection because of reset */
-#define	ECONNABORTED	103	/* Software caused connection abort */
-#define	ECONNRESET	104	/* Connection reset by peer */
-#define	ENOBUFS		105	/* No buffer space available */
-#define	EISCONN		106	/* Transport endpoint is already connected */
-#define	ENOTCONN	107	/* Transport endpoint is not connected */
-#define	ESHUTDOWN	108	/* Cannot send after transport endpoint shutdown */
-#define	ETOOMANYREFS	109	/* Too many references: cannot splice */
-#define	ETIMEDOUT	110	/* Connection timed out */
-#define	ECONNREFUSED	111	/* Connection refused */
-#define	EHOSTDOWN	112	/* Host is down */
-#define	EHOSTUNREACH	113	/* No route to host */
-#define	EALREADY	114	/* Operation already in progress */
-#define	EINPROGRESS	115	/* Operation now in progress */
-#define	ESTALE		116	/* Stale NFS file handle */
-#define	EUCLEAN		117	/* Structure needs cleaning */
-#define	ENOTNAM		118	/* Not a XENIX named type file */
-#define	ENAVAIL		119	/* No XENIX semaphores available */
-#define	EISNAM		120	/* Is a named type file */
-#define	EREMOTEIO	121	/* Remote I/O error */
-#define	EDQUOT		122	/* Quota exceeded */
-
-#define	ENOMEDIUM	123	/* No medium found */
-#define	EMEDIUMTYPE	124	/* Wrong medium type */
-#define	ECANCELED	125	/* Operation Canceled */
-#define	ENOKEY		126	/* Required key not available */
-#define	EKEYEXPIRED	127	/* Key has expired */
-#define	EKEYREVOKED	128	/* Key has been revoked */
-#define	EKEYREJECTED	129	/* Key was rejected by service */
-
-/* for robust mutexes */
-#define	EOWNERDEAD	130	/* Owner died */
-#define	ENOTRECOVERABLE	131	/* State not recoverable */
-
-#endif
diff --git a/doc/legacy/errno.list.macosx.txt b/doc/legacy/errno.list.macosx.txt
deleted file mode 100644
index 2dff28cd379..00000000000
--- a/doc/legacy/errno.list.macosx.txt
+++ /dev/null
@@ -1,1513 +0,0 @@
-/* Copyright 2000-2005 The Apache Software Foundation or its licensors, as
- * applicable.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-#ifndef APR_ERRNO_H
-#define APR_ERRNO_H
-
-/**
- * @file apr_errno.h
- * @brief APR Error Codes
- */
-
-#include "apr.h"
-
-#if APR_HAVE_ERRNO_H
-#include <errno.h>
-#endif
-
-#ifdef __cplusplus
-extern "C" {
-#endif /* __cplusplus */
-
-/**
- * @defgroup apr_errno Error Codes
- * @ingroup APR
- * @{
- */
-
-/**
- * Type for specifying an error or status code.
- */
-typedef int apr_status_t;
-
-/**
- * Return a human readable string describing the specified error.
- * @param statcode The error code the get a string for.
- * @param buf A buffer to hold the error string.
- * @param bufsize Size of the buffer to hold the string.
- */
-APR_DECLARE(char *) apr_strerror(apr_status_t statcode, char *buf,
-                                 apr_size_t bufsize);
-
-#if defined(DOXYGEN)
-/**
- * @def APR_FROM_OS_ERROR(os_err_type syserr)
- * Fold a platform specific error into an apr_status_t code.
- * @return apr_status_t
- * @param e The platform os error code.
- * @warning  macro implementation; the syserr argument may be evaluated
- *      multiple times.
- */
-#define APR_FROM_OS_ERROR(e) (e == 0 ? APR_SUCCESS : e + APR_OS_START_SYSERR)
-
-/**
- * @def APR_TO_OS_ERROR(apr_status_t statcode)
- * @return os_err_type
- * Fold an apr_status_t code back to the native platform defined error.
- * @param e The apr_status_t folded platform os error code.
- * @warning  macro implementation; the statcode argument may be evaluated
- *      multiple times.  If the statcode was not created by apr_get_os_error
- *      or APR_FROM_OS_ERROR, the results are undefined.
- */
-#define APR_TO_OS_ERROR(e) (e == 0 ? APR_SUCCESS : e - APR_OS_START_SYSERR)
-
-/** @def apr_get_os_error()
- * @return apr_status_t the last platform error, folded into apr_status_t, on most platforms
- * @remark This retrieves errno, or calls a GetLastError() style function, and
- *      folds it with APR_FROM_OS_ERROR.  Some platforms (such as OS2) have no
- *      such mechanism, so this call may be unsupported.  Do NOT use this
- *      call for socket errors from socket, send, recv etc!
- */
-
-/** @def apr_set_os_error(e)
- * Reset the last platform error, unfolded from an apr_status_t, on some platforms
- * @param e The OS error folded in a prior call to APR_FROM_OS_ERROR()
- * @warning This is a macro implementation; the statcode argument may be evaluated
- *      multiple times.  If the statcode was not created by apr_get_os_error
- *      or APR_FROM_OS_ERROR, the results are undefined.  This macro sets
- *      errno, or calls a SetLastError() style function, unfolding statcode
- *      with APR_TO_OS_ERROR.  Some platforms (such as OS2) have no such
- *      mechanism, so this call may be unsupported.
- */
-
-/** @def apr_get_netos_error()
- * Return the last socket error, folded into apr_status_t, on all platforms
- * @remark This retrieves errno or calls a GetLastSocketError() style function,
- *      and folds it with APR_FROM_OS_ERROR.
- */
-
-/** @def apr_set_netos_error(e)
- * Reset the last socket error, unfolded from an apr_status_t
- * @param e The socket error folded in a prior call to APR_FROM_OS_ERROR()
- * @warning This is a macro implementation; the statcode argument may be evaluated
- *      multiple times.  If the statcode was not created by apr_get_os_error
- *      or APR_FROM_OS_ERROR, the results are undefined.  This macro sets
- *      errno, or calls a WSASetLastError() style function, unfolding
- *      socketcode with APR_TO_OS_ERROR.
- */
-
-#endif /* defined(DOXYGEN) */
-
-/**
- * APR_OS_START_ERROR is where the APR specific error values start.
- */
-#define APR_OS_START_ERROR     20000
-/**
- * APR_OS_ERRSPACE_SIZE is the maximum number of errors you can fit
- *    into one of the error/status ranges below -- except for
- *    APR_OS_START_USERERR, which see.
- */
-#define APR_OS_ERRSPACE_SIZE 50000
-/**
- * APR_OS_START_STATUS is where the APR specific status codes start.
- */
-#define APR_OS_START_STATUS    (APR_OS_START_ERROR + APR_OS_ERRSPACE_SIZE)
-/**
- * APR_OS_START_USERERR are reserved for applications that use APR that
- *     layer their own error codes along with APR's.  Note that the
- *     error immediately following this one is set ten times farther
- *     away than usual, so that users of apr have a lot of room in
- *     which to declare custom error codes.
- */
-#define APR_OS_START_USERERR    (APR_OS_START_STATUS + APR_OS_ERRSPACE_SIZE)
-/**
- * APR_OS_START_USEERR is obsolete, defined for compatibility only.
- * Use APR_OS_START_USERERR instead.
- */
-#define APR_OS_START_USEERR     APR_OS_START_USERERR
-/**
- * APR_OS_START_CANONERR is where APR versions of errno values are defined
- *     on systems which don't have the corresponding errno.
- */
-#define APR_OS_START_CANONERR  (APR_OS_START_USERERR \
-                                 + (APR_OS_ERRSPACE_SIZE * 10))
-/**
- * APR_OS_START_EAIERR folds EAI_ error codes from getaddrinfo() into
- *     apr_status_t values.
- */
-#define APR_OS_START_EAIERR    (APR_OS_START_CANONERR + APR_OS_ERRSPACE_SIZE)
-/**
- * APR_OS_START_SYSERR folds platform-specific system error values into
- *     apr_status_t values.
- */
-#define APR_OS_START_SYSERR    (APR_OS_START_EAIERR + APR_OS_ERRSPACE_SIZE)
-
-/** no error. */
-#define APR_SUCCESS 0
-
-/**
- * @defgroup APR_Error APR Error Values
- * <PRE>
- * <b>APR ERROR VALUES</b>
- * APR_ENOSTAT      APR was unable to perform a stat on the file
- * APR_ENOPOOL      APR was not provided a pool with which to allocate memory
- * APR_EBADDATE     APR was given an invalid date
- * APR_EINVALSOCK   APR was given an invalid socket
- * APR_ENOPROC      APR was not given a process structure
- * APR_ENOTIME      APR was not given a time structure
- * APR_ENODIR       APR was not given a directory structure
- * APR_ENOLOCK      APR was not given a lock structure
- * APR_ENOPOLL      APR was not given a poll structure
- * APR_ENOSOCKET    APR was not given a socket
- * APR_ENOTHREAD    APR was not given a thread structure
- * APR_ENOTHDKEY    APR was not given a thread key structure
- * APR_ENOSHMAVAIL  There is no more shared memory available
- * APR_EDSOOPEN     APR was unable to open the dso object.  For more
- *                  information call apr_dso_error().
- * APR_EGENERAL     General failure (specific information not available)
- * APR_EBADIP       The specified IP address is invalid
- * APR_EBADMASK     The specified netmask is invalid
- * APR_ESYMNOTFOUND Could not find the requested symbol
- * </PRE>
- *
- * <PRE>
- * <b>APR STATUS VALUES</b>
- * APR_INCHILD        Program is currently executing in the child
- * APR_INPARENT       Program is currently executing in the parent
- * APR_DETACH         The thread is detached
- * APR_NOTDETACH      The thread is not detached
- * APR_CHILD_DONE     The child has finished executing
- * APR_CHILD_NOTDONE  The child has not finished executing
- * APR_TIMEUP         The operation did not finish before the timeout
- * APR_INCOMPLETE     The operation was incomplete although some processing
- *                    was performed and the results are partially valid
- * APR_BADCH          Getopt found an option not in the option string
- * APR_BADARG         Getopt found an option that is missing an argument
- *                    and an argument was specified in the option string
- * APR_EOF            APR has encountered the end of the file
- * APR_NOTFOUND       APR was unable to find the socket in the poll structure
- * APR_ANONYMOUS      APR is using anonymous shared memory
- * APR_FILEBASED      APR is using a file name as the key to the shared memory
- * APR_KEYBASED       APR is using a shared key as the key to the shared memory
- * APR_EINIT          Ininitalizer value.  If no option has been found, but
- *                    the status variable requires a value, this should be used
- * APR_ENOTIMPL       The APR function has not been implemented on this
- *                    platform, either because nobody has gotten to it yet,
- *                    or the function is impossible on this platform.
- * APR_EMISMATCH      Two passwords do not match.
- * APR_EABSOLUTE      The given path was absolute.
- * APR_ERELATIVE      The given path was relative.
- * APR_EINCOMPLETE    The given path was neither relative nor absolute.
- * APR_EABOVEROOT     The given path was above the root path.
- * APR_EBUSY          The given lock was busy.
- * APR_EPROC_UNKNOWN  The given process wasn't recognized by APR
- * </PRE>
- * @{
- */
-/** @see APR_STATUS_IS_ENOSTAT */
-#define APR_ENOSTAT        (APR_OS_START_ERROR + 1)
-/** @see APR_STATUS_IS_ENOPOOL */
-#define APR_ENOPOOL        (APR_OS_START_ERROR + 2)
-/* empty slot: +3 */
-/** @see APR_STATUS_IS_EBADDATE */
-#define APR_EBADDATE       (APR_OS_START_ERROR + 4)
-/** @see APR_STATUS_IS_EINVALSOCK */
-#define APR_EINVALSOCK     (APR_OS_START_ERROR + 5)
-/** @see APR_STATUS_IS_ENOPROC */
-#define APR_ENOPROC        (APR_OS_START_ERROR + 6)
-/** @see APR_STATUS_IS_ENOTIME */
-#define APR_ENOTIME        (APR_OS_START_ERROR + 7)
-/** @see APR_STATUS_IS_ENODIR */
-#define APR_ENODIR         (APR_OS_START_ERROR + 8)
-/** @see APR_STATUS_IS_ENOLOCK */
-#define APR_ENOLOCK        (APR_OS_START_ERROR + 9)
-/** @see APR_STATUS_IS_ENOPOLL */
-#define APR_ENOPOLL        (APR_OS_START_ERROR + 10)
-/** @see APR_STATUS_IS_ENOSOCKET */
-#define APR_ENOSOCKET      (APR_OS_START_ERROR + 11)
-/** @see APR_STATUS_IS_ENOTHREAD */
-#define APR_ENOTHREAD      (APR_OS_START_ERROR + 12)
-/** @see APR_STATUS_IS_ENOTHDKEY */
-#define APR_ENOTHDKEY      (APR_OS_START_ERROR + 13)
-/** @see APR_STATUS_IS_EGENERAL */
-#define APR_EGENERAL       (APR_OS_START_ERROR + 14)
-/** @see APR_STATUS_IS_ENOSHMAVAIL */
-#define APR_ENOSHMAVAIL    (APR_OS_START_ERROR + 15)
-/** @see APR_STATUS_IS_EBADIP */
-#define APR_EBADIP         (APR_OS_START_ERROR + 16)
-/** @see APR_STATUS_IS_EBADMASK */
-#define APR_EBADMASK       (APR_OS_START_ERROR + 17)
-/* empty slot: +18 */
-/** @see APR_STATUS_IS_EDSOPEN */
-#define APR_EDSOOPEN       (APR_OS_START_ERROR + 19)
-/** @see APR_STATUS_IS_EABSOLUTE */
-#define APR_EABSOLUTE      (APR_OS_START_ERROR + 20)
-/** @see APR_STATUS_IS_ERELATIVE */
-#define APR_ERELATIVE      (APR_OS_START_ERROR + 21)
-/** @see APR_STATUS_IS_EINCOMPLETE */
-#define APR_EINCOMPLETE    (APR_OS_START_ERROR + 22)
-/** @see APR_STATUS_IS_EABOVEROOT */
-#define APR_EABOVEROOT     (APR_OS_START_ERROR + 23)
-/** @see APR_STATUS_IS_EBADPATH */
-#define APR_EBADPATH       (APR_OS_START_ERROR + 24)
-/** @see APR_STATUS_IS_EPATHWILD */
-#define APR_EPATHWILD      (APR_OS_START_ERROR + 25)
-/** @see APR_STATUS_IS_ESYMNOTFOUND */
-#define APR_ESYMNOTFOUND   (APR_OS_START_ERROR + 26)
-/** @see APR_STATUS_IS_EPROC_UNKNOWN */
-#define APR_EPROC_UNKNOWN  (APR_OS_START_ERROR + 27)
-/** @see APR_STATUS_IS_ENOTENOUGHENTROPY */
-#define APR_ENOTENOUGHENTROPY (APR_OS_START_ERROR + 28)
-/** @} */
-
-/**
- * @defgroup APR_STATUS_IS Status Value Tests
- * @warning For any particular error condition, more than one of these tests
- *      may match. This is because platform-specific error codes may not
- *      always match the semantics of the POSIX codes these tests (and the
- *      corresponding APR error codes) are named after. A notable example
- *      are the APR_STATUS_IS_ENOENT and APR_STATUS_IS_ENOTDIR tests on
- *      Win32 platforms. The programmer should always be aware of this and
- *      adjust the order of the tests accordingly.
- * @{
- */
-/**
- * APR was unable to perform a stat on the file
- * @warning always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_ENOSTAT(s)        ((s) == APR_ENOSTAT)
-/**
- * APR was not provided a pool with which to allocate memory
- * @warning always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_ENOPOOL(s)        ((s) == APR_ENOPOOL)
-/** APR was given an invalid date  */
-#define APR_STATUS_IS_EBADDATE(s)       ((s) == APR_EBADDATE)
-/** APR was given an invalid socket */
-#define APR_STATUS_IS_EINVALSOCK(s)     ((s) == APR_EINVALSOCK)
-/** APR was not given a process structure */
-#define APR_STATUS_IS_ENOPROC(s)        ((s) == APR_ENOPROC)
-/** APR was not given a time structure */
-#define APR_STATUS_IS_ENOTIME(s)        ((s) == APR_ENOTIME)
-/** APR was not given a directory structure */
-#define APR_STATUS_IS_ENODIR(s)         ((s) == APR_ENODIR)
-/** APR was not given a lock structure */
-#define APR_STATUS_IS_ENOLOCK(s)        ((s) == APR_ENOLOCK)
-/** APR was not given a poll structure */
-#define APR_STATUS_IS_ENOPOLL(s)        ((s) == APR_ENOPOLL)
-/** APR was not given a socket */
-#define APR_STATUS_IS_ENOSOCKET(s)      ((s) == APR_ENOSOCKET)
-/** APR was not given a thread structure */
-#define APR_STATUS_IS_ENOTHREAD(s)      ((s) == APR_ENOTHREAD)
-/** APR was not given a thread key structure */
-#define APR_STATUS_IS_ENOTHDKEY(s)      ((s) == APR_ENOTHDKEY)
-/** Generic Error which can not be put into another spot */
-#define APR_STATUS_IS_EGENERAL(s)       ((s) == APR_EGENERAL)
-/** There is no more shared memory available */
-#define APR_STATUS_IS_ENOSHMAVAIL(s)    ((s) == APR_ENOSHMAVAIL)
-/** The specified IP address is invalid */
-#define APR_STATUS_IS_EBADIP(s)         ((s) == APR_EBADIP)
-/** The specified netmask is invalid */
-#define APR_STATUS_IS_EBADMASK(s)       ((s) == APR_EBADMASK)
-/* empty slot: +18 */
-/**
- * APR was unable to open the dso object.
- * For more information call apr_dso_error().
- */
-#if defined(WIN32)
-#define APR_STATUS_IS_EDSOOPEN(s)       ((s) == APR_EDSOOPEN \
-                       || APR_TO_OS_ERROR(s) == ERROR_MOD_NOT_FOUND)
-#else
-#define APR_STATUS_IS_EDSOOPEN(s)       ((s) == APR_EDSOOPEN)
-#endif
-/** The given path was absolute. */
-#define APR_STATUS_IS_EABSOLUTE(s)      ((s) == APR_EABSOLUTE)
-/** The given path was relative. */
-#define APR_STATUS_IS_ERELATIVE(s)      ((s) == APR_ERELATIVE)
-/** The given path was neither relative nor absolute. */
-#define APR_STATUS_IS_EINCOMPLETE(s)    ((s) == APR_EINCOMPLETE)
-/** The given path was above the root path. */
-#define APR_STATUS_IS_EABOVEROOT(s)     ((s) == APR_EABOVEROOT)
-/** The given path was bad. */
-#define APR_STATUS_IS_EBADPATH(s)       ((s) == APR_EBADPATH)
-/** The given path contained wildcards. */
-#define APR_STATUS_IS_EPATHWILD(s)      ((s) == APR_EPATHWILD)
-/** Could not find the requested symbol.
- * For more information call apr_dso_error().
- */
-#if defined(WIN32)
-#define APR_STATUS_IS_ESYMNOTFOUND(s)   ((s) == APR_ESYMNOTFOUND \
-                       || APR_TO_OS_ERROR(s) == ERROR_PROC_NOT_FOUND)
-#else
-#define APR_STATUS_IS_ESYMNOTFOUND(s)   ((s) == APR_ESYMNOTFOUND)
-#endif
-/** The given process was not recognized by APR. */
-#define APR_STATUS_IS_EPROC_UNKNOWN(s)  ((s) == APR_EPROC_UNKNOWN)
-
-/** APR could not gather enough entropy to continue. */
-#define APR_STATUS_IS_ENOTENOUGHENTROPY(s) ((s) == APR_ENOTENOUGHENTROPY)
-
-/** @} */
-
-/**
- * @addtogroup APR_Error
- * @{
- */
-/** @see APR_STATUS_IS_INCHILD */
-#define APR_INCHILD        (APR_OS_START_STATUS + 1)
-/** @see APR_STATUS_IS_INPARENT */
-#define APR_INPARENT       (APR_OS_START_STATUS + 2)
-/** @see APR_STATUS_IS_DETACH */
-#define APR_DETACH         (APR_OS_START_STATUS + 3)
-/** @see APR_STATUS_IS_NOTDETACH */
-#define APR_NOTDETACH      (APR_OS_START_STATUS + 4)
-/** @see APR_STATUS_IS_CHILD_DONE */
-#define APR_CHILD_DONE     (APR_OS_START_STATUS + 5)
-/** @see APR_STATUS_IS_CHILD_NOTDONE */
-#define APR_CHILD_NOTDONE  (APR_OS_START_STATUS + 6)
-/** @see APR_STATUS_IS_TIMEUP */
-#define APR_TIMEUP         (APR_OS_START_STATUS + 7)
-/** @see APR_STATUS_IS_INCOMPLETE */
-#define APR_INCOMPLETE     (APR_OS_START_STATUS + 8)
-/* empty slot: +9 */
-/* empty slot: +10 */
-/* empty slot: +11 */
-/** @see APR_STATUS_IS_BADCH */
-#define APR_BADCH          (APR_OS_START_STATUS + 12)
-/** @see APR_STATUS_IS_BADARG */
-#define APR_BADARG         (APR_OS_START_STATUS + 13)
-/** @see APR_STATUS_IS_EOF */
-#define APR_EOF            (APR_OS_START_STATUS + 14)
-/** @see APR_STATUS_IS_NOTFOUND */
-#define APR_NOTFOUND       (APR_OS_START_STATUS + 15)
-/* empty slot: +16 */
-/* empty slot: +17 */
-/* empty slot: +18 */
-/** @see APR_STATUS_IS_ANONYMOUS */
-#define APR_ANONYMOUS      (APR_OS_START_STATUS + 19)
-/** @see APR_STATUS_IS_FILEBASED */
-#define APR_FILEBASED      (APR_OS_START_STATUS + 20)
-/** @see APR_STATUS_IS_KEYBASED */
-#define APR_KEYBASED       (APR_OS_START_STATUS + 21)
-/** @see APR_STATUS_IS_EINIT */
-#define APR_EINIT          (APR_OS_START_STATUS + 22)
-/** @see APR_STATUS_IS_ENOTIMPL */
-#define APR_ENOTIMPL       (APR_OS_START_STATUS + 23)
-/** @see APR_STATUS_IS_EMISMATCH */
-#define APR_EMISMATCH      (APR_OS_START_STATUS + 24)
-/** @see APR_STATUS_IS_EBUSY */
-#define APR_EBUSY          (APR_OS_START_STATUS + 25)
-/** @} */
-
-/**
- * @addtogroup APR_STATUS_IS
- * @{
- */
-/**
- * Program is currently executing in the child
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code */
-#define APR_STATUS_IS_INCHILD(s)        ((s) == APR_INCHILD)
-/**
- * Program is currently executing in the parent
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_INPARENT(s)       ((s) == APR_INPARENT)
-/**
- * The thread is detached
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_DETACH(s)         ((s) == APR_DETACH)
-/**
- * The thread is not detached
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_NOTDETACH(s)      ((s) == APR_NOTDETACH)
-/**
- * The child has finished executing
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_CHILD_DONE(s)     ((s) == APR_CHILD_DONE)
-/**
- * The child has not finished executing
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_CHILD_NOTDONE(s)  ((s) == APR_CHILD_NOTDONE)
-/**
- * The operation did not finish before the timeout
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_TIMEUP(s)         ((s) == APR_TIMEUP)
-/**
- * The operation was incomplete although some processing was performed
- * and the results are partially valid.
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_INCOMPLETE(s)     ((s) == APR_INCOMPLETE)
-/* empty slot: +9 */
-/* empty slot: +10 */
-/* empty slot: +11 */
-/**
- * Getopt found an option not in the option string
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_BADCH(s)          ((s) == APR_BADCH)
-/**
- * Getopt found an option not in the option string and an argument was
- * specified in the option string
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_BADARG(s)         ((s) == APR_BADARG)
-/**
- * APR has encountered the end of the file
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_EOF(s)            ((s) == APR_EOF)
-/**
- * APR was unable to find the socket in the poll structure
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_NOTFOUND(s)       ((s) == APR_NOTFOUND)
-/* empty slot: +16 */
-/* empty slot: +17 */
-/* empty slot: +18 */
-/**
- * APR is using anonymous shared memory
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_ANONYMOUS(s)      ((s) == APR_ANONYMOUS)
-/**
- * APR is using a file name as the key to the shared memory
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_FILEBASED(s)      ((s) == APR_FILEBASED)
-/**
- * APR is using a shared key as the key to the shared memory
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_KEYBASED(s)       ((s) == APR_KEYBASED)
-/**
- * Ininitalizer value.  If no option has been found, but
- * the status variable requires a value, this should be used
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_EINIT(s)          ((s) == APR_EINIT)
-/**
- * The APR function has not been implemented on this
- * platform, either because nobody has gotten to it yet,
- * or the function is impossible on this platform.
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_ENOTIMPL(s)       ((s) == APR_ENOTIMPL)
-/**
- * Two passwords do not match.
- * @warning
- * always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_EMISMATCH(s)      ((s) == APR_EMISMATCH)
-/**
- * The given lock was busy
- * @warning always use this test, as platform-specific variances may meet this
- * more than one error code
- */
-#define APR_STATUS_IS_EBUSY(s)          ((s) == APR_EBUSY)
-
-/** @} */
-
-/**
- * @addtogroup APR_Error APR Error Values
- * @{
- */
-/* APR CANONICAL ERROR VALUES */
-/** @see APR_STATUS_IS_EACCES */
-#ifdef EACCES
-#define APR_EACCES EACCES
-#else
-#define APR_EACCES         (APR_OS_START_CANONERR + 1)
-#endif
-
-/** @see APR_STATUS_IS_EXIST */
-#ifdef EEXIST
-#define APR_EEXIST EEXIST
-#else
-#define APR_EEXIST         (APR_OS_START_CANONERR + 2)
-#endif
-
-/** @see APR_STATUS_IS_ENAMETOOLONG */
-#ifdef ENAMETOOLONG
-#define APR_ENAMETOOLONG ENAMETOOLONG
-#else
-#define APR_ENAMETOOLONG   (APR_OS_START_CANONERR + 3)
-#endif
-
-/** @see APR_STATUS_IS_ENOENT */
-#ifdef ENOENT
-#define APR_ENOENT ENOENT
-#else
-#define APR_ENOENT         (APR_OS_START_CANONERR + 4)
-#endif
-
-/** @see APR_STATUS_IS_ENOTDIR */
-#ifdef ENOTDIR
-#define APR_ENOTDIR ENOTDIR
-#else
-#define APR_ENOTDIR        (APR_OS_START_CANONERR + 5)
-#endif
-
-/** @see APR_STATUS_IS_ENOSPC */
-#ifdef ENOSPC
-#define APR_ENOSPC ENOSPC
-#else
-#define APR_ENOSPC         (APR_OS_START_CANONERR + 6)
-#endif
-
-/** @see APR_STATUS_IS_ENOMEM */
-#ifdef ENOMEM
-#define APR_ENOMEM ENOMEM
-#else
-#define APR_ENOMEM         (APR_OS_START_CANONERR + 7)
-#endif
-
-/** @see APR_STATUS_IS_EMFILE */
-#ifdef EMFILE
-#define APR_EMFILE EMFILE
-#else
-#define APR_EMFILE         (APR_OS_START_CANONERR + 8)
-#endif
-
-/** @see APR_STATUS_IS_ENFILE */
-#ifdef ENFILE
-#define APR_ENFILE ENFILE
-#else
-#define APR_ENFILE         (APR_OS_START_CANONERR + 9)
-#endif
-
-/** @see APR_STATUS_IS_EBADF */
-#ifdef EBADF
-#define APR_EBADF EBADF
-#else
-#define APR_EBADF          (APR_OS_START_CANONERR + 10)
-#endif
-
-/** @see APR_STATUS_IS_EINVAL */
-#ifdef EINVAL
-#define APR_EINVAL EINVAL
-#else
-#define APR_EINVAL         (APR_OS_START_CANONERR + 11)
-#endif
-
-/** @see APR_STATUS_IS_ESPIPE */
-#ifdef ESPIPE
-#define APR_ESPIPE ESPIPE
-#else
-#define APR_ESPIPE         (APR_OS_START_CANONERR + 12)
-#endif
-
-/**
- * @see APR_STATUS_IS_EAGAIN
- * @warning use APR_STATUS_IS_EAGAIN instead of just testing this value
- */
-#ifdef EAGAIN
-#define APR_EAGAIN EAGAIN
-#elif defined(EWOULDBLOCK)
-#define APR_EAGAIN EWOULDBLOCK
-#else
-#define APR_EAGAIN         (APR_OS_START_CANONERR + 13)
-#endif
-
-/** @see APR_STATUS_IS_EINTR */
-#ifdef EINTR
-#define APR_EINTR EINTR
-#else
-#define APR_EINTR          (APR_OS_START_CANONERR + 14)
-#endif
-
-/** @see APR_STATUS_IS_ENOTSOCK */
-#ifdef ENOTSOCK
-#define APR_ENOTSOCK ENOTSOCK
-#else
-#define APR_ENOTSOCK       (APR_OS_START_CANONERR + 15)
-#endif
-
-/** @see APR_STATUS_IS_ECONNREFUSED */
-#ifdef ECONNREFUSED
-#define APR_ECONNREFUSED ECONNREFUSED
-#else
-#define APR_ECONNREFUSED   (APR_OS_START_CANONERR + 16)
-#endif
-
-/** @see APR_STATUS_IS_EINPROGRESS */
-#ifdef EINPROGRESS
-#define APR_EINPROGRESS EINPROGRESS
-#else
-#define APR_EINPROGRESS    (APR_OS_START_CANONERR + 17)
-#endif
-
-/**
- * @see APR_STATUS_IS_ECONNABORTED
- * @warning use APR_STATUS_IS_ECONNABORTED instead of just testing this value
- */
-
-#ifdef ECONNABORTED
-#define APR_ECONNABORTED ECONNABORTED
-#else
-#define APR_ECONNABORTED   (APR_OS_START_CANONERR + 18)
-#endif
-
-/** @see APR_STATUS_IS_ECONNRESET */
-#ifdef ECONNRESET
-#define APR_ECONNRESET ECONNRESET
-#else
-#define APR_ECONNRESET     (APR_OS_START_CANONERR + 19)
-#endif
-
-/** @see APR_STATUS_IS_ETIMEDOUT
- *  @deprecated */
-#ifdef ETIMEDOUT
-#define APR_ETIMEDOUT ETIMEDOUT
-#else
-#define APR_ETIMEDOUT      (APR_OS_START_CANONERR + 20)
-#endif
-
-/** @see APR_STATUS_IS_EHOSTUNREACH */
-#ifdef EHOSTUNREACH
-#define APR_EHOSTUNREACH EHOSTUNREACH
-#else
-#define APR_EHOSTUNREACH   (APR_OS_START_CANONERR + 21)
-#endif
-
-/** @see APR_STATUS_IS_ENETUNREACH */
-#ifdef ENETUNREACH
-#define APR_ENETUNREACH ENETUNREACH
-#else
-#define APR_ENETUNREACH    (APR_OS_START_CANONERR + 22)
-#endif
-
-/** @see APR_STATUS_IS_EFTYPE */
-#ifdef EFTYPE
-#define APR_EFTYPE EFTYPE
-#else
-#define APR_EFTYPE        (APR_OS_START_CANONERR + 23)
-#endif
-
-/** @see APR_STATUS_IS_EPIPE */
-#ifdef EPIPE
-#define APR_EPIPE EPIPE
-#else
-#define APR_EPIPE         (APR_OS_START_CANONERR + 24)
-#endif
-
-/** @see APR_STATUS_IS_EXDEV */
-#ifdef EXDEV
-#define APR_EXDEV EXDEV
-#else
-#define APR_EXDEV         (APR_OS_START_CANONERR + 25)
-#endif
-
-/** @see APR_STATUS_IS_ENOTEMPTY */
-#ifdef ENOTEMPTY
-#define APR_ENOTEMPTY ENOTEMPTY
-#else
-#define APR_ENOTEMPTY     (APR_OS_START_CANONERR + 26)
-#endif
-
-/** @} */
-
-#if defined(OS2) && !defined(DOXYGEN)
-
-#define APR_FROM_OS_ERROR(e) (e == 0 ? APR_SUCCESS : e + APR_OS_START_SYSERR)
-#define APR_TO_OS_ERROR(e)   (e == 0 ? APR_SUCCESS : e - APR_OS_START_SYSERR)
-
-#define INCL_DOSERRORS
-#define INCL_DOS
-
-/* Leave these undefined.
- * OS2 doesn't rely on the errno concept.
- * The API calls always return a result codes which
- * should be filtered through APR_FROM_OS_ERROR().
- *
- * #define apr_get_os_error()   (APR_FROM_OS_ERROR(GetLastError()))
- * #define apr_set_os_error(e)  (SetLastError(APR_TO_OS_ERROR(e)))
- */
-
-/* A special case, only socket calls require this;
- */
-#define apr_get_netos_error()   (APR_FROM_OS_ERROR(errno))
-#define apr_set_netos_error(e)  (errno = APR_TO_OS_ERROR(e))
-
-/* And this needs to be greped away for good:
- */
-#define APR_OS2_STATUS(e) (APR_FROM_OS_ERROR(e))
-
-/* These can't sit in a private header, so in spite of the extra size,
- * they need to be made available here.
- */
-#define SOCBASEERR              10000
-#define SOCEPERM                (SOCBASEERR+1)             /* Not owner */
-#define SOCESRCH                (SOCBASEERR+3)             /* No such process */
-#define SOCEINTR                (SOCBASEERR+4)             /* Interrupted system call */
-#define SOCENXIO                (SOCBASEERR+6)             /* No such device or address */
-#define SOCEBADF                (SOCBASEERR+9)             /* Bad file number */
-#define SOCEACCES               (SOCBASEERR+13)            /* Permission denied */
-#define SOCEFAULT               (SOCBASEERR+14)            /* Bad address */
-#define SOCEINVAL               (SOCBASEERR+22)            /* Invalid argument */
-#define SOCEMFILE               (SOCBASEERR+24)            /* Too many open files */
-#define SOCEPIPE                (SOCBASEERR+32)            /* Broken pipe */
-#define SOCEOS2ERR              (SOCBASEERR+100)           /* OS/2 Error */
-#define SOCEWOULDBLOCK          (SOCBASEERR+35)            /* Operation would block */
-#define SOCEINPROGRESS          (SOCBASEERR+36)            /* Operation now in progress */
-#define SOCEALREADY             (SOCBASEERR+37)            /* Operation already in progress */
-#define SOCENOTSOCK             (SOCBASEERR+38)            /* Socket operation on non-socket */
-#define SOCEDESTADDRREQ         (SOCBASEERR+39)            /* Destination address required */
-#define SOCEMSGSIZE             (SOCBASEERR+40)            /* Message too long */
-#define SOCEPROTOTYPE           (SOCBASEERR+41)            /* Protocol wrong type for socket */
-#define SOCENOPROTOOPT          (SOCBASEERR+42)            /* Protocol not available */
-#define SOCEPROTONOSUPPORT      (SOCBASEERR+43)            /* Protocol not supported */
-#define SOCESOCKTNOSUPPORT      (SOCBASEERR+44)            /* Socket type not supported */
-#define SOCEOPNOTSUPP           (SOCBASEERR+45)            /* Operation not supported on socket */
-#define SOCEPFNOSUPPORT         (SOCBASEERR+46)            /* Protocol family not supported */
-#define SOCEAFNOSUPPORT         (SOCBASEERR+47)            /* Address family not supported by protocol family */
-#define SOCEADDRINUSE           (SOCBASEERR+48)            /* Address already in use */
-#define SOCEADDRNOTAVAIL        (SOCBASEERR+49)            /* Can't assign requested address */
-#define SOCENETDOWN             (SOCBASEERR+50)            /* Network is down */
-#define SOCENETUNREACH          (SOCBASEERR+51)            /* Network is unreachable */
-#define SOCENETRESET            (SOCBASEERR+52)            /* Network dropped connection on reset */
-#define SOCECONNABORTED         (SOCBASEERR+53)            /* Software caused connection abort */
-#define SOCECONNRESET           (SOCBASEERR+54)            /* Connection reset by peer */
-#define SOCENOBUFS              (SOCBASEERR+55)            /* No buffer space available */
-#define SOCEISCONN              (SOCBASEERR+56)            /* Socket is already connected */
-#define SOCENOTCONN             (SOCBASEERR+57)            /* Socket is not connected */
-#define SOCESHUTDOWN            (SOCBASEERR+58)            /* Can't send after socket shutdown */
-#define SOCETOOMANYREFS         (SOCBASEERR+59)            /* Too many references: can't splice */
-#define SOCETIMEDOUT            (SOCBASEERR+60)            /* Connection timed out */
-#define SOCECONNREFUSED         (SOCBASEERR+61)            /* Connection refused */
-#define SOCELOOP                (SOCBASEERR+62)            /* Too many levels of symbolic links */
-#define SOCENAMETOOLONG         (SOCBASEERR+63)            /* File name too long */
-#define SOCEHOSTDOWN            (SOCBASEERR+64)            /* Host is down */
-#define SOCEHOSTUNREACH         (SOCBASEERR+65)            /* No route to host */
-#define SOCENOTEMPTY            (SOCBASEERR+66)            /* Directory not empty */
-
-/* APR CANONICAL ERROR TESTS */
-#define APR_STATUS_IS_EACCES(s)         ((s) == APR_EACCES \
-                || (s) == APR_OS_START_SYSERR + ERROR_ACCESS_DENIED \
-                || (s) == APR_OS_START_SYSERR + ERROR_SHARING_VIOLATION)
-#define APR_STATUS_IS_EEXIST(s)         ((s) == APR_EEXIST \
-                || (s) == APR_OS_START_SYSERR + ERROR_OPEN_FAILED \
-                || (s) == APR_OS_START_SYSERR + ERROR_FILE_EXISTS \
-                || (s) == APR_OS_START_SYSERR + ERROR_ALREADY_EXISTS \
-                || (s) == APR_OS_START_SYSERR + ERROR_ACCESS_DENIED)
-#define APR_STATUS_IS_ENAMETOOLONG(s)   ((s) == APR_ENAMETOOLONG \
-                || (s) == APR_OS_START_SYSERR + ERROR_FILENAME_EXCED_RANGE \
-                || (s) == APR_OS_START_SYSERR + SOCENAMETOOLONG)
-#define APR_STATUS_IS_ENOENT(s)         ((s) == APR_ENOENT \
-                || (s) == APR_OS_START_SYSERR + ERROR_FILE_NOT_FOUND \
-                || (s) == APR_OS_START_SYSERR + ERROR_PATH_NOT_FOUND \
-                || (s) == APR_OS_START_SYSERR + ERROR_NO_MORE_FILES \
-                || (s) == APR_OS_START_SYSERR + ERROR_OPEN_FAILED)
-#define APR_STATUS_IS_ENOTDIR(s)        ((s) == APR_ENOTDIR)
-#define APR_STATUS_IS_ENOSPC(s)         ((s) == APR_ENOSPC \
-                || (s) == APR_OS_START_SYSERR + ERROR_DISK_FULL)
-#define APR_STATUS_IS_ENOMEM(s)         ((s) == APR_ENOMEM)
-#define APR_STATUS_IS_EMFILE(s)         ((s) == APR_EMFILE \
-                || (s) == APR_OS_START_SYSERR + ERROR_TOO_MANY_OPEN_FILES)
-#define APR_STATUS_IS_ENFILE(s)         ((s) == APR_ENFILE)
-#define APR_STATUS_IS_EBADF(s)          ((s) == APR_EBADF \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_HANDLE)
-#define APR_STATUS_IS_EINVAL(s)         ((s) == APR_EINVAL \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_PARAMETER \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_FUNCTION)
-#define APR_STATUS_IS_ESPIPE(s)         ((s) == APR_ESPIPE \
-                || (s) == APR_OS_START_SYSERR + ERROR_NEGATIVE_SEEK)
-#define APR_STATUS_IS_EAGAIN(s)         ((s) == APR_EAGAIN \
-                || (s) == APR_OS_START_SYSERR + ERROR_NO_DATA \
-                || (s) == APR_OS_START_SYSERR + SOCEWOULDBLOCK \
-                || (s) == APR_OS_START_SYSERR + ERROR_LOCK_VIOLATION)
-#define APR_STATUS_IS_EINTR(s)          ((s) == APR_EINTR \
-                || (s) == APR_OS_START_SYSERR + SOCEINTR)
-#define APR_STATUS_IS_ENOTSOCK(s)       ((s) == APR_ENOTSOCK \
-                || (s) == APR_OS_START_SYSERR + SOCENOTSOCK)
-#define APR_STATUS_IS_ECONNREFUSED(s)   ((s) == APR_ECONNREFUSED \
-                || (s) == APR_OS_START_SYSERR + SOCECONNREFUSED)
-#define APR_STATUS_IS_EINPROGRESS(s)    ((s) == APR_EINPROGRESS \
-                || (s) == APR_OS_START_SYSERR + SOCEINPROGRESS)
-#define APR_STATUS_IS_ECONNABORTED(s)   ((s) == APR_ECONNABORTED \
-                || (s) == APR_OS_START_SYSERR + SOCECONNABORTED)
-#define APR_STATUS_IS_ECONNRESET(s)     ((s) == APR_ECONNRESET \
-                || (s) == APR_OS_START_SYSERR + SOCECONNRESET)
-/* XXX deprecated */
-#define APR_STATUS_IS_ETIMEDOUT(s)         ((s) == APR_ETIMEDOUT \
-                || (s) == APR_OS_START_SYSERR + SOCETIMEDOUT)
-#undef APR_STATUS_IS_TIMEUP
-#define APR_STATUS_IS_TIMEUP(s)         ((s) == APR_TIMEUP \
-                || (s) == APR_OS_START_SYSERR + SOCETIMEDOUT)
-#define APR_STATUS_IS_EHOSTUNREACH(s)   ((s) == APR_EHOSTUNREACH \
-                || (s) == APR_OS_START_SYSERR + SOCEHOSTUNREACH)
-#define APR_STATUS_IS_ENETUNREACH(s)    ((s) == APR_ENETUNREACH \
-                || (s) == APR_OS_START_SYSERR + SOCENETUNREACH)
-#define APR_STATUS_IS_EFTYPE(s)         ((s) == APR_EFTYPE)
-#define APR_STATUS_IS_EPIPE(s)          ((s) == APR_EPIPE \
-                || (s) == APR_OS_START_SYSERR + ERROR_BROKEN_PIPE \
-                || (s) == APR_OS_START_SYSERR + SOCEPIPE)
-#define APR_STATUS_IS_EXDEV(s)          ((s) == APR_EXDEV \
-                || (s) == APR_OS_START_SYSERR + ERROR_NOT_SAME_DEVICE)
-#define APR_STATUS_IS_ENOTEMPTY(s)      ((s) == APR_ENOTEMPTY \
-                || (s) == APR_OS_START_SYSERR + ERROR_DIR_NOT_EMPTY \
-                || (s) == APR_OS_START_SYSERR + ERROR_ACCESS_DENIED)
-
-/*
-    Sorry, too tired to wrap this up for OS2... feel free to
-    fit the following into their best matches.
-
-    { ERROR_NO_SIGNAL_SENT,     ESRCH           },
-    { SOCEALREADY,              EALREADY        },
-    { SOCEDESTADDRREQ,          EDESTADDRREQ    },
-    { SOCEMSGSIZE,              EMSGSIZE        },
-    { SOCEPROTOTYPE,            EPROTOTYPE      },
-    { SOCENOPROTOOPT,           ENOPROTOOPT     },
-    { SOCEPROTONOSUPPORT,       EPROTONOSUPPORT },
-    { SOCESOCKTNOSUPPORT,       ESOCKTNOSUPPORT },
-    { SOCEOPNOTSUPP,            EOPNOTSUPP      },
-    { SOCEPFNOSUPPORT,          EPFNOSUPPORT    },
-    { SOCEAFNOSUPPORT,          EAFNOSUPPORT    },
-    { SOCEADDRINUSE,            EADDRINUSE      },
-    { SOCEADDRNOTAVAIL,         EADDRNOTAVAIL   },
-    { SOCENETDOWN,              ENETDOWN        },
-    { SOCENETRESET,             ENETRESET       },
-    { SOCENOBUFS,               ENOBUFS         },
-    { SOCEISCONN,               EISCONN         },
-    { SOCENOTCONN,              ENOTCONN        },
-    { SOCESHUTDOWN,             ESHUTDOWN       },
-    { SOCETOOMANYREFS,          ETOOMANYREFS    },
-    { SOCELOOP,                 ELOOP           },
-    { SOCEHOSTDOWN,             EHOSTDOWN       },
-    { SOCENOTEMPTY,             ENOTEMPTY       },
-    { SOCEPIPE,                 EPIPE           }
-*/
-
-#elif defined(WIN32) && !defined(DOXYGEN) /* !defined(OS2) */
-
-#define APR_FROM_OS_ERROR(e) (e == 0 ? APR_SUCCESS : e + APR_OS_START_SYSERR)
-#define APR_TO_OS_ERROR(e)   (e == 0 ? APR_SUCCESS : e - APR_OS_START_SYSERR)
-
-#define apr_get_os_error()   (APR_FROM_OS_ERROR(GetLastError()))
-#define apr_set_os_error(e)  (SetLastError(APR_TO_OS_ERROR(e)))
-
-/* A special case, only socket calls require this:
- */
-#define apr_get_netos_error()   (APR_FROM_OS_ERROR(WSAGetLastError()))
-#define apr_set_netos_error(e)   (WSASetLastError(APR_TO_OS_ERROR(e)))
-
-/* APR CANONICAL ERROR TESTS */
-#define APR_STATUS_IS_EACCES(s)         ((s) == APR_EACCES \
-                || (s) == APR_OS_START_SYSERR + ERROR_ACCESS_DENIED \
-                || (s) == APR_OS_START_SYSERR + ERROR_CANNOT_MAKE \
-                || (s) == APR_OS_START_SYSERR + ERROR_CURRENT_DIRECTORY \
-                || (s) == APR_OS_START_SYSERR + ERROR_DRIVE_LOCKED \
-                || (s) == APR_OS_START_SYSERR + ERROR_FAIL_I24 \
-                || (s) == APR_OS_START_SYSERR + ERROR_LOCK_VIOLATION \
-                || (s) == APR_OS_START_SYSERR + ERROR_LOCK_FAILED \
-                || (s) == APR_OS_START_SYSERR + ERROR_NOT_LOCKED \
-                || (s) == APR_OS_START_SYSERR + ERROR_NETWORK_ACCESS_DENIED \
-                || (s) == APR_OS_START_SYSERR + ERROR_SHARING_VIOLATION)
-#define APR_STATUS_IS_EEXIST(s)         ((s) == APR_EEXIST \
-                || (s) == APR_OS_START_SYSERR + ERROR_FILE_EXISTS \
-                || (s) == APR_OS_START_SYSERR + ERROR_ALREADY_EXISTS)
-#define APR_STATUS_IS_ENAMETOOLONG(s)   ((s) == APR_ENAMETOOLONG \
-                || (s) == APR_OS_START_SYSERR + ERROR_FILENAME_EXCED_RANGE \
-                || (s) == APR_OS_START_SYSERR + WSAENAMETOOLONG)
-#define APR_STATUS_IS_ENOENT(s)         ((s) == APR_ENOENT \
-                || (s) == APR_OS_START_SYSERR + ERROR_FILE_NOT_FOUND \
-                || (s) == APR_OS_START_SYSERR + ERROR_PATH_NOT_FOUND \
-                || (s) == APR_OS_START_SYSERR + ERROR_OPEN_FAILED \
-                || (s) == APR_OS_START_SYSERR + ERROR_NO_MORE_FILES)
-#define APR_STATUS_IS_ENOTDIR(s)        ((s) == APR_ENOTDIR \
-                || (s) == APR_OS_START_SYSERR + ERROR_PATH_NOT_FOUND \
-                || (s) == APR_OS_START_SYSERR + ERROR_BAD_NETPATH \
-                || (s) == APR_OS_START_SYSERR + ERROR_BAD_NET_NAME \
-                || (s) == APR_OS_START_SYSERR + ERROR_BAD_PATHNAME \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_DRIVE)
-#define APR_STATUS_IS_ENOSPC(s)         ((s) == APR_ENOSPC \
-                || (s) == APR_OS_START_SYSERR + ERROR_DISK_FULL)
-#define APR_STATUS_IS_ENOMEM(s)         ((s) == APR_ENOMEM \
-                || (s) == APR_OS_START_SYSERR + ERROR_ARENA_TRASHED \
-                || (s) == APR_OS_START_SYSERR + ERROR_NOT_ENOUGH_MEMORY \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_BLOCK \
-                || (s) == APR_OS_START_SYSERR + ERROR_NOT_ENOUGH_QUOTA \
-                || (s) == APR_OS_START_SYSERR + ERROR_OUTOFMEMORY)
-#define APR_STATUS_IS_EMFILE(s)         ((s) == APR_EMFILE \
-                || (s) == APR_OS_START_SYSERR + ERROR_TOO_MANY_OPEN_FILES)
-#define APR_STATUS_IS_ENFILE(s)         ((s) == APR_ENFILE)
-#define APR_STATUS_IS_EBADF(s)          ((s) == APR_EBADF \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_HANDLE \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_TARGET_HANDLE)
-#define APR_STATUS_IS_EINVAL(s)         ((s) == APR_EINVAL \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_ACCESS \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_DATA \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_FUNCTION \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_HANDLE \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_PARAMETER \
-                || (s) == APR_OS_START_SYSERR + ERROR_NEGATIVE_SEEK)
-#define APR_STATUS_IS_ESPIPE(s)         ((s) == APR_ESPIPE \
-                || (s) == APR_OS_START_SYSERR + ERROR_SEEK_ON_DEVICE \
-                || (s) == APR_OS_START_SYSERR + ERROR_NEGATIVE_SEEK)
-#define APR_STATUS_IS_EAGAIN(s)         ((s) == APR_EAGAIN \
-                || (s) == APR_OS_START_SYSERR + ERROR_NO_DATA \
-                || (s) == APR_OS_START_SYSERR + ERROR_NO_PROC_SLOTS \
-                || (s) == APR_OS_START_SYSERR + ERROR_NESTING_NOT_ALLOWED \
-                || (s) == APR_OS_START_SYSERR + ERROR_MAX_THRDS_REACHED \
-                || (s) == APR_OS_START_SYSERR + ERROR_LOCK_VIOLATION \
-                || (s) == APR_OS_START_SYSERR + WSAEWOULDBLOCK)
-#define APR_STATUS_IS_EINTR(s)          ((s) == APR_EINTR \
-                || (s) == APR_OS_START_SYSERR + WSAEINTR)
-#define APR_STATUS_IS_ENOTSOCK(s)       ((s) == APR_ENOTSOCK \
-                || (s) == APR_OS_START_SYSERR + WSAENOTSOCK)
-#define APR_STATUS_IS_ECONNREFUSED(s)   ((s) == APR_ECONNREFUSED \
-                || (s) == APR_OS_START_SYSERR + WSAECONNREFUSED)
-#define APR_STATUS_IS_EINPROGRESS(s)    ((s) == APR_EINPROGRESS \
-                || (s) == APR_OS_START_SYSERR + WSAEINPROGRESS)
-#define APR_STATUS_IS_ECONNABORTED(s)   ((s) == APR_ECONNABORTED \
-                || (s) == APR_OS_START_SYSERR + WSAECONNABORTED)
-#define APR_STATUS_IS_ECONNRESET(s)     ((s) == APR_ECONNRESET \
-                || (s) == APR_OS_START_SYSERR + ERROR_NETNAME_DELETED \
-                || (s) == APR_OS_START_SYSERR + WSAECONNRESET)
-/* XXX deprecated */
-#define APR_STATUS_IS_ETIMEDOUT(s)         ((s) == APR_ETIMEDOUT \
-                || (s) == APR_OS_START_SYSERR + WSAETIMEDOUT \
-                || (s) == APR_OS_START_SYSERR + WAIT_TIMEOUT)
-#undef APR_STATUS_IS_TIMEUP
-#define APR_STATUS_IS_TIMEUP(s)         ((s) == APR_TIMEUP \
-                || (s) == APR_OS_START_SYSERR + WSAETIMEDOUT \
-                || (s) == APR_OS_START_SYSERR + WAIT_TIMEOUT)
-#define APR_STATUS_IS_EHOSTUNREACH(s)   ((s) == APR_EHOSTUNREACH \
-                || (s) == APR_OS_START_SYSERR + WSAEHOSTUNREACH)
-#define APR_STATUS_IS_ENETUNREACH(s)    ((s) == APR_ENETUNREACH \
-                || (s) == APR_OS_START_SYSERR + WSAENETUNREACH)
-#define APR_STATUS_IS_EFTYPE(s)         ((s) == APR_EFTYPE \
-                || (s) == APR_OS_START_SYSERR + ERROR_EXE_MACHINE_TYPE_MISMATCH \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_DLL \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_MODULETYPE \
-                || (s) == APR_OS_START_SYSERR + ERROR_BAD_EXE_FORMAT \
-                || (s) == APR_OS_START_SYSERR + ERROR_INVALID_EXE_SIGNATURE \
-                || (s) == APR_OS_START_SYSERR + ERROR_FILE_CORRUPT \
-                || (s) == APR_OS_START_SYSERR + ERROR_BAD_FORMAT)
-#define APR_STATUS_IS_EPIPE(s)          ((s) == APR_EPIPE \
-                || (s) == APR_OS_START_SYSERR + ERROR_BROKEN_PIPE)
-#define APR_STATUS_IS_EXDEV(s)          ((s) == APR_EXDEV \
-                || (s) == APR_OS_START_SYSERR + ERROR_NOT_SAME_DEVICE)
-#define APR_STATUS_IS_ENOTEMPTY(s)      ((s) == APR_ENOTEMPTY \
-                || (s) == APR_OS_START_SYSERR + ERROR_DIR_NOT_EMPTY)
-
-#elif defined(NETWARE) && defined(USE_WINSOCK) && !defined(DOXYGEN) /* !defined(OS2) && !defined(WIN32) */
-
-#define APR_FROM_OS_ERROR(e) (e == 0 ? APR_SUCCESS : e + APR_OS_START_SYSERR)
-#define APR_TO_OS_ERROR(e)   (e == 0 ? APR_SUCCESS : e - APR_OS_START_SYSERR)
-
-#define apr_get_os_error()    (errno)
-#define apr_set_os_error(e)   (errno = (e))
-
-/* A special case, only socket calls require this: */
-#define apr_get_netos_error()   (APR_FROM_OS_ERROR(WSAGetLastError()))
-#define apr_set_netos_error(e)  (WSASetLastError(APR_TO_OS_ERROR(e)))
-
-/* APR CANONICAL ERROR TESTS */
-#define APR_STATUS_IS_EACCES(s)         ((s) == APR_EACCES)
-#define APR_STATUS_IS_EEXIST(s)         ((s) == APR_EEXIST)
-#define APR_STATUS_IS_ENAMETOOLONG(s)   ((s) == APR_ENAMETOOLONG)
-#define APR_STATUS_IS_ENOENT(s)         ((s) == APR_ENOENT)
-#define APR_STATUS_IS_ENOTDIR(s)        ((s) == APR_ENOTDIR)
-#define APR_STATUS_IS_ENOSPC(s)         ((s) == APR_ENOSPC)
-#define APR_STATUS_IS_ENOMEM(s)         ((s) == APR_ENOMEM)
-#define APR_STATUS_IS_EMFILE(s)         ((s) == APR_EMFILE)
-#define APR_STATUS_IS_ENFILE(s)         ((s) == APR_ENFILE)
-#define APR_STATUS_IS_EBADF(s)          ((s) == APR_EBADF)
-#define APR_STATUS_IS_EINVAL(s)         ((s) == APR_EINVAL)
-#define APR_STATUS_IS_ESPIPE(s)         ((s) == APR_ESPIPE)
-
-#define APR_STATUS_IS_EAGAIN(s)         ((s) == APR_EAGAIN \
-                || (s) ==                       EWOULDBLOCK \
-                || (s) == APR_OS_START_SYSERR + WSAEWOULDBLOCK)
-#define APR_STATUS_IS_EINTR(s)          ((s) == APR_EINTR \
-                || (s) == APR_OS_START_SYSERR + WSAEINTR)
-#define APR_STATUS_IS_ENOTSOCK(s)       ((s) == APR_ENOTSOCK \
-                || (s) == APR_OS_START_SYSERR + WSAENOTSOCK)
-#define APR_STATUS_IS_ECONNREFUSED(s)   ((s) == APR_ECONNREFUSED \
-                || (s) == APR_OS_START_SYSERR + WSAECONNREFUSED)
-#define APR_STATUS_IS_EINPROGRESS(s)    ((s) == APR_EINPROGRESS \
-                || (s) == APR_OS_START_SYSERR + WSAEINPROGRESS)
-#define APR_STATUS_IS_ECONNABORTED(s)   ((s) == APR_ECONNABORTED \
-                || (s) == APR_OS_START_SYSERR + WSAECONNABORTED)
-#define APR_STATUS_IS_ECONNRESET(s)     ((s) == APR_ECONNRESET \
-                || (s) == APR_OS_START_SYSERR + WSAECONNRESET)
-/* XXX deprecated */
-#define APR_STATUS_IS_ETIMEDOUT(s)       ((s) == APR_ETIMEDOUT \
-                || (s) == APR_OS_START_SYSERR + WSAETIMEDOUT \
-                || (s) == APR_OS_START_SYSERR + WAIT_TIMEOUT)
-#undef APR_STATUS_IS_TIMEUP
-#define APR_STATUS_IS_TIMEUP(s)         ((s) == APR_TIMEUP \
-                || (s) == APR_OS_START_SYSERR + WSAETIMEDOUT \
-                || (s) == APR_OS_START_SYSERR + WAIT_TIMEOUT)
-#define APR_STATUS_IS_EHOSTUNREACH(s)   ((s) == APR_EHOSTUNREACH \
-                || (s) == APR_OS_START_SYSERR + WSAEHOSTUNREACH)
-#define APR_STATUS_IS_ENETUNREACH(s)    ((s) == APR_ENETUNREACH \
-                || (s) == APR_OS_START_SYSERR + WSAENETUNREACH)
-#define APR_STATUS_IS_ENETDOWN(s)       ((s) == APR_OS_START_SYSERR + WSAENETDOWN)
-#define APR_STATUS_IS_EFTYPE(s)         ((s) == APR_EFTYPE)
-#define APR_STATUS_IS_EPIPE(s)          ((s) == APR_EPIPE)
-#define APR_STATUS_IS_EXDEV(s)          ((s) == APR_EXDEV)
-#define APR_STATUS_IS_ENOTEMPTY(s)      ((s) == APR_ENOTEMPTY)
-
-#else /* !defined(NETWARE) && !defined(OS2) && !defined(WIN32) */
-
-/*
- *  os error codes are clib error codes
- */
-#define APR_FROM_OS_ERROR(e)  (e)
-#define APR_TO_OS_ERROR(e)    (e)
-
-#define apr_get_os_error()    (errno)
-#define apr_set_os_error(e)   (errno = (e))
-
-/* A special case, only socket calls require this:
- */
-#define apr_get_netos_error() (errno)
-#define apr_set_netos_error(e) (errno = (e))
-
-/**
- * @addtogroup APR_STATUS_IS
- * @{
- */
-
-/** permission denied */
-#define APR_STATUS_IS_EACCES(s)         ((s) == APR_EACCES)
-/** file exists */
-#define APR_STATUS_IS_EEXIST(s)         ((s) == APR_EEXIST)
-/** path name is too long */
-#define APR_STATUS_IS_ENAMETOOLONG(s)   ((s) == APR_ENAMETOOLONG)
-/**
- * no such file or directory
- * @remark
- * EMVSCATLG can be returned by the automounter on z/OS for
- * paths which do not exist.
- */
-#ifdef EMVSCATLG
-#define APR_STATUS_IS_ENOENT(s)         ((s) == APR_ENOENT \
-                                      || (s) == EMVSCATLG)
-#else
-#define APR_STATUS_IS_ENOENT(s)         ((s) == APR_ENOENT)
-#endif
-/** not a directory */
-#define APR_STATUS_IS_ENOTDIR(s)        ((s) == APR_ENOTDIR)
-/** no space left on device */
-#ifdef EDQUOT
-#define APR_STATUS_IS_ENOSPC(s)         ((s) == APR_ENOSPC \
-                                      || (s) == EDQUOT)
-#else
-#define APR_STATUS_IS_ENOSPC(s)         ((s) == APR_ENOSPC)
-#endif
-/** not enough memory */
-#define APR_STATUS_IS_ENOMEM(s)         ((s) == APR_ENOMEM)
-/** too many open files */
-#define APR_STATUS_IS_EMFILE(s)         ((s) == APR_EMFILE)
-/** file table overflow */
-#define APR_STATUS_IS_ENFILE(s)         ((s) == APR_ENFILE)
-/** bad file # */
-#define APR_STATUS_IS_EBADF(s)          ((s) == APR_EBADF)
-/** invalid argument */
-#define APR_STATUS_IS_EINVAL(s)         ((s) == APR_EINVAL)
-/** illegal seek */
-#define APR_STATUS_IS_ESPIPE(s)         ((s) == APR_ESPIPE)
-
-/** operation would block */
-#if !defined(EWOULDBLOCK) || !defined(EAGAIN)
-#define APR_STATUS_IS_EAGAIN(s)         ((s) == APR_EAGAIN)
-#elif (EWOULDBLOCK == EAGAIN)
-#define APR_STATUS_IS_EAGAIN(s)         ((s) == APR_EAGAIN)
-#else
-#define APR_STATUS_IS_EAGAIN(s)         ((s) == APR_EAGAIN \
-                                      || (s) == EWOULDBLOCK)
-#endif
-
-/** interrupted system call */
-#define APR_STATUS_IS_EINTR(s)          ((s) == APR_EINTR)
-/** socket operation on a non-socket */
-#define APR_STATUS_IS_ENOTSOCK(s)       ((s) == APR_ENOTSOCK)
-/** Connection Refused */
-#define APR_STATUS_IS_ECONNREFUSED(s)   ((s) == APR_ECONNREFUSED)
-/** operation now in progress */
-#define APR_STATUS_IS_EINPROGRESS(s)    ((s) == APR_EINPROGRESS)
-
-/**
- * Software caused connection abort
- * @remark
- * EPROTO on certain older kernels really means ECONNABORTED, so we need to
- * ignore it for them.  See discussion in new-httpd archives nh.9701 & nh.9603
- *
- * There is potentially a bug in Solaris 2.x x<6, and other boxes that
- * implement tcp sockets in userland (i.e. on top of STREAMS).  On these
- * systems, EPROTO can actually result in a fatal loop.  See PR#981 for
- * example.  It's hard to handle both uses of EPROTO.
- */
-#ifdef EPROTO
-#define APR_STATUS_IS_ECONNABORTED(s)    ((s) == APR_ECONNABORTED \
-                                       || (s) == EPROTO)
-#else
-#define APR_STATUS_IS_ECONNABORTED(s)    ((s) == APR_ECONNABORTED)
-#endif
-
-/** Connection Reset by peer */
-#define APR_STATUS_IS_ECONNRESET(s)      ((s) == APR_ECONNRESET)
-/** Operation timed out
- *  @deprecated */
-#define APR_STATUS_IS_ETIMEDOUT(s)      ((s) == APR_ETIMEDOUT)
-/** no route to host */
-#define APR_STATUS_IS_EHOSTUNREACH(s)    ((s) == APR_EHOSTUNREACH)
-/** network is unreachable */
-#define APR_STATUS_IS_ENETUNREACH(s)     ((s) == APR_ENETUNREACH)
-/** inappropriate file type or format */
-#define APR_STATUS_IS_EFTYPE(s)          ((s) == APR_EFTYPE)
-/** broken pipe */
-#define APR_STATUS_IS_EPIPE(s)           ((s) == APR_EPIPE)
-/** cross device link */
-#define APR_STATUS_IS_EXDEV(s)           ((s) == APR_EXDEV)
-/** Directory Not Empty */
-#define APR_STATUS_IS_ENOTEMPTY(s)       ((s) == APR_ENOTEMPTY || \
-                                          (s) == APR_EEXIST)
-/** @} */
-
-#endif /* !defined(NETWARE) && !defined(OS2) && !defined(WIN32) */
-
-/** @} */
-
-#ifdef __cplusplus
-}
-#endif
-
-#endif  /* ! APR_ERRNO_H */
-/*
- * Copyright (c) 2000 Apple Computer, Inc. All rights reserved.
- *
- * @APPLE_LICENSE_HEADER_START@
- *
- * This file contains Original Code and/or Modifications of Original Code
- * as defined in and that are subject to the Apple Public Source License
- * Version 2.0 (the 'License'). You may not use this file except in
- * compliance with the License. Please obtain a copy of the License at
- * http://www.opensource.apple.com/apsl/ and read it before using this
- * file.
- *
- * The Original Code and all software distributed under the License are
- * distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER
- * EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
- * INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT.
- * Please see the License for the specific language governing rights and
- * limitations under the License.
- *
- * @APPLE_LICENSE_HEADER_END@
- */
-#include <sys/errno.h>
-
-
-/*
- * Copyright (c) 2000-2002 Apple Computer, Inc. All rights reserved.
- *
- * @APPLE_OSREFERENCE_LICENSE_HEADER_START@
- *
- * This file contains Original Code and/or Modifications of Original Code
- * as defined in and that are subject to the Apple Public Source License
- * Version 2.0 (the 'License'). You may not use this file except in
- * compliance with the License. The rights granted to you under the License
- * may not be used to create, or enable the creation or redistribution of,
- * unlawful or unlicensed copies of an Apple operating system, or to
- * circumvent, violate, or enable the circumvention or violation of, any
- * terms of an Apple operating system software license agreement.
- *
- * Please obtain a copy of the License at
- * http://www.opensource.apple.com/apsl/ and read it before using this file.
- *
- * The Original Code and all software distributed under the License are
- * distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER
- * EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
- * INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT.
- * Please see the License for the specific language governing rights and
- * limitations under the License.
- *
- * @APPLE_OSREFERENCE_LICENSE_HEADER_END@
- */
-/* Copyright (c) 1995 NeXT Computer, Inc. All Rights Reserved */
-/*
- * Copyright (c) 1982, 1986, 1989, 1993
- *	The Regents of the University of California.  All rights reserved.
- * (c) UNIX System Laboratories, Inc.
- * All or some portions of this file are derived from material licensed
- * to the University of California by American Telephone and Telegraph
- * Co. or Unix System Laboratories, Inc. and are reproduced herein with
- * the permission of UNIX System Laboratories, Inc.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- * 3. All advertising materials mentioning features or use of this software
- *    must display the following acknowledgement:
- *	This product includes software developed by the University of
- *	California, Berkeley and its contributors.
- * 4. Neither the name of the University nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- *	@(#)errno.h	8.5 (Berkeley) 1/21/94
- */
-
-#ifndef	_SYS_ERRNO_H_
-#define	_SYS_ERRNO_H_
-
-#include <sys/cdefs.h>
-__BEGIN_DECLS
-extern int * __error(void);
-#define errno (*__error())
-__END_DECLS
-
-/*
- * Error codes
- */
-
-#define	EPERM		1		/* Operation not permitted */
-#define	ENOENT		2		/* No such file or directory */
-#define	ESRCH		3		/* No such process */
-#define	EINTR		4		/* Interrupted system call */
-#define	EIO		5		/* Input/output error */
-#define	ENXIO		6		/* Device not configured */
-#define	E2BIG		7		/* Argument list too long */
-#define	ENOEXEC		8		/* Exec format error */
-#define	EBADF		9		/* Bad file descriptor */
-#define	ECHILD		10		/* No child processes */
-#define	EDEADLK		11		/* Resource deadlock avoided */
-					/* 11 was EAGAIN */
-#define	ENOMEM		12		/* Cannot allocate memory */
-#define	EACCES		13		/* Permission denied */
-#define	EFAULT		14		/* Bad address */
-#if !defined(_POSIX_C_SOURCE) || defined(_DARWIN_C_SOURCE)
-#define	ENOTBLK		15		/* Block device required */
-#endif
-#define	EBUSY		16		/* Device / Resource busy */
-#define	EEXIST		17		/* File exists */
-#define	EXDEV		18		/* Cross-device link */
-#define	ENODEV		19		/* Operation not supported by device */
-#define	ENOTDIR		20		/* Not a directory */
-#define	EISDIR		21		/* Is a directory */
-#define	EINVAL		22		/* Invalid argument */
-#define	ENFILE		23		/* Too many open files in system */
-#define	EMFILE		24		/* Too many open files */
-#define	ENOTTY		25		/* Inappropriate ioctl for device */
-#define	ETXTBSY		26		/* Text file busy */
-#define	EFBIG		27		/* File too large */
-#define	ENOSPC		28		/* No space left on device */
-#define	ESPIPE		29		/* Illegal seek */
-#define	EROFS		30		/* Read-only file system */
-#define	EMLINK		31		/* Too many links */
-#define	EPIPE		32		/* Broken pipe */
-
-/* math software */
-#define	EDOM		33		/* Numerical argument out of domain */
-#define	ERANGE		34		/* Result too large */
-
-/* non-blocking and interrupt i/o */
-#define	EAGAIN		35		/* Resource temporarily unavailable */
-#define	EWOULDBLOCK	EAGAIN		/* Operation would block */
-#define	EINPROGRESS	36		/* Operation now in progress */
-#define	EALREADY	37		/* Operation already in progress */
-
-/* ipc/network software -- argument errors */
-#define	ENOTSOCK	38		/* Socket operation on non-socket */
-#define	EDESTADDRREQ	39		/* Destination address required */
-#define	EMSGSIZE	40		/* Message too long */
-#define	EPROTOTYPE	41		/* Protocol wrong type for socket */
-#define	ENOPROTOOPT	42		/* Protocol not available */
-#define	EPROTONOSUPPORT	43		/* Protocol not supported */
-#if !defined(_POSIX_C_SOURCE) || defined(_DARWIN_C_SOURCE)
-#define	ESOCKTNOSUPPORT	44		/* Socket type not supported */
-#endif /* (!_POSIX_C_SOURCE || _DARWIN_C_SOURCE) */
-#define ENOTSUP		45		/* Operation not supported */
-#if !__DARWIN_UNIX03 && !defined(KERNEL)
-/*
- * This is the same for binary and source copmpatability, unless compiling
- * the kernel itself, or compiling __DARWIN_UNIX03; if compiling for the
- * kernel, the correct value will be returned.  If compiling non-POSIX
- * source, the kernel return value will be converted by a stub in libc, and
- * if compiling source with __DARWIN_UNIX03, the conversion in libc is not
- * done, and the caller gets the expected (discrete) value.
- */
-#define	EOPNOTSUPP	 ENOTSUP	/* Operation not supported on socket */
-#endif /* !__DARWIN_UNIX03 && !KERNEL */
-
-#if !defined(_POSIX_C_SOURCE) || defined(_DARWIN_C_SOURCE)
-#define	EPFNOSUPPORT	46		/* Protocol family not supported */
-#endif /* (_POSIX_C_SOURCE && !_DARWIN_C_SOURCE) */
-#define	EAFNOSUPPORT	47		/* Address family not supported by protocol family */
-#define	EADDRINUSE	48		/* Address already in use */
-#define	EADDRNOTAVAIL	49		/* Can't assign requested address */
-
-/* ipc/network software -- operational errors */
-#define	ENETDOWN	50		/* Network is down */
-#define	ENETUNREACH	51		/* Network is unreachable */
-#define	ENETRESET	52		/* Network dropped connection on reset */
-#define	ECONNABORTED	53		/* Software caused connection abort */
-#define	ECONNRESET	54		/* Connection reset by peer */
-#define	ENOBUFS		55		/* No buffer space available */
-#define	EISCONN		56		/* Socket is already connected */
-#define	ENOTCONN	57		/* Socket is not connected */
-#if !defined(_POSIX_C_SOURCE) || defined(_DARWIN_C_SOURCE)
-#define	ESHUTDOWN	58		/* Can't send after socket shutdown */
-#define	ETOOMANYREFS	59		/* Too many references: can't splice */
-#endif /* (_POSIX_C_SOURCE && !_DARWIN_C_SOURCE) */
-#define	ETIMEDOUT	60		/* Operation timed out */
-#define	ECONNREFUSED	61		/* Connection refused */
-
-#define	ELOOP		62		/* Too many levels of symbolic links */
-#define	ENAMETOOLONG	63		/* File name too long */
-
-/* should be rearranged */
-#if !defined(_POSIX_C_SOURCE) || defined(_DARWIN_C_SOURCE)
-#define	EHOSTDOWN	64		/* Host is down */
-#endif /* (_POSIX_C_SOURCE && !_DARWIN_C_SOURCE) */
-#define	EHOSTUNREACH	65		/* No route to host */
-#define	ENOTEMPTY	66		/* Directory not empty */
-
-/* quotas & mush */
-#if !defined(_POSIX_C_SOURCE) || defined(_DARWIN_C_SOURCE)
-#define	EPROCLIM	67		/* Too many processes */
-#define	EUSERS		68		/* Too many users */
-#endif /* (_POSIX_C_SOURCE && !_DARWIN_C_SOURCE) */
-#define	EDQUOT		69		/* Disc quota exceeded */
-
-/* Network File System */
-#define	ESTALE		70		/* Stale NFS file handle */
-#if !defined(_POSIX_C_SOURCE) || defined(_DARWIN_C_SOURCE)
-#define	EREMOTE		71		/* Too many levels of remote in path */
-#define	EBADRPC		72		/* RPC struct is bad */
-#define	ERPCMISMATCH	73		/* RPC version wrong */
-#define	EPROGUNAVAIL	74		/* RPC prog. not avail */
-#define	EPROGMISMATCH	75		/* Program version wrong */
-#define	EPROCUNAVAIL	76		/* Bad procedure for program */
-#endif /* (_POSIX_C_SOURCE && !_DARWIN_C_SOURCE) */
-
-#define	ENOLCK		77		/* No locks available */
-#define	ENOSYS		78		/* Function not implemented */
-
-#if !defined(_POSIX_C_SOURCE) || defined(_DARWIN_C_SOURCE)
-#define	EFTYPE		79		/* Inappropriate file type or format */
-#define	EAUTH		80		/* Authentication error */
-#define	ENEEDAUTH	81		/* Need authenticator */
-
-/* Intelligent device errors */
-#define	EPWROFF		82	/* Device power is off */
-#define	EDEVERR		83	/* Device error, e.g. paper out */
-#endif /* (_POSIX_C_SOURCE && !_DARWIN_C_SOURCE) */
-
-#define	EOVERFLOW	84		/* Value too large to be stored in data type */
-
-/* Program loading errors */
-#if !defined(_POSIX_C_SOURCE) || defined(_DARWIN_C_SOURCE)
-#define EBADEXEC	85	/* Bad executable */
-#define EBADARCH	86	/* Bad CPU type in executable */
-#define ESHLIBVERS	87	/* Shared library version mismatch */
-#define EBADMACHO	88	/* Malformed Macho file */
-#endif /* (_POSIX_C_SOURCE && !_DARWIN_C_SOURCE) */
-
-#define	ECANCELED	89		/* Operation canceled */
-
-#define EIDRM		90		/* Identifier removed */
-#define ENOMSG		91		/* No message of desired type */
-#define EILSEQ		92		/* Illegal byte sequence */
-#if !defined(_POSIX_C_SOURCE) || defined(_DARWIN_C_SOURCE)
-#define ENOATTR		93		/* Attribute not found */
-#endif /* (_POSIX_C_SOURCE && !_DARWIN_C_SOURCE) */
-
-#define EBADMSG		94		/* Bad message */
-#define EMULTIHOP	95		/* Reserved */
-#define	ENODATA		96		/* No message available on STREAM */
-#define ENOLINK		97		/* Reserved */
-#define ENOSR		98		/* No STREAM resources */
-#define ENOSTR		99		/* Not a STREAM */
-#define	EPROTO		100		/* Protocol error */
-#define ETIME		101		/* STREAM ioctl timeout */
-
-#if __DARWIN_UNIX03 || defined(KERNEL)
-/* This value is only discrete when compiling __DARWIN_UNIX03, or KERNEL */
-#define	EOPNOTSUPP	102		/* Operation not supported on socket */
-#endif /* __DARWIN_UNIX03 || KERNEL */
-
-#define ENOPOLICY	103		/* No such policy registered */
-
-#if !defined(_POSIX_C_SOURCE) || defined(_DARWIN_C_SOURCE)
-#define	ELAST		103		/* Must be equal largest errno */
-#endif /* (_POSIX_C_SOURCE && !_DARWIN_C_SOURCE) */
-
-#endif /* _SYS_ERRNO_H_ */
diff --git a/doc/legacy/errno.list.solaris.txt b/doc/legacy/errno.list.solaris.txt
deleted file mode 100644
index 23601e9d374..00000000000
--- a/doc/legacy/errno.list.solaris.txt
+++ /dev/null
@@ -1,206 +0,0 @@
-/*
- * CDDL HEADER START
- *
- * The contents of this file are subject to the terms of the
- * Common Development and Distribution License, Version 1.0 only
- * (the "License").  You may not use this file except in compliance
- * with the License.
- *
- * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
- * or http://www.opensolaris.org/os/licensing.
- * See the License for the specific language governing permissions
- * and limitations under the License.
- *
- * When distributing Covered Code, include this CDDL HEADER in each
- * file and include the License file at usr/src/OPENSOLARIS.LICENSE.
- * If applicable, add the following below this CDDL HEADER, with the
- * fields enclosed by brackets "[]" replaced with your own identifying
- * information: Portions Copyright [yyyy] [name of copyright owner]
- *
- * CDDL HEADER END
- */
-/*
- * Copyright 2000 Sun Microsystems, Inc.  All rights reserved.
- * Use is subject to license terms.
- */
-
-/*	Copyright (c) 1983, 1984, 1985, 1986, 1987, 1988, 1989 AT&T	*/
-/*	  All Rights Reserved  	*/
-
-/*
- * University Copyright- Copyright (c) 1982, 1986, 1988
- * The Regents of the University of California
- * All Rights Reserved
- *
- * University Acknowledgment- Portions of this document are derived from
- * software developed by the University of California, Berkeley, and its
- * contributors.
- */
-
-#ifndef _SYS_ERRNO_H
-#define	_SYS_ERRNO_H
-
-#pragma ident	"@(#)errno.h	1.22	05/06/08 SMI"
-
-#ifdef	__cplusplus
-extern "C" {
-#endif
-
-/*
- * Error codes
- */
-
-#define	EPERM	1	/* Not super-user			*/
-#define	ENOENT	2	/* No such file or directory		*/
-#define	ESRCH	3	/* No such process			*/
-#define	EINTR	4	/* interrupted system call		*/
-#define	EIO	5	/* I/O error				*/
-#define	ENXIO	6	/* No such device or address		*/
-#define	E2BIG	7	/* Arg list too long			*/
-#define	ENOEXEC	8	/* Exec format error			*/
-#define	EBADF	9	/* Bad file number			*/
-#define	ECHILD	10	/* No children				*/
-#define	EAGAIN	11	/* Resource temporarily unavailable	*/
-#define	ENOMEM	12	/* Not enough core			*/
-#define	EACCES	13	/* Permission denied			*/
-#define	EFAULT	14	/* Bad address				*/
-#define	ENOTBLK	15	/* Block device required		*/
-#define	EBUSY	16	/* Mount device busy			*/
-#define	EEXIST	17	/* File exists				*/
-#define	EXDEV	18	/* Cross-device link			*/
-#define	ENODEV	19	/* No such device			*/
-#define	ENOTDIR	20	/* Not a directory			*/
-#define	EISDIR	21	/* Is a directory			*/
-#define	EINVAL	22	/* Invalid argument			*/
-#define	ENFILE	23	/* File table overflow			*/
-#define	EMFILE	24	/* Too many open files			*/
-#define	ENOTTY	25	/* Inappropriate ioctl for device	*/
-#define	ETXTBSY	26	/* Text file busy			*/
-#define	EFBIG	27	/* File too large			*/
-#define	ENOSPC	28	/* No space left on device		*/
-#define	ESPIPE	29	/* Illegal seek				*/
-#define	EROFS	30	/* Read only file system		*/
-#define	EMLINK	31	/* Too many links			*/
-#define	EPIPE	32	/* Broken pipe				*/
-#define	EDOM	33	/* Math arg out of domain of func	*/
-#define	ERANGE	34	/* Math result not representable	*/
-#define	ENOMSG	35	/* No message of desired type		*/
-#define	EIDRM	36	/* Identifier removed			*/
-#define	ECHRNG	37	/* Channel number out of range		*/
-#define	EL2NSYNC 38	/* Level 2 not synchronized		*/
-#define	EL3HLT	39	/* Level 3 halted			*/
-#define	EL3RST	40	/* Level 3 reset			*/
-#define	ELNRNG	41	/* Link number out of range		*/
-#define	EUNATCH 42	/* Protocol driver not attached		*/
-#define	ENOCSI	43	/* No CSI structure available		*/
-#define	EL2HLT	44	/* Level 2 halted			*/
-#define	EDEADLK	45	/* Deadlock condition.			*/
-#define	ENOLCK	46	/* No record locks available.		*/
-#define	ECANCELED 47	/* Operation canceled			*/
-#define	ENOTSUP	48	/* Operation not supported		*/
-
-/* Filesystem Quotas */
-#define	EDQUOT	49	/* Disc quota exceeded			*/
-
-/* Convergent Error Returns */
-#define	EBADE	50	/* invalid exchange			*/
-#define	EBADR	51	/* invalid request descriptor		*/
-#define	EXFULL	52	/* exchange full			*/
-#define	ENOANO	53	/* no anode				*/
-#define	EBADRQC	54	/* invalid request code			*/
-#define	EBADSLT	55	/* invalid slot				*/
-#define	EDEADLOCK 56	/* file locking deadlock error		*/
-
-#define	EBFONT	57	/* bad font file fmt			*/
-
-/* Interprocess Robust Locks */
-#define	EOWNERDEAD	58	/* process died with the lock */
-#define	ENOTRECOVERABLE	59	/* lock is not recoverable */
-
-/* stream problems */
-#define	ENOSTR	60	/* Device not a stream			*/
-#define	ENODATA	61	/* no data (for no delay io)		*/
-#define	ETIME	62	/* timer expired			*/
-#define	ENOSR	63	/* out of streams resources		*/
-
-#define	ENONET	64	/* Machine is not on the network	*/
-#define	ENOPKG	65	/* Package not installed		*/
-#define	EREMOTE	66	/* The object is remote			*/
-#define	ENOLINK	67	/* the link has been severed		*/
-#define	EADV	68	/* advertise error			*/
-#define	ESRMNT	69	/* srmount error			*/
-
-#define	ECOMM	70	/* Communication error on send		*/
-#define	EPROTO	71	/* Protocol error			*/
-
-/* Interprocess Robust Locks */
-#define	ELOCKUNMAPPED	72	/* locked lock was unmapped */
-
-#define	ENOTACTIVE 73	/* Facility is not active		*/
-#define	EMULTIHOP 74	/* multihop attempted			*/
-#define	EBADMSG 77	/* trying to read unreadable message	*/
-#define	ENAMETOOLONG 78	/* path name is too long		*/
-#define	EOVERFLOW 79	/* value too large to be stored in data type */
-#define	ENOTUNIQ 80	/* given log. name not unique		*/
-#define	EBADFD	81	/* f.d. invalid for this operation	*/
-#define	EREMCHG	82	/* Remote address changed		*/
-
-/* shared library problems */
-#define	ELIBACC	83	/* Can't access a needed shared lib.	*/
-#define	ELIBBAD	84	/* Accessing a corrupted shared lib.	*/
-#define	ELIBSCN	85	/* .lib section in a.out corrupted.	*/
-#define	ELIBMAX	86	/* Attempting to link in too many libs.	*/
-#define	ELIBEXEC 87	/* Attempting to exec a shared library.	*/
-#define	EILSEQ	88	/* Illegal byte sequence.		*/
-#define	ENOSYS	89	/* Unsupported file system operation	*/
-#define	ELOOP	90	/* Symbolic link loop			*/
-#define	ERESTART 91	/* Restartable system call		*/
-#define	ESTRPIPE 92	/* if pipe/FIFO, don't sleep in stream head */
-#define	ENOTEMPTY 93	/* directory not empty			*/
-#define	EUSERS	94	/* Too many users (for UFS)		*/
-
-/* BSD Networking Software */
-	/* argument errors */
-#define	ENOTSOCK	95	/* Socket operation on non-socket */
-#define	EDESTADDRREQ	96	/* Destination address required */
-#define	EMSGSIZE	97	/* Message too long */
-#define	EPROTOTYPE	98	/* Protocol wrong type for socket */
-#define	ENOPROTOOPT	99	/* Protocol not available */
-#define	EPROTONOSUPPORT	120	/* Protocol not supported */
-#define	ESOCKTNOSUPPORT	121	/* Socket type not supported */
-#define	EOPNOTSUPP	122	/* Operation not supported on socket */
-#define	EPFNOSUPPORT	123	/* Protocol family not supported */
-#define	EAFNOSUPPORT	124	/* Address family not supported by */
-				/* protocol family */
-#define	EADDRINUSE	125	/* Address already in use */
-#define	EADDRNOTAVAIL	126	/* Can't assign requested address */
-	/* operational errors */
-#define	ENETDOWN	127	/* Network is down */
-#define	ENETUNREACH	128	/* Network is unreachable */
-#define	ENETRESET	129	/* Network dropped connection because */
-				/* of reset */
-#define	ECONNABORTED	130	/* Software caused connection abort */
-#define	ECONNRESET	131	/* Connection reset by peer */
-#define	ENOBUFS		132	/* No buffer space available */
-#define	EISCONN		133	/* Socket is already connected */
-#define	ENOTCONN	134	/* Socket is not connected */
-/* XENIX has 135 - 142 */
-#define	ESHUTDOWN	143	/* Can't send after socket shutdown */
-#define	ETOOMANYREFS	144	/* Too many references: can't splice */
-#define	ETIMEDOUT	145	/* Connection timed out */
-#define	ECONNREFUSED	146	/* Connection refused */
-#define	EHOSTDOWN	147	/* Host is down */
-#define	EHOSTUNREACH	148	/* No route to host */
-#define	EWOULDBLOCK	EAGAIN
-#define	EALREADY	149	/* operation already in progress */
-#define	EINPROGRESS	150	/* operation now in progress */
-
-/* SUN Network File System */
-#define	ESTALE		151	/* Stale NFS file handle */
-
-
-#ifdef	__cplusplus
-}
-#endif
-
-#endif	/* _SYS_ERRNO_H */
diff --git a/doc/legacy/fdl.texi b/doc/legacy/fdl.texi
deleted file mode 100644
index e33c687cdfb..00000000000
--- a/doc/legacy/fdl.texi
+++ /dev/null
@@ -1,454 +0,0 @@
-
-@c @node GNU Free Documentation License
-@c @appendixsec GNU Free Documentation License
-
-@cindex FDL, GNU Free Documentation License
-@center Version 1.2, November 2002
-
-@display
-Copyright @copyright{} 2000,2001,2002 Free Software Foundation, Inc.
-59 Temple Place, Suite 330, Boston, MA  02111-1307, USA
-
-Everyone is permitted to copy and distribute verbatim copies
-of this license document, but changing it is not allowed.
-@end display
-
-@enumerate 0
-@item
-PREAMBLE
-
-The purpose of this License is to make a manual, textbook, or other
-functional and useful document @dfn{free} in the sense of freedom: to
-assure everyone the effective freedom to copy and redistribute it,
-with or without modifying it, either commercially or noncommercially.
-Secondarily, this License preserves for the author and publisher a way
-to get credit for their work, while not being considered responsible
-for modifications made by others.
-
-This License is a kind of ``copyleft'', which means that derivative
-works of the document must themselves be free in the same sense.  It
-complements the GNU General Public License, which is a copyleft
-license designed for free software.
-
-We have designed this License in order to use it for manuals for free
-software, because free software needs free documentation: a free
-program should come with manuals providing the same freedoms that the
-software does.  But this License is not limited to software manuals;
-it can be used for any textual work, regardless of subject matter or
-whether it is published as a printed book.  We recommend this License
-principally for works whose purpose is instruction or reference.
-
-@item
-APPLICABILITY AND DEFINITIONS
-
-This License applies to any manual or other work, in any medium, that
-contains a notice placed by the copyright holder saying it can be
-distributed under the terms of this License.  Such a notice grants a
-world-wide, royalty-free license, unlimited in duration, to use that
-work under the conditions stated herein.  The ``Document'', below,
-refers to any such manual or work.  Any member of the public is a
-licensee, and is addressed as ``you''.  You accept the license if you
-copy, modify or distribute the work in a way requiring permission
-under copyright law.
-
-A ``Modified Version'' of the Document means any work containing the
-Document or a portion of it, either copied verbatim, or with
-modifications and/or translated into another language.
-
-A ``Secondary Section'' is a named appendix or a front-matter section
-of the Document that deals exclusively with the relationship of the
-publishers or authors of the Document to the Document's overall
-subject (or to related matters) and contains nothing that could fall
-directly within that overall subject.  (Thus, if the Document is in
-part a textbook of mathematics, a Secondary Section may not explain
-any mathematics.)  The relationship could be a matter of historical
-connection with the subject or with related matters, or of legal,
-commercial, philosophical, ethical or political position regarding
-them.
-
-The ``Invariant Sections'' are certain Secondary Sections whose titles
-are designated, as being those of Invariant Sections, in the notice
-that says that the Document is released under this License.  If a
-section does not fit the above definition of Secondary then it is not
-allowed to be designated as Invariant.  The Document may contain zero
-Invariant Sections.  If the Document does not identify any Invariant
-Sections then there are none.
-
-The ``Cover Texts'' are certain short passages of text that are listed,
-as Front-Cover Texts or Back-Cover Texts, in the notice that says that
-the Document is released under this License.  A Front-Cover Text may
-be at most 5 words, and a Back-Cover Text may be at most 25 words.
-
-A ``Transparent'' copy of the Document means a machine-readable copy,
-represented in a format whose specification is available to the
-general public, that is suitable for revising the document
-straightforwardly with generic text editors or (for images composed of
-pixels) generic paint programs or (for drawings) some widely available
-drawing editor, and that is suitable for input to text formatters or
-for automatic translation to a variety of formats suitable for input
-to text formatters.  A copy made in an otherwise Transparent file
-format whose markup, or absence of markup, has been arranged to thwart
-or discourage subsequent modification by readers is not Transparent.
-An image format is not Transparent if used for any substantial amount
-of text.  A copy that is not ``Transparent'' is called ``Opaque''.
-
-Examples of suitable formats for Transparent copies include plain
-@sc{ascii} without markup, Texinfo input format, La@TeX{} input
-format, @acronym{SGML} or @acronym{XML} using a publicly available
-@acronym{DTD}, and standard-conforming simple @acronym{HTML},
-PostScript or @acronym{PDF} designed for human modification.  Examples
-of transparent image formats include @acronym{PNG}, @acronym{XCF} and
-@acronym{JPG}.  Opaque formats include proprietary formats that can be
-read and edited only by proprietary word processors, @acronym{SGML} or
-@acronym{XML} for which the @acronym{DTD} and/or processing tools are
-not generally available, and the machine-generated @acronym{HTML},
-PostScript or @acronym{PDF} produced by some word processors for
-output purposes only.
-
-The ``Title Page'' means, for a printed book, the title page itself,
-plus such following pages as are needed to hold, legibly, the material
-this License requires to appear in the title page.  For works in
-formats which do not have any title page as such, ``Title Page'' means
-the text near the most prominent appearance of the work's title,
-preceding the beginning of the body of the text.
-
-A section ``Entitled XYZ'' means a named subunit of the Document whose
-title either is precisely XYZ or contains XYZ in parentheses following
-text that translates XYZ in another language.  (Here XYZ stands for a
-specific section name mentioned below, such as ``Acknowledgements'',
-``Dedications'', ``Endorsements'', or ``History''.)  To ``Preserve the Title''
-of such a section when you modify the Document means that it remains a
-section ``Entitled XYZ'' according to this definition.
-
-The Document may include Warranty Disclaimers next to the notice which
-states that this License applies to the Document.  These Warranty
-Disclaimers are considered to be included by reference in this
-License, but only as regards disclaiming warranties: any other
-implication that these Warranty Disclaimers may have is void and has
-no effect on the meaning of this License.
-
-@item
-VERBATIM COPYING
-
-You may copy and distribute the Document in any medium, either
-commercially or noncommercially, provided that this License, the
-copyright notices, and the license notice saying this License applies
-to the Document are reproduced in all copies, and that you add no other
-conditions whatsoever to those of this License.  You may not use
-technical measures to obstruct or control the reading or further
-copying of the copies you make or distribute.  However, you may accept
-compensation in exchange for copies.  If you distribute a large enough
-number of copies you must also follow the conditions in section 3.
-
-You may also lend copies, under the same conditions stated above, and
-you may publicly display copies.
-
-@item
-COPYING IN QUANTITY
-
-If you publish printed copies (or copies in media that commonly have
-printed covers) of the Document, numbering more than 100, and the
-Document's license notice requires Cover Texts, you must enclose the
-copies in covers that carry, clearly and legibly, all these Cover
-Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on
-the back cover.  Both covers must also clearly and legibly identify
-you as the publisher of these copies.  The front cover must present
-the full title with all words of the title equally prominent and
-visible.  You may add other material on the covers in addition.
-Copying with changes limited to the covers, as long as they preserve
-the title of the Document and satisfy these conditions, can be treated
-as verbatim copying in other respects.
-
-If the required texts for either cover are too voluminous to fit
-legibly, you should put the first ones listed (as many as fit
-reasonably) on the actual cover, and continue the rest onto adjacent
-pages.
-
-If you publish or distribute Opaque copies of the Document numbering
-more than 100, you must either include a machine-readable Transparent
-copy along with each Opaque copy, or state in or with each Opaque copy
-a computer-network location from which the general network-using
-public has access to download using public-standard network protocols
-a complete Transparent copy of the Document, free of added material.
-If you use the latter option, you must take reasonably prudent steps,
-when you begin distribution of Opaque copies in quantity, to ensure
-that this Transparent copy will remain thus accessible at the stated
-location until at least one year after the last time you distribute an
-Opaque copy (directly or through your agents or retailers) of that
-edition to the public.
-
-It is requested, but not required, that you contact the authors of the
-Document well before redistributing any large number of copies, to give
-them a chance to provide you with an updated version of the Document.
-
-@item
-MODIFICATIONS
-
-You may copy and distribute a Modified Version of the Document under
-the conditions of sections 2 and 3 above, provided that you release
-the Modified Version under precisely this License, with the Modified
-Version filling the role of the Document, thus licensing distribution
-and modification of the Modified Version to whoever possesses a copy
-of it.  In addition, you must do these things in the Modified Version:
-
-@enumerate A
-@item
-Use in the Title Page (and on the covers, if any) a title distinct
-from that of the Document, and from those of previous versions
-(which should, if there were any, be listed in the History section
-of the Document).  You may use the same title as a previous version
-if the original publisher of that version gives permission.
-
-@item
-List on the Title Page, as authors, one or more persons or entities
-responsible for authorship of the modifications in the Modified
-Version, together with at least five of the principal authors of the
-Document (all of its principal authors, if it has fewer than five),
-unless they release you from this requirement.
-
-@item
-State on the Title page the name of the publisher of the
-Modified Version, as the publisher.
-
-@item
-Preserve all the copyright notices of the Document.
-
-@item
-Add an appropriate copyright notice for your modifications
-adjacent to the other copyright notices.
-
-@item
-Include, immediately after the copyright notices, a license notice
-giving the public permission to use the Modified Version under the
-terms of this License, in the form shown in the Addendum below.
-
-@item
-Preserve in that license notice the full lists of Invariant Sections
-and required Cover Texts given in the Document's license notice.
-
-@item
-Include an unaltered copy of this License.
-
-@item
-Preserve the section Entitled ``History'', Preserve its Title, and add
-to it an item stating at least the title, year, new authors, and
-publisher of the Modified Version as given on the Title Page.  If
-there is no section Entitled ``History'' in the Document, create one
-stating the title, year, authors, and publisher of the Document as
-given on its Title Page, then add an item describing the Modified
-Version as stated in the previous sentence.
-
-@item
-Preserve the network location, if any, given in the Document for
-public access to a Transparent copy of the Document, and likewise
-the network locations given in the Document for previous versions
-it was based on.  These may be placed in the ``History'' section.
-You may omit a network location for a work that was published at
-least four years before the Document itself, or if the original
-publisher of the version it refers to gives permission.
-
-@item
-For any section Entitled ``Acknowledgements'' or ``Dedications'', Preserve
-the Title of the section, and preserve in the section all the
-substance and tone of each of the contributor acknowledgements and/or
-dedications given therein.
-
-@item
-Preserve all the Invariant Sections of the Document,
-unaltered in their text and in their titles.  Section numbers
-or the equivalent are not considered part of the section titles.
-
-@item
-Delete any section Entitled ``Endorsements''.  Such a section
-may not be included in the Modified Version.
-
-@item
-Do not retitle any existing section to be Entitled ``Endorsements'' or
-to conflict in title with any Invariant Section.
-
-@item
-Preserve any Warranty Disclaimers.
-@end enumerate
-
-If the Modified Version includes new front-matter sections or
-appendices that qualify as Secondary Sections and contain no material
-copied from the Document, you may at your option designate some or all
-of these sections as invariant.  To do this, add their titles to the
-list of Invariant Sections in the Modified Version's license notice.
-These titles must be distinct from any other section titles.
-
-You may add a section Entitled ``Endorsements'', provided it contains
-nothing but endorsements of your Modified Version by various
-parties---for example, statements of peer review or that the text has
-been approved by an organization as the authoritative definition of a
-standard.
-
-You may add a passage of up to five words as a Front-Cover Text, and a
-passage of up to 25 words as a Back-Cover Text, to the end of the list
-of Cover Texts in the Modified Version.  Only one passage of
-Front-Cover Text and one of Back-Cover Text may be added by (or
-through arrangements made by) any one entity.  If the Document already
-includes a cover text for the same cover, previously added by you or
-by arrangement made by the same entity you are acting on behalf of,
-you may not add another; but you may replace the old one, on explicit
-permission from the previous publisher that added the old one.
-
-The author(s) and publisher(s) of the Document do not by this License
-give permission to use their names for publicity for or to assert or
-imply endorsement of any Modified Version.
-
-@item
-COMBINING DOCUMENTS
-
-You may combine the Document with other documents released under this
-License, under the terms defined in section 4 above for modified
-versions, provided that you include in the combination all of the
-Invariant Sections of all of the original documents, unmodified, and
-list them all as Invariant Sections of your combined work in its
-license notice, and that you preserve all their Warranty Disclaimers.
-
-The combined work need only contain one copy of this License, and
-multiple identical Invariant Sections may be replaced with a single
-copy.  If there are multiple Invariant Sections with the same name but
-different contents, make the title of each such section unique by
-adding at the end of it, in parentheses, the name of the original
-author or publisher of that section if known, or else a unique number.
-Make the same adjustment to the section titles in the list of
-Invariant Sections in the license notice of the combined work.
-
-In the combination, you must combine any sections Entitled ``History''
-in the various original documents, forming one section Entitled
-``History''; likewise combine any sections Entitled ``Acknowledgements'',
-and any sections Entitled ``Dedications''.  You must delete all
-sections Entitled ``Endorsements.''
-
-@item
-COLLECTIONS OF DOCUMENTS
-
-You may make a collection consisting of the Document and other documents
-released under this License, and replace the individual copies of this
-License in the various documents with a single copy that is included in
-the collection, provided that you follow the rules of this License for
-verbatim copying of each of the documents in all other respects.
-
-You may extract a single document from such a collection, and distribute
-it individually under this License, provided you insert a copy of this
-License into the extracted document, and follow this License in all
-other respects regarding verbatim copying of that document.
-
-@item
-AGGREGATION WITH INDEPENDENT WORKS
-
-A compilation of the Document or its derivatives with other separate
-and independent documents or works, in or on a volume of a storage or
-distribution medium, is called an ``aggregate'' if the copyright
-resulting from the compilation is not used to limit the legal rights
-of the compilation's users beyond what the individual works permit.
-When the Document is included in an aggregate, this License does not
-apply to the other works in the aggregate which are not themselves
-derivative works of the Document.
-
-If the Cover Text requirement of section 3 is applicable to these
-copies of the Document, then if the Document is less than one half of
-the entire aggregate, the Document's Cover Texts may be placed on
-covers that bracket the Document within the aggregate, or the
-electronic equivalent of covers if the Document is in electronic form.
-Otherwise they must appear on printed covers that bracket the whole
-aggregate.
-
-@item
-TRANSLATION
-
-Translation is considered a kind of modification, so you may
-distribute translations of the Document under the terms of section 4.
-Replacing Invariant Sections with translations requires special
-permission from their copyright holders, but you may include
-translations of some or all Invariant Sections in addition to the
-original versions of these Invariant
author	Pranith Kumar K <pkarampu@redhat.com>	2014-02-15 17:54:38 +0530
committer	Anand Avati <avati@redhat.com>	2014-02-24 18:56:06 -0800
commit	868287aa5f70bdfcf7f56663e042d74fe1acf0db (patch)
tree	f84a67b72a8dac9c7d9eb80e80cc83d65d59165e /extras/hook-scripts
parent	3a5914e421d10bca363f020b748e8648a4819495 (diff)